Controlling multiple robots for collaborative item pick-and-place

Abstract

A robotic system for controlling multiple robots to cooperatively pick and place objects is disclosed. In various embodiments, the robotic system includes a first robotic arm having a first end effector, a second robotic arm having a second end effector, and a control computer configured to pick and place a plurality of objects using the first robotic arm and the second robotic arm, including cooperatively operating to pick and place one or more of the plurality of objects using the first robotic arm and the second robotic arm.

Description

CROSS-REFERENCE TO OTHER APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 274,465, filed November 1, 2021, entitled "CONTROLLING MULTIPLE ROBOTS TO COOPERATIVELY PICK AND PLACE ITEMS," which is incorporated herein by reference for all purposes. [Background technology]

[0002] Robots have been provided to perform a variety of tasks, such as manipulating objects. For example, a robotic arm having an end effector may be utilized to pick and place items. Examples of commercial applications of such robots include sorting, kitting, palletizing, depalletizing, loading and unloading trucks or containers, etc.

[0003] In some contexts, the objects to be manipulated vary greatly in size, weight, packaging, and other attributes. Typically, a robotic arm is rated to handle up to a maximum size, weight, etc. of an object. In some contexts, conventional approaches may require a robotic arm that can manipulate the largest, heaviest, and / or otherwise most difficult object that may be asked to be manipulated. [Brief explanation of the drawings]

[0004] Various embodiments of the present invention are disclosed in the following detailed description and the accompanying drawings.

[0005] [Figure 1] FIG. 1 is a block diagram illustrating an embodiment of a robotic system configured to control multiple robots to collaboratively perform tasks.

[0006] [Figure 2A]FIG. 1 illustrates an example of a collaborative pick-and-place task performed in one embodiment of the robotic system disclosed herein. [Figure 2B] FIG. 1 illustrates an example of a collaborative pick-and-place task performed in one embodiment of the robotic system disclosed herein. [Figure 2C] FIG. 1 illustrates an example of a collaborative pick-and-place task performed in one embodiment of the robotic system disclosed herein.

[0007] [Figure 3] FIG. 1 is a block diagram illustrating an embodiment of a robot control system.

[0008] [Figure 4] FIG. 1 is a state diagram illustrating an embodiment of a robotic system configured to control multiple robots to collaboratively perform tasks.

[0009] [Figure 5A] 1 is a flowchart illustrating an embodiment of a process for collaboratively performing a task as a "leader" robot in an embodiment of a robotic system herein.

[0010] [Figure 5B] 1 is a flowchart illustrating an embodiment of a process for collaboratively performing tasks as "follower" robots in an embodiment of a robotic system herein.

[0011] [Figure 6A] FIG. 1 illustrates an embodiment of a robotic system configured to cooperatively pick and place an object using two or more robots.

[0012] [Figure 6B] FIG. 1 illustrates an embodiment of a robotic system configured to cooperatively pick and place an object using two or more robots.

[0013] [Figure 7] 1 is a flow chart illustrating one embodiment of a process for cooperatively picking and placing an object using two or more robots. DETAILED DESCRIPTION OF THE INVENTION

[0014] The present invention may be embodied in various forms, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor configured to execute instructions stored in and / or provided by a memory coupled to the processor. These embodiments, or any other form the present invention may take, may be referred to herein as technology. In general, the order of steps in a disclosed process may be varied within the scope of the present invention. Unless otherwise noted, components, such as a processor or memory, described as configured to perform a task may be implemented as general components temporarily configured to perform the task at a given time, or as specific components manufactured to perform the task. As used herein, the term “processor” refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0015] The following is a detailed description of one or more embodiments of the present invention with reference to figures that illustrate the principles of the invention. While the present invention has been described in connection with such embodiments, it is not limited to any particular embodiment. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications, and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. These details are for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For simplicity, technical matters that are well known in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the present invention.

[0016] A system for coordinating and controlling the use of multiple robots to cooperatively pick and place packages is disclosed. In various embodiments, the system disclosed herein may have one or more of the following technical features: The system detects objects that are picked collaboratively. Integrated computer vision system to identify obstacles and prioritize the picking of packages from stacks for safe collaborative operation. ·Planning and executing collaborative picking of a single object by multiple robots. A control architecture that enables robots to work together to lift and place objects in a safe and controlled manner. · Robots can act individually (alone) when not performing collaborative tasks.

[0017] In various embodiments, multiple robotic arms are used to cooperatively pick and place a single package. In some embodiments, a robotic singulation (or other pick / place) system detects that an object should be cooperatively picked using two or more robots due to, for example, mismatches in object size, weight, previous failed pick attempts, visual categorization, and / or affordances between the object and individual robot grippers.

[0018] In various embodiments, when both robots stop picking independently, the system determines the best way to pick the object, ensuring the grasp point is within the robot's reach and on opposite sides of the object. The robots remove any surrounding packaging that may prevent the robot from picking the desired package. The robots plan paths to independently reach pick locations on either side of the object. Once both robots are in place, the leader robot begins to back away and the follower robot maintains contact with the box using force control while maintaining its position / orientation relative to the leader bot, allowing them to cooperatively lift and move heavy or oversized objects.

[0019] Further techniques implemented in various embodiments include, but are not limited to, one or more of the following: Haptic manipulation to enhance unseen parts of objects. To ensure that multiple robots can find collision-free picks for the same object, sometimes one of the robots must pick on a side of the object that is invisible to the robot. In some embodiments, tactile perception from the gripper is used to explore the back side of the object without seeing it in order to find a stable and collision-free pick location. Pushing to improve object visibility and grasp stability. Sometimes, not all sides of an object are visible to the robots to achieve a collaborative pick, and a robot needs to change the object's position / orientation to make pickable positions visible to multiple robots. This change may be made by pushing with one robot or multiple robots to identify a more stable grasp point. Collision avoidance between multiple robots. It is desirable to coordinate the movements of multiple robots with their corresponding environments in a collision-free manner.

[0020] FIG. 1 is a block diagram illustrating one embodiment of a robotic system configured to control multiple robots to collaboratively perform a task. In the illustrated example, system and environment 100 includes a first robotic arm 102 with a suction-type end effector 104 and a second robotic arm 106 with a suction-type end effector 108. As shown, robotic arm 102 and robotic arm 106 are positioned to collaboratively perform a pick-and-place task on a large box 110. A control computer 112 is configured to wirelessly communicate with robotic arm 102, robotic arm 106, and one or more cameras or other sensors 114 within a workspace. Image data received from camera 114 may be used by control computer 112 to generate a three-dimensional view of the workspace and send commands and information to robotic arm 102 and robotic arm 106, as needed, to facilitate the collaborative pick-and-place task, for example.

[0021] 2A-2C illustrate an example of a collaborative pick-and-place task performed in one embodiment of the robotic system disclosed herein. In the example shown in FIG. 2A, robotic arm 202 with suction-type end effector 204 and robotic arm 206 with suction-type end effector 208 are positioned to begin collaboratively performing a pick-and-place task on large box 210, similar to the start state shown in FIG. 1. In various embodiments, robotic arm 202 may be the "leader" and robotic arm 206 may be the "follower" in the collaborative pick-and-place disclosed herein. The "leader" may be selected by any suitable method (e.g., by assigning the role of "leader" to the robot that initiated the collaborative task, by randomly assigning the role to one or other of the participating robots, by "election" or other selection method).

[0022] To initiate an operation, in various embodiments, the "leader" robotic arm 202 moves its end effector 204 to the position shown in the figure and then grasps the box, for example, by moving the end effector 204 into contact or near contact with the side of the box 210 and creating suction. A signal may be sent to the other robot (and / or a process controlling the other robot) to indicate that the leader has completed its grasp. The follower (e.g., robotic arm 206 in this example) then grasps the box 210, for example, on the side opposite the side on which the leader (i.e., robotic arm 202) grasped the box 210. The follower records a transformation based on the position and orientation of the leader's end effector 204 and the relevant dimensions of the box 210. For example, a vision system and / or other sensors may be used to measure dimensions or to recognize the box 210 (e.g., specifically and / or by type) and determine the dimensions using the item and / or type information (e.g., by lookup).

[0023] As shown in FIG. 2B , by this point, both robots (202, 206) have grasped the box 210. The leader (in this example, robot arm 202) performs calculations independently of the follower robot arm 206 to move the box along a trajectory determined by the robot arm 202 (and / or its associated control process). In various embodiments, the follower robot (in this example, robot arm 206) receives (e.g., periodically, continuously, etc.) position and orientation information of the end effector 204 of the leader robot arm 202. The follower robot arm 206 (and / or its associated control process) uses the position and orientation information of the leader robots (202, 204) and the previously determined and recorded transformations to calculate a new target position and orientation for the follower end effector 208, and calculates and applies torques to the motors that make up the robot arm 206, as necessary, to minimize the error (difference) between the current position and orientation of the follower end effector 208 and the (most recently updated) target.

[0024] 2C, once the object (box 210) is placed in its destination location, the leader robot (robot arm 202) releases its grip and signals the follower that the pick-and-place task is complete. In response, the follower (robot arm 206) releases its grip, allowing both robots (202, 206) to perform other tasks, such as (returning) to individually pick and place a smaller / lighter object and / or collaboratively performing a next pick-and-place task for another larger or heavier object.

[0025] Figure 3 is a block diagram illustrating one embodiment of a robotic control system. In various embodiments, the robotic control system 302 of Figure 3 includes or is included in the control computer 112 of Figure 1. In various embodiments, one or more modules or subsystems that make up the robotic control system 302 of Figure 3 may be distributed across multiple computational nodes, such as the computers and / or processors that make up one or more of the control computer 112, robotic arm 102, and / or robotic arm 106 of Figure 1.

[0026] In the depicted example, robot control system 302 includes a hierarchical planner, scheduler, and / or control module with a robot collaboration facilitation module 304 configured to facilitate the collaborative performance of a task by two or more robots as disclosed herein, and robot-specific controllers 306 and 308. For example, controller 306 for robot 1 may be associated with robot arm 102 of Figure 1 and / or robot arm 202 of Figures 2A-2C, while controller 308 for robot 2 may be associated with robot arm 106 of Figure 1 and / or robot arm 206 of Figures 2A-2C.

[0027] In various embodiments, each robot associated with a respective robot 1 controller 306 and robot 2 controller 308 may operate independently, such as to pick and place an object that the robots can manipulate individually. In various embodiments, a collaborative task using two or more robots may be initiated and / or executed by one or more communications sent between the robot 1 controller 306 and the robot 2 controller 308, by bilateral communications between the robot collaboration facilitation module 304 and the respective robot 1 controller 306 and robot 2 controller 308, and / or by communications between all three (or four or more) entities.

[0028] In the depicted example, the robot control system 302 further includes a computer vision subsystem 310 configured to receive image and depth data from one or more 3D cameras and / or other sensors (such as camera 114 in FIG. 1 ) and use the received data to generate and / or update a three-dimensional view of the workspace. The output of the computer vision subsystem 310 may be provided to one or more of the robot collaboration facilitation module 304, the controller for robot 1 306, and the controller for robot 2 308 to enable the robots to collaboratively initiate and execute tasks to pick and place items. For example, the image data may be used to determine if a box or other object is too large and / or heavy to be picked and placed by a single robot. The three-dimensional view of the workspace and objects therein may be used, for example, to determine a respective grasping strategy and / or position for each robot, determine multiple collision-free trajectories for moving each robot's end effector to its corresponding pick location, and determine collision-free trajectories for collaboratively moving the objects to a destination location where the objects will be placed.

[0029] 4 is a state diagram illustrating one embodiment of a robotic system configured to control multiple robots to collaboratively perform a task. In various embodiments, state diagram 400 of FIG. 4 may be implemented by and / or in connection with a robot configured to collaboratively perform a task using two or more robots. In some embodiments, state diagram 400 of FIG. 4 may be implemented by control computer 112 of FIG. 1 and / or one or more of robot collaboration facilitating module 304, controller for robot 1 306, and controller for robot 2 308 of FIG. 3.

[0030] In the illustrated example, in state 402, the robots work individually to perform a task. For example, the robots may individually pick and place items to fill boxes or other containers in a kitting operation, place items on a conveyor belt or other transport in a sorting operation, stack items on a pallet, etc. Upon receiving an indication that assistance (help) is needed to perform the task (404), such as an indication that an item that needs to be recognized and picked and placed is too large to be grasped and moved by a single robot, the robots and / or controllers transition to state 406, where collaborative execution of the task begins. For example, a communication may be sent to another robot (e.g., from controller 306 for robot 1 to controller 308 for robot 2 in FIG. 3 ) or to a higher-level planner / scheduler (e.g., to robot collaboration facilitation module 304 in FIG. 3 ), or the higher-level planner / scheduler may recognize the need for collaborative execution of the task and initiate the transition to state 406.

[0031] In the illustrated example, the robot and / or controller may return to working independently at state 402 via "cancel assistance" transition 408. For example, the robot / controller and / or higher level planner / scheduler may determine that the task has already been performed by and / or assigned to one or more other robots.

[0032] In some embodiments, in the "begin collaboration" state 406, the robot / controller initiating the collaborative execution of a task communicates directly or indirectly with the helper robot, for example, by requesting assistance. Another robot may be assigned to assist and / or agree to assist. The robot may be assigned to assist and / or agree to assist at some future time or upon the occurrence of a future condition (such as the completion of a task already started by the helper robot and / or a task with a higher priority). For example, to facilitate the collaborative task, a task of removing other objects from around a large or heavy object may have a higher priority and therefore be completed first. When the helper robot is ready to perform the collaborative task, the helper robot notifies the task initiator, directly or indirectly (e.g., via a higher-level planner / scheduler (e.g., robot collaboration facilitation module 304 in FIG. 3 )), that the helper robot is ready, prompting a transition 410 to the "begin collaboration" state 412. A helper may transition directly from the working independently state 402 to the "initiate collaboration" state 412 through the "provide assistance" transition 414 in the illustrated example.

[0033] Once all participating robots are ready in the "Begin Collaboration" state 412, a "leader" is determined, if necessary, and the leader transitions (416) to the "Acting Leader" state 418, while the followers transition (420) to the "Acting Follower" state 422. In the "Acting Leader" state 418 and the "Acting Follower" state 422, the leader and followers collaborate as disclosed herein to cooperatively perform a task, such as picking and placing a large or heavy object, as in the example shown in Figures 2A-2C. Once the task is completed, the leader and followers return (424, 426) to the "Working Individually" state 402 to resume their individual work.

[0034] 5A is a flow chart illustrating an embodiment of a process for collaboratively performing a task as a "leader" robot in an embodiment of a robotic system herein. In various embodiments, the process 500 of FIG. 5A may be performed by a robot controller associated with a robot participating as a "leader" in the collaborative performance of a task by two or more robots disclosed herein.

[0035] In the illustrated example, at step 502, an instruction is received to initiate a collaborative task (with one or more other robots) in the role of a “leader.” For example, an instruction may be received to collaboratively perform a pick-and-place task. At step 504, the leader determines a location to grasp an object and plans a trajectory to safely move its end effector to the position to grasp the object, and at step 506, the leader moves the end effector along the trajectory to the grasping position. At step 508, the leader determines a trajectory (independently of any other robots) to move the object to the associated destination. For example, a model of the robot and its kinematics, as well as images and / or other information about the workspace (e.g., configuration data, CAD files, etc.), one or more attributes of the object (e.g., dimensions, stiffness, etc.), and image / sensor data may be used to plan the trajectory. At step 510, an indication is received from one or more "follower" robots, with which the robots must cooperate, that one or more follower robots are ready to begin collaborative execution of a task. In response, at step 512, the "leader" robot moves its end effector (and the object being jointly grasped by the leader and follower) to a destination along a trajectory determined by the leader. At step 514, once the object is placed at its destination, the leader robot releases its grasp and notifies the follower robots that the task is complete. In various embodiments, the leader then resumes operating independently.

[0036] 5B is a flow chart illustrating an embodiment of a process for collaboratively performing a task as a "follower" robot in an embodiment of a robotic system herein. In various embodiments, process 520 of FIG. 5B may be performed by a robot controller associated with a robot participating as a "follower" in the collaborative performance of a task by two or more robots disclosed herein.

[0037] In the illustrated example, at step 522, an instruction is received to perform a task collaboratively with one or more other robots in the role of a "follower," as disclosed herein. At step 524, the follower determines a grasp point (e.g., a point on the side of the object opposite the side indicated by the "leader" to be grasped) and plans a trajectory to move to a position to grasp the object at that point. At step 526, the follower moves its end effector to the determined grasp position and grasps the object, for example, in response to receiving an indication that the leader has completed its grasp. At step 528, position and orientation information of the leader's end effector is received, and the follower uses this information along with information about the object (e.g., the size of the object in the dimension separating the leader's and follower's end effectors) to calculate a transformation. In various embodiments, the transformation comprises a matrix or other mathematical construct that can be applied to the position and orientation of the leader's end effector (typically expressed in the leader's frame of reference) to provide a corresponding position and orientation for the follower's end effector that maintains the relative position and orientation of the follower's end effector with respect to the leader's end effector as the end effectors and the object grasped therebetween are moved through the workspace to the object's placement destination. At step 530, the follower robot notifies the leader that the follower is "ready" (e.g., the follower is ready to grasp an object, calculate a transformation, and maintain its position relative to (e.g., opposite) the leader's end effector).

[0038] In step 532, once the leader robot begins moving along the trajectory independently determined by the leader, the follower utilizes the transformations it has calculated and the position and orientation information of the leader's end effector that it continuously receives as the leader's end effector is moved through the workspace. For example, for each of at least a portion of the positions and / or orientations of the leader's end effector received, the follower calculates a new target position and / or orientation for that end effector and applies torques to its motors that it determines are necessary to minimize the error (e.g., difference) between the current position and / or orientation of its end effector and the current target.

[0039] In step 534, the follower receives an indication (eg, from the leader) that the collaborative task is "complete," and in response, the follower releases the grip and process 520 ends.

[0040] In various embodiments, the techniques disclosed herein are used to collaboratively perform pick-and-place tasks, for example, in connection with singulation / sorting, kitting, palletization or depalletization, and / or loading or unloading trucks or other containers.

[0041] FIG. 6A illustrates an embodiment of a robotic system configured to collaboratively pick and place objects using two or more robots. In the illustrated example, system and environment 600A demonstrates the use of the technology disclosed herein in the context of a singulation / sorting operation. Items of various sizes, shapes, and other attributes arrive via an inlet transport 602, such as a gravity-fed chute or ramp and / or an inlet conveyor belt or similar structure. In the illustrated example, robots 202, 206 of FIGS. 2A-2C are utilized in an independent operating mode to pick items from inlet transport 602 and place them one-by-one onto conveyor 604. Image data from camera 606 may be used to generate a three-dimensional view of the workspace, enabling robots 202, 206 to identify and prioritize target objects, develop a plan and strategy for grasping the objects, and pick and place the objects.

[0042] 6A, the robots 202, 206 operate independently but cooperatively. For example, the robots 202, 206 may alternate between picking from the inlet transport 602 and placing onto the conveyor 604. When one (e.g., 202) is picking from the inlet transport 602, the other is placing onto the conveyor 604, and vice versa.

[0043] Figure 6B illustrates an embodiment of a robotic system configured to cooperatively pick and place an object using two or more robots. In the example and state shown in Figure 6B, robots 202, 206 are being used to cooperatively pick and place a large box that arrives at a pick area of ​​an inlet transport 602. Image data from camera 606 may be used to detect the large box and / or determine (e.g., by lookup) the weight and / or other attributes of the box that indicate the need to cooperatively pick and place the box using two robots 202, 206.

[0044] In various embodiments, robots 202, 206 cooperate as disclosed herein to pick and place a large box. For example, robot 202 may act as a "leader" robot and perform process 500 of Figure 5A, while robot 206 may function as a "follower" robot and perform process 520 of Figure 5B, or vice versa.

[0045] Once the placing operation of the large box onto the conveyor 604 shown in FIG. 6B is completed, the robots 202, 206 may resume their individual operations as described above.

[0046] FIG. 7 is a flowchart illustrating one embodiment of a process for collaboratively picking and placing an object using two or more robots. In various embodiments, process 700 of FIG. 7 may be implemented by one or more computers (such as control computer 112 of FIG. 1 ) and / or by one or more other computers and / or processors comprising the robotic systems disclosed herein. In the illustrated example, at step 702, a determination is made as to the need to collaboratively perform a pick-and-place task using two or more robots. For example, a computer (such as control computer 112 of FIG. 1 ) and / or a controller or other control module associated with an individual robot and / or a higher-level controller in a hierarchical controller may make that determination. At step 704, two or more robots are assigned (e.g., by themselves, by a coordinator, etc.) to collaboratively perform the task, and for each, a corresponding pick location (on the object) and position (e.g., of the end effector and / or robot arm) are determined (or attempted to be determined). If, at step 706, it is determined that a sufficiently clear view is not available to allow a pick location to be determined for one or more of the robots, at step 708, one or more robots may be used to move other objects out of the way to provide a clearer view. If a pick location is visible (step 706), or an object is moved to provide a clear view (step 708), at step 710, a determination is made as to whether all participating robots have a clear path or trajectory for each to move to their corresponding pick location without collision. If any robot does not have a clear path (step 712), at step 714, one or more robots are used as needed to move objects to provide a clear path. For example, a target object may be pushed or pulled out of a pile of cluttered items.Alternatively, an object adjacent to the target object or in the path between the end effector and its pick location may be moved out of the way by the robot or by another robot that needs to clear the path to the pick location.

[0047] In some cases, a robot may not be able to clearly see its pick location, and in some embodiments, such a robot may use force sensors or other tactile feedback to feel its way into position to grasp the object.

[0048] Once all participating robots have a clear path to their pick locations (steps 712, 714), the robots work to cooperatively perform the pick-and-place task as disclosed herein. For example, one robot may act as a "leader" and perform process 500 of Figure 5A, while another robot acts as a "follower" and performs process 520 of Figure 5B.

[0049] In various embodiments, the techniques disclosed herein may be used to cooperatively pick and place objects using two or more robots, such as objects that are too heavy, soft, bulky, etc. to be picked and placed by a single robot.

[0050] Although the above-described embodiments have been described in some detail for ease of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and are not intended to be limiting. [Application Example 1] A robot system, a first robot arm having a first end effector; a second robot arm having a second end effector; a control computer configured to pick and place one or more of the plurality of objects using the first robotic arm and the second robotic arm, including cooperatively working to pick and place one or more of the plurality of objects using the first robotic arm and the second robotic arm; A system comprising: [Application Example 2] The system described in Application Example 1, further comprising a camera arranged to generate image data associated with a workspace in which the first robot arm and the second robot arm are arranged. [Application Example 3] A system as described in Application Example 2, wherein the control computer is configured to use the image data to determine how to use the first robot arm and the second robot arm to work cooperatively to pick and place a given object. [Application Example 4] A system according to Application Example 3, wherein the determination is based at least in part on attributes of the object determined directly or indirectly using the image data. [Application Example 5] The system according to Application Example 1, wherein the control computer includes two or more processors distributed across two or more nodes. [Application Example 6] A system as described in Application Example 1, wherein the control computer executes a hierarchical planner comprising individual robot controllers for each of the first robot arm and the second robot arm, and a higher-level controller configured to coordinate the operation of the first robot arm in cooperation with the second robot arm to cooperatively pick and place one or more of the objects. [Application Example 7] A system as described in Application Example 1, wherein the first robot arm is operated in a leader mode and the second robot arm is operated in a follower mode to perform a given task for collaboratively picking and placing a given item. [Application Example 8] A system described in Application Example 7, wherein the first robot arm, when in the leader mode, is configured to independently plan a trajectory for moving a given object to be cooperatively picked and placed, grasp the given object, and move the given object along the planned trajectory. [Application Example 9] A system as described in Application Example 8, wherein the second robot arm, when in the follower mode, is configured to grasp the given object, calculate a transformation based at least in part on the position and orientation of the first end effector, and notify the first robot arm that the second robot arm is ready to move the given object. [Application Example 10] A system as described in Application Example 1, wherein the first robot arm and the second robot arm are configured to pick and place objects individually when not working cooperatively to pick and place one or more of the objects. [Application Example 11] A system as described in Application Example 1, wherein the control computer is configured to move one or more of the plurality of objects out of a line of sight from a camera or other sensor to a given object based at least in part on a determination that a more unobstructed view of the given object to be collaboratively picked and placed is required to collaboratively perform a pick-and-place task relating to the given object. [Application Example 12] A system as described in Application Example 1, wherein the control computer is configured to use one or both of the first robot arm and the second robot arm to pull or push the given object into the field of view of a camera or other sensor to facilitate the task of cooperatively picking and placing the given object using the first robot arm and the second robot arm. [Application Example 13] A system as described in Application Example 1, wherein the control computer is configured to use one or both of the first robot arm and the second robot arm to move one or more of the plurality of objects out of the way so that one or both of the first robot arm and the second robot arm can be used to grasp a given object. [Application Example 14] A system as described in Application Example 1, wherein the control computer is configured to use the first robot arm and the second robot arm to alternately pick and place objects included in the plurality of objects when the control computer is not working cooperatively to pick and place a given object using the first robot arm and the second robot arm. [Application Example 15] A system as described in Application Example 1, wherein the control computer is configured to use a force sensor or other tactile feedback to grasp an object by touching it using the first robot arm or the second robot arm, the object being in a position invisible to the control computer. [Application Example 16] A system as described in Application Example 1, wherein the control computer is configured to ensure that the first robot arm and the second robot arm do not collide with each other or with obstacles in the workspace when performing a pick-and-place task. [Application Example 17] A method, 1. A method comprising: picking and placing one or more of a plurality of objects using a first robotic arm and a second robotic arm, the method comprising: using the first robotic arm and a second robotic arm to cooperatively work to pick and place one or more of the plurality of objects. [Application Example 18] A method as described in Application Example 17, further comprising using image data from a camera to determine how to use the first robot arm and the second robot arm to work cooperatively to pick and place a given object. [Application Example 19] A method as described in Application Example 17, wherein working cooperatively to pick and place a given object using the first robot arm and the second robot arm includes using the first robot arm in a leader mode and using the second robot arm in a follower mode. [Application Example 20] A computer program product embodied in a non-transitory computer-readable medium, 1. A computer program product comprising computer instructions for picking and placing one or more of a plurality of objects using a first robotic arm and a second robotic arm, the computer program product comprising: using a first robotic arm and a second robotic arm to cooperatively work to pick and place one or more of the plurality of objects.

Claims

1. 1. A robotic system comprising: a first robot arm having a first end effector; a second robotic arm having a second end effector; a control computer configured to pick and place one or more of a plurality of objects using the first robotic arm and the second robotic arm, including cooperatively working to pick and place one or more of the plurality of objects using the first robotic arm and the second robotic arm; Equipped with the control computer is configured to use one or both of the first robotic arm and the second robotic arm to move one or more of the plurality of objects out of the way to enable one or both of the first robotic arm and the second robotic arm to be used to grasp a given object.

2. 10. The system of claim 1, further comprising a camera positioned to generate image data associated with a workspace in which the first robotic arm and the second robotic arm are positioned.

3. 3. The system of claim 2, wherein the control computer is configured to utilize the image data to determine how to use the first robotic arm and the second robotic arm to cooperatively work to pick and place a given object.

4. The system of claim 3 , wherein the determination is based at least in part on attributes of the object determined directly or indirectly using the image data.

5. 10. The system of claim 1, wherein the control computer comprises two or more processors distributed across two or more nodes.

6. 2. The system of claim 1, wherein the control computer executes a hierarchical planner comprising a separate robot controller for each of the first robot arm and the second robot arm, and a higher-level controller configured to coordinate operation of the first robot arm in cooperation with the second robot arm to cooperatively pick and place the one or more of the objects.

7. 10. The system of claim 1, wherein the first robotic arm is operated in a leader mode and the second robotic arm is operated in a follower mode to perform a given task for cooperatively picking and placing a given item.

8. 8. The system of claim 7, wherein when in the leader mode, the first robotic arm is configured to independently plan a trajectory for moving a given object to be cooperatively picked and placed, grasp the given object, and move the given object along the planned trajectory.

9. 9. The system of claim 8, wherein the second robotic arm, when in the follower mode, is configured to grasp the given object, calculate a transformation based at least in part on a position and orientation of the first end effector, and notify the first robotic arm that the second robotic arm is ready to move the given object.

10. 10. The system of claim 1, wherein the first robotic arm and the second robotic arm are configured to individually pick and place objects when not working cooperatively to pick and place the one or more of the objects.

11. 1. A robotic system comprising: a first robot arm having a first end effector; a second robotic arm having a second end effector; a control computer configured to pick and place one or more of a plurality of objects using the first robotic arm and the second robotic arm, including cooperatively working to pick and place one or more of the plurality of objects using the first robotic arm and the second robotic arm; Equipped with The control computer is configured to move one or more of the plurality of objects out of a line of sight from a camera or other sensor to a given object to be collaboratively picked and placed based at least in part on a determination that a more unobstructed view of the given object is needed to collaboratively perform a pick-and-place task with respect to the given object.

12. 1. A robotic system comprising: a first robot arm having a first end effector; a second robotic arm having a second end effector; a control computer configured to pick and place one or more of a plurality of objects using the first robotic arm and the second robotic arm, including cooperatively working to pick and place one or more of the plurality of objects using the first robotic arm and the second robotic arm; Equipped with The control computer is configured to use one or both of the first robotic arm and the second robotic arm to pull or push the given object into the field of view of a camera or other sensor to facilitate a task of cooperatively picking and placing the given object using the first robotic arm and the second robotic arm.

13. 1. A robotic system comprising: a first robot arm having a first end effector; a second robotic arm having a second end effector; a control computer configured to pick and place one or more of a plurality of objects using the first robotic arm and the second robotic arm, including cooperatively working to pick and place one or more of the plurality of objects using the first robotic arm and the second robotic arm; Equipped with the control computer is configured to alternately pick and place objects within the plurality of objects using the first robotic arm and the second robotic arm when the control computer is not cooperatively working to pick and place a given object using the first robotic arm and the second robotic arm.

14. 1. A robotic system comprising: a first robot arm having a first end effector; a second robotic arm having a second end effector; a control computer configured to pick and place one or more of a plurality of objects using the first robotic arm and the second robotic arm, including cooperatively working to pick and place one or more of the plurality of objects using the first robotic arm and the second robotic arm; Equipped with the control computer is configured to utilize force sensors or other tactile feedback to touch and grasp an object with the first robotic arm or the second robotic arm, the object being in a position invisible to the control computer.

15. 2. The system of claim 1, wherein the control computer is configured to ensure that the first robot arm and the second robot arm do not collide with each other or with obstacles in a workspace in which the first robot arm and the second robot arm are located when performing a pick-and-place task.

16. 1. A method comprising: picking and placing one or more of a plurality of objects using a first robotic arm and a second robotic arm, including cooperatively operating with the first robotic arm and a second robotic arm to pick and place the one or more of the plurality of objects; wherein picking and placing the plurality of objects using the first robot arm and the second robot arm comprises moving one or more of the plurality of objects out of the way using one or both of the first robot arm and the second robot arm to allow one or both of the first robot arm and the second robot arm to be used to grasp a given object.

17. 17. The method of claim 16, further comprising utilizing image data from a camera to determine using the first robotic arm and the second robotic arm to cooperatively work to pick and place a given object.

18. 17. The method of claim 16, wherein using the first robotic arm and the second robotic arm to cooperatively work to pick and place a given object includes utilizing the first robotic arm in a leader mode and utilizing the second robotic arm in a follower mode.

19. A computer program product embodied in a non-transitory computer-readable medium, computer instructions for picking and placing one or more of a plurality of objects using a first robotic arm and a second robotic arm, the computer instructions including cooperatively operating with the first robotic arm and the second robotic arm to pick and place one or more of the plurality of objects; and wherein picking and placing the plurality of objects using the first robotic arm and the second robotic arm comprises moving one or more of the plurality of objects out of the way using one or both of the first robotic arm and the second robotic arm to allow one or both of the first robotic arm and the second robotic arm to be used to grasp a given object.

Images