Controlling multiple robots to cooperatively unload trucks or other containers

Abstract

A robotics system is disclosed for controlling multiple robots to cooperatively load / unload a truck or other container. In various embodiments, image data is received and utilized to control a first robotic arm and a second robotic arm to cooperatively load or unload objects from a truck or other container, including using the first robotic arm and the second robotic arm together to cooperatively grasp each of one or more objects of a plurality of objects and move them along corresponding trajectories to load or unload one or more of the objects.

Description

CROSS-REFERENCE TO OTHER APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 274,464, filed November 1, 2021, entitled "CONTROLLING MULTIPLE ROBOTS TO COOPERATIVELY UNLOAD A TRUCK OR OTHER CONTAINER," which is incorporated herein by reference for all purposes. [Background technology]

[0002] Robots have been provided to perform a variety of tasks, such as manipulating objects. For example, a robotic arm having an end effector may be utilized to pick and place items. Examples of commercial applications of such robots include sorting, kitting, palletizing, depalletizing, loading and unloading trucks or containers, etc.

[0003] In some contexts, the objects to be manipulated vary greatly in size, weight, packaging, and other attributes. Typically, a robotic arm is rated to handle up to a maximum size, weight, etc. of an object. In some contexts, conventional approaches may require a robotic arm that can manipulate the largest, heaviest, and / or otherwise most difficult object that may be asked to be manipulated.

[0004] In some contexts, such as loading and unloading a truck or other container, the workspace constrains the movement of the robot. For example, the system must limit the range of motion of the robot arm to prevent any part of the robot from colliding with the interior walls of the truck or container or other obstacles within the workspace. Furthermore, to work inside a truck or other container, the robot must be able to fit and move within the constrained interior space, limiting the size and lifting capabilities of individual robots deployed to work within such spaces. [Brief explanation of the drawings]

[0005] Various embodiments of the present invention are disclosed in the following detailed description and the accompanying drawings.

[0006] [Figure 1] FIG. 1 is a block diagram illustrating an embodiment of a robotic system configured to control multiple robots to collaboratively perform tasks.

[0007] [Figure 2A] FIG. 1 illustrates an example of a collaborative pick-and-place task performed in one embodiment of the robotic system disclosed herein. [Figure 2B] FIG. 1 illustrates an example of a collaborative pick-and-place task performed in one embodiment of the robotic system disclosed herein. [Figure 2C] FIG. 1 illustrates an example of a collaborative pick-and-place task performed in one embodiment of the robotic system disclosed herein.

[0008] [Figure 3] FIG. 1 is a block diagram illustrating an embodiment of a robot control system.

[0009] [Figure 4] FIG. 1 is a state diagram illustrating an embodiment of a robotic system configured to control multiple robots to collaboratively perform tasks.

[0010] [Figure 5A] 1 is a flowchart illustrating an embodiment of a process for collaboratively performing a task as a "leader" robot in an embodiment of a robotic system herein.

[0011] [Figure 5B] 1 is a flowchart illustrating an embodiment of a process for collaboratively performing tasks as "follower" robots in an embodiment of a robotic system herein.

[0012] [Figure 6A] FIG. 1 illustrates an embodiment of a system for controlling multiple robots to cooperatively load / unload a truck or other container.

[0013] [Figure 6B] FIG. 1 illustrates an embodiment of a system for controlling multiple robots to cooperatively load / unload a truck or other container.

[0014] [Figure 6C] FIG. 1 illustrates an embodiment of a system for controlling multiple robots to cooperatively load / unload a truck or other container.

[0015] [Figure 7A] FIG. 1 illustrates an example of robots working independently in one embodiment of a system for controlling multiple robots to cooperatively load / unload a truck or other container.

[0016] [Figure 7B] FIG. 1 illustrates an example of robots working together in one embodiment of a system for controlling multiple robots to cooperatively load / unload a truck or other container. DETAILED DESCRIPTION OF THE INVENTION

[0017] The present invention may be embodied in various forms, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor configured to execute instructions stored in and / or provided by a memory coupled to the processor. These embodiments, or any other form the present invention may take, may be referred to herein as technology. In general, the order of steps in a disclosed process may be varied within the scope of the present invention. Unless otherwise noted, components, such as a processor or memory, described as configured to perform a task may be implemented as general components temporarily configured to perform the task at a given time, or as specific components manufactured to perform the task. As used herein, the term “processor” refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0018] The following is a detailed description of one or more embodiments of the present invention with reference to figures that illustrate the principles of the invention. While the present invention has been described in connection with such embodiments, it is not limited to any particular embodiment. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications, and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. These details are for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For simplicity, technical matters that are well known in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the present invention.

[0019] A robotics system is disclosed for controlling and utilizing multiple robots (e.g., two or more robotic arms) to cooperatively unload or load a truck or other container. In various embodiments, the robotic arms, mounted on a frame or other structure, are positioned within the truck or other container. To unload, the robots pick items from the truck and place them on a robotically controlled conveyor or other transport. For heavy or bulky items, two or more robots are used cooperatively to pick the items and place them on a conveyor or other destination. Once an area of ​​the truck or other container has been unloaded (e.g., all items within the robot's reach have been unloaded, but additional items remain), the frame and the robotic arms mounted thereon, as well as the conveyor, are extended (robotically) further into the truck or other container to move the items comprising the next set of items within the reach of the robotic arms. To load, the frame / robot and conveyor are extended into the truck or other container near the far / rear end of the truck / container, and the robots work cooperatively to load items into the truck / container as they arrive via the conveyor.

[0020] In various embodiments, the systems disclosed herein include two robots along with one or more conveyors between which packages are placed (picked).

[0021] Most common robotic arms mounted close to a wall tend to collide with parts of their arms (e.g., the robot's elbow) against the wall when moving the robot's endpoint / load to a position on the other side of the wall across from the robot. The problem is particularly bad when using the most common types of industrial robots with kinematics based on the first three joints in a roll-pitch-pitch configuration. The problem is exacerbated when two of these arms are positioned alongside an intervening conveyor, thus utilizing them to move much closer to the wall of the track than a single robot / conveyor system. The following description corresponds to the robot positions and tasks that are part of this system as implemented in various embodiments.

[0022] In various embodiments, each robot base is mounted on a mounting surface that is angled away from the wall near the truck or other container. A mounting surface angle of 10° to 20° from horizontal, in various embodiments, significantly reduces the potential for collisions between the robot's elbow and the wall of the truck. This significantly increases the workspace available for picking packages in front of the robot and for placing boxes on one or more conveyors near the centerline of the truck. In various embodiments, this arrangement of each robot's workspace significantly increases the available collaborative workspace (without colliding with walls) for two robots to cooperatively pick packages as described above.

[0023] In various embodiments, the robot moves or is moved into the container / truck, unloads the packages to make space, and then moves or is moved further into the truck. In some embodiments, the robot is attached to a conveyor belt that moves with the robot and onto which the robot places the packages it picks for removal from the truck. Alternatively, there may be a central aisle between the robots, and a series of robotic vehicles may park there to receive packages placed onto them for transport from the truck, and / or a robot-controlled conveyor may be located and controlled there.

[0024] A vision system identifies pickable packages in the scene and provides them to the robots. The robots then prioritize picking packages to prevent collisions between robots so they don't get too close to each other. Motion planning allows the robots to plan paths that avoid robot collisions. In some embodiments, a hierarchical planner / scheduler allows each robot to plan its own movements knowing what the other robots are planning to do, and potential collisions can be identified to avoid and / or trigger re-planning by one or both robots.

[0025] If there is only one output belt to place onto, the robots alternate placing to prevent collisions. If each robot has its own output belt, the robots are allowed to operate at their own speed and do not need to synchronize their placing operations.

[0026] In some embodiments, for objects stacked on top of a truck / container, the robot may pick them from the side. Lower objects are typically picked from above. Large or irregular objects are picked collaboratively by both robots.

[0027] When loading a truck or container, in various embodiments, the robot first moves or is moved completely into the container / truck. Once the robot loads the packages and builds the load, it moves or is moved out of the truck. In various embodiments, the robot may be attached to and / or otherwise work in conjunction with a robotically controlled conveyor belt, which moves with the robot and from which the robot picks packages to load into the truck. Packages may be fed to a conveyor located between the robots from a conveyor or other transport structure extending from the truck / container to a loading dock or other area outside the truck. Packages may be fed to the input transport structure by, for example, a human worker or another robot.

[0028] Picking from a jumbled / mixed pile of packages, such as can occur when unloading a truck packed with various items, can cause other packages to shift / tip / fall, In some embodiments, a vision system is used to identify the new configuration and target the next package before the next pick is made.

[0029] In some embodiments, for purposes of access when gripping out-of-reach packages (such as packages at the top of a stack of packages), the system may identify, plan for, and pick packages that are not at the top in order to move the higher out-of-reach packages (e.g., by gravity) to a lower pickable position.

[0030] FIG. 1 is a block diagram illustrating one embodiment of a robotic system configured to control multiple robots to collaboratively perform a task. In the illustrated example, system and environment 100 includes a first robotic arm 102 with a suction-type end effector 104 and a second robotic arm 106 with a suction-type end effector 108. As shown, robotic arm 102 and robotic arm 106 are positioned to collaboratively perform a pick-and-place task on a large box 110. A control computer 112 is configured to wirelessly communicate with robotic arm 102, robotic arm 106, and one or more cameras or other sensors 114 within a workspace. Image data received from camera 114 may be used by control computer 112 to generate a three-dimensional view of the workspace and send commands and information to robotic arm 102 and robotic arm 106, as needed, to facilitate the collaborative pick-and-place task, for example.

[0031] 2A-2C illustrate an example of a collaborative pick-and-place task performed in one embodiment of the robotic system disclosed herein. In the example shown in FIG. 2A, robotic arm 202 with suction-type end effector 204 and robotic arm 206 with suction-type end effector 208 are positioned to begin collaboratively performing a pick-and-place task on large box 210, similar to the start state shown in FIG. 1. In various embodiments, robotic arm 202 may be the "leader" and robotic arm 206 may be the "follower" in the collaborative pick-and-place disclosed herein. The "leader" may be selected by any suitable method (e.g., by assigning the role of "leader" to the robot that initiated the collaborative task, by randomly assigning the role to one or other of the participating robots, by "election" or other selection method).

[0032] To initiate an operation, in various embodiments, the "leader" robotic arm 202 moves its end effector 204 to the position shown in the figure and then grasps the box, for example, by moving the end effector 204 into contact or near contact with the side of the box 210 and creating suction. A signal may be sent to the other robot (and / or a process controlling the other robot) to indicate that the leader has completed its grasp. The follower (e.g., robotic arm 206 in this example) then grasps the box 210, for example, on the side opposite the side on which the leader (i.e., robotic arm 202) grasped the box 210. The follower records a transformation based on the position and orientation of the leader's end effector 204 and the relevant dimensions of the box 210. For example, a vision system and / or other sensors may be used to measure dimensions or to recognize the box 210 (e.g., specifically and / or by type) and determine the dimensions using the item and / or type information (e.g., by lookup).

[0033] As shown in FIG. 2B , by this point, both robots (202, 206) have grasped the box 210. The leader (in this example, robot arm 202) performs calculations independently of the follower robot arm 206 to move the box along a trajectory determined by the robot arm 202 (and / or its associated control process). In various embodiments, the follower robot (in this example, robot arm 206) receives (e.g., periodically, continuously, etc.) position and orientation information of the end effector 204 of the leader robot arm 202. The follower robot arm 206 (and / or its associated control process) uses the position and orientation information of the leader robots (202, 204) and the previously determined and recorded transformations to calculate a new target position and orientation for the follower end effector 208, and calculates and applies torques to the motors that make up the robot arm 206, as necessary, to minimize the error (difference) between the current position and orientation of the follower end effector 208 and the (most recently updated) target.

[0034] 2C, once the object (box 210) is placed at the destination location, the leader robot (robot arm 202) releases its grip and signals the follower that the pick-and-place task is complete. In response, the follower (robot arm 206) releases its grip, allowing both robots (202, 206) to perform other tasks, such as individually picking and placing a smaller / lighter object and / or collaboratively performing a next pick-and-place task for another larger or heavier object.

[0035] Figure 3 is a block diagram illustrating one embodiment of a robotic control system. In various embodiments, the robotic control system 302 of Figure 3 includes or is included in the control computer 112 of Figure 1. In various embodiments, one or more modules or subsystems that make up the robotic control system 302 of Figure 3 may be distributed across multiple computational nodes, such as the computers and / or processors that make up one or more of the control computer 112, robotic arm 102, and / or robotic arm 106 of Figure 1.

[0036] In the depicted example, robot control system 302 includes a hierarchical planner, scheduler, and / or control module with a robot collaboration facilitation module 304 configured to facilitate the collaborative performance of a task by two or more robots as disclosed herein, and robot-specific controllers 306 and 308. For example, controller 306 for robot 1 may be associated with robot arm 102 of Figure 1 and / or robot arm 202 of Figures 2A-2C, while controller 308 for robot 2 may be associated with robot arm 106 of Figure 1 and / or robot arm 206 of Figures 2A-2C.

[0037] In various embodiments, each robot associated with a respective robot 1 controller 306 and robot 2 controller 308 may operate independently, such as to pick and place an object that the robots can manipulate individually. In various embodiments, a collaborative task using two or more robots may be initiated and / or executed by one or more communications sent between the robot 1 controller 306 and the robot 2 controller 308, by bilateral communications between the robot collaboration facilitation module 304 and the respective robot 1 controller 306 and robot 2 controller 308, and / or by communications between all three (or four or more) entities.

[0038] In the depicted example, the robot control system 302 further includes a computer vision subsystem 310 configured to receive image and depth data from one or more 3D cameras and / or other sensors (such as camera 114 in FIG. 1 ) and use the received data to generate and / or update a three-dimensional view of the workspace. The output of the computer vision subsystem 310 may be provided to one or more of the robot collaboration facilitation module 304, the controller for robot 1 306, and the controller for robot 2 308 to enable the robots to collaboratively initiate and execute tasks to pick and place items. For example, the image data may be used to determine if a box or other object is too large and / or heavy to be picked and placed by a single robot. The three-dimensional view of the workspace and objects therein may be used, for example, to determine a respective grasping strategy and / or position for each robot, determine multiple collision-free trajectories for moving each robot's end effector to its corresponding pick location, and determine collision-free trajectories for collaboratively moving the objects to a destination location where the objects will be placed.

[0039] 4 is a state diagram illustrating one embodiment of a robotic system configured to control multiple robots to collaboratively perform a task. In various embodiments, state diagram 400 of FIG. 4 may be implemented by and / or in connection with a robot configured to collaboratively perform a task using two or more robots. In some embodiments, state diagram 400 of FIG. 4 may be implemented by control computer 112 of FIG. 1 and / or one or more of robot collaboration facilitating module 304, controller for robot 1 306, and controller for robot 2 308 of FIG. 3.

[0040] In the illustrated example, in state 402, the robots work individually to perform a task. For example, the robots may individually pick and place items to fill boxes or other containers in a kitting operation, place items on a conveyor belt or other transport in a sorting operation, stack items on a pallet, etc. Upon receiving an indication that assistance (help) is needed to perform the task (404), such as an indication that an item that needs to be recognized and picked and placed is too large to be grasped and moved by a single robot, the robots and / or controllers transition to state 406, where collaborative execution of the task begins. For example, a communication may be sent to another robot (e.g., from controller 306 for robot 1 to controller 308 for robot 2 in FIG. 3 ) or to a higher-level planner / scheduler (e.g., to robot collaboration facilitation module 304 in FIG. 3 ), or the higher-level planner / scheduler may recognize the need for collaborative execution of the task and initiate the transition to state 406.

[0041] In the illustrated example, the robot and / or controller may return to working independently at state 402 via "cancel assistance" transition 408. For example, the robot / controller and / or higher level planner / scheduler may determine that the task has already been performed by and / or assigned to one or more other robots.

[0042] In some embodiments, in the "begin collaboration" state 406, the robot / controller initiating the collaborative execution of a task communicates directly or indirectly with the helper robot, for example, by requesting assistance. Another robot may be assigned to assist and / or agree to assist. The robot may be assigned to assist and / or agree to assist at some future time or upon the occurrence of a future condition (such as the completion of a task already started by the helper robot and / or a task with a higher priority). For example, to facilitate the collaborative task, a task of removing other objects from around a large or heavy object may have a higher priority and therefore be completed first. When the helper robot is ready to perform the collaborative task, the helper robot notifies the task initiator, directly or indirectly (e.g., via a higher-level planner / scheduler (e.g., robot collaboration facilitation module 304 in FIG. 3 )), that the helper robot is ready, prompting a transition 410 to the "begin collaboration" state 412. A helper may transition directly from the working independently state 402 to the "initiate collaboration" state 412 through the "provide assistance" transition 414 in the illustrated example.

[0043] Once all participating robots are ready in the "Begin Collaboration" state 412, a "leader" is determined, if necessary, and the leader transitions (416) to the "Acting Leader" state 418, while the followers transition (420) to the "Acting Follower" state 422. In the "Acting Leader" state 418 and the "Acting Follower" state 422, the leader and followers collaborate as disclosed herein to cooperatively perform a task, such as picking and placing a large or heavy object, as in the example shown in Figures 2A-2C. Once the task is completed, the leader and followers return (424, 426) to the "Working Individually" state 402 to resume their individual work.

[0044] 5A is a flow chart illustrating an embodiment of a process for collaboratively performing a task as a "leader" robot in an embodiment of a robotic system herein. In various embodiments, the process 500 of FIG. 5A may be performed by a robot controller associated with a robot participating as a "leader" in the collaborative performance of a task by two or more robots disclosed herein.

[0045] In the illustrated example, at step 502, an instruction is received to initiate a collaborative task (with one or more other robots) in the role of a “leader.” For example, an instruction may be received to collaboratively perform a pick-and-place task. At step 504, the leader determines a location to grasp an object and plans a trajectory to safely move its end effector to the position to grasp the object, and at step 506, the leader moves the end effector along the trajectory to the grasping position. At step 508, the leader determines a trajectory (independently of any other robots) to move the object to the associated destination. For example, a model of the robot and its kinematics, as well as images and / or other information about the workspace (e.g., configuration data, CAD files, etc.), one or more attributes of the object (e.g., dimensions, stiffness, etc.), and image / sensor data may be used to plan the trajectory. At step 510, an indication is received from one or more "follower" robots, with which the robots must cooperate, that one or more follower robots are ready to begin collaborative execution of a task. In response, at step 512, the "leader" robot moves its end effector (and the object being jointly grasped by the leader and follower) to a destination along a trajectory determined by the leader. At step 514, once the object is placed at its destination, the leader robot releases its grasp and notifies the follower robots that the task is complete. In various embodiments, the leader then resumes operating independently.

[0046] 5B is a flow chart illustrating an embodiment of a process for collaboratively performing a task as a "follower" robot in an embodiment of a robotic system herein. In various embodiments, process 520 of FIG. 5B may be performed by a robot controller associated with a robot participating as a "follower" in the collaborative performance of a task by two or more robots disclosed herein.

[0047] In the illustrated example, at step 522, an instruction is received to perform a task collaboratively with one or more other robots in the role of a "follower," as disclosed herein. At step 524, the follower determines a grasp point (e.g., a point on the side of the object opposite the side indicated by the "leader" to be grasped) and plans a trajectory to move to a position to grasp the object at that point. At step 526, the follower moves its end effector to the determined grasp position and grasps the object, for example, in response to receiving an indication that the leader has completed its grasp. At step 528, position and orientation information of the leader's end effector is received, and the follower uses this information along with information about the object (e.g., the size of the object in the dimension separating the leader's and follower's end effectors) to calculate a transformation. In various embodiments, the transformation comprises a matrix or other mathematical construct that can be applied to the position and orientation of the leader's end effector (typically expressed in the leader's frame of reference) to provide a corresponding position and orientation for the follower's end effector that maintains the relative position and orientation of the follower's end effector with respect to the leader's end effector as the end effectors and the object grasped therebetween are moved through the workspace to the object's placement destination. At step 530, the follower robot notifies the leader that the follower is "ready" (e.g., the follower is ready to grasp an object, calculate a transformation, and maintain its position relative to (e.g., opposite) the leader's end effector).

[0048] In step 532, once the leader robot begins moving along the trajectory independently determined by the leader, the follower utilizes the transformations it has calculated and the position and orientation information of the leader's end effector that it continuously receives as the leader's end effector is moved through the workspace. For example, for each of at least a portion of the positions and / or orientations of the leader's end effector received, the follower calculates a new target position and / or orientation for that end effector and applies torques to its motors that it determines are necessary to minimize the error (e.g., difference) between the current position and / or orientation of its end effector and the current target.

[0049] In step 534, the follower receives an indication (eg, from the leader) that the collaborative task is "complete," and in response, the follower releases the grip and process 520 ends.

[0050] FIG. 6A illustrates one embodiment of a system for controlling multiple robots to cooperatively load and unload a truck or other container. In the illustrated example, the system and environment 600 includes a truck or other container 602 with a number of boxes stacked inside. While the boxes shown in FIG. 6A are uniform in size and have consistent dimensions, in various embodiments, the load may include objects with various sizes, weights, stiffness, packaging types, and other attributes. While the boxes shown in FIG. 6A are neatly stacked, in various embodiments, the load may include a disorganized pile of items or a mix of relatively neat and less neat items.

[0051] FIG. 6A includes a rear view (bottom left) and a side view (bottom right) of a robotic truck loading / unloading system 604. In the illustrated example, the robotic truck loading / unloading system 604 comprises a frame 606 disposed on a movable and / or mobile chassis 608. In some embodiments, the chassis 608 allows the robotic truck loading / unloading system 604 to be manually moved into position, such as by being pushed into position by a human or robotic operator. In some embodiments, the chassis 608 is self-propelled and may be robotically positioned (e.g., advanced into or out of a truck or other container). For example, once manually positioned at the deepest portion of a truck or container (for loading) or at a rear access opening (for unloading), the chassis 608 may be robotically propelled forward as needed to continue unloading or reverse to a new position to continue loading.

[0052] 6A , in the illustrated example, a first robot 610 and a second robot 612 are positioned on either side of an opening (hole) defined by a frame 606. In various embodiments, the robots 610, 612 are controlled to cooperatively load / unload a truck or other container as disclosed herein. A 3D camera 614 mounted on the top cross member of the frame 606 provides image and depth information that can be utilized by a computer vision system (such as the computer vision subsystem 310 of FIG. 3 ) to provide a three-dimensional view of the interior workspace where the robotic truck loading / unloading system 604 is currently using the robots 610, 612 to cooperatively load or unload a truck.

[0053] FIG. 6B illustrates one embodiment of a system for controlling multiple robots to cooperatively load and unload trucks or other containers. In the illustrated example and state, the robotic truck loading / unloading system 604 of FIG. 6A is positioned at the rear opening of a fully loaded truck 602. For example, the truck 602 may have arrived at a warehouse, distribution center, or the like and backed into a loading dock or loading bay. A robot-controlled conveyor 620 is positioned to deliver boxes unloaded from the truck 602 by the robotic truck loading / unloading system 604 and placed on the conveyor 620 to a downstream location. In some embodiments, the conveyor 620 comprises components and / or systems separate from the robotic truck loading / unloading system 603, e.g., moved into position between the robots 610 and 612 after the robotic truck loading / unloading system 604 has been moved into position. In other embodiments, the conveyor 620 comprises an integral part of the robotic truck loading / unloading system 604 and remains in place between the robots 610, 612 when the chassis 608 is moved to position the robotic truck loading / unloading system 604, for example, as shown.

[0054] 6B, conveyor 620 provides boxes onto a further conveyor 622 that carries boxes (e.g., boxes 624, 626) to further downstream locations where other workers (e.g., humans, other robots) perform further tasks on the boxes unloaded from truck 602, such as unpacking the boxes, further moving the boxes to a storage or staging location, sending them for transportation to other destinations, etc.

[0055] FIG. 6C illustrates one embodiment of a system for controlling multiple robots to cooperatively load and unload a truck or other container. In FIG. 6C, truck 602 is outlined in dashed lines, and robotic truck loading / unloading system 604 is shown deep within truck 602, continuing its unloading operation. As shown, boxes of various sizes remain stacked within truck 602. Boxes 624 and 626 are being picked from within the truck by robots 610, 612 and placed onto conveyor 620, which, in this example, carries the boxes off truck 602 and places them on conveyor 622.

[0056] In various embodiments, as the robotic truck loading / unloading system 604 continues unloading the truck 602, once the boxes within its reach have been unloaded, if more boxes remain to be unloaded, the robotic truck loading / unloading system 604 advances further into the truck 602, e.g., under robotic control and self-propulsion, to position the robots 610, 612 to reach the next layer or other range or set of boxes or other objects.

[0057] In various embodiments, the robotic truck loading / unloading systems disclosed herein (such as robotic truck loading / unloading system 604) are configured to use two or more robots to collaboratively perform tasks associated with loading and unloading truck (or other container) operations. For example, robots 610, 612 may work independently to load and unload objects that can be safely handled by a single robot, but may also be used collaboratively to perform tasks to load and unload large, bulky, and / or heavy boxes or other objects.

[0058] FIG. 7A illustrates an example of robots working independently in one embodiment of a system for controlling multiple robots to collaboratively load / unload trucks or other containers. In the illustrated example, robots 610 and 612 are being used to independently load / unload objects (such as boxes 702 and 704) in an independent operating mode, such as state 402 in FIG. 4 . In various embodiments, robots 610 and 612 work independently but collaboratively when operating in the “independent” operating mode. For example, in various embodiments, robots 610 and 612 alternate picking and placing items to reduce the risk of collisions and the need to wait to avoid collisions. While one robot (e.g., 610) is placing an item on conveyor 620, the other robot (e.g., 612) may be reaching out to grab the next item to be grabbed by that robot. By the time the latter robot is ready to place, the former robot has left the conveyor's placement area. In some embodiments, each robot 610, 612 takes into account the other robot's planned trajectory when planning its own trajectory. For example, each may plan a trajectory that avoids crossing the other robot's intended trajectory. In some embodiments, if a risk of collision is detected, one or both robots may enter a short, independently determined, randomly-length waiting period before resuming operation.

[0059] FIG. 7B illustrates an example of robots working cooperatively in one embodiment of a system for controlling multiple robots to cooperatively load / unload trucks or other containers. In the illustrated example, robots 610 and 612 are used to cooperatively load / unload a large box 706. In the illustrated example and state, robots 610 and 612 are grasping box 706 on both sides. The techniques disclosed above are used by one or more of robotic truck loading / unloading system 604, robot 610, and robot 612 to move box 706 to perform a requested task (e.g., stacking box 706 on one or more other boxes for loading into truck 602 or placing box 706 on conveyor 620 (not shown in FIG. 7B for simplicity) for unloading). For example, robot 610 may act as a "leader" and perform process 500 of FIG. 5A, while robot 612 may act as a "follower" and perform process 520 of FIG. 5B, or vice versa.

[0060] In various embodiments, the techniques disclosed herein may be used to control multiple robots to cooperatively load / unload trucks or other containers.

[0061] Although the above-described embodiments have been described in some detail for ease of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and are not intended to be limiting. [Application Example 1] A robot system, a frame defining an opening having dimensions related to interior dimensions defined by one or more of the top, bottom, and side walls of the truck or other container; a first robot arm and a second robot arm attached to the frame with the opening therebetween; a camera attached to the frame; a control computer configured to utilize the first robotic arm and the second robotic arm and image data received from the camera to load or unload one or more objects from the truck or other container, including using the first robotic arm and the second robotic arm cooperatively to grasp one or more objects of a plurality of objects and move the one or more objects along a trajectory to load or unload the one or more objects; A system comprising: [Application Example 2] A system according to Application Example 1, wherein the control computer executes a first process for operating the first robot arm in a leader operation mode and a second process for operating the second robot arm in a follower operation mode to cooperatively grasp and move the one or more objects along a trajectory. [Application Example 3] A system as described in Application Example 1, wherein the first robot arm and the second robot arm are configured to be used independently to load or unload objects when not being used cooperatively to grasp and move the one or more objects along a trajectory. [Application Example 4] A system according to Application Example 3, wherein the first robot arm and the second robot arm are configured to operate independently to avoid collisions therebetween. [Application Example 5] The system according to Application Example 1, further comprising a chassis to which the frame is attached. [Application Example 6] The system according to Application Example 5, wherein the chassis is self-propelled. [Application Example 7] A system as described in Application Example 6, wherein the control computer is configured to advance the chassis into the truck or other container to access objects located deeper within the truck or other container. [Application Example 8] A system as described in Application Example 6, wherein the control computer is configured to retract the chassis from the truck or other container to position the robot system in loading objects into the next available set of locations within the truck or other container. [Application Example 9] The system according to Application Example 1, further comprising a conveyor disposed between the first robot arm and the second robot arm. [Application Example 10] A system as described in Application Example 9, wherein the control computer is configured to operate the conveyor to move items to a position where they are loaded by the first robot and the second robot. [Application Example 11] A system as described in Application Example 9, wherein the control computer is configured to operate the conveyor to move items dropped off using the first robot arm and the second robot arm to a downstream position. [Application Example 12] A system as described in Application Example 1, wherein the first robot arm is operated in a leader operating mode and the second robot arm is operated in a follower operating mode in which a transformation is applied to a continuous set of information about the position and orientation of a first end effector of the first robot arm to determine an updated target position and orientation of a second end effector of the second robot arm. [Application Example 13] A system as described in Application Example 1, wherein the control computer is configured to use one or both of the first robot arm and the second robot arm to remove a first object from a lower layer of the stack, thereby moving a second object in a higher layer above the lower layer to a position where one or both of the first robot arm and the second robot arm can reach the object. [Application Example 14] A system as described in Application Example 1, wherein the control computer is configured to use the first robot arm and the second robot arm to cooperatively grasp a given object and move it along a corresponding trajectory. [Application Example 15] A system as described in Application Example 14, wherein the control computer is configured to use image data received from the camera to determine how to cooperatively grasp a given object using the first robot arm and the second robot arm and move it along a corresponding trajectory. [Application Example 16] A method, Receives image data from the camera, utilizing the image data to control the first robotic arm and the second robotic arm to load or unload one or more of the objects onto or from a truck or other container, including using the first robotic arm and the second robotic arm together cooperatively to grasp each of one or more of the objects and move them along corresponding trajectories to load or unload one or more of the objects; A method comprising: [Application Example 17] A method as described in Application Example 16, wherein the first robot arm is used in a leader operating mode and the second robot arm is used in a follower operating mode to cooperatively grasp and move the one or more objects along a trajectory. [Application Example 18] A method as described in Application Example 16, wherein the first robot arm and the second robot arm are mounted on a chassis, and the method further comprises advancing the chassis into the truck or other container to access objects located deeper within the truck or other container. [Application Example 19] A method as described in Application Example 16, further comprising operating a conveyor to move items to a position where they are loaded by the first robot and the second robot. [Application Example 20] A computer program product embodied in a non-transitory computer-readable medium, computer instructions for receiving image data from the camera; computer instructions for utilizing the image data to control the first robotic arm and the second robotic arm to load or unload one or more of the objects onto or from a truck or other container, including using the first robotic arm and the second robotic arm together cooperatively to grasp each of one or more objects of the plurality of objects and move the arms along a corresponding trajectory to load or unload one or more of the objects; A computer program product comprising:

Claims

1. 1. A robotic system comprising: a frame defining an opening having dimensions related to interior dimensions defined by one or more of the top, bottom, and side walls of the truck or other container; a first robot arm and a second robot arm attached to the frame with the opening therebetween; a camera attached to the frame; a control computer configured to utilize the first robotic arm and the second robotic arm and image data received from the camera to load or unload one or more objects from the truck or other container, including using the first robotic arm and the second robotic arm cooperatively to grasp one or more objects of a plurality of objects and move the one or more objects along a trajectory to load or unload the one or more objects; Equipped with the control computer is configured to use one or both of the first robot arm and the second robot arm to remove a first object from a lower layer of the stack, thereby moving a second object in a higher layer above the lower layer to a position where one or both of the first robot arm and the second robot arm can reach the object.

2. 2. The system of claim 1, wherein the control computer executes a first process to operate the first robotic arm in a leader operational mode and a second process to operate the second robotic arm in a follower operational mode to cooperatively grasp and move the one or more objects along a trajectory.

3. 10. The system of claim 1, wherein the first robotic arm and the second robotic arm are configured to be used independently to load or unload objects when not being used cooperatively to grasp and move the one or more objects along a trajectory.

4. 4. The system of claim 3, wherein the first robotic arm and the second robotic arm are configured to operate independently to avoid collisions therebetween.

5. The system of claim 1 further comprising a chassis to which the frame is mounted.

6. 6. The system of claim 5, wherein the chassis is self-propelled.

7. 7. The system of claim 6, wherein the control computer is configured to advance the chassis into the truck or other container to access objects located deeper within the truck or other container.

8. 7. The system of claim 6, wherein the control computer is configured to retract the chassis from the truck or other container to position the robotic system in loading objects into a next available set of locations in the truck or other container.

9. 10. The system of claim 1, further comprising a conveyor disposed between the first robotic arm and the second robotic arm.

10. 10. The system of claim 9, wherein the control computer is configured to operate the conveyor to move items to a position to be loaded by the first robotic arm and the second robotic arm.

11. 10. The system of claim 9, wherein the control computer is configured to operate the conveyor to move items deposited using the first robotic arm and the second robotic arm to a downstream position.

12. 10. The system of claim 1, wherein the first robotic arm is operated in a leader operational mode and the second robotic arm is operated in a follower operational mode in which a transformation is applied to a continuous set of information about the position and orientation of a first end effector of the first robotic arm to determine an updated target position and orientation of a second end effector of the second robot arm.

13. 10. The system of claim 1, wherein the control computer is configured to use the first robotic arm and the second robotic arm to cooperatively grasp a given object and move it along a corresponding trajectory.

14. 14. The system of claim 13, wherein the control computer is configured to utilize image data received from the camera to determine when to cooperatively grasp a given object with the first robotic arm and the second robotic arm and move it along a corresponding trajectory.

15. 1. A method comprising: Receives image data from the camera, utilizing the image data to control the first robotic arm and the second robotic arm to load or unload one or more of the objects onto or from a truck or other container, including using the first robotic arm and the second robotic arm together to cooperatively grasp each of one or more objects of a plurality of objects and move them along corresponding trajectories to load or unload one or more of the objects; controlling the first robot arm and the second robot arm includes using one or both of the first robot arm and the second robot arm to remove a first object from a lower layer of a stack, thereby moving a second object in a higher layer above the lower layer to a position where one or both of the first robot arm and the second robot arm can reach the object.

16. 16. The method of claim 15, wherein the first robotic arm is used in a leader operational mode and the second robotic arm is used in a follower operational mode to cooperatively grasp and move the one or more objects along a trajectory.

17. 16. The method of claim 15, wherein the first robotic arm and the second robotic arm are mounted on a chassis, the method further comprising advancing the chassis into the truck or other container to access objects located deeper within the truck or other container.

18. 16. The method of claim 15, further comprising operating a conveyor to move items to a location to be loaded by the first robotic arm and the second robotic arm.

19. A computer program product embodied in a non-transitory computer-readable medium, computer instructions for receiving image data from the camera; computer instructions for utilizing the image data to control the first robotic arm and the second robotic arm to load or unload one or more of the objects into or from a truck or other container, including using the first robotic arm and the second robotic arm together cooperatively to grasp each of one or more objects of a plurality of objects and move the arms along a corresponding trajectory to load or unload one or more of the objects; Equipped with controlling the first robot arm and the second robot arm includes using one or both of the first robot arm and the second robot arm to remove a first object from a lower layer of a stack, thereby moving a second object in a higher layer above the lower layer to a position where one or both of the first robot arm and the second robot arm can reach the object.

Images