Simultaneous and synchronized pick-and-place by multiple robots

Abstract

The robotic system receives a request from a first agent included in the plurality of robot control agents to provide a motion plan for performing a first pick-and-place task assigned to the first agent, and determines a first motion plan for the first agent, at least in part, by considering a second motion plan associated with a second agent included in the plurality of robot control agents for performing a second pick-and-place task assigned to the second agent, such as by considering a swept volume associated with at least a remaining uncompleted portion of a second motion plan as an obstacle that the first agent will not collide with during execution of the first motion plan.

Description

CROSS-REFERENCE TO OTHER APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 435,085, filed December 23, 2022, entitled "DUAL ROBOT SIMULTANEOUS AND SYNCHRONOUS PICK AND PLACE," which is incorporated herein by reference for all purposes. [Background technology]

[0002] Robots have been provided to perform a variety of tasks, such as handling boxes and other items, including, but not limited to, stacking or removing items from pallets, loading or unloading items from trucks or other containers, singulation or sorting operations, picking items from shelves to form kits, or picking and placing items along a kitting line, or the like.

[0003] To increase throughput, it may be advantageous to use multiple robots to perform such tasks. When multiple robots share a common workspace, care must be taken to pick and place objects from the overlapping robot workspaces without collisions. In a typical approach, this is achieved by having a single robot lock the entire workspace at any one time, resulting in only that one robot occupying the workspace. Under such an approach, robots other than the robot with the overlapping workspace locked remain inactive, or at least are not active in the overlapping portion of the workspace, locking the shared workspace and waiting their turn to operate there. [Brief explanation of the drawings]

[0004] Various embodiments of the present invention are disclosed in the following detailed description and the accompanying drawings.

[0005] [Figure 1] FIG. 1 is a block diagram illustrating one embodiment of a system for picking and placing using multiple robotic agents simultaneously and synchronously.

[0006] [Figure 2] 1 is a flow chart illustrating one embodiment of a process for picking and placing using multiple robotic agents simultaneously and synchronously.

[0007] [Figure 3] FIG. 1 is a block diagram illustrating an embodiment of a motion orchestrator with a system for picking and placing using multiple robotic agents simultaneously and synchronously.

[0008] [Figure 4] 1 is a flow chart illustrating one embodiment of a process for picking and placing using multiple robotic agents simultaneously and synchronously.

[0009] [Figure 5A] A diagram showing an example of a pick / place scenario using multiple robotic agents simultaneously and synchronously.

[0010] [Figure 5B] A diagram showing an example of a pick / place scenario using multiple robotic agents simultaneously and synchronously.

[0011] [Figure 5C] A diagram showing an example of a pick / place scenario using multiple robotic agents simultaneously and synchronously.

[0012] [Figure 5D] A diagram showing an example of a pick / place scenario using multiple robotic agents simultaneously and synchronously.

[0013] [Figure 6A]FIG. 1 is a block diagram illustrating an embodiment of a control loop comprising a system for picking and placing using multiple robotic agents simultaneously and synchronously.

[0014] [Figure 6B] FIG. 1 illustrates an embodiment of a robotic system in which each robot / agent has its own synchronous robot motion control loop. DETAILED DESCRIPTION OF THE INVENTION

[0015] The present invention may be embodied in various forms, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor configured to execute instructions stored in and / or provided by a memory coupled to the processor. These embodiments, or any other form the present invention may take, may be referred to herein as technology. In general, the order of steps in a disclosed process may be varied within the scope of the present invention. Unless otherwise noted, components, such as a processor or memory, described as configured to perform a task may be implemented as general components temporarily configured to perform the task at a given time, or as specific components manufactured to perform the task. As used herein, the term “processor” refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0016] The following is a detailed description of one or more embodiments of the present invention with reference to figures that illustrate the principles of the invention. While the present invention has been described in connection with such embodiments, it is not limited to any particular embodiment. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications, and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. These details are for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For simplicity, technical matters that are well known in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the present invention.

[0017] Techniques are disclosed for utilizing multiple robots to work simultaneously in a shared workspace without collision. In various embodiments, the robots' movements are synchronized to ensure no collisions occur. In some embodiments, a swept volume associated with the predicted movement of a first robot and its payload (e.g., a grasped item) through (the remainder of) a first trajectory assigned to the first robot is considered when determining a second trajectory to be assigned to a second robot to perform the task. The second trajectory is calculated to avoid intersecting the swept volume, for example. In some embodiments, the swept volume associated with the first robot and its first trajectory is included in a "collision world" or other representation of obstacles that need to be avoided when determining a second trajectory for the second robot. In various embodiments, the swept volume is updated based on one or more of progress information reported by the first robot (e.g., the distance the first robot has traveled along the first trajectory) and perception system information (e.g., computer vision information based on cameras or other sensors in the workspace and / or on the robot). For example, an updated swept volume may be determined based on updated information from the first robot and / or perception, allowing a second (or subsequent) trajectory to be determined through a space that the first robot has already traversed and / or is predicted to traverse by the time a second robot following a second trajectory reaches the same space.

[0018] The terms "robot" and "agent" are used interchangeably herein to refer to robotically controlled devices such as robotic arms, Cartesian robots, mobile robots (such as robotic arms mounted on a mobile chassis or robotic transport device (such as a robotic cart or platform or forklift)), robotically controlled conveyors or other transport structures, robotic actuation structures for gating, dividing, blocking, separating, releasing, or otherwise moving one or more items or streams or other sets of items. In various embodiments, combinations of robotically controlled "agents" may be utilized and operated synchronously without collision as disclosed herein.

[0019] In various embodiments, the systems disclosed herein include a centralized orchestrator through which each agent is driven to consider, during execution, the planned movements of other agents and their raw progress along their movement trajectories. In various embodiments, any agent can request a movement from the centralized orchestrator, which will provide a collision-free trajectory (movement plan) that is optimal with respect to time and takes into account the current and future states of all other agents along their respective trajectories.

[0020] In various embodiments, each trajectory is precisely executed and controlled not only in space but also in time to avoid collisions.

[0021] In various embodiments, the centralized orchestrator periodically receives information from the perception system so that it plans the agent's movements with the most up-to-date state of objects that the agent may collide with.

[0022] FIG. 1 is a block diagram illustrating one embodiment of a system for simultaneously and synchronously picking and placing multiple robotic agents. In the illustrated example, robotic system 100 includes a first robot (agent) 102 and a second robot 104 mounted (in this example, at fixed locations) on either side of a pick conveyor 106. In some embodiments, pick conveyor 106 includes a robotically controlled transport structure that robotically moves items to positions where the robots 102 and 104 (e.g., in this example, robot arms with grippers at their ends) can grasp the items. For example, a vision system (or other perception system) may monitor the pick area of ​​pick conveyor 106 and advance pick conveyor 106 as needed to move additional items into the area to be picked. In the illustrated example, reach circles 108 and 110 indicate the reachable areas of robots 102 and 104, respectively.

[0023] In various embodiments, the system 100 is configured to use the robots 102, 104 to pick items from a pick area of ​​the pick conveyor 106 and place each item in a corresponding destination location in a place structure or area 112. Examples of place structures or areas 112 include, but are not limited to, output conveyors or other transport structures (e.g., in singulation or sortation applications), pallets, bins, containers, or other receptacles (e.g., in palletization or kitting applications), trucks or other shipping containers, etc.

[0024] In the illustrated example, reach circles 108 and 110 overlap at a first overlap area of ​​pick conveyor 106 and a second overlap area of ​​destination area or structure 112. While such overlap allows robots 102, 104 to be used to both pick items from and place items into the overlap area, such overlap also creates a risk of collision. In various embodiments, the techniques disclosed herein are utilized to enable high-throughput, collision-free item picking / placing using robots 102, 104.

[0025] In the scenario shown in FIG. 1, a series of eight boxes (numbered "1" through "8") are present on the pick conveyor 106. In this example, robot 102 is tasked with moving box "1" from the source location shown in the figure to the destination location shown in dashed line via trajectory 114. Similarly, robot 104 is tasked with moving box "3" from the source location shown in the figure to the destination location shown in dashed line via trajectory 116. In this case, both robots 102, 104 pick from the area where their reach circles 108, 110 overlap, but depending on the location of the boxes and the destination locations, the robots may work simultaneously to pick their assigned boxes and move them to their destination locations via their assigned trajectories. In other cases, the techniques disclosed herein may be used to determine and assign trajectories for each robot 102, 104 to perform its assigned tasks without risk of collision. Examples include, but are not limited to, when one of the robots 102, 104 is assigned to move first, for example, to move itself and / or an item it is assigned to pick / place out of the way in order to create a collision-free path for the other robot to perform its assigned task. These and other scenarios are described in more detail below.

[0026] Referring to the control elements at the bottom of FIG. 1 , the system 100 includes a first robot controller 120 associated with the first robot 102 and a second robot controller 130 associated with the second robot 104. Each of the robot controllers 120, 130 receives high-level tasks 122, 132 assigned to that robot, for example, by a high-level scheduler or application. For example, in the example shown at the top of FIG. 1 , an item and task selection process, based on, for example, computer vision information, may have determined that boxes "1" and "3" are to be picked / placed next and each moved to a corresponding destination location as shown at the top of FIG. 1 . The same process may have determined that the task of moving box "1" is assigned to robot 102 and the task of moving box "3" is assigned to robot 104. Each respective controller 120, 130 includes states and associated processes 124, 134 for moving and remaining in a reset position (in some embodiments, the reset pose of both robots 102, 104 may be assumed to be a fixed reset pose in each robot's non-overlapping operating workspace) until tasked, states and actions 126, 136 for picking an assigned item, for example, by moving from a reset to a pick position along an assigned trajectory, with the end effector positioned in a position and orientation associated with the strategy for grasping the item, and place states and actions 128, 138.

[0027] In a typical conventional system, the controllers 120, 130 of each robot 102, 104 may operate independently to pick and place assigned items. To work in a shared portion of the workspace, the robots 102, 104 may acquire or be given a "lock" to operate exclusively in the shared area, with the other robots waiting while the lock is in effect. Using the technology disclosed herein, in various embodiments, the robots 102, 104 may be operated synchronously / concurrently so that each performs tasks in a manner that does not risk collision with the other robots.

[0028] 1 , in the illustrated example, upon being assigned a task 122, 132, each robot controller 120, 130 requests a trajectory from the motion orchestrator 140 for the robot 102, 104 associated with that controller 120, 130 to pick and place the item (122, 132) that it has been assigned to pick and place. The motion orchestrator determines whether a collision-free trajectory can be assigned, and if so, what the trajectory should be, using one or more of the following: (1) input from a perception system 142 (e.g., a computer vision system) that assesses the state of the workspace, e.g., items present in the workspace and agents operating within the workspace, based on data generated by a 3D camera or other sensor present in the workspace; (2) a robot model 144 representing factors including the motion dynamics and / or other capabilities and / or limitations of each robot 102, 104 and / or other robot-controlled agent within the workspace; and (3) progress reports from the robots or other agents that have been assigned a plan / trajectory for making the move.

[0029] Based on the perceptions, inferences, and / or reported states of the workspace and the agents operating therein, the motion orchestrator 140 uses the techniques disclosed herein to generate a corresponding response for each received outstanding request, which in various embodiments may include a rejection of the task (e.g., a collision-free trajectory or motion plan does not exist and / or cannot be determined), a wait command, and, ideally, a motion plan that defines a collision-free trajectory for performing the task assigned to the robot with which the request is associated.

[0030] As mentioned above, in various embodiments, a robot or other agent to which motion orchestrator 140 has provided a motion plan may report its progress. The progress report may be used to update the "collision world" used to process subsequent and / or outstanding requests from other agents, for example, by reducing the "swept volume" as described above to exclude space already traveled by the robot (including its components and grasped items, if any).

[0031] 1, there are two robots 102, 104, although in other embodiments, three or more robots may be present in the same workspace. In the example shown in FIG. 1, the robots 102, 104 and other agents (e.g., 106) are shown in fixed locations, although in other embodiments, one or more mobile robots or other mobile agents may be utilized and controlled as disclosed herein. In various embodiments, the techniques disclosed herein may be used to control two or more robots and / or other agents that may otherwise collide or otherwise interfere with each other to provide synchronous / simultaneous collision-free movement with high utilization and throughput.

[0032] 2 is a flow chart illustrating one embodiment of a process for simultaneously and synchronously picking and placing multiple robotic agents. In various embodiments, process 200 of FIG. 2 may be performed by a control computer, such as a computer configured to provide a motion orchestrator (such as motion orchestrator 140 of FIG. 1). In the illustrated example, at step 202, an ordered list of pickable items is received. For example, in the example shown in FIG. 1, a list of pickable items numbered consecutively from "1" to "8" may be determined.

[0033] In some embodiments, the ordered list is a list determined by the perception system as the most preferred boxes to pick first. Example criteria utilized include, but are not limited to, proximity to the place location (faster to pick and place, aids in the forward movement of the conveyor), overlapping or being underneath another box, etc. With further reference to FIG. 1 , an example ordered list received in operation 202 is {1, 2, 3, 4, 5, 6}, indicating that those six boxes can be picked up by both robots 102, 104 in the preferred order, as determined by the perception system.

[0034] In step 204, a placement order is determined. For example, all possible placement pairs (or sets of n items, where n=2 in this example, if n agents are present and available) may be considered, and a particular pair (e.g., {1, 3} in the example shown in FIG. 1 ) may be selected to be picked / placed first. In various embodiments, a decision-making algorithm chooses pairs based on one or more criteria associated with the operation / application being performed with the robotic system. For example, in the case of packing containers or stacking onto pallets, pairs that result in higher packing density and stability of the packed boxes may be selected (e.g., based on size, weight, dimensions, stiffness, or other attributes). A respective placement location is determined for each item / box. In this example, the pair {1, 3} is selected to be placed (first / next), and each placement location is within the reach of one or both of the robots.

[0035] In step 206, robots (agents) are assigned to specific tasks. For example, robot 102 may be assigned to pick / place box "1," while robot 104 is assigned to pick / place box "3." In this example, the pick location for each box is within the reach of both robots, but the place location for box "1" is only within the reach of robot 102, and the place location for box "3" is only within the reach of robot 104. In other examples where the place locations for both items are within the reach of both robots, other selection criteria may be applied, such as which robot has a clear path to pick a given item, which robot is closer to the item, etc.

[0036] In various embodiments, steps 202, 204, and 206 of FIG. 2 may be performed by one or more higher level processes (eg, running on a control computer).

[0037] At step 208, each agent assigned a task makes a request to the motion orchestrator to obtain a motion plan (e.g., a trajectory) for executing its task. At step 210, the motion orchestrator determines collision-free paths (motion plans, trajectories, etc.) for both (or all n) robots / agents. In various embodiments, the robots / agents' motions are planned to maximize the collective availability and / or throughput of all robots / agents. In some embodiments, motions are planned so that the robots / agents operate simultaneously as safely as possible. If a safe and collision-free motion plan including simultaneous motions is not possible, the most globally efficient / optimal set of motion plans may be generated that requires the robots / agents to operate (at least partially) sequentially. At step 212, each collision-free path determined by the motion orchestrator is returned to the respective robot / agent controller (e.g., 120, 130 in FIG. 1 ).

[0038] In various embodiments, the motion plan generated in step 210 may include a plan for a robot / agent to move sequentially if one or more of the following is true: ● When the robot collides at the end of its motion (for example, at the pick position when moving from reset to pick). In some embodiments, partially synchronized picks / places may be considered, for example, a motion plan may be returned that includes a first robot (e.g., 102) picking from reset while a second robot (e.g., 104) moves from pick to place. The place locations for a pair of boxes are within overlapping workspaces. Although there may be paths where both robots do not collide when placing in overlapping reachable areas, this increases the complexity of finding such a solution, so in various embodiments, a motion plan for sequential pick / place may be generated in such cases (where the place locations are within overlapping workspaces).

[0039] In contrast, situations in which simultaneous / synchronous pick-and-place motion plans may be determined in step 210 include when the place location of the first box in the pair is exclusively within the reach circle of the first robot and the place location of the second box is exclusively within the reach circle of the second (or nth) robot.

[0040] FIG. 3 is a block diagram illustrating one embodiment of a motion orchestrator comprising a system for simultaneously and synchronously performing pick / place using multiple robotic agents. In the illustrated example, motion orchestrator 140 of FIG. 1 receives a request to perform a pick / place task (e.g., pick a particular item from its pick location, move it to its place location, and place it there) from, for example, robot controllers 120, 130. At 302, motion orchestrator 140 determines whether the final pose (i.e., the final pose of the requesting robot and the item it is carrying) is collision-free. If not collision-free, for example, if another robot / agent is en route to or through the same location, the request is rejected. In various embodiments, rejection of the request may trigger exception handling, such as assignment of a different or modified task to place the item at a different location or to pick / place a different item.

[0041] To make decisions and / or perform other actions in 302, the motion orchestrator 140 utilizes the composite state 304 of all motions of all robots / agents in the workspace. Such information may be obtained from motion plans (paths, trajectories) assigned to the robots / agents by the motion orchestrator 140, progress reported by such agents, and information from the perception systems 142.

[0042] If the final pose is determined to be collision-free in step 302, then at 306, the motion orchestrator 140 determines a path for each request, taking into account all possible swept obstacles across all current and future states. For example, if requests are received to pick / place boxes "1" and "3" in the example above, the motion orchestrator 140 may consider multiple possible motion plans (paths) for each robot and, for each, calculate the swept obstacles associated with that path. At 308, the paths and associated swept obstacles are considered to determine an optimal collision-free path (or set of collision-free paths) for each request, to return as a response 310. The selected and assigned path is used to update 312 the composite state information 304, for example, to include initial swept obstacle information for the selected / assigned path. Progress reports from the robot / agent and / or perception information 142 may then be used to update the swept obstacles (e.g., to exclude regions of space that the robot / agent has already traveled).

[0043] In various embodiments, motion planning for a collision-free path, as disclosed herein, may include one or more of the following: Find collision-free paths for each robot using a motion planning method to find collision-free robot states, and find trajectories from the start to the goal. -Construct a collision world and check each robot state and whether it collides with obstacles in the space. In some embodiments, each of the robot's links and the box / item it is carrying is represented as a simple convex shape at each joint configuration that corresponds to a point on the path. · Represent the robot link with the box in a given pose as an obstacle. The swept obstacle space is defined as the union of these objects in the collision universe at each pose along the trajectory. The resolution of these obstacles can be coarsely represented and does not need to be high resolution. The higher the resolution, the closer the robots can move to each other without colliding. This resolution can be as wide as an entire robot link or as small as a buffer around the obstacles during collision checking. This swept obstacle represents the volume of space occupied by the robot arm following the path. Use this swept obstacle as part of a collision universe for other robots to plan paths that avoid collisions with it. If a path exists, it means that as long as both robots follow the determined path, their paths will not overlap and the robots can run synchronously. Path discovery and obstacle generation can be done in parallel with the robot executing motion, thereby hiding the time required for synchronous motion planning. This makes finding collision-free trajectories faster and post-processing them smoother. Planning time can range from 5-40 ms per query in some embodiments. Input: Robot start pose, robot goal pose, collision world, shape of object held by robot. · Output: A set of attitudes representing a collision-free trajectory. This motion plan may be utilized to determine synchronous robot motion using the swept obstacle technique disclosed herein. · A swept obstacle workspace ("world") is generated for a given path of the robot.

[0044] FIG. 4 is a flowchart illustrating one embodiment of a process for simultaneously and synchronously performing pick / place using multiple robotic agents. In various embodiments, process 400 of FIG. 4 may be performed by a motion orchestrator (such as motion orchestrator 140 of FIG. 1). In the illustrated example, at step 402, a set of poses constituting a collision-free path (trajectory) determined for the robot / agent is used to generate a swept obstacle space. When progression along the path / trajectory is repeated at step 404, at step 406, the swept obstacle space determined at step 402 is reduced, e.g., to exclude space already traveled by the robot / agent. The next update of the swept obstacle space occurs when / if a subsequent progress report is received (steps 404, 406) until process 400 is completed (step 408), e.g., until the robot / agent reports and / or is observed to have completed movement through its assigned trajectory.

[0045] In various embodiments, future states may be considered when determining a collision-free trajectory. For example, in some embodiments, a motion orchestrator disclosed herein may predict and consider future states (e.g., states in which a first robot has traveled through a first set of states that constitute its assigned trajectory) when planning a path for a second robot. In some cases, a leader robot / agent may be selected and assigned a first trajectory that includes a first set of states (postures, positions, expected times, etc.). The trajectory of the second robot may be determined based on the predicted future states of the first robot. For example, the trajectory of the second “follower” robot may include a wait period, e.g., upon reset, to allow the first robot to travel through a first portion of its trajectory, after which the second robot moves through its assigned trajectory. The first and second trajectories may overlap in space, but by designating the first robot as the “leader,” the trajectories do not overlap in space and time, resulting in a collision-free path for both.

[0046] In various embodiments, the leader and / or follower robots / agents may be determined and / or controlled as follows: ●The movement orchestrator is general in the sense that the robot / agent can autonomously request movement at any time, i.e. the agent can decide where it wants to move and the orchestrator will reject / accept the agent's request from that point onwards. However, in some cases, it may be beneficial to introduce the concept of a leader and follower to further optimize the coordinated motion in the case of two robots. One robot (the leader) can lead the other robot (the follower), which means that the follower robot is subject to the swept volume obstacles of the leader robot. • There can be many strategies that can be applied to determine leaders and followers. Here we present a simple strategy. • Leader and follower decisions should take into account: ○Starting and finishing positions ○ The following robot movements, in this case reset to pick, pick to place, and place to reset. ●Movement of the first and second robots from reset to pick In this case, it is preferable that the robot picking the furthest from the place position is the leader. The closest robot is the one that will not be blocked in its next move from pick to place if the follower becomes the leader, for the simple reason that it will not be blocked in its next move from pick to place. ●About the pick-to-place movement The robot closest to the place position is the leader. This ensures that the robot is not prevented from continuing to place.

[0047] Figure 5A illustrates an example pick / place scenario using multiple robotic agents simultaneously and synchronously. In the illustrated example, robots 502, 504 are configured to pick items from a conveyor 506 and place the items in a place area 508. Robot 502 is assigned to pick box "1" from pick location 510 and place it in place location 512, while robot 504 is assigned to pick box "3" from pick location 514 and place it in place location 516. In the scenario shown in Figure 5A, the discovered paths 518, 520 and their swept obstacle workspaces are non-overlapping and can be executed together.

[0048] FIG. 5B illustrates an example pick / place scenario using multiple robotic agents simultaneously and synchronously. In this illustration, the upper portion represents the state at time zero, when robot 504 has not yet begun moving box "1" along trajectory 538 assigned to move box "1" from pick location 528 to place location 530. Robot 502 has been assigned trajectory 536 to move box "3" from pick location 523 to place location 534. In this scenario, the swept obstacle spaces 531, 532 associated with each trajectory 536, 538 overlap, making it impossible for robots 502, 504 to simultaneously follow their assigned trajectories without colliding. As a result, in this example, robot 504 is selected to begin executing its motion plan 538 first. As shown in the center portion of Figure 5B, robot 502 begins its movement as robot 504 moves through the first portion of its trajectory 538 (i.e., states s(start) and s(1)). By the time robot 502 begins its movement, as shown in the center portion of Figure 5B, swept obstacles 531 associated with the remainder of robot 504's trajectory 538 have been reduced to updated / remaining swept obstacles 533, which no longer overlap with swept obstacles 532 of trajectory 536 assigned to robot 502.

[0049] In some embodiments, further optimization may be achieved by assigning a partial path to robot 502 to start "early" (i.e., before robot 504 progresses to state s(2)), for example, by following a partial path that avoids overlapping robot's 504's original swept obstacle 531. Robot 502 returns to original trajectory 536 once robot's 504's swept obstacle is cleared.

[0050] FIG. 5C illustrates an example pick / place scenario using multiple robotic agents simultaneously and synchronously. In the illustrated example, robot 502 is attempting to pick box "3" (544), which is farthest from its place location (i.e., farther than box "1" (542)). In various embodiments, parallel motions can be performed as long as the final poses of each pick (i.e., robot 502 with respect to box "3" and robot 504 with respect to box "1") do not overlap. For example, because box "3" is far from both robots 502 and 504 and its place location, robot 502 may be designated the leader for the pick (reset-to-pick motion), but once both have completed the pick, robot 504 may be instructed to place (pick-to-place motion) first, thereby moving out of the way and allowing robot 502 to follow the most (or more) efficient or direct pick-to-place path.

[0051] FIG. 5D illustrates an example pick / place scenario using multiple robotic agents simultaneously and synchronously. In the illustrated example, two different motions are sequenced. For example, robot 504 may be performing its pick-to-place motion while robot 502 is executing its reset-to-pick motion to pick box "3" (570). By the time robot 502 is ready to move from pick to place, robot 504 has already moved to the position and state shown in FIG. 5D. That is, box "1" has been moved from pick position 562, associated with original swept obstacle 566, to the intermediate position in the diagram with associated updated swept obstacle 568. Robot 502 then has an obstacle-free path to pick and move box "3," while robot 504 continues placing box "1" at place position 564.

[0052] 6A is a block diagram illustrating one embodiment of a control loop comprising a system for picking and placing using multiple robotic agents simultaneously and synchronously. In the illustrated example, each robot 602, 604 is controlled ("motion space control") by an associated control loop 606, 608 operating at 1 kHz, while a synchronous dual (or other multiple) robot motion control loop 610 implementing the techniques disclosed herein operates at 500 Hz.

[0053] In various embodiments, the systems disclosed herein precisely control and follow desired trajectories for both (or all n) of the robots / agents. In the example shown in FIG. 6A, this is achieved by commanding both robots / agents at smaller differential time steps (e.g., at a rate of 500 Hz). In various embodiments, the robot / agent trajectories are splined and discretized to allow them to be followed with sub-centimeter accuracy, where each point in the trajectory has a pose defined in terms of the position and orientation of the end effector mapped to the corresponding joint configuration.

[0054] 6B illustrates an embodiment of a robotic system in which each robot / agent has its own synchronous robotic motion control loop. In the illustrated example, each robot / agent 622, 624 is controlled ("motion space control") by an associated control loop 626, 628 operating at 1 kHz, and each has its own synchronous robotic motion control loop 630, 632 implementing the techniques disclosed herein and operating at 500 Hz.

[0055] In various embodiments, two or more robots / agents may be operated synchronously / concurrently using a single shared outer motion control loop (e.g., 610 in FIG. 6A ) in certain situations. For example, if a robot includes two robot arms on either side of a robot-controlled central conveyor, and the robot arms are used to cooperatively pick / place large and / or heavy items, the motion plans provided by the motion orchestrator for all three robots / agents (robot arm 1, robot arm 2, and the conveyor) may be implemented under the monitoring and control of a single outer control loop. In contrast, for robots / agents that operate independently but within the same workspace, each may be assigned a motion plan, and the execution of the plan may be performed and monitored by a corresponding robot / agent-specific outer control loop, as in the example shown in FIG. 6B . In some such embodiments, a single outer-outer control loop may be provided to coordinate the execution of each motion plan synchronously by the robot / agent-specific robot / agent motion control loops 630, 632 shown in FIG. 6B .

[0056] In some embodiments, the approach shown in Figure 6A or the approach shown in Figure 6B may be dynamically selected and configured depending on the robots / agents and the pick / place or other tasks assigned to them. When close coordination is required, a single outer control loop may be used, as in Figure 6A, because the robots / agents are working in close proximity to each other and / or to perform a collaborative task. In contrast, when the same robots / agents are assigned to perform separate, independent tasks that do not require close coordination, each may be dynamically assigned to be controlled by a dedicated corresponding outer control loop, as in Figure 6B.

[0057] 6A and 6B illustrate two robots / agents, the same principles may be applied to any number of robots / agents. In some embodiments, the approaches shown in Figures 6A and 6B may be mixed and matched, depending, for example, on the relative degree of coordination required between two or more robots / agents in a given group.

[0058] While the specific examples described herein describe position control (e.g., defining a trajectory in terms of the position and orientation of the robot and its components and payload), in various embodiments, the techniques disclosed herein may be used to define, follow, or control a trajectory using velocity control or other derivatives of position (acceleration, jerk, etc.).

[0059] In various embodiments, the techniques disclosed herein enable maximizing the utilization of all robots (or other agents) through a computationally simplified approach, thereby maximizing total system throughput. Robot idle time is minimized because a robot does not need to remain idle or be utilized suboptimally while another agent has the shared workspace locked.

[0060] Although the above-described embodiments have been described in some detail for ease of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and are not intended to be limiting.

Claims

1. 1. A robotic system comprising: a communication interface; a processor connected to the communication interface; Equipped with The processor: receiving, via the communication interface, a request from a first agent included in a plurality of robot control agents associated with a shared workspace, the request providing a motion plan for executing a first pick-and-place task assigned to the first agent; determining a first motion plan for a first agent to perform the first pick-and-place task assigned to the first agent, such as by considering a second motion plan associated with a second agent included in the plurality of robot controlling agents for performing a second pick-and-place task assigned to the second agent; the first motion plan includes the first agent at least partially performing the first pick-and-place task while the second agent is performing the second pick-and-place task, and the second motion plan associated with the second agent is considered, at least in part, by considering a swept volume associated with at least a remaining, uncompleted portion of the second motion plan as an obstacle that the first agent will not collide with during execution of the first motion plan.

2. The system of claim 1 , wherein the first agent comprises a robotic arm.

3. The system of claim 1 , wherein the first agent comprises a mobile robot.

4. 10. The system of claim 1, wherein the swept volume is calculated at least in part by representing one or both of the elements comprising the second agent and the payload to be picked and placed by the second agent as simple convex shapes.

5. 5. The system of claim 4, wherein the second motion plan includes a set of end effector positions and orientations and associated joint configurations at each of a plurality of state points that make up the second motion plan.

6. 6. The system of claim 5, wherein the swept volume represents a space occupied by the second agent or any portion of the payload as the second agent moves the payload through the workspace according to the second motion plan.

7. 2. The system of claim 1, wherein the processor is further configured to instruct the first agent to begin implementing the first motion plan at a time associated with the second agent completing at least a first portion of the second motion plan.

8. 3. The system of claim 2, wherein the processor is configured to receive, via the communication interface, a progress report from the second agent indicating progress made by the second agent in implementing the second movement plan.

9. 2. The system of claim 1, wherein the first motion plan and the second motion plan are determined simultaneously before the first motion plan is assigned to the first agent or before the two motion plans are assigned to the second agent.

10. 10. The system of claim 9, wherein the processor is further configured to select the second agent to begin executing the second motion plan before the first agent begins executing the first motion plan.

11. The system of claim 1 , wherein the processor is further configured to determine an ordered list of items to be picked and placed within the workspace.

12. 12. The system of claim 11, wherein the processor is further configured to determine a placement order in which items are to be picked and placed based at least in part on the ordered list.

13. 13. The system of claim 12, wherein the placement order is determined, at least in part, by considering a plurality of n-tuples of items, where n corresponds to a number of agents in the plurality of robotic control agents that can pick and place items in the workspace, and selecting an n-tuple of items to be picked or placed first or next.

14. 14. The system of claim 13, wherein the processor is further configured to determine whether the items comprising the selected n-tuple are candidates to be synchronously picked and placed.

15. 15. The system of claim 14, wherein the processor is configured to determine that the items comprising the selected n-set are not candidates to be synchronously picked and placed based at least in part on determining that the respective pick and / or place locations of any two or more items comprising the n-set overlap.

16. The system of claim 1 , wherein the first movement plan is determined based at least in part on state information received from a computer vision or other perception system.

17. The system of claim 1 , wherein the request is received from a first robot controller associated with the first agent.

18. 18. The system of claim 17, wherein the first robotic controller operates a control loop at a rate of about 1000 Hz, and the processor is configured to monitor and control synchronous execution of the first and second motion plans at a rate of about 500 Hz.

19. 1. A method comprising: receiving, via a communication interface, a request from a first agent included in a plurality of robot control agents associated with a shared workspace to provide a motion plan for executing a first pick-and-place task assigned to the first agent; determining, with a processor, a first motion plan for a first agent to perform the first pick-and-place task assigned to the first agent, such as by considering a second motion plan associated with a second agent included in the plurality of robot controlling agents for performing a second pick-and-place task assigned to the first agent; Equipped with the first motion plan includes the first agent at least partially performing the first pick-and-place task while the second agent is performing the second pick-and-place task, and the second motion plan associated with the second agent is considered, at least in part, by considering a swept volume associated with at least a remaining, uncompleted portion of the second motion plan as an obstacle that the first agent will not collide with during execution of the first motion plan.

20. A computer program product embodied in a non-transitory computer-readable medium, computer instructions for receiving, via a communications interface, a request from a first agent included in a plurality of robotic control agents associated with a shared workspace to be provided with a motion plan for executing a first pick-and-place task assigned to the first agent; computer instructions for determining a first motion plan for a first agent to perform the first pick-and-place task assigned to the first agent, such as by considering a second motion plan associated with a second agent included in the plurality of robotic controlling agents for performing a second pick-and-place task assigned to the second agent; Equipped with the first motion plan includes the first agent at least partially performing the first pick-and-place task while the second agent is performing the second pick-and-place task, and the second motion plan associated with the second agent is considered, at least in part, by considering a swept volume associated with at least a remaining, uncompleted portion of the second motion plan as an obstacle that the first agent will not collide with during execution of the first motion plan.

Images