Robust motion planning and / or control for multi-robot environments

Abstract

Motion planning generates a motion plan for a robot operating in a shared workspace, the motion plan including a nominal trajectory (e.g., a specified trajectory). An acceptable lag time is determined for the nominal trajectory, where compliance with the acceptable lag time during actual operation can ensure collision-free operation. Actual operation of the robot (e.g., an actual trajectory) is monitored for compliance with the corresponding acceptable lag time. Optionally, remedial measures are selected and / or taken as necessary (e.g., actual lag time approaches or exceeds a margin or threshold, such as the corresponding acceptable lag time). Motion planning can employ the determined acceptable lag time to select trajectories, thereby providing the robot with more robust operation.

Description

[0001] Cross Reference to Related Applications

[0002] This patent application claims priority to U.S. Patent Application No. 63 / 358,422 filed July 5, 2022, the disclosure of which is incorporated by reference herein in its entirety for all purposes. TECHNICAL FIELD

[0003] The present disclosure relates generally to motion planning and operation of robots operating in a shared workspace, and to the computationally efficient generation of robust motion plans that can be executed by the robots in an efficient manner (e.g., reducing or even eliminating collisions and unexpected halts), as well as the monitoring of the actual execution of the motion plans, and optionally taking remedial measures if necessary. BACKGROUND

[0004] Description of Related Art

[0005] Applications exist in which two or more robots are employed to operate in a shared workspace. For example, two or more robots can be employed to perform a task on or with one or more objects or workpieces in a shared workspace, such as screwing bolts onto a chassis, where portions of the robots can overlap in range of motion.

[0006] Planning generally involves task planning and motion planning. For example, task planning can determine that, in order to perform a given task, a robot needs to move between two poses (e.g., from a start pose to an end pose). In contrast, motion planning focuses on how to move the robot between two poses.

[0007] Motion planning is a fundamental problem in robot control and robotics. Motion planning specifies a trajectory that a robot can follow to transition from a start pose, configuration, or state to a goal pose, configuration, or state, typically to accomplish a task without colliding with any obstacles in a shared space, or to reduce the likelihood of colliding with any obstacles in a shared space. Challenges in motion planning involve the ability to perform motion planning quickly (i.e., in real-time), while possibly taking into account changes in the environment (e.g., changes in the position or orientation of obstacles in a shared workspace). Challenges also include performing motion planning using relatively low-cost devices, with relatively low energy consumption and limited computing power and / or storage space (e.g., memory circuitry, such as on-chip circuitry).

[0008] Operating two or more robots in a shared workspace (also referred to as a workcell or multi-robot operating environment) presents a class of particular problems. For example, motion plans should take into account and avoid situations in which the robots or robot accessories of the robots can interfere with each other during performance of a task.

[0009] One approach to operating multiple robots in a shared workspace can be referred to as a task-level approach. An engineer can manually ensure that the robots do not collide by defining portions of the shared workspace in which the robots can collide with each other (i.e., interference zones), and programming the individual robots such that only one robot is in an interference zone of the shared workspace at any given point in time. For example, when a first robot begins to move into an interference zone of the shared workspace, the first robot sets a flag. A controller (e.g., a programmable logic controller (PLC)) reads the flag and prevents other robots from moving into the interference zone of the shared workspace until the first robot disarms the flag upon exiting the interference zone. This approach is intuitive and easy to understand, but is often difficult to implement and time consuming, and can not yield optimal results. Since this approach leaves at least one of the robots idle for a significant amount of time, it necessarily results in a lower work throughput, even though the idle robot can technically perform useful work in the shared workspace.

[0010] In the traditional approach, a team of engineers typically decomposes the problem and optimizes the resulting smaller sub-problems independently from each other (e.g., assigning tasks to robots, sequencing the tasks assigned to each robot, motion planning for each robot). This can take the form of iterative simulation of motion to ensure that the robots / robot attachments do not collide with each other, which can require hours of computation time and can not yield an optimal solution. Furthermore, if a modification to the shared workspace causes a change in the actual trajectory of one of the robots / robot attachments, the entire workflow must be re-validated. It is clear that this approach is not optimal and often requires experts to go through a slow iterative process in an attempt to find a combination of solutions that yields good results. In any case, this approach often cannot ensure collision-free operation when the robots are operated under real-world conditions. SUMMARY

[0011] Various methods and apparatuses are described herein for generating motion plans for robots operating in a shared workspace, the motion plans including nominal trajectories (e.g., specified trajectories) having associated acceptable lag times, where compliance with the acceptable lag times during actual operation can ensure collision-free operation (i.e., collision of one robot with itself and collision of one robot with other robots of the system of robots). For example, the nominal trajectories can represent corresponding collision-free paths. Various methods and apparatuses are described herein that monitor actual operation of the robots (e.g., actual trajectories) relative to the acceptable lag times, and optionally take one or more remedial measures when necessary (e.g., actual lag times approach or exceed a threshold, such as a corresponding acceptable lag time).

[0012] A particularly advantageous approach is described in International Patent Application PCT / US2021 / 013610 (Publication No. WO2021150439A1), which describes a system that generates optimized multi-robot motion plans. More specifically, motion plans are optimized by minimizing or attempting to minimize some cost function related to system performance (e.g., the probability or likelihood of collision). Since a motion plan specifies a nominal trajectory for a robotic system, each motion plan is also time-based. A nominal trajectory is a specified ordered sequence of poses, configurations, or states of each movable part of one or more robots of a robotic system, and is parameterized by time (e.g., at every time unit of the entire trajectory). For example, a nominal trajectory can represent a corresponding collision-free path. Since a robotic system can include multiple individual robots, a motion plan can be “multi-robot.”

[0013] A motion plan can be represented with a set of discrete robot trajectories: Trj(t) = {trj_r(t) for r e {1…N}, t e {0, AT, 2AT, …, T}}, where N represents the total number of robots of a robotic system, AT represents a sample time step, and T represents the total duration of the motion plan. For a sufficiently small value of AT, Trj(t) ensures collision- free with respect to all robots of the robotic system. In other words, a motion specified by a set of trajectories Trj(t) of a robotic system with two or more robots does not result in self-collision (i.e., collision of one robot with itself and collision of one robot with other robots of the robotic system). Although collision assessment can be performed with respect to other objects or obstacles in the shared workspace, this set of trajectories Trj(t) of the robotic system is not necessarily collision-free with respect to other objects or obstacles in the environment. This self-collision-free condition can be easily checked or reviewed in simulation; and once checked or reviewed, it can be considered safe to execute the motion prescribed by the motion plan.

[0014] However, in practice, during actual execution of a motion plan by a physical robotic system comprising two or more robots, many factors can adversely affect synchronization between the robots. For example, these factors can include: signal communication, control delays, sensor noise, numerical approximation errors, etc. This can cause the actual physical robotic system to deviate from the trajectories Trj(t) specified by the motion plan. In addition to this "reality gap" between the simulated environment used to generate and review one or more motion plans and the actual operation of a real physical robotic system, many other factors can also cause a robotic system to deviate from a motion plan that provides a set of nominal trajectories. For example, in some cases, the execution of a task can take more time to complete than expected, and therefore, a robot can need to stay or dwell at a particular location (e.g., a goal location, and thus in a particular pose, configuration, or state) for a longer time than specified by the trajectories in the motion plan. Deviating from a particular motion plan that has been reviewed and deemed safe can be dangerous and can compromise the safety of the robotic system.

[0015] A set of trajectories Trj(t) specified by a motion plan is named herein as a set of nominal trajectories to indicate that these trajectories are the specified trajectories. An individual trajectory specified by a motion plan is named herein as a nominal trajectory. A set of trajectories executed by actual physical motion of a robot is named herein as a set of actual trajectories to distinguish from the specified trajectories or nominal trajectories. An individual trajectory executed by actual physical motion of a robot is named herein as an actual trajectory.

[0016] The methods described herein advantageously extend the safety of operation regime beyond the safety of operation regime associated with a set of nominal trajectories or a nominal trajectory of one or more motion plans, for example, by computing a neighborhood of one or more nominal trajectories of a motion plan in which the robotic system can operate without causing the self-collisions described above. For each robot that will operate in the shared workspace, a maximum acceptable lag time (e.g., a time delay or "lag") of one or more nominal trajectories that the robot can execute will be computed. During online execution (i.e., run-time), the actual lag time of each robot is monitored. If the actual lag time is within a defined margin or threshold, it is deemed safe to continue executing the motion plan since self-collision free movement between the robots is ensured. If the actual lag time of any of the robots exceeds the corresponding margin or threshold, safety is compromised since self-collision free operation can no longer be ensured. Optionally, the processor-based system can select and / or take one or more remedial measures in such a case. BRIEF DESCRIPTION OF DRAWINGS

[0017] In the drawings, like reference numerals identify like elements or acts. The sizes and relative positions of elements in the drawings are not necessarily drawn to scale. For example, the angles and shapes of various elements can have been exaggerated, some parts can have been arbitrarily enlarged or minimized, and / or some elements can have been arbitrarily positioned relative to one another for the sake of illustration. In addition, particular shapes and / or relative positions of the elements in the drawings are not intended to convey any information regarding the actual shape, relative position, and / or size of the particular elements in any particular embodiment. Rather, the particular shapes and / or relative positions of the elements in the drawings serve merely to facilitate understanding of the drawings themselves and / or the underlying teachings included herein.

[0018] Figure 1 is a schematic diagram of a configuration optimization system for a shared workspace and execution optimization to configure robots, in accordance with one illustrated embodiment, in which multiple robots operate in the shared workspace to perform a task.

[0019] Figure 2 is a functional block diagram of at least a first robot and a robot control system, in accordance with at least another illustrated embodiment, in which the robot control system is communicatively coupled to control operation of the first robot, in which the robot control system includes a motion planner that advantageously determines and employs acceptable lag times for respective nominal trajectories of the robot in motion planning, and optionally monitors actual operation of the at least first robot, compares actual lag times to a margin or threshold (e.g., acceptable lag times), and optionally takes remedial action, if necessary.

[0020] Figure 3 shows an operational method for a processor-based system to perform motion planning to control operation of robots that are to operate in a shared workspace, in accordance with at least one illustrated embodiment, in which acceptable lag times are determined for each robot based on nominal trajectories of other robots without having to account for effects of lag times in the nominal trajectories of the other robots, and motion plans are generated or selected that specify nominal trajectories that are associated with respective acceptable lag times.

[0021] Figure 4 shows an operational method for a processor-based system to perform motion planning to control operation of robots that are to operate in a shared workspace, in accordance with at least one illustrated embodiment, in which acceptable lag times are determined for each robot based on nominal trajectories of other robots while having to account for effects of lag times in the nominal trajectories of the other robots, and motion plans are generated or selected that specify nominal trajectories that are associated with respective acceptable lag times.

[0022] Figure 5 shows an operational method for a processor-based system, in accordance with at least one illustrated embodiment, to generate swept volumes for one or more trajectories of each robot of a plurality of robots that are to operate in a shared workspace, optionally, the method can be used to perform Figure 3 ,4 and / or the method of 6.

[0023] Figure 6 A method of operation of a processor-based system is shown, in accordance with at least one illustrated embodiment, for performing collision assessment for a plurality of robots and a plurality of trajectories for each of the robots, where movement of the robots or portions thereof along the trajectories is represented as swept volumes, and the collision assessment can be used to determine acceptable latency times for one or more robots operating in a shared workspace.

[0024] Figure 7 A method of operation of a processor-based system is shown, in accordance with at least one illustrated embodiment, for generating or selecting motion plans for one or more robots to operate in a shared workspace, based at least in part on acceptable latency times, and optionally based at least in part on other criteria such as represented as costs or cost functions.

[0025] Figure 8 A method of operation of a processor-based system is shown, in accordance with at least one illustrated embodiment, for controlling operation of robots operating in a shared workspace based at least in part on acceptable latency times. DETAILED DESCRIPTION

[0026] In the following description, certain specific details are set forth in order to provide a thorough understanding of various disclosed implementations. However, one skilled in the relevant art will recognize that the implementations can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures associated with computer systems, robots, actuator systems, and / or communication networks have not been described in detail to avoid unnecessarily obscuring descriptions of the implementations. In still other instances, well-known computer vision methods and techniques for generating perception data and volume representations for one or more objects and the like have not been described in detail to avoid unnecessarily obscuring descriptions of the implementations.

[0027] Unless the context requires otherwise, throughout the specification and claims which follow, the word "comprise" and variations thereof, such as "comprising" and "comprises," are to be construed in an open, inclusive sense, that is, as "including, but not limited to."

[0028] References throughout this specification to “one implementation” or “an implementation” or “one embodiment” or “an embodiment” mean that a particular feature, structure, or characteristic described in connection with the implementation is included in at least one implementation or embodiment. The appearances of the phrases “in one implementation” or “in an implementation” or “in one embodiment” or “in an embodiment” in various places in the specification are not necessarily all referring to the same implementation or embodiment. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more implementations or embodiments.

[0029] Unless the content clearly dictates otherwise, an element recited in the specification and the appended claims without a quantity modifier is meant to be present at least one of that element. It should also be noted that the term “or” is generally employed in its sense including “and / or” unless the content clearly dictates otherwise.

[0030] As used in this specification and the appended claims, the term “optimize” and variants thereof mean that an improved result is being prepared, generated, or produced, or has been prepared, generated, or produced. These terms are used in a relative sense and do not necessarily mean that an absolutely optimal value has been prepared, generated, or produced.

[0031] As used in this specification and the appended claims, the term “workspace” or “shared workspace” is used to refer to an operating environment in which two or more robots operate, one or more portions of the shared workspace being volumes in which the robots can potentially collide with each other, and thus these portions can be named interference zones. The operating environment can include obstacles (i.e., items that the robots are to avoid colliding with) and / or workpieces (i.e., items that the robots need to interact with or act on or over).

[0032] As used in this specification and the appended claims, the term “task” is used to refer to a robot task of transitioning a robot from a pose A to a pose B, preferably without colliding with obstacles in its environment. The task can involve grasping or ungrasping an item, moving or setting down an item, rotating an item, or retrieving or placing an item. The transition from pose A to pose B can optionally include a transition between one or more intermediate poses.

[0033] As used in the present specification and the appended claims, the term "a trajectory" or "trajectories" is used to refer to an ordered sequence of poses or configurations or states of one or more robots, parameterized by a time parameter, through which the one or more robots or at least a portion thereof can move, e.g., to perform a task. A trajectory is preferably represented in the configuration space (also referred to as C-space) of the respective robot, but can alternatively also be represented in the real space or real-world space of the shared workspace. A trajectory can include pauses, changes or even reversals of direction of one or more poses, and is not necessarily smooth in terms of direction, time, velocity, or acceleration.

[0034] As used in the present specification and the appended claims, the term "a nominal trajectory" or "nominal trajectories" is used to refer to a "specified" trajectory or trajectories. As used in the present specification and the appended claims, the term "an actual trajectory" or "actual trajectories" is used in contrast to "a nominal trajectory" or "nominal trajectories" to refer to the physical motion of a physical robot actually performed. In some cases, "an actual trajectory" or "actual trajectories" can match "a nominal trajectory" or "nominal trajectories", although in many cases there will be a difference between the two, e.g., due to "an actual trajectory" or "actual trajectories" lagging behind "a nominal trajectory" or "nominal trajectories" in time.

[0035] As used in the present specification and the appended claims, the term "self-collision" when used or referred to in the context of a robot system comprising two or more robots includes both i) a collision of a robot with itself, and ii) a collision of a robot with other robots of the robot system. As used in the present specification and the appended claims, the term "self-collision" when used or referred to in the context of a single robot includes a collision of the robot with itself.

[0036] The title and summary of the disclosure provided herein are merely for convenience and do not interpret the scope or meaning of the embodiments.

[0037] The methods described herein advantageously extend the safety of robotic operation beyond that ensured by nominal trajectories of motion plans by determining acceptable lag times for physical robots of a robotic system to operate while still ensuring that such operation of the robotic system does not occur with self-collision. In at least some implementations, for each robot of a robotic system that will operate in a shared workspace, a maximum acceptable lag time (e.g., time delay or "lag") is computed for each of one or more nominal trajectories. Advantageously, this maximum acceptable lag time can be computed at a configuration time prior to the robot executing a motion plan. During online execution (i.e., runtime, after the configuration time), the actual motion of each robot is monitored, including monitoring the actual lag time of each robot along its actual trajectory. If the actual lag time value is within a defined margin or threshold (e.g., acceptable lag time), then it can be considered safe to continue executing the motion plan as no self-collision movement can be ensured. If the actual lag time of a robot exceeds the corresponding margin or threshold, then safety can be compromised as no self-collision operation can be ensured. For example, the safety of all robots can be considered compromised even if only one robot is experiencing an actual lag time that exceeds the corresponding margin or threshold. Optionally, a processor-based system can select and / or take one or more remedial actions in such a case.

[0038] Figure 1 A robotic system 100 is shown in accordance with one illustrated implementation, which includes a plurality of robots 102a, 102b, 102c (collectively, 102) that operate in a shared workspace 104 (also referred to as a multi-robot environment) to perform tasks. In the illustrated implementation, the robots 102a, 102b, 102c are configured to perform a set of tasks. The tasks can be specified as a task plan. The task plan can specify T tasks that need to be performed by N robots. The task plan can be modeled as a vector for each robot, where the vector is an ordered list of tasks to be performed by the corresponding robot (e.g., {task 7, task 2, task 9}). The task vector can also optionally include a dwell duration that specifies a duration of time that the robot or portion thereof should dwell at a given configuration or goal. The task vector can also specify a home pose and / or other "functional poses" that are not directly related to solving the task (e.g., a "yield" or storage pose). The poses can be specified in C-space of the robot. Figure 1 In the robotic system 100, as part of the optimization, an acceptable lag time is determined and provided to one or more robot control systems along with the optimized motion plan that includes the nominal trajectories of the robots.

[0039] The plurality of robots can be configured to perform a set of tasks. The tasks can be specified as a task plan. The task plan can specify T tasks that need to be performed by N robots. The task plan can be modeled as a vector for each robot, where the vector is an ordered list of tasks to be performed by the corresponding robot (e.g., {task 7, task 2, task 9}). The task vector can also optionally include a dwell duration that specifies a duration of time that the robot or portion thereof should dwell at a given configuration or goal. The task vector can also specify a home pose and / or other "functional poses" that are not directly related to solving the task (e.g., a "yield" or storage pose). The poses can be specified in C-space of the robot.

[0040] The robots 102 can take any of a variety of forms. Generally, the robots 102 will take the form of one or more robot attachments 103 (only one labeled) and a base 105 (only one labeled) from which the robot attachments 103 extend. The robots 102 can include one or more links having one or more joints, and actuators (e.g., electric motors, stepper motors, solenoids, pneumatic actuators, or hydraulic actuators) coupled and operable to move the links in response to control or drive signals. For example, a pneumatic actuator can include one or more pistons, cylinders, valves, gas reservoirs, and / or pressure sources (e.g., compressors, blowers). For example, a hydraulic actuator can include one or more pistons, cylinders, valves, liquid reservoirs (e.g., low-compressibility hydraulic fluid), and / or pressure sources (e.g., compressors, blowers). The robot system 100 can employ other forms of robots 102, e.g., autonomous vehicles.

[0041] The shared workspace 104 generally represents a three-dimensional space in which the robots 102 can operate and move, although in certain limited implementations the shared workspace 104 can represent a two-dimensional space. The shared workspace 104 is a volume or region in which at least a portion of the robots 102 can overlap in space and time, or would collide if motion were not controlled to avoid collision. Notably, the shared workspace 104 is a physical space or volume, the location and orientation of which can be conveniently represented via a Cartesian coordinate system, e.g., with respect to a reference frame (e.g., the reference frame represented by orthogonal axes X, Y, and Z shown in FIG. 1) that is fixed in space. Figure 1 Notably, the reference frame of the shared workspace 104 is distinct from the respective “configuration space” or “C-space” of any of the robots 102, which is generally represented by a set of joint positions, orientations, or configurations in the respective reference frame of any of the robots 102.

[0042] As described herein, the robot 102a or a portion thereof can constitute an obstacle when considered from the perspective of another robot 102b (i.e., when motion planning is performed for the other robot 102b). The shared workspace 104 can also include other obstacles, e.g., mechanical components (e.g., a conveyor 106), pillars, posts, walls, ceilings, floors, tables, people, and / or animals. The shared workspace 104 can also include one or more work items or workpieces that the robots 102 manipulate (e.g., one or more packages, packages, fasteners, tools, items, or other objects) as part of performing a task.

[0043] Optionally, the robot system 100 includes one or more processor-based multi-robot configuration optimization systems 108 (Figure 1 One or more optional multi-robot configuration optimization systems 108 are shown. The one or more optional multi-robot configuration optimization systems 108 receive a set of inputs 109 and generate as output 111 one or more solutions specifying configurations of the robots 102, including a workcell layout (e.g., respective base positions and orientations of each of the robots 102), one or more task plans (e.g., a respective task plan for each of the robots 102), and optionally one or more motion plans specifying or including one or more nominal trajectories of the robots 102a-c (e.g., a respective nominal trajectory for each of the robots 102) and an acceptable lag time for each of the nominal trajectories. The one or more components of the output 111 can be optimized at least to some extent.

[0044] The one or more optional multi-robot configuration optimization systems 108 can include a population generator 110, a multi-robot environment simulator 112, a multi-robot optimization engine 114, and an acceptable lag time evaluator 115.

[0045] The population generator 110 generates a set of candidate solutions 116 based on the provided inputs 109. The candidate solutions 116 represent possible solutions to the configuration problem (i.e., how to configure the robots 102 in the shared workcell 104 to complete a set of tasks). Any given candidate solution 116 can or can not actually be feasible. That is, an initial candidate can be invalid (e.g., a robot is in an impossible place, has an unreachable goal, or an infeasible task plan that would result in a collision). In certain implementations, the population generator can attempt to find better candidate solutions.

[0046] The multi-robot environment simulator 112 models the workcell or multi-robot environment based on each candidate solution to determine certain properties, such as an amount of time required to complete the tasks, a probability or rate of collision when completing the tasks, or a feasibility or infeasibility of a particular configuration specified by the candidate solution. For example, the multi-robot environment simulator 112 can generate a cost via one or more cost functions that reflect these properties in terms of cost.

[0047] For example, the cost or cost function can represent a probability or likelihood of collision. Alternatively, the cost or cost function can represent one or more of: an acceptable lag time or "robustness," a severity of collision, an energy expenditure or loss and / or a time or lag to perform or complete a motion corresponding to a nominal trajectory. In some implementations, the cost or cost function represents a determined acceptable lag time for a given nominal trajectory of a given robot. The determined acceptable lag time represents a maximum or approximately maximum or optimized lag time that can be incurred while executing the corresponding nominal trajectory while still maintaining, ensuring, or even guaranteeing at least collision-free movement with respect to itself and with respect to other robots operating in the shared workspace or workcell. Thus, the lag time can represent an amount of lag that can be introduced in the actual execution of or in the actual execution of a nominal trajectory without sacrificing a safety factor (e.g., collision-free operation), thereby thus enhancing the robustness of the corresponding motion plan. For example, the safety factor can be checked or reviewed via simulation of the robot operation using the nominal trajectory. Thus, if the actual trajectory of the robot lags behind the nominal trajectory by more than a specified margin or threshold (e.g., the acceptable lag time), then collision-free operation can no longer be ensured.

[0048] The multi-robot optimization engine 114 evaluates candidate solutions based at least in part on the associated costs 119 and advantageously optimizes in coordination across a set of two or more non-homogeneous parameters, e.g., across two or more of: respective base positions and orientations of the robots, assignment of tasks to respective robots, respective target sequences of the robots, and / or respective trajectories or paths (e.g., collision-free trajectories or paths) between consecutive targets. Straight-line trajectories between consecutive targets can be used for simplicity of illustration, but the trajectories need not be straight-line trajectories.

[0049] The input 109 can include one or more static environment models representing or characterizing the operating environment or shared workspace 104, e.g., representing a floor, walls, ceiling, pillars, other obstacles, etc. The operating environment or shared workspace 104 can be represented by one or more models, e.g., geometric models (e.g., point clouds) representing a floor, walls, ceiling, obstacles, and other objects in the operating environment. For example, the models can be represented in Cartesian coordinates.

[0050] Input 109 can include one or more robot models representing or characterizing each of robots 102, e.g., specifying geometry and kinematics, e.g., dimensions or lengths, number of links, number of joints, joint types, motion ranges, velocity limits, acceleration limits, or jerk limits. Robots 102 can be represented by one or more robot geometry models defining the geometry of a given robot 102a-c, e.g., in terms of joints, degrees of freedom, dimensions (e.g., link lengths), and / or respective C-space of robot 102a-c.

[0051] Input 109 can include one or more sets of tasks to be performed, e.g., represented as target goals (e.g., poses, configurations, states, or positions or locations). For example, the tasks can be represented in terms of end poses, end configurations, or end states and / or intermediate poses, intermediate configurations, or intermediate states of respective robots 102a-c. For example, poses, configurations, or states can be defined in terms of joint positions and joint angles / rotations (e.g., joint poses, joint coordinates) of respective robots 102a-c. Optionally, input 109 can include one or more dwell times specifying a nominal amount of time that a robot or portion thereof should dwell at a given goal in order to complete a task (e.g., tightening a screw or nut, picking up and placing an object, the goal being to sort a pile of objects into two or more different types of object piles by two or more robots operating in a common workspace).

[0052] In some implementations, the one or more multi-robot configuration optimization systems 108 are capable of generating, for each robot, a nominal trajectory for performing one or more tasks. A nominal trajectory is a “prescribed” trajectory in which each nominal trajectory includes a time-parameterized ordered set or sequence of poses, configurations, or states of the robot between an initial or starting pose, configuration, or state of the nominal trajectory and a final or ending pose, configuration, and state of the nominal trajectory, each with a corresponding timing. The poses or configurations are preferably represented in a configuration space (also referred to as C-space) of the respective robot, or alternatively in a real space or real-world space of a workspace. The timing can be specified in relative terms (e.g., timing defined by a relative offset from the timing of an immediately preceding pose) or in absolute terms (e.g., timing defined by a duration from the start of the execution of the trajectory, and e.g., relative to a common clock). In at least some cases, a nominal trajectory can specify or include one or more pauses in the motion or path of the robot or portions thereof, and / or can specify a reversal of the direction or path of the motion of the robot or portions thereof, otherwise the motion can not be smooth in direction or in time. Applicants note that while any given trajectory can correspond to smooth motion of the robot, the term “trajectory” as used herein is not limited to this and will generally specify motion that is not smooth, nor will it define a straight path for the robot or portions thereof. For example, a nominal trajectory can specify a time-parameterized ordered set or sequence of poses through which the robot or portions thereof move to complete a task or portion of a task. The performance of any given task can employ one or more nominal trajectories. As described herein, the actual motion or actual trajectory of the robot or portions thereof can deviate from the respective nominal trajectory, e.g., due to unexpected delays in transitioning between poses (e.g., due to a need to linger or dwell at a target object for longer than expected).

[0053] The acceptable lag time evaluator 115 evaluates each candidate lag time for a given nominal trajectory to determine an acceptable lag time that ensures collision-free operation even if the actual trajectory of the corresponding robot lags behind the nominal trajectory by no more than the acceptable lag time. In a preferred approach, the acceptable lag time evaluator 115 takes into account not only the effect of a lag in the nominal trajectory of a given robot, but also one or more lags or effects of the corresponding nominal trajectories of other robots operating in the shared workspace. Thus, the acceptable lag time evaluator 115 is able to identify an acceptable lag time, or in other words, a time lag, for each nominal trajectory assuming the worst case scenario in which the actual trajectories of all robots operating in the shared workspace experience their respective acceptable lag times. Thus, for example, the acceptable lag time evaluator 115 is able to determine a maximum acceptable lag time for each robot that still ensures collision-free operation even assuming that all robots experience their respective maximum acceptable lag times. Several approaches to determining an acceptable lag time are described herein.

[0054] The inputs 109 can optionally include a limit on the number of robots that can be configured in the shared workspace 104. The inputs 109 can optionally include a limit on the number of tasks or goals that can be assigned to a given robot 102a-c, herein named a task capacity, that can be configured in the shared workspace 104, e.g., to limit the complexity of the configuration problem to ensure that the configuration problem is solvable, or solvable within some acceptable time period using available computational resources, or to pre-eliminate certain solutions that are deemed too slow because tasks or goals are significantly over-assigned to a given robot 102a-c. The inputs 109 can optionally include one or more bounds or constraints on variables or other parameters. The inputs 109 can optionally include a total number of iteration loops or a time limit on iterations that can be used to refine candidate solutions, e.g., to ensure that the configuration problem is solvable or solvable within some acceptable time period using available computational resources.

[0055] The robot system 100 can optionally include one or more robot control systems 118 Figure 1 (e.g., only one is shown in the figure) communicatively coupled to control the robots 102. For example, the one or more robot control systems 118 can provide control signals (e.g., drive signals) to various actuators to move the robots 102 between various configurations to various specified goals to perform specified tasks.

[0056] The robot system 100 can optionally include one or more motion planners 120 Figure 1A motion planner 120 (only one of which is shown) is communicatively coupled to control the robot 102. As described elsewhere herein, the one or more motion planners 120 generate, produce, select, or refine motion plans for the robot 102, e.g., taking into account small deviations in time from motion plans provided by the multi-robot optimization engine 114, or taking into account unexpected appearance of obstacles (e.g., a human entering the operating environment or shared workspace 104). The optional motion planner 120 is operable to dynamically generate motion plans to cause the robot 102 to perform tasks in the operating environment. The motion planner 120, as well as other structures and / or operations, can employ those described in U.S. Patent Application Serial No. 62 / 865,431, filed June 24, 2019.

[0057] In cases where a motion planner 120 is included, the motion planner 120 is optionally communicatively coupled to receive, as input, perception data provided, e.g., by a perception subsystem (not shown). The perception data is representative of static and / or dynamic objects in the shared workspace 104 that were previously unknown. The perception data can be raw data sensed via one or more sensors (e.g., cameras, stereo cameras, time-of-flight cameras, lidar) and / or raw data converted by the perception subsystem into a digital representation of obstacles, which can generate respective discretizations of representations of environments in which the robot 102 will operate to perform tasks for a variety of different scenarios.

[0058] Figure 1Various communication paths are shown as lines between various structures, in some cases with arrows indicating the direction of input 109 and output 111. For example, the communication paths can take the form of one or more wired communication paths (e.g., electrical conductors, signal buses, or optical fibers) and / or one or more wireless communication paths (e.g., via RF or microwave radios and antennas, infrared transceivers). For example, the communication channels can include one or more transmitters, receivers, transceivers, radios, routers, wired ports (e.g., Ethernet ports), etc. The general operation of the robotic system 100, and in particular the general operation of the one or more multi-robot configuration optimization systems 108, is shown and described in International Patent Application PCT / US2021 / 013610 (Publication No. WO 2021 / 150439), which is not repeated herein for the sake of brevity. Only some of the more significant differences in operation are described herein, e.g., in implementations the one or more multi-robot configuration optimization systems 108 perform various methods described herein to determine acceptable lag times, generate or select motion plans based at least in part on the determined acceptable lag times, and / or the one or more robot control systems 118 perform various methods described herein to monitor actual lag times of the robots executing the motion plans, compare the actual lag times to margins or thresholds (e.g., acceptable lag times), and / or control the robots accordingly, e.g., take one or more remedial actions in the event one or more actual lag times exceed one or more acceptable lag times or related thresholds.

[0059] Figure 2 A robotic system is shown in accordance with at least one illustrated implementation in which a first robot control system 200a includes a first motion planner 204a that generates a first motion plan 206a to control operation of a first robot 202, and optionally provides the first motion plan 206a and / or a motion representation as an obstacle to other motion planners 204b of other robot control systems 200b via at least one communication channel (indicated by the approximate arrows, e.g., transmitters, receivers, transceivers, radios, routers, Ethernet) to control operation of other robots Figure 2 (not shown) in accordance with the first motion plan 206a. In Figure 2 the robot control systems 200a, 200b, acceptable lag times are determined by the motion planners along with the motion plans 206a, 206b, which include one or more nominal trajectories to be executed by one or more robots 202. In contrast to the robotic system 100 Figure 1 (not shown) in accordance with the first motion plan 206a. In Figure 2 the robot control systems 200a, 200b, acceptable lag times are determined by the motion planners along with the motion plans 206a, 206b, which include one or more nominal trajectories to be executed by one or more robots 202. In contrast to the robotic system 100 Figure 2The robot control systems 200a and 200b monitor the actual lag time and, optionally, select and / or take remedial measures when necessary.

[0060] Similarly, one or more other motion planners 204b of one or more other robot control systems 200b generate one or more other motion plans 206b to control one or more other robots. Figure 2 (Not shown in the diagram) The operation, and optionally, the provision of one or more other motion plans 206b to the first motion planner 204a and one or more other motion planners in one or more other robot control systems 200b. Motion planners 204a and 204b may also optionally receive motion completion information 209, indicating when the motion of each robot 202 has been completed. This allows motion planners 204a and 204b to generate new or updated motion plans based on the current or updated state of the shared workspace. For example, after the first robot 202 has completed part or all of a set of motions (which constitute part of the task performed by the first robot 102), a portion of the shared workspace may become blocked, unblocked, or otherwise made available for the second robot to perform a task. Additionally or alternatively, motion planners 204a and 204b are able to receive information (e.g., images, occupancy grids, joint positions, and joint angles / rotations) collected by various sensors or generated by other motion planners 204b, indicating when a portion of the shared workspace may become blocked, unblocked, or otherwise become available for the second robot to perform a task after the first robot 202 has completed part or all of a set of movements (which constitute part of the task performed by the first robot 102).

[0061] As described herein, motion plans 206a and 206b specify a nominal trajectory for each robot, for example, to perform one or more tasks. As previously described, a nominal trajectory is a “specified” trajectory, wherein each nominal trajectory comprises a time-parameterized ordered set or sequence of robot poses, configurations, or states, each pose, configuration, and state having a corresponding timing. The poses, configurations, or states are preferably represented in the configuration space (also known as C-space) of the respective robot.

[0062] Robot control systems 200a and 200b can be optionally communicatively coupled, for example via at least one communication channel (indicated by approximate arrows, such as transmitter, receiver, transceiver, radio, router, Ethernet), to optionally receive motion planning graph 208 and / or sweep volume representation 211 from one or more sources 212. According to one illustrated embodiment, one or more sources 212 of motion planning graph 208 and / or sweep volume representation 211 can be separate from and distinct from motion planners 204a and 204b. For example, one or more sources 212 of motion planning graph 208 and / or sweep volume representation 211 can be one or more processor-based computer systems (e.g., server computers), which can be operated or controlled by the respective manufacturer of robot 202 or some other entity. Motion planning graph 208 can each include a set of nodes 214 ( Figure 2 Only two are marked in the text) and a set of edges 216 ( Figure 2 (Only two are shown in the diagram). Node 214 represents the pose, configuration, or state of the corresponding robot, and edge 216 couples the corresponding node pair 214 to node 214, representing a legal or valid transition between poses, configurations, and states. For example, poses, configurations, or states can be defined in the robot's corresponding configuration space (C-space), representing the set of joint positions, orientations, poses, or coordinates of each joint in the joints of the corresponding robot 202. Therefore, each node 214 can represent the pose, configuration, or state of the robot 202 or a portion thereof, which is entirely defined by the pose configuration or state of the joints constituting the robot 202. The motion planning graph 208 can be determined, set up, or defined before runtime (i.e., before performing the task), such as during pre-runtime or configuration time. An optional sweep volume representation 211 represents the corresponding volume that the robot 202 or a portion thereof will occupy when performing motion or transitions between poses corresponding to the corresponding edges 216 of the motion planning graph 208. The optional swept volume representation 211 can be represented in any of a variety of forms, such as, for example, voxels, Euclidean distance fields, or hierarchical structures of geometric objects. This advantageously allows some of the most computationally intensive work to be performed before runtime, when responsiveness is not of particular concern. While swept volume is used in this paper, it is exemplary and can be applied to any of a variety of other collision evaluation methods.

[0063] Each robot 202 may optionally include a base ( Figure 2The base can be stationary in the environment or movable in the environment (e.g., an autonomous or semi-autonomous vehicle). Each robot 202 can optionally include a set of links, joints, end-of-arm tools or end effectors, and / or actuators 218a, 218b, 218c (three shown, collectively 218) operable to move the links about the joints. The set of links, joints, end-of-arm tools or end effectors generally includes one or more accessories of the robot that are movably coupled to the base of the robot. Each robot 202 can optionally include one or more motion controllers (e.g., motor controllers) 220 (only one shown) that receive control signals (e.g., in the form of motion plans 206a) and provide drive signals to drive the actuators 218. Alternatively, the motion controllers 220 can be separate from the robots 202 and communicatively coupled to the robots 202. For example, as described in International Patent Application PCT / US2021 / 013610 (Publication No. WO 2021 / 150439), each robot 202 can be positioned and oriented in the shared workspace based on an optimized workcell layout.

[0064] Each robot 202 (only one shown in FIG. 1) can have a respective robot control system 200a, 200b, or alternatively, one robot control system 200a can perform motion planning for two or more robots 202. For purposes of illustration, one robot control system 200 will be described in detail. Those skilled in the art will recognize that the description can apply to similar or even identical additional instances of other robot control systems 200. Figure 2

[0065] The first robot control system 200a can include one or more processors 222, and one or more associated non-transitory computer or processor-readable storage media, e.g., system memory 224a, one or more disk drives 224b, and / or memory or registers of the processors 222 (not shown). The non-transitory computer or processor-readable storage media (e.g., system memory 224a, one or more disk drives 224b) are communicatively coupled to the one or more processors 222a via one or more communication channels (e.g., system bus 234). The system bus 234 can take any known bus structures or architectures, including a memory bus having a memory controller, a peripheral bus, and / or a local bus. One or more of such components can also, or alternatively, communicate with each other via one or more other communication channels, e.g., one or more parallel cables, serial cables, or wireless network channels capable of high-speed communication, e.g., Universal Serial Bus (“USB”) 3.0, Peripheral Component Interconnect Express (PCIe), or via a network, e.g., an Ethernet network, a Wi-Fi network, a Bluetooth network, or a cellular network. ​

[0066] The first robot control system 200a can also be communicatively coupled to one or more remote computer systems, such as server computers (e.g., source 212), desktop computers, laptop computers, ultra-portable computers, tablet computers, smartphones, wearable computers, and / or sensors. Figure 2 (Not shown in the diagram), these remote computer systems are directly or indirectly communicatively coupled to the various components of the first robot control system 200a, for example, via interface 227. Remote computing systems, such as server computers (e.g., source 212), can be used to program, configure, control, or otherwise interact with the first robot control system 200a and its various components, or to input data (e.g., motion planning graph 208, sweep volume representation 211, task specification 215, or even candidate paths or nominal trajectories). This connection can be via one or more communication channels 210, such as one or more wide area networks (WANs), for example, Ethernet, or the Internet using Internet Protocol. As described above, pre-run calculations can be performed by a system separate from the first robot control system 200a or the first robot 202, while runtime calculations can be performed by one or more processors 222 of the first robot control system 200a while one or more robots are performing tasks. In some implementations, one or more of the robot control systems 200a and 200b may be mounted on the respective robot (e.g., the first robot 202).

[0067] As described above, the first robot control system 200a may include one or more processors 222 (i.e., circuitry), non-transitory storage media (e.g., system memory 224a, one or more disk drives 224b), and a system bus 234 coupling various system components. The processors 222 may be any logic processing unit, such as one or more central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), etc. The system memory 224a may include read-only memory (“ROM”) 226, random access memory (“RAM”) 228, flash memory 230, EEPROM (not shown), or any combination thereof. A basic input / output system (“BIOS”) 232 capable of forming a portion of the ROM 226 contains basic routines that facilitate the transfer of information between elements within the first robot control system 200a, for example, during startup.

[0068] For example, the one or more disk drives 224b can be hard disk drives for reading from and writing to magnetic disks, solid state (e.g., flash) drives for reading from and writing to solid state memory, and / or optical disk drives for reading from and writing to removable optical disks. In various different embodiments, the first robotic control system 200a can also include any combination of such disk drives. The one or more disk drives 224b can communicate with the one or more processors 222 via the system bus 234. As known to those of skill in the relevant art, the one or more disk drives 224b can include an interface or controller (not shown) coupled between such drives and the system bus 234. The one or more disk drives 224b and their associated computer-readable media provide non-volatile storage for the first robotic control system 200a of computer- or processor-readable and / or executable instructions, data structures, program modules, and other data. Those of skill in the relevant art will appreciate that other types of computer-readable media that can store data that is accessible by a computer, such as WORM drives, RAID drives, cassettes, digital video disks ("DVDs"), Bernoulli cartridges, RAM, ROM, smart cards, and the like, can be used.

[0069] Executable instructions and data can be stored in the system memory 224a, such as an operating system 236, one or more application programs 238, other applications or modules 240, and program data 242. The application programs 238 can include processor-executable instructions that cause the one or more processors 222 to perform one or more of the following.

[0070] The application programs 238 can include processor-executable instructions that cause the one or more processors 222 to receive or generate a discretized representation of a shared workspace in which the robot 202 is to operate, including obstacles and / or target objects or workpieces in the shared workspace, where planned motions of other robots can be represented as obstacles.

[0071] The application programs 238 can include processor-executable instructions that cause the one or more processors 222 to generate a motion plan 206a, 206b that specifies a nominal trajectory. Each of the nominal trajectories generally defines a respective ordered sequence of poses, configurations, or states of the robot or portions thereof, and the sequences are parameterized by time.

[0072] The application 238 can include processor-executable instructions that cause the one or more processors 222 to determine respective acceptable lag times for the nominal trajectories. An acceptable lag time is generally an amount of time by which actual motion of the robot 202 (e.g., an actual trajectory) can lag or differ from a nominal time specified by a respective nominal trajectory while ensuring at least no-self-collision operation of the robot 202 relative to one or more other robots operating in a shared workspace. In some implementations, the no-self-collision operation is conditioned on no other robots experiencing any lag time during execution, although preferably the no-self-collision operation is conditioned on all robots operating within respective acceptable lag times for their respective nominal trajectories, thus accounting for each robot experiencing a lag time in executing a respective nominal trajectory.

[0073] To generate motion plans, generate nominal trajectories, and / or determine acceptable lag times, the application 238 can include processor-executable instructions that cause the one or more processors 222 to invoke or otherwise perform collision assessments. Collision assessments are often referred to as “collision detection” or “collision checking,” even though the assessment often determines a probability or likelihood of a collision, and often occurs prior to actual movement of a robot, as opposed to detecting an actual physical collision of a robot during physical movement of the robot. Collision assessments are alternatively referred to herein as “collision detection,” “collision checking,” or “collision analysis.”

[0074] The application 238 can include processor-executable instructions that cause the one or more processors 222 to set a cost value or cost function for edges in a motion planning graph, e.g., reflecting a determined probability or likelihood of a collision occurring, and optionally other parameters. The application 238 can include processor-executable instructions that cause the one or more processors 222 to set a cost value or cost function for nominal trajectories, e.g., reflecting respective acceptable lag times, and alternatively additionally reflecting a determined probability or likelihood of a collision occurring, and optionally other parameters.

[0075] The application programs 238 can include processor-executable instructions that cause the one or more processors 222 to evaluate available nominal trajectories generated from the motion planning graph; identify (e.g., select, determine, generate) a nominal trajectory based on, e.g., a cost or cost function, and / or identify or generate a motion plan that can be executed by the one or more robots to cause the robots to perform a motion, e.g., to further perform one or more tasks by the one or more robots. The application programs 238 can include processor-executable instructions that cause the one or more processors 222 to optionally store the determined motion plan and / or provide instructions to cause the one or more robots to perform or otherwise move in accordance with the motion plan. Motion planning and motion plan construction (e.g., collision evaluation or detection, setting, updating, or adjusting a cost or cost function, e.g., based at least in part on collision evaluation or detection, and optionally in part on a determined acceptable lag time), as well as nominal trajectory generation, analysis or evaluation of candidate nominal trajectories (e.g., selecting between two nominal trajectories based at least in part on a respective acceptable lag time), can be performed as described herein (e.g., with reference to the methods of 5, 6, 7, and 8) and as described in the references incorporated by reference herein. Collision detection or evaluation can employ various structures, techniques, and algorithms described herein and elsewhere. Figure 3 , 4 , 5, 6, 7, and 8). Collision detection or evaluation can employ various structures, techniques, and algorithms described herein and elsewhere.

[0076] The application programs 238 can also include one or more machine-readable and machine-executable instructions that cause the one or more processors 222 to monitor robot motion (e.g., actual trajectories) and evaluate actual delays or actual lag times of these actual trajectories compared to a margin or threshold (e.g., an acceptable lag time or based on an acceptable lag time) of a respective nominal trajectory. Optionally, the application programs 238 can also include one or more machine-readable and machine-executable instructions that cause the one or more processors 222 to select and / or take one or more remedial measures (e.g., cause the one or more robots to slow down, cause the one or more robots to stop, and / or cause the one or more robots to speed up) if necessary (e.g., if the actual lag time approaches or exceeds the acceptable lag time).

[0077] The application programs 238 can also include one or more machine-readable and machine-executable instructions that optionally cause the one or more processors 222 to monitor the robots in the environment to determine when a path along an actual trajectory becomes unblocked or clear, and cause the robots to move toward a goal in response to the path along the actual trajectory becoming unblocked or clear.

[0078] The application programs 238 can additionally include one or more machine- readable and machine-executable instructions that cause the one or more processors 222 to perform other operations, e.g., optionally processing sensory data (captured via sensors). The application programs 238 can additionally include one or more machine-executable instructions that cause the one or more processors 222 to perform various other methods described herein and in references incorporated by reference herein.

[0079] In various embodiments, one or more of the above-described operations can be performed by one or more remote processing devices or computers coupled via the interface 227 over the communication channel 210 (e.g., a network).

[0080] While shown in the Figure 2 system memory 224a, the operating system 236, the application programs 238, the other programs / modules 240, and the program data 242 can be stored on other non-transitory computer- or processor-readable media (e.g., one or more disk drives 224b).

[0081] The motion planner 204a of the first robotic control system 200a can include dedicated motion planner hardware, or can be implemented in whole or in part via the one or more processors 222 and processor-executable instructions stored in the system memory 224a and / or the one or more disk drives 224b.

[0082] The motion planner 204a can include or implement a motion converter 250, a path generator 252, a collision evaluator 253, a cost setter 254, an optional path analyzer 255, a trajectory generator 256, an acceptable lag time evaluator 257, and an optional nominal trajectory analyzer 258. Each of these can be implemented via one or more processors (e.g., circuitry) executing logic, e.g., executable software instructions, firmware instructions, hardwired logic, or any combination thereof.

[0083] The motion converter 250 converts motion of objects (e.g., other robots, people) into a representation of obstacles. The motion converter 250 receives motion plans 206b or other motion representations from other motion planners 204b.

[0084] The motion converter 250 can include a trajectory predictor 251 to predict a trajectory of a transient object (e.g., other robots, other objects, including, for example, humans), for example, when the trajectory of the transient object is not known (e.g., when the object is another robot but no motion plan for the other robot is received, the object is not another robot, for example, is a human). For example, the trajectory predictor 251 can assume that the object will continue existing movement in two directions, speed, and acceleration without change. For example, the trajectory predictor 251 can account for expected changes in motion or path of the object, for example, when the path of the object will result in a collision, thus, the object can be expected to stop or change direction, or when the object’s destination is known, thus, the object can be expected to stop when the destination is reached. In at least some cases, the trajectory predictor 251 can use a learned behavior model of the object to predict the trajectory of the object. For example, the trajectory predictor 251 can extrapolate known motions and expected changes to generate a predicted trajectory of the transient object.

[0085] The motion converter 250 can then optionally determine a region or volume corresponding to the known and / or one or more extrapolated motions of the object. For example, the motion converter can convert the motion to a corresponding swept volume, i.e., a volume swept by the corresponding robot or portion thereof in moving or transitioning between poses represented by the motion plan, for example, by generating a volumetric representation of the robot or portion thereof, and projecting the volumetric representation along a path defined by the trajectory of the robot or portion thereof. Also for example, the motion converter can convert the motion to a corresponding swept volume, for example, by generating a volumetric representation of the object (e.g., a non-robotic object such as a human), and projecting the volumetric representation of the object along a path defined by the known and / or extrapolated trajectory of the object. Advantageously, the motion planner 204a can simply queue the obstacles (e.g., swept volumes), and can not need to determine, track, or indicate a time of the corresponding motion or swept volume. While generally described as a motion converter 250 for a first robot 202 to convert motions of other robots (not shown in FIG. 2) to obstacles, in some implementations, other robotic control systems 200b of other robots operating in a shared workspace can provide obstacle representations (e.g., swept volumes) of particular motions to the motion planner 204a of the first robot 202. Figure 2

[0086] ​The path generator 252 generates a path from one pose, configuration, or state (e.g., a start pose, configuration, or state) to another pose or configuration or state (e.g., an end pose, configuration, state; or a goal pose, configuration, and state). The path generator 252 can determine or identify one or more feasible paths from a start (e.g., a start node or start pose or start state) to a goal (e.g., a goal node or goal pose or goal state; an end node or end pose or end state). For example, the path generator 252 can determine or identify an ordered sequence of nodes in the motion planning graph that provides a complete path from a start or current node to a goal or end node (i.e., a set of ordered nodes for which there is a respective valid transition between each pair of consecutive nodes in the complete path, e.g., represented by the existence of an edge coupling the nodes of the pair). As described above, each node can correspond to a respective pose, configuration, or state of a respective robot. The path generator 252 can use or perform any kind of path finding method, technique, and / or algorithm. For example, the path generator 252 can employ various methods, techniques, and / or algorithms to randomly or pseudo-randomly generate a path, select a sequence of nodes in the motion planning graph, where the nodes are connected to the next node in the sequence via an edge (i.e., an edge representing a valid transition between the poses represented by the connected nodes). The path generator 252 can generate a relatively large number of candidate paths between the start node and the end node, and thus a large number of candidate paths between the start pose, configuration, or state and the end pose, configuration, or state. In some implementations, the path generator 252 can determine or identify feasible paths or trajectories independent of a cost (e.g., a cost representing a probability or likelihood of experiencing a collision along the path), and thus create a set of feasible paths or candidate paths that can later be evaluated based at least in part on the probability or likelihood of experiencing a collision. In other cases, the path generator 252 can consider a cost (e.g., a cost representing a probability or likelihood of experiencing a collision along the path) in determining or identifying feasible paths or trajectories.

[0087] The collision evaluator 253 performs collision evaluation, also referred to as collision detection or collision analysis. In particular, the collision evaluator 253 optionally performs collision evaluation as part of determining whether a candidate path representing a transition or motion of a given robot 202 or portion thereof specified by a nominal trajectory will result in or is likely to result in a collision with an obstacle. As described above, the motion of other robots can advantageously be represented as obstacles. Thus, the collision evaluator 253 can determine whether the motion of one robot will result in or is likely to result in a collision with another robot moving in the shared workspace.

[0088] As described herein, collision assessment, detection, or analysis can be performed not only on candidate paths, but additionally or alternatively on nominal trajectories (e.g., specified trajectories), and in particular on nominal trajectories with various lag times introduced (e.g., to simulate actual trajectories that can lag behind nominal trajectories). In at least some implementations, collision assessment, detection, or analysis of nominal trajectories with various lag times can be performed for each of two or more robots (e.g., pairs of robots). Collision assessment, detection, or analysis can be performed for one or more nominal trajectories of each robot in a pair of robots. Collision assessment, detection, or analysis can be performed for each of those nominal trajectories (e.g., one, two, or more) with zero lag times and non-zero lag times, e.g., to evaluate pairs of trajectories for each permutation of lag times for the pair. As described herein, collision assessment, detection, or analysis can not necessarily produce a binary result, but can produce a non-binary value, e.g., representing a probability or likelihood of a collision between a pair of robots resulting from a path or trajectory.

[0089] In some implementations, collision assessor 253 implements software-based collision assessment, detection, or analysis, e.g., to perform bounding-box-bounding-box collision assessment, detection, or analysis based on a hierarchy of geometric (e.g., sphere) representations of volumes swept by a robot (e.g., first robot 202) or portions thereof in the course of movement. In some implementations, collision assessor 253 implements hardware-based collision assessment, detection, or analysis, e.g., to represent obstacles with a set of dedicated hardware logic circuits and to stream representations of motion through the dedicated hardware logic circuits. In hardware-based collision assessment, detection, or analysis, a collision detector can employ one or more configurable arrays of circuits, e.g., one or more FPGAs 259, and can optionally produce a Boolean collision assessment.

[0090] Cost setter 254 can set, update, and / or adjust costs or cost functions associated with transitions (e.g., edges in a motion planning graph) or motions (e.g., trajectories of a motion plan). For example, cost setter 254 can set, update, and / or adjust costs or cost functions based at least in part on collision assessment, detection, or analysis. For example, cost setter 254 can set relatively high cost values for edges or trajectories representing transitions between nodes or motions that result in or are likely to result in collisions. Also for example, cost setter 254 can set relatively low cost values for edges or trajectories representing transitions between nodes or motions that do not result in or are not likely to result in collisions. Setting, updating, and / or adjusting costs or cost functions can include setting, updating, or adjusting costs or cost functions that are logically associated with respective edges or trajectories via some data structure (e.g., a field in a record, a pointer in a list, a table).

[0091] In some embodiments, for example, the cost setter 254 can optionally set, update, or adjust the cost or cost function to at least partially represent a determined acceptable lag time for the respective nominal trajectory. This can advantageously enable the first robot control system 200a to select a nominal trajectory from a set of available or candidate nominal trajectories for the robot to perform a given task, e.g., to select a nominal trajectory with the longest or maximum determined acceptable lag time for generating a motion plan that enables more robust operation when the robot is subjected to real-world conditions while performing the task.

[0092] In some embodiments, additionally or alternatively, the cost setter 254 can set, update, or adjust the cost or cost function to at least partially represent one or more other parameters, such as one or more of: a severity of collision, a consumption or depletion of energy, and / or a time or delay of performance or completion.

[0093] The optional path analyzer 255 can use the motion planning graph 208 and the cost or cost function to determine or identify one or more suitable paths and / or select a single path (i.e., a selected path, e.g., an optimal or optimized path). For example, the optional path analyzer 255 can identify one or more paths that satisfy some specified criteria (e.g., a cost within a threshold upper limit) or even select a single path (e.g., select a single lowest cost path) from a set of candidate feasible paths determined or identified by the path generator 252. The identified paths can be named suitable paths because these paths represent options with an acceptably low probability or likelihood of collision. For example, the optional path analyzer 255 can constitute a lowest cost path optimizer that determines a lowest or relatively low cost path between two nodes (thus, between two poses, configurations, or states represented by respective nodes in the motion planning graph). The optional path analyzer 255 can use or perform any kind of path finding algorithm (e.g., a lowest cost path finding algorithm) that takes into account the cost values associated with each edge, which represent the probability or likelihood of collision, and optionally one or more other parameters (e.g., a severity of collision, a consumption or depletion of energy, and / or a time or delay of performance or completion). In some embodiments, cost-based optimization can alternatively or additionally be applied to nominal trajectories to advantageously enable the representation of an acceptable lag time in the cost value or cost function, as described herein, e.g., in addition to the probability or likelihood of collision, and optionally in addition to or instead of one or more of: a severity of collision, a consumption or depletion of energy, and / or a time or delay of performance or completion.

[0094] The nominal trajectory generator 256 is capable of generating nominal trajectories that the robot can follow, e.g., to accomplish a task. As described herein, the trajectory comprises an ordered sequence of poses or configurations or states parameterized by time through which the robot or at least part thereof can move, e.g., to accomplish a task. The nominal trajectory generator 256 can generate a trajectory for all generated paths, or only for selected paths or only for one selected path if the optional path analyzer 255 is employed.

[0095] The acceptable lag time evaluator 257 evaluates various candidate lag times for a given nominal trajectory to determine an acceptable lag time, i.e., a lag time for which no self-collision operation is ensured if the actual trajectory of the corresponding robot does not lag the nominal trajectory by more than the acceptable lag time. In a preferred approach, the acceptable lag time evaluator 257 takes into account not only the lag impact in the nominal trajectory of a given robot, but also the lag impact of the corresponding nominal trajectories of other robots operating in the shared workspace. Thus, the acceptable lag time evaluator 257 identifies an acceptable lag time for each nominal trajectory that assumes the worst case scenario in which the actual trajectories of all robots operating in the shared workspace experience their respective acceptable lag times. Thus, for example, the acceptable lag time evaluator 257 can determine an optimized, e.g., maximum, lag time for each robot that ensures no self-collision operation. Several approaches to determining an acceptable lag time are described herein, e.g., with respect to method 300 Figure 3 ), method 400 Figure 4 ), method 500 Figure 5 ), method 600 Figure 6 ), and / or method 700 Figure 7 .

[0096] The optional nominal trajectory analyzer 258 can use cost values or cost functions or some other objective function to determine, identify or select one or more suitable trajectories and / or to determine, identify or select a single trajectory (i.e., a selected trajectory, e.g., an optimal or optimized trajectory). For example, the nominal trajectory analyzer 258 can determine, identify or select one or more trajectories that satisfy some specified criteria (e.g., cost is within a threshold upper limit) or even a single trajectory (e.g., the trajectory with the lowest cost) from a set of feasible or candidate trajectories generated by the trajectory generator 256 (e.g., determine, identify or select the lowest cost trajectory). For example, the nominal trajectory analyzer 258 can constitute a lowest cost trajectory optimizer that determines a lowest or relatively low cost trajectory between two poses, configurations or states represented by respective nodes in the motion planning graph. For example, the nominal trajectory analyzer 258 can select a set of trajectories that optimize robust operation for all robots, e.g., by maximizing the sum of all lag times for all robots, for a given task or a set of given tasks to be performed by the robots, within a given time period. The nominal trajectory analyzer 258 can use or perform any kind of algorithm, e.g., a lowest cost trajectory finding algorithm, while taking into account the cost or cost function associated with each trajectory, where the cost or cost function represents one or more of a probability or likelihood of collision and an acceptable lag time, and optionally, a severity of collision, an expenditure or loss of energy and / or a time or delay of execution or completion.

[0097] Various algorithms and structures can be used to determine a lowest cost path and / or a lowest cost trajectory, including algorithms and structures that implement the Bellman-Ford algorithm, but other algorithms and structures can also be used, including but not limited to any such process that determines a lowest cost path or a lowest cost trajectory as a path between two nodes in the motion planning graph 208, or a trajectory specified by an ordered sequence of poses parameterized by time, such that the sum of the costs or weights of the edges or motions is minimized. This process improves motion planning techniques for the robots 102( Figure 1 )、202( Figure 2 ) by determining acceptable lag times, optionally selecting trajectories based on the determined acceptable lag times, monitoring actual trajectories to assess whether actual lag times are approaching or exceeding a margin or threshold (e.g., an acceptable lag time), and optionally selecting and / or taking remedial measures if necessary.

[0098] Although not shown, the motion planner 204a can optionally include a look-ahead evaluator that can cause an improvement to be taken, e.g., in response to a determination of the existence or occurrence of a blockage or potential blockage condition, or a determination that a blockage or potential blockage location will occur when the given robot and / or other robots move along respective trajectories. In at least some implementations, the look-ahead evaluator is able to determine or select a type of improvement to be taken, e.g., select from a set of different types of improvements based on one or more criteria. As described in U.S. Patent Application 63 / 327,917, filed April 6, 2022, one or more of various types of improvements can be implemented (wherein referred to as remedial actions). For example, a new, revised, or replacement first motion plan can be generated to move the given robot to the first goal based on an analysis of a second motion plan that moves the given robot from the first goal. Also, for example, a new, revised, or replacement motion plan can be generated for another robot that is or can be blocking the given robot. Also, for example, a new order of a set of goals can be determined or generated, the set of goals including the first goal and at least a second goal. This can be determined or generated pseudo-randomly or based on one or more heuristics (e.g., always attempt to move the first goal one position downstream relative to the order of the set of goals being modified).

[0099] Although not shown, the motion planner 204a can optionally include an optional multi-path analyzer that is able to analyze a total cost or aggregate cost associated with the trajectories of two or more motion plans (e.g., a total cost of the first motion plan and the second motion plan), e.g., as described in U.S. Patent Application 63 / 327,917, filed April 6, 2022. This can be used to identify a combination of motion plans that has a lowest total cost. For example, the multi-path analyzer can consider a total cost or aggregate cost of two or more options of the first motion plan in conjunction with the second motion plan, e.g., using or performing any kind of lowest cost lookup algorithm while considering a cost value that represents a likelihood of a relevant collision, and optionally one or more of the following: an acceptable lag time, a severity of a collision, a consumption or depletion of energy, and / or a time or delay to perform or complete a transformation associated with a respective edge representation.

[0100] Optionally, the motion planner 204a can include a trimmer 260. The trimmer 260 can receive information representing the completion of motions by other robots, which is referred to herein as a motion completion message 209. Alternatively, a flag can be set to indicate completion. In response, the trimmer 260 can remove obstacles or portions of obstacles representing motions that have now been completed. This can allow a new motion plan to be generated for a given robot, which can be more efficient, or allow a given robot to perform a task that was previously prevented by the motion of another robot. This approach advantageously allows the motion transformer 250 to ignore the timing of motions when generating an obstacle representation of the motions, while still achieving better throughput than using other techniques. The motion planner 204a can also send a signal, cue, or trigger to cause the collision evaluator 253 to perform new collision detection or evaluation with the obstacles modified, to produce an updated motion planning graph in which the edge weights or costs associated with the edges have been modified, and to cause the cost setter 254, optional path analyzer 255, and nominal trajectory analyzer 258 to update the cost values and to determine new or revised paths, trajectories, and / or motion plans accordingly.

[0101] The motion planner 204a can optionally include an environment transformer 263 that transforms output from an optional sensor 262 (e.g., a digital camera) (e.g., a digitized representation of the environment) into a representation of obstacles. Thus, the motion planner 204a is able to perform motion planning that takes into account transient objects (e.g., people, animals, etc.) in the environment.

[0102] The one or more processors processor 222 and / or the motion planner 204a can be, or can include, any logic processing unit, such as one or more central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic controllers (PLCs), etc. Non-limiting examples of commercially available computer systems include, but are not limited to: Celeron, Core, Core 2, Itanium, and Xeon series microprocessors provided by Intel Corporation of America; K8, K10, Bulldozer, and Bobcat series microprocessors provided by Advanced Micro Devices, Inc. of America; A5, A6, and A7 series microprocessors provided by Apple Inc. of America; Snapdragon series microprocessors provided by Qualcomm Incorporated of America; and SPARC series microprocessors provided by Oracle Corporation of America. Figure 2The construction and the operation of the various structures shown in the figures can implement or employ structures, techniques, and algorithms described in or similar to those described in International Patent Application No. PCT / US2017 / 036880, filed June 9, 2017; International Patent Application Publication No. WO 2016 / 122840, filed January 5, 2016; U.S. Patent Application No. 62 / 616783, filed January 12, 2018; International Patent Application PCT / US2021 / 013610, published as WO 2021 / 150439; and / or U.S. Patent Application 63 / 327917, filed April 6, 2022.

[0103] While not required, many implementations will be described in the general context of computer-executable instructions, such as program application modules, objects, or macros, stored on computer or processor readable media and executed by one or more computers or processors, which can be implemented as various structures for obstacle representation, collision assessment, and other motion planning operations.

[0104] Motion planning operations can include, but are not limited to, generating or converting one, more, or all of: a geometric model-based robot geometry representation, a task specification 215, and an optional representation of a volume occupied by the robot at various states or poses and / or during movement between states or poses (e.g., a swept volume), in digital form, such as a point cloud, a Euclidean distance field, a data structure format (e.g., a hierarchical format, a non-hierarchical format), and / or a curve (e.g., a polynomial or spline representation). Motion planning operations can optionally include, but are not limited to, generating or converting one, more, or all of: representing an obstacle in static or persistent form and / or perception data for a static or transient obstacle in digital form, such as a point cloud, a Euclidean distance field, a data structure format (e.g., a hierarchical format, a non-hierarchical format), and / or a curve (e.g., a polynomial or spline representation).

[0105] Motion planning operations can include, but are not limited to, assessing, detecting, determining, or predicting collisions between various poses, configurations, or states of the robot, or motion of the robot along respective trajectories between states or poses, using various collision assessment techniques or algorithms (e.g., software-based, hardware-based).

[0106] In some implementations, motion planning operations can include, but are not limited to: determining one or more motion planning graphs, motion plans, or roadmaps with nominal trajectories; acceptable latencies for the nominal trajectories; storing the determined planning graphs, motion plans, or roadmaps or acceptable latencies, and / or providing the planning graphs, motion plans, or roadmaps or acceptable latencies to control operation of one or more robots, and optionally monitoring operation of the robots, and selecting and / or taking remedial measures as necessary.

[0107] In one embodiment, the function call or similar process performs collision detection or evaluation, and returns a Boolean value in response. The collision evaluator 253 can be implemented via one or more field programmable gate arrays (FPGAs) 259 and / or one or more application specific integrated circuits (ASICs) to enable low latency, relatively low power consumption, and increased amount of information that can be processed while performing collision detection.

[0108] In various embodiments, these operations can be performed entirely in hardware circuitry, as software stored in a memory such as system memory 224a, and executed by one or more hardware processors 222a, e.g., one or more microprocessors, digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing unit (GPU) processors, programmable logic controllers (PLCs), electrically programmable read only memories (EEPROMs), or as a combination of hardware circuitry and software stored in memory.

[0109] Various aspects of sensing, graph construction, collision detection, and path searching that can be employed in whole or in part are also described in International Patent Application No. PCT / US2017 / 036880, filed June 9, 2017; International Patent Application No. WO 2016 / 122840, filed January 5, 2016; U.S. Patent Application No. 62 / 616783, filed January 12, 2018; U.S. Patent Application No. 62 / 856548, filed June 3, 2019; International Patent Application No. PCT / US2020 / 039193 (Publication No. WO 2020 / 263861), filed June 23, 2020; International Patent Application No. PCT / US2021 / 013610, Publication No. WO 2021 / 150439; and / or U.S. Patent Application No. 63 / 327917, filed April 6, 2022. Those skilled in the relevant art will appreciate that the illustrated embodiments, as well as other embodiments, can be practiced with other system configurations and arrangements and / or other computing system configurations and arrangements, including robots, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, personal computers (“PCs”), networked PCs, mini-computers, mainframe computers, etc. These embodiments or examples, or portions thereof (e.g., at configuration time and runtime), can be practiced in a distributed computing environment where tasks or modules are performed by remote processing devices, which are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices or media. However, where certain types of information are stored and how they are stored can be important for helping to improve motion planning.

[0110] For example, various motion planning solutions “bake in” a roadmap (i.e., a motion planning graph) into a processor (e.g., FPGA 259), and each edge in the roadmap corresponds to a non-reconfigurable Boolean circuit of the processor. One problem with designs that “bake in” a planning graph into a processor is that the processor circuit is limited in that it cannot store multiple or large planning graphs, and is typically not reconfigurable for different robots.

[0111] One solution is to provide a reconfigurable design that puts planning graph information into memory. This approach stores information in memory rather than baking it into circuitry. Another approach is to use templated reconfigurable circuitry instead of memory.

[0112] As described above, some information (e.g., a robot geometry model) can be captured, received, input, or provided during configuration time (i.e., before run time). The received information can be processed during configuration time to generate processed information (e.g., a motion planning graph 208) to speed up operations or reduce computational complexity during run time.

[0113] During run time, collision detection can be performed for the entire environment (i.e., a shared workspace), including determining whether any part of a robot will or is expected to collide with another part of the robot itself, with other robots or parts thereof, with persistent or static obstacles in the environment, or with transient obstacles in the environment (e.g., people) having unknown trajectories, for any pose or movement between poses.

[0114] The first robot control system 200a or some other processor-based system can include a lag monitor 264 (interchangeably referred to as a lag time monitor 264) and / or a remedial action selector 266. The lag time monitor 264 and / or the remedial action selector 266 operate during run time (e.g., during operation of one or more robots to complete one or more tasks), monitor operation of the robots, and select and take one or more remedial actions if necessary (e.g., if an actual lag time approaches or exceeds a margin or threshold (e.g., a corresponding acceptable lag time), slow down or stop one or more robots).

[0115] The lag time monitor 264 monitors the actual lag time of the actual trajectory executed by the robot and compares the actual lag time to a margin or threshold (e.g., a margin or threshold representing or even equal to the corresponding acceptable lag time) for the corresponding nominal trajectory. As previously described, while in some cases the actual trajectory can match the nominal trajectory, in many cases the actual trajectory will not match the nominal trajectory, typically with at least some of the poses, configurations, states, or motions of the actual trajectory being executed with a lag in timing from those specified by the nominal trajectory. The lag time monitor 264 determines the actual lag time, as well as whether the amount of monitoring of the actual lag time of the actual trajectory of the robot approaches or exceeds the margin or threshold for the corresponding robot (e.g., the corresponding determined acceptable lag time for the corresponding nominal trajectory). For example, if the actual lag time for any of the robots operating in the shared workspace exceeds the corresponding threshold (e.g., approaches or exceeds the corresponding acceptable lag time), the lag time monitor 264 can provide an indication (e.g., set a flag, send information, and / or invoke the remedial action selector 266). In some embodiments, the margin or threshold can be set to include a desired safety factor, e.g., less than a set amount or a defined percentage of the determined acceptable lag time for the corresponding nominal trajectory. Thus, in some embodiments, the margin or threshold can be set to equal the corresponding acceptable lag time, while in other embodiments the margin or threshold can be less than (e.g., less than 10%) the corresponding acceptable lag time, such that detection and remedial action is taken before a self-collision can occur.

[0116] The optional remedial action selector 266 selects one or more appropriate remedial actions and / or causes the selected remedial action(s) to be taken in response to determining that the monitored amount of lag time exceeds the corresponding threshold (e.g., the determined acceptable lag time) for the corresponding trajectory of any of the robots. Determining that the actual lag time exceeds the corresponding threshold (e.g., the determined acceptable lag time) means that the collision-free movement is no longer assured, and thus one or more remedial actions can need to be taken.

[0117] The remedial action can include one or more of the following: stopping movement of one or more robots, slowing movement of one or more robots, and / or speeding up movement of one or more robots. For example, movement of one, two, more, or even all robots can be stopped. Also, for example, movement of one, two, more, or even all robots can be slowed, e.g., by different amounts relative to one another. Also, for example, movement of one, two, more, or even all robots can be sped up, e.g., by different amounts relative to one another. Also, for example, movement of one or more robots can be stopped while movement of one or more robots is slowed. Also, for example, movement of one or more robots can be stopped while movement of one or more robots is sped up or held constant. Also, for example, movement of one or more robots can be slowed while movement of one or more robots is sped up or held constant. Also, for example, movement of one or more robots can be stopped while movement of one or more robots is slowed and while movement of one or more robots is sped up or held constant. To take at least one remedial action, the processor-based system can send instructions to one or more actuators (e.g., via the actuators 218a-c of the motion controller 220) to cause movement of the robots to stop, slow, speed up, or even proceed along a respective trajectory specified by the respective nominal trajectory. This can then restart movement of the robots, e.g., from where movement stopped or slowed, although in some cases can include returning to a start of the respective nominal trajectory and resuming from there.

[0118] Figure 3An operational method 300 of a processor-based system that performs motion planning to control robots that will operate in a shared workspace is shown, in accordance with at least one illustrated embodiment. In accordance with at least one illustrated embodiment, the processor-based system includes one or more processors that execute processor-executable instructions to determine, for each robot, an acceptable lag time based on nominal trajectories of other robots without having to account for the effects of lag time in the nominal trajectories of the other robots, and generate or select a motion plan that specifies a nominal trajectory that is associated with the corresponding acceptable lag time. For example, the method 300 can be performed during a configuration or "pre-run" time, i.e., a time during which some or all of the motion planning occurs, e.g., to generate the nominal trajectories and / or motion plans for each robot. The configuration or "pre-run" time can occur before the robots move in accordance with the nominal trajectories of the corresponding robots, e.g., before a run time, where the run time is a time during which the one or more robots execute the corresponding motion plans. This advantageously allows some of the most computationally intensive work to be performed before the run time, when responsiveness is not as much of a concern. In at least some embodiments, the method 300 or portions thereof are performed during a run time, i.e., a time during which the one or more robots are performing tasks, after the configuration time or pre-run time. In other embodiments, the method 300 is performed during a run time, i.e., a time during which the one or more robots are performing tasks, after the configuration time or pre-run time.

[0119] The method 300 begins, at 302, e.g., in response to a start-up or power-up of the system or components thereof, a receipt of information or data, or a call or invocation of a routine or program.

[0120] While Figure 3 Not shown in the method 300, but that can be performed, is that the method 300 can generate paths; perform collision assessments or detections or analyses on the paths; set, update or adjust the costs of the edges of the paths; and optionally perform analyses to identify or select paths based on the costs (e.g., the lowest cost path). This is described above with reference to the method 200, and at least some aspects of the references cited herein are described. These are omitted from the method 300 for brevity. Figure 2 Not shown in the method 300, but that can be performed, is that the method 300 can generate paths; perform collision assessments or detections or analyses on the paths; set, update or adjust the costs of the edges of the paths; and optionally perform analyses to identify or select paths based on the costs (e.g., the lowest cost path). This is described above with reference to the method 200, and at least some aspects of the references cited herein are described. These are omitted from the method 300 for brevity.

[0121] Optionally, in 304, at least one processor of the processor-based system generates or accesses (e.g., receives, retrieves) a set of nominal trajectories for each robot I. As described previously, a nominal trajectory is a “prescribed” trajectory in which each trajectory includes an ordered set or sequence of poses, configurations, or states of the robot that extends between two poses, configurations, and states (e.g., a start pose, configuration, or state; an end pose, configuration, state) and includes a respective timing for each pose or configuration. The timing can be specified in relative terms (timing is defined by a relative offset from a previous pose) or absolute terms (timing is defined by a relative offset from the start of trajectory execution, e.g., relative to a common clock). Applicants note that while any given trajectory can correspond to smooth motion of the robot, the term “trajectory” as used herein is not limited to this and will generally specify motion that is not smooth, nor will it define straight paths for the robot or portions thereof. In at least some cases, a nominal trajectory can specify or include one or more pauses in motion of the robot or portions thereof, and / or can specify changes in direction or path direction of motion of the robot or portions thereof even reversals, changes in rate, changes in velocity, and changes in acceleration, and can not be smooth motion in terms of direction, rate, velocity, or time. For example, a nominal trajectory can specify a time-parameterized ordered set or sequence of poses through which the robot or portions thereof move to perform a task or partial task. Execution of any given task can employ one or more nominal trajectories. As described herein, actual motion or actual trajectories of the robot or portions thereof can deviate from the corresponding nominal trajectories, e.g., due to unexpected delays in transitioning between poses (e.g., due to a need to loiter or dwell on a target object for longer than expected).

[0122] Any of a variety of techniques and / or algorithms can be employed to generate nominal trajectories, e.g., sample-based motion planners (SBMPs) such as probabilistic roadmaps (PRMs) or rapidly-exploring random trees (RRTs, RTT*s), stable sparse RRT*s (SST*s), and / or fast-matching trees (FMTs). In at least one implementation, nominal trajectories can be generated via the systems, methods, and techniques described in International Patent Application PCT / US2021 / 013610 (Publication No. WO 2021150439A1). The teachings of the present patent application are not limited to a particular form of nominal trajectory generation. Advantageously, the teachings herein include any combination or permutation of the following aspects: computationally efficient use of nominal trajectories to generate motion plans with associated acceptable lag times, monitoring of deviations of actual motion from nominal trajectories in terms of lag times, optionally taking remedial action for deviations that exceed respective acceptable lag times, and / or optionally using determined lag times to select a set of nominal trajectories for a motion plan to enhance robustness of operation.

[0123] At 306, the at least one processor of the processor-based system initializes a robot counter I, e.g., sets the robot counter equal to the integer value of 1. At 308, the at least one processor of the processor-based system executes an outer iteration loop, performing an iteration for each of the two or more robots that will operate in the shared workspace, or until a stopping condition is reached (e.g., a maximum lag time is determined that ensures collision-free movement).

[0124] At 310, the at least one processor of the processor-based system initializes a nominal trajectory counter J, e.g., sets the nominal trajectory counter equal to the integer value of 1.

[0125] At 312, the at least one processor of the processor-based system executes an inner iteration loop, performing an iteration for each nominal trajectory of a given robot, at least until a stopping condition is reached.

[0126] At 314, the at least one processor of the processor-based system determines a respective acceptable lag time for a current nominal trajectory (i.e., current in the inner iteration loop) of a current robot (i.e., current in the outer iteration loop). The respective acceptable lag time reflects a maximum acceptable delay or lag in the current nominal trajectory J that still guarantees collision-free movement of the current robot relative to other robots that are each moving in accordance with a respective nominal trajectory when the current nominal trajectory of the current robot is executed (the respective acceptable lag time is introduced into the current nominal trajectory of the current robot). The acceptable lag time for a given robot I guarantees that no self-collision will occur as long as all other robots are operating in accordance with their respective nominal trajectories (i.e., lag time = 0). In other words, as long as the respective current actual lag time of a given robot is less than or equal to (i.e., not greater than) the respective acceptable lag time of that robot, and the other robots are operating in accordance with their nominal trajectories, the robots will not collide with each other (i.e., collision-free movement between the robots is guaranteed).

[0127] At 316, the at least one processor of the processor-based system determines whether each of the nominal trajectories of the given robot has been considered, e.g., determines whether the nominal trajectory counter J is equal to the total number of nominal trajectories of the given robot (e.g., J = M?). If each of the nominal trajectories of the given robot has not been considered, control passes to 318, where the nominal trajectory counter is incremented (e.g., J = J + 1), and then control passes back to 312 to consider the next nominal trajectory of the given robot. If each of the nominal trajectories of the given robot has been considered (e.g., J = M), control passes directly to 320.

[0128] Optionally, in 320, the at least one processor of the processor-based system selects a maximum of the set of determined acceptable latencies determined for the given robot.

[0129] In 322, the at least one processor of the processor-based system provides one or more of the determined acceptable latencies for the given robot (e.g., provides the selected maximum of the determined acceptable latencies) for use in determining a motion plan for at least the given robot and / or controlling operation of the given robot. For example, this can include providing the one or more determined acceptable latencies to different processors, or transferring the one or more determined acceptable latencies to different registers of a processor.

[0130] In 324, the at least one processor of the processor-based system determines whether every robot in the set of robots has been considered, e.g., determines whether a robot counter I is equal to a total number of robots (e.g., I = N?). If every robot in the set of robots has not been considered (e.g., I < N), then control passes to 326, in which the at least one processor of the processor-based system iterates the robot counter (e.g., I = I + 1), and then control returns to 308 to consider the next robot. If every robot in the set of robots has been processed (e.g., I = N), then control passes directly to 328.

[0131] In 328, the at least one processor of the processor-based system generates or selects a respective motion plan for each robot. The motion plan can be or can represent a nominal trajectory of the robot associated with the maximum acceptable latency. The motion plan can be or can represent a nominal trajectory of one or more robots. This approach can advantageously enhance the robustness of the motion plans generated by the motion planning, and thereby improve operation of the robots executing the resulting motion plans.

[0132] In 330, the at least one processor of the processor-based system provides the respective motion plan to each robot or motion controller to cause the robot to move in accordance with the respective motion plan.

[0133] In 332, the method 300 can terminate, e.g., until invoked again. While the method 300 is described in an ordered sequence, in many implementations, various acts or operations will be performed concurrently or in parallel, and / or can include additional acts and / or omit some acts.

[0134] Figure 4An operational method 400 of a processor-based system that performs motion planning to control robots that will operate in a shared workspace is shown, in accordance with at least one illustrated embodiment. The processor-based system includes one or more processors that execute processor-executable instructions to determine, for each robot, an acceptable lag time based on nominal trajectories of other robots, and at the same time must take into account the effect of lag times in the nominal trajectories of the other robots, and generate or select a motion plan that specifies a nominal trajectory that is associated with the corresponding acceptable lag time. For example, the method 400 can be performed during a configuration or "pre-run" time, i.e., a time during which some or all of the motion planning occurs, e.g., to generate a nominal trajectory and / or a motion plan for each robot. The configuration or "pre-run" time can occur before the robots move in accordance with the nominal trajectories of the corresponding robots, e.g., before a run time, where the run time is a time during which the one or more robots execute the corresponding motion plans. This advantageously allows some of the most computationally intensive work to be performed before the run time, when responsiveness is not as much of a concern. In at least some embodiments, the method 400 or portions thereof are performed during a run time, i.e., a time during which the one or more robots are performing tasks, after the configuration time or pre-run time. In some other embodiments, the method 400 is performed during a run time, i.e., a time during which the one or more robots are performing tasks, after the configuration time or pre-run time.

[0135] The method 400 begins, at 402, e.g., in response to a start-up or power-up of the system or components thereof, a receipt of information or data, or a call or invocation of a routine or program.

[0136] While Figure 4 Not shown in the method 400, but which can be performed, are generating paths; performing collision assessments or detections or analyses of the paths; setting, updating or adjusting costs of edges of the paths; and optionally performing analyses to identify or select paths (e.g., a lowest cost path) based on the costs. The above is described with reference to the method 300, and at least some aspects of the references cited herein. These are omitted from the method 400 for brevity. Figure 2 This is described above, and at least some aspects of the references cited herein. These are omitted from the method 400 for brevity.

[0137] Optionally, in 404, at least one processor of the processor-based system generates or accesses (e.g., receives, retrieves) a set of nominal trajectories for each robot (i). As described previously, a nominal trajectory is a “prescribed” trajectory in which each trajectory includes an ordered set or sequence of poses, configurations, and states of the robot between two poses, configurations, or states, and includes a respective timing for each pose or configuration. The timing can be specified in relative terms (timing is defined by a relative offset from a previous pose) or absolute terms (timing is defined by a relative offset from the start of execution of the trajectory, e.g., relative to a common clock). Applicants note that while any given trajectory can correspond to smooth motion of the robot, the term “trajectory” as used herein is not limited to this and will generally specify motion that is not smooth, nor will it define straight paths for the robot or portions thereof. In at least some cases, a nominal trajectory can specify or include one or more pauses in motion of the robot or portions thereof, and / or can specify changes in direction or path direction of motion of the robot or portions thereof even reversals, changes in rate, changes in velocity, and changes in acceleration, and can not be smooth motion in terms of direction, rate, velocity, or time. For example, a nominal trajectory can specify a time-parameterized ordered set or sequence of poses through which the robot or portions thereof move to perform a task or partial task. Execution of any given task can employ one or more nominal trajectories. As described herein, actual motion or actual trajectories of the robot or portions thereof can deviate from the respective nominal trajectories, e.g., due to unexpected delays in transitioning between poses (e.g., due to a need to loiter or dwell on a target object for longer than expected).

[0138] Any of a variety of techniques and / or algorithms can be employed to generate nominal trajectories, e.g., sample-based motion planners (SBMPs), e.g., probabilistic roadmaps (PRMs) or rapidly-exploring random trees (RRTs, RTT*s), stable sparse RRT*s (SST*s), and / or fast-matching trees (FMTs). In at least one implementation, nominal trajectories can be generated via the systems, methods, and techniques described in International Patent Application PCT / US2021 / 013610 (Publication No. WO 2021150439A1). The teachings of the present patent application are not limited to a particular form of generation of nominal trajectories. Advantageously, the teachings herein include any combination or permutation of the following aspects: computationally efficient use of nominal trajectories to generate motion plans with associated acceptable lag times, monitoring of deviations of actual motion from nominal trajectories in terms of lag times, optionally taking remedial action for deviations that exceed a respective acceptable number of lags, and / or optionally using determined lag times to select a set of nominal trajectories for a motion plan to enhance robustness of operation.

[0139] At 406, the at least one processor of the processor-based system initializes a robot counter I, e.g., sets the robot counter equal to the integer value of 1. At 408, the at least one processor of the processor-based system executes an outer iteration loop, performing an iteration for each robot of the two or more robots that will operate in the shared workspace, or until a stopping condition is reached (e.g., determining a maximum lag time that ensures collision-free motion between the robots).

[0140] At 410, the at least one processor of the processor-based system initializes a nominal trajectory counter J, e.g., sets the nominal trajectory counter equal to the integer value of 1. At 412, the at least one processor of the processor-based system executes an inner iteration loop, performing an iteration for each nominal trajectory of the given robot, at least until a stopping condition is reached.

[0141] At 414, the at least one processor of the processor-based system determines a respective acceptable lag time for the current nominal trajectory (i.e., current in the inner iteration loop) of the current robot (i.e., current in the outer iteration loop). The respective acceptable lag time reflects a maximum acceptable delay or lag in the current nominal trajectory J that still guarantees collision-free motion of the current robot relative to the other robots that are each moving in accordance with a respective nominal trajectory (the respective acceptable lag time is introduced into the current robot’s nominal trajectory) when the current robot executes the nominal trajectory (the respective acceptable lag time is introduced into the current robot’s nominal trajectory). The acceptable lag time for the given robot I guarantees that no collision will occur as long as the current actual lag time of all other robots is less than their respective acceptable lag time. That is, the other robots do not have to run on their respective nominal trajectories (i.e., lag time = 0). In other words, the robots will not collide with each other or themselves (i.e., collision-free motion between the robots is guaranteed) as long as the respective current actual lag time of each of the robots operating in the shared workspace is less than or equal to (i.e., not greater than) the respective acceptable lag time of the robot.

[0142] At 416, the at least one processor of the processor-based system determines whether each of the nominal trajectories of the given robot has been considered, e.g., determines whether the nominal trajectory counter J is equal to the total number of nominal trajectories of the given robot (e.g., J = M?). If each of the nominal trajectories of the given robot has not been considered, then control passes to 418, where the nominal trajectory counter is incremented (e.g., J = J + 1), and then control returns to 412 to consider the next nominal trajectory of the given robot. If each of the nominal trajectories of the given robot has been considered (e.g., J = M), then control passes directly to 420.

[0143] Optionally, in 420, the at least one processor of the processor-based system selects a maximum of the set of determined acceptable latencies determined for the given robot. In 422, the at least one processor of the processor-based system provides one or more determined acceptable latencies (e.g., the selected maximum of the determined acceptable latencies) for the given robot for use in determining a motion plan for at least the given robot and / or controlling operation of the given robot. For example, this can include providing the one or more determined acceptable latencies to different processors, or transferring the one or more determined acceptable latencies to different registers of a processor.

[0144] In 424, the at least one processor of the processor-based system determines whether every robot has been considered, e.g., determines whether a robot counter I is equal to a total number of robots (e.g., I = N?). If every robot has not been considered (e.g., I < N), control passes to 426, in which the at least one processor of the processor-based system iterates the robot counter (e.g., I = I + 1), and then control returns to 408 to consider the next robot. If every robot has been processed (e.g., I = N), control passes directly to 428.

[0145] In 428, the at least one processor of the processor-based system generates or selects a respective motion plan for each robot. The motion plan can be or can represent a nominal trajectory of the robot associated with the maximum acceptable latency. The motion plan can be or can represent a nominal trajectory of one or more robots. This approach can advantageously enhance the robustness of the motion plans generated by the motion planning, and thereby improve operation of the robots executing the resulting motion plans.

[0146] In 430, the at least one processor of the processor-based system provides the respective motion plan to each robot or each motion controller to cause the robot to move in accordance with the respective motion plan.

[0147] In 432, the method 400 can terminate, e.g., until invoked again. While the method 400 is described in an ordered sequence, in many implementations, various acts or operations will be performed concurrently or in parallel, and / or can include additional acts and / or omit some acts.

[0148] As described herein, to determine an acceptable lag time, the processor-based system can perform a collision assessment to determine which of a set of candidate lag times will or can result in a collision. One approach to collision assessment is to use a swept volume of the robot to perform the collision assessment. The swept volume represents the volume swept by the robot or portion thereof as it moves along a given trajectory. The swept volume is specified by the geometry and kinematics of the given robot and by the start and end positions specified by the given trajectory. Notably, the swept volume itself is not affected by the particular times at which the motions specified by the trajectory are performed. Thus, the swept volume can advantageously be determined prior to the run time at which the robot performs or executes the task. Thus, the processor-based system can generate a swept volume for each of a plurality (e.g., a multitude) of paths for each robot that will operate in the shared workspace from which the trajectories are generated for the robots. The processor-based system can use the swept volumes to perform the collision assessment. For example, for each of two or more robots, the processor-based system can perform a collision assessment between i) at least a portion of a respective sample trajectory of a respective nominal trajectory of the robot into which at least one respective lag time is introduced, and ii) at least a portion of a respective sample of each of the respective trajectories of the other of the two or more robots into which at least one respective lag time is introduced. In response to determining that a collision will or can occur between the robot and at least one other of the two or more robots (e.g., represented as a probability or likelihood), the processor-based system can identify, for one or more nominal trajectories of the robot, a respective lag time that is less than the respective lag time that resulted in the determination that a collision will occur as the respective acceptable lag time. The acceptable lag time can represent a maximum acceptable delay in timing relative to the pose of the nominal trajectory that still guarantees that movement of the robot through the sequence of poses specified by the nominal trajectory remains collision free (for the robot system, self-collision free) relative to movement of the other of the two or more robots. The following Figure 5 and Figure 6 Non-limiting approaches to determining acceptable lag times using swept volumes are described.

[0149] Figure 5An operational method 500 of a processor-based system for generating swept volumes for one or more trajectories of each robot of a plurality of robots to be operated in a shared workspace is shown, in accordance with at least one illustrated embodiment. In accordance with at least one illustrated embodiment, the processor-based system includes one or more processors that execute processor-executable instructions to generate the swept volumes. For example, the method 500 can be performed during a configuration or “pre-run” time, i.e., a time during which some or all of motion planning occurs, e.g., to generate a nominal trajectory and / or motion plan for each robot. For example, the configuration or “pre-run” time can be prior to a run time, where the run time is a time during which the one or more robots execute the respective motion plans. This advantageously allows some of the most computationally intensive work to be performed prior to the run time, as responsiveness is not as much of a concern at this time. In at least some embodiments, the robot configuration method 500 or portions thereof are performed during the run time (i.e., the time during which the one or more robots are performing a task) after the configuration time or pre-run time. Optionally, the configuration time operations can be performed via a different processor-based system than the processor-based system that performs the run time operations. For example, the method 500 can optionally be used to perform the methods of 3, Figure 3 , 4 and / or 6, e.g., as part of performing collision detection (see method 600, Figure 6 ) in determining the respective acceptable lag times 314( Figure 3 ) and 414( Figure 4 ).

[0150] The method 500 of generating swept volumes begins at 502, e.g., in response to a startup or power-up of the system or components thereof, a receipt of information or data, or a call or invocation of a routine or program.

[0151] In 504, at least one processor of the processor-based system initializes a robot counter I (e.g., I = 1). In 506, at least one processor of the processor-based system performs an outer robot processing loop. The outer robot processing loop allows at least one processor of the processor-based system to perform the generation of swept volumes for each robot of the robots to be operated in the shared workspace.

[0152] In 508, at least one processor of the processor-based system initializes a trajectory counter J (e.g., J = 1). In 510, at least one processor of the processor-based system performs an inner nominal trajectory processing loop. The inner nominal trajectory processing loop is nested within the outer robot processing loop. The inner nominal trajectory processing loop allows at least one processor of the processor-based system to generate a respective swept volume for each nominal trajectory of one or more nominal trajectories of a given robot.

[0153] At 512, the at least one processor of the processor-based system generates a swept volume representation representing a volume swept by at least a portion of a given robot I as it moves between poses of a set or sequence of poses specified by a given nominal trajectory J. Thus, for each of two or more robots, and for each of one or more nominal trajectories, the processor-based system can generate a swept volume representation representing a volume swept by at least a portion of the robot between a set of poses specified by the nominal trajectory, from at least one time of the nominal trajectory to another time of the nominal trajectory. The swept volume can be generated using any of a variety of techniques, including digitally representing the robot or portion thereof with one or more (e.g., layered) data structures or point clouds, and projecting the digital representation along a path or trajectory to generate a digital representation (e.g., a set of voxels) of the swept volume.

[0154] At 514, the at least one processor of the processor-based system determines whether all trajectories of a given robot have been processed (e.g., J = M?). If all trajectories of the given robot have not been processed, control passes to 516, where a trajectory counter is incremented (e.g., J = J + 1), and control then returns to the top of the inner nominal trajectory processing loop 510. If all nominal trajectories have been processed (e.g., J = M), control passes directly to 518.

[0155] At 518, the at least one processor of the processor-based system determines whether all robots have been processed (e.g., I = N?). If all robots have not been processed, control passes to 520, where a robot counter is incremented (e.g., I = I + 1), and control then returns to the top of the outer robot processing loop 506. If all robots have been processed (e.g., I = N), control passes directly to 522.

[0156] At 522, the method 500 can terminate, e.g., until invoked again. While the method 500 is described in an ordered sequence, in many implementations various acts or operations will be performed concurrently or in parallel, and / or can include additional acts and / or omit some acts.

[0157] Figure 6An operational method 600 of a processor-based system for performing collision assessments on a plurality of robots and a plurality of trajectories of each of the robots is shown, in accordance with at least one illustrated embodiment. The method 600 can be used to perform collision assessments to identify a feasible path between a start node and an end node in a motion planning graph. Additionally or alternatively, the method 600 can be used to perform collision assessments to determine respective acceptable lag times for one or more nominal trajectories of robots operating in a shared workspace. In accordance with at least one illustrated embodiment, the processor-based system includes one or more processors that execute processor-executable instructions to perform collision assessments, e.g., using swept volumes. A swept volume represents or models a volume swept by a robot or portion thereof, e.g., in transitioning between various poses, configurations, or states along a path represented in a motion planning graph between a start node and an end node, or e.g., in moving along a nominal trajectory between a start pose, configuration, or state to an end pose, configuration, or state. For example, the method 600 can advantageously use swept volumes generated via the method 500. For example, the method 600 can be performed during a configuration or “pre-run” time, i.e., a time during which some or all of a motion planning occurs, e.g., to generate nominal trajectories and / or motion plans for each robot. For example, the configuration or “pre-run” time can precede a run time, where the run time is a time during which the one or more robots execute respective motion plans. This advantageously allows some of the most computationally intensive work to be performed prior to the run time, as responsiveness is not as much of a concern at this time. In at least some embodiments, the robot configures the method 600 or portions thereof to be performed during the run time (i.e., the time during which the one or more robots are performing a task) after the configuration time or pre-run time.

[0158] The method 600 of determining acceptable lag times begins at 602, e.g., in response to a startup or power-up of the system or components thereof, a receipt of information or data, or a call or invocation of a routine or program.

[0159] In 604, at least one processor of the processor-based system initiates an external robot pair collision assessment loop. The external robot pair collision assessment loop allows the at least one processor of the processor-based system to assess or estimate a probability or likelihood of a collision occurring between robots in each pair of robots that will operate in a shared workspace. The external robot pair collision assessment loop can account for collisions that can occur in each combination or permutation of robot pairs.

[0160] In 606, the at least one processor of the processor-based system initializes a trajectory counter J (e.g., J = 1). In 608, the at least one processor of the processor-based system executes an inner nominal trajectory collision assessment loop. The inner nominal trajectory collision assessment loop is nested within the outer robot collision assessment loop. The inner nominal trajectory collision assessment loop allows the at least one processor of the processor-based system to assess or estimate a collision probability or likelihood for each of the one or more nominal trajectories of a given robot pair.

[0161] In 610, the at least one processor of the processor-based system initializes a hysteresis time counter K (e.g., K = 1). In 612, the at least one processor of the processor-based system executes a nested inner hysteresis time collision assessment loop. The nested inner hysteresis time collision assessment loop is nested within the inner nominal trajectory collision assessment loop. The nested inner hysteresis time collision assessment loop allows the at least one processor of the processor-based system to assess or estimate the impact of each of a plurality of possible hysteresis times (i.e., a candidate hysteresis time from a set of candidate hysteresis times) when introduced to a given nominal trajectory. Thus, for example, a nominal trajectory that introduces a hysteresis time equal to zero and a nominal trajectory that introduces a plurality (one, two, or more) of non-zero hysteresis times can be assessed to determine the probability or likelihood of a collision occurring if the trajectories corresponding to the nominal trajectories with the respective hysteresis times introduced are executed. Assessments can be made for successively longer hysteresis times.

[0162] At 614, the at least one processor of the processor-based system performs a collision assessment of the respective trajectories with the introduced lag times (e.g., for zero and non-zero lag times) using the generated swept volumes. For example, the at least one processor of the processor-based system can determine whether respective swept volumes corresponding to the respective volumes swept by a robot of a pair of robots when executing a nominal trajectory with an introduced lag time equal to zero and an introduced plurality of non-zero lag times intersect. For example, for each robot of the two or more robots, the processor-based system can perform a collision assessment between i) at least a portion of a respective sample trajectory representing a respective nominal trajectory of the robot with at least one respective lag time introduced (e.g., a zero lag time and at least one non-zero lag time), and ii) at least one portion of each respective sample of a respective trajectory of each of the other robots of the two or more robots, where at least one respective lag time (e.g., a zero lag time and at least one non-zero lag time) is introduced into the respective trajectory of each of the other robots of the two or more robots. Thus, for each robot of the robots and each nominal trajectory of the nominal trajectories, all permutations of “candidate” lag times can be assessed, e.g., to identify when a probability or likelihood of a collision equals or exceeds some collision detection threshold.

[0163] While collision assessment is described in terms of swept volumes, the teachings herein are not necessarily limited thereto, and the methods and systems can employ various other collision assessment or detection techniques or algorithms.

[0164] In 616, the at least one processor of the processor-based system determines whether the robots of a given pair will or can cause a collision based on the collision assessment. In at least some implementations, the collision assessment can produce a non-binary value representing a collision opportunity or probability. A determination of whether a collision will or can occur can be made based on the non-binary value being above a defined collision detection threshold (e.g., collision probability > 50%; collision probability > 10%; collision probability > 5%). In 617, in response to determining that a collision has been detected or is likely to occur (e.g., collision probability is equal to or greater than the collision detection threshold), the processor-based system can identify and / or store a previous lag time value that did not cause a potential collision, and optionally exit the nested inner lag time collision assessment loop 612, returning control to the top of the inner nominal trajectory collision assessment loop 608. Thus, for example, in response to determining that a collision will or can occur between a robot and at least one other robot of the two or more robots, for one or more nominal trajectories of at least one of the two or more other robots for which a collision has been determined to occur, the processor-based system identifies a respective lag time that is less than a respective lag time for which a determination was made that a collision would occur as the respective acceptable lag time, where the acceptable lag reflects a maximum acceptable delay in timing of the poses of the nominal trajectory that still guarantees that movement of the robot through the sequence of poses specified by the respective nominal trajectory remains collision free with respect to movement of the other robot of the two or more robots. In response to determining that a collision has been detected or is likely to be detected (e.g., collision probability is equal to or greater than the collision detection threshold), control passes directly to 618.

[0165] In 618, the at least one processor of the processor-based system determines whether all lag times for the current trajectory of the current pair of robots have been considered (e.g., K = P?). If all lag times have not been considered, the lag time counter K is incremented at 620 (e.g., K = K + 1) to select the next candidate lag time from the set of candidate lag times for evaluation, and control returns to the top of the nested inner lag time collision assessment loop 612. If all lag times have been considered, control passes directly to 622.

[0166] In 622, the at least one processor of the processor-based system determines whether all trajectories for the given pair of robots have been processed (e.g., J = M?). If all trajectories for the given pair of robots have not been processed, the trajectory counter is incremented at 624 (e.g., J = J + 1), and control returns to the top of the inner nominal trajectory collision assessment loop 608. If all nominal trajectories have been processed (e.g., J = M), control passes to 626.

[0167] In 626, at least one processor of the processor-based system determines whether there are additional pairs of robots to process. The method 600 can process each permutation of pairs of robots, each permutation of trajectories for those pairs of robots, and each permutation of candidate lag times, at least until a stopping condition is reached (e.g., an unacceptably high probability of collision is found). Thus, for any given robot, the determined acceptable lag time (e.g., maximum lag time) can be a function of the respective selected trajectories of every other robot operating in the shared workspace and the determined acceptable lag times associated with those selected trajectories. In this sense, the term "maximum lag time" does not necessarily mean an absolute maximum lag time for the given robot in isolation, but can mean the maximum lag time for the given robot taking into account the trajectories and associated lag times of the other robots operating in the shared workspace. If there are additional pairs of robots to process, control passes to 628, in which a new pair of robots is selected, and then control passes to the top of the outer pair of robots collision evaluation loop. If there are no remaining pairs of robots to process, control passes directly to 630.

[0168] In 630, at least one processor of the processor-based system returns the respective acceptable lag times for the one or more nominal trajectories. For example, the acceptable lag times can be stored to a non-transitory processor-readable medium, e.g., in one or more data structures associated with the nominal trajectories. Control then passes to 632.

[0169] In 632, the method 600 can terminate, e.g., until invoked again. While the method 600 is described in an ordered flow, in many implementations various actions or operations will be performed concurrently or in parallel, and / or can include additional actions and / or omit some actions.

[0170] To determine respective acceptable lag times for nominal trajectories, for example, a processor-based system can iterate through each of a plurality of candidate lag times in order from a relatively small candidate lag time to a relatively large candidate lag time, at least until a stopping condition is reached, iterating through each of a plurality of times that a nominal trajectory covers, based on a current one of the candidate lag times, check for a collision between one of the two or more robots and at least one other of the two or more robots. In response to determining that a collision between one of the two or more robots and at least one other of the two or more robots will occur or is likely to occur, the processor-based system can set respective acceptable lag times for nominal trajectories of the robots for which a collision has been determined to occur to a nearest prior candidate lag time (e.g., a lag time for which a collision probability is below a collision detection threshold indicative of an acceptable risk of collision). To check for a collision, for example, the processor-based system can determine whether a swept volume of one of a pair of robots intersects a swept volume of another of the pair of robots. For example, in response to determining that a swept volume of one of the pair of robots intersects a swept volume of another of the pair of robots, and identifying the robots for which a collision was determined to occur, the processor-based system can generate a collision indication.

[0171] An example method of determining and applying acceptable lag times to provide a safety margin is set out below.

[0172] First, a motion plan comprising a set of nominal trajectories (Trj(t)) is generated, e.g., using the optimization described in International Patent Application PCT / US2021 / 013610 (Publication No. WO2021150439A1). Thus, for each mobile robot r in the motion plan, an acceptable lag time (e.g., a maximum safe lag value (max_lag_r)) is computed with respect to the motion plan. It is important to note the following: (i) the acceptable lag time value (max_lag) can be different for each robot r (e.g., max_lag_r1≠ max_lag_r2); (ii) the acceptable lag time value (max_lag) depends strictly on the set of nominal trajectories Trj(t) and the relative position of the robot r; and (iii) static obstacles and static robots, if any, do not affect the acceptable lag time value (max_lag). In this example, the set of nominal trajectories Trj(t) and the acceptable lag time values (max_lag_r) are computed sequentially and preferably offline within a configuration time before run-time; however, this is not intended to limit the methods described herein. A central clock (t) is implemented during execution (e.g., online; during run-time), which is common to all robots running in the shared workspace. For each robot r, a lag (lag_r) with respect to the nominal motion plan is computed at a high frequency, that is, the monitoring of the actual lag time is performed at a sufficiently high frequency with respect to the motion speed of the robot to avoid undesired actual collisions.

[0173] In the ideal case, where each robot r is perfectly synchronized with the motion plan, the actual lag of the robot is zero (lag_r = 0). However, in the general case, one or more robots can fall behind the plan, where the current robot state is given by the trajectory trj_r(t - nAT) and the actual lag lag_r = nAT.

[0174] The processor-based system monitors the actual lag values to ensure the safety of the robot system. If all actual lag values are within the safety margin specified by the acceptable lag time (e.g., lag_r < max_lag_r), then it is considered safe to continue execution since the no-self-collision condition is ensured. Otherwise, the safety of the system is considered compromised since the no-self-collision condition cannot be ensured.

[0175] To keep the robots in a safe state at all times, the system can monitor the actual lag values and simply stop the movement of the entire robot system when the safety margin is exceeded (e.g., when the margin or threshold is exceeded). To safely restart operation, the robots are able to move to any (safe) state of the set of nominal trajectories Trj(t*) and the central clock is reset to t*.

[0176] Less disruptive solutions can include taking remedial or corrective action on robots that are still within the safety margin but are relatively close to the limit. For example, the motion of a robot that is about to reach a limit can be accelerated (speed up) to reduce the value of its actual lag time. This action will directly increase the distance from the limit and improve the level of safety. Alternatively, other robots can be slowed down. This action can exploit the time dilation of the central clock to maintain consistency. Hybrid actions can also be very effective.

[0177] An exemplary processor-executable algorithm to determine the safety margin is set out in the pseudo-code below.

[0178] For better understanding, the algorithm can be divided into 4 parts:

[0179] • main loop: it iterates over the candidate lag values from 0 to L on the nominal trajectory Trj(t).

[0180] • CheckLagCollision(): evaluates the impact of a candidate lag value applied at a specific time t. Given a set of candidate lag values for each of the robots, it determines whether any of the robots collide at a given time t. In the absence of lag, a robot occupies a volume determined by its current pose. Currently, in the presence of lag, a robot can sweep a volume based on the trajectory it takes from time t-lag to time t.

[0181] • ComputeSweptVolume_r(tl, t2): computes the volume swept by robot r along a segment of its trajectory from time tl to time t2 (trj_r(tl...t2)).

[0182] • VolumesExcept(): geometric search for the intersection between two volumes.

[0183] The main loop() and CheckLagCollision() will be detailed below. ComputeSweptVolume_r() and VolumesIntersect() are not detailed as they can be implemented with many algorithms.

[0184] The inputs include: trj[l...N]: a set of collision-free trajectories for N robots; AT: the sampling time; T: the maximum duration of robot activity; AL: the sampled lag time values to search; and L: the maximum lag value to search (L<=T).

[0185] The output includes: a maximum lag value max_lag[1...N] for each robot that guarantees collision-free motion between the robots

[0186] The internal variables include: lag_candidate: a lag value evaluated in the current iteration; lag[1...N]: a lag value evaluated in the current iteration for each robot; lag_prev[1...N]: a lag value evaluated in the previous iteration for each robot; found_max-lag[1...N]: a Boolean value for each robot that stops the search.

[0187]

[0188] Figure 7 An operational method 700 of a processor-based system for generating or selecting motion plans for one or more robots to operate in a shared workspace based at least in part on acceptable lag times, and optionally other criteria, is shown in accordance with at least one illustrated embodiment. The lag times, and optionally the other criteria, can be expressed as a cost or cost function. For example, the method 700 can be performed during a configuration or “pre-run” time, i.e., a time during which some or all of the motion planning occurs, e.g., to generate nominal trajectories and / or motion plans for each robot. The configuration or “pre-run” time can precede a run time, i.e., a time during which the one or more robots execute the corresponding motion plans. This advantageously allows some of the most computationally intensive work to be performed prior to the run time, when responsiveness is not as much of a concern. In at least some embodiments, the method 700, or portions thereof, are performed during the run time, i.e., the time during which the one or more robots are performing a task, after the configuration time or pre-run time.

[0189] The method 700 of determining acceptable lag times begins at 702, e.g., in response to a startup or power-up of the system or components thereof, a receipt of information or data, or a call or invocation of a routine or program.

[0190] Optionally, in 704, at least one processor of the processor-based system generates or accesses (e.g., receives, retrieves) a nominal trajectory for each robot. As described previously, a nominal trajectory is a “prescribed” trajectory, where each trajectory includes an ordered sequence of poses or configurations of the robot, and a respective timing for each pose or configuration. As described herein, the actual motion or actual trajectory of a robot or portion thereof can deviate from the corresponding nominal trajectory, e.g., due to unexpected delays in transitioning between poses (e.g., due to a need to linger or dwell on a target object for longer than expected).

[0191] As previously described, any of a variety of techniques and / or algorithms can be employed to generate nominal trajectories, e.g., sample-based motion planners (SBMPs), such as probabilistic roadmaps (PRMs) or rapidly-exploring random trees (RRTs, RTT*s), stable sparse RRT*s (SST*s), and / or fast-matching trees (FMTs). Advantageously, the teachings herein include computationally efficiently using a determined hysteresis time to select a set of nominal trajectories for a motion plan to enhance robustness of operation.

[0192] In 706, the at least one processor of the processor-based system initializes a robot counter I, e.g., setting the robot counter equal to the integer value 1. In 708, the at least one processor of the processor-based system executes an outer robot processing iteration loop, iterating over each of the two or more robots that will operate in the shared workspace, or until a stopping condition is reached (e.g., a maximum hysteresis time is determined that ensures collision-free motion).

[0193] In 710, the at least one processor of the processor-based system initializes a nominal trajectory counter J, e.g., setting the nominal trajectory counter equal to the integer value 1. In 712, the at least one processor of the processor-based system executes an inner nominal trajectory processing iteration loop, iterating over each nominal trajectory for the given robot. The inner nominal trajectory processing iteration loop 712 is nested in the outer robot processing iteration loop 708.

[0194] At 714, at least one processor of the processor-based system determines a respective acceptable lag time for the current nominal trajectory J of the current robot I. The respective acceptable lag time reflects a maximum acceptable delay or lag in the current nominal trajectory J that still guarantees collision-free movement of the current robot I with respect to each of the other robots moving in accordance with their respective nominal trajectories, at least when the current robot I (the current robot of the outer robot processing iteration loop) executes the current nominal trajectory J (the current nominal trajectory of the inner current nominal trajectory iteration loop) with the respective acceptable lag time introduced into the current nominal trajectory J of the current robot I. Preferably, this can be evaluated with respect to the nominal trajectories of the other robots introducing various candidate lag times (e.g., zero and non-zero lag times) or even other robots introducing respective acceptable lag times, if known. As long as all other robots are operating on or within their respective nominal trajectories (i.e., lag time = 0), or all other robots are operating on or within their respective nominal trajectories with the respective lag times introduced into the respective nominal trajectories, the acceptable lag time for a given robot I can guarantee that no self-collision will occur. Thus, in at least one embodiment, as long as the respective current actual lag time of a given robot is less than the respective acceptable lag time of the robot, and the other robots are operating in accordance with their nominal trajectories, the robots will not collide with each other (guaranteeing that no self-collision will occur between the robots). Thus, in at least one preferred embodiment, as long as the respective current actual lag time of all robots is less than the respective acceptable lag time of each robot, the robots will not collide with each other (guaranteeing that no self-collision will occur between the robots).

[0195] At 716, at least one processor of the processor-based system determines whether each of the nominal trajectories of the given robot has been considered, e.g., determines whether the nominal trajectory counter J is equal to the total number of nominal trajectories of the given robot (e.g., J = M?). If each of the nominal trajectories of the given robot has not been considered, control passes to 718, where the nominal trajectory counter is incremented (e.g., J = J + 1), and control returns to the top of the inner nominal trajectory processing iteration loop 712 to consider the next nominal trajectory of the given robot I. If each of the nominal trajectories of the given robot I has been considered (e.g., J = M), control passes directly to 720.

[0196] In 720, the at least one processor of the processor-based system selects a maximum of the set of determined acceptable lag times determined for the given robot I. In 722, the at least one processor of the processor-based system provides one or more of the determined acceptable lag times (e.g., the selected maximum of the determined acceptable lag times) for the given robot for use in determining a motion plan for at least the given robot. This can include, for example, providing the one or more determined acceptable lag times to a different processor, or transmitting the one or more determined acceptable lag times to a different register of the processor, or otherwise storing it in a non-transitory processor-readable medium.

[0197] In 724, the at least one processor of the processor-based system determines whether every robot has been considered, e.g., determines whether the robot counter I is equal to the total number of robots (e.g., I = N?). If every robot has not yet been considered (e.g., I < N), then control passes to 726, in which the at least one processor of the processor-based system iterates the robot counter (e.g., I = I + 1), and then control returns to the top of the outer iteration robot processing loop 708 to consider the next robot. If every robot has been processed (e.g., I = N), then control passes directly to 728.

[0198] In 728, the at least one processor of the processor-based system generates or selects a respective motion plan for each robot. The motion plan can be or can represent a respective nominal trajectory of the robot associated with the maximum acceptable lag time. This can advantageously enhance the robustness of the motion plan generated by the motion planning, and thereby improve the operation of the robot executing the resulting motion plan.

[0199] When multiple trajectories are considered for each robot, the acceptable lag time determined for any given robot (e.g., the maximum lag time) can be a function of the respective selected trajectory of each of the other robots operating in the shared workspace, and the determined acceptable lag time associated with those selected trajectories. Thus, as described herein, in at least some implementations, the system determines, for each robot and each candidate trajectory of that robot, the acceptable lag time given all other possible combinations of candidate trajectories and candidate lag times of all other robots. Once complete, the system (e.g., the nominal trajectory analyzer 258 of FIG. 2) can select a set of trajectories for all robots operating in the shared workspace. The system (e.g., the nominal trajectory analyzer 258 of FIG. 2) can partition the set of trajectories, for example, by optimizing an objective function (e.g., the maximum sum of all lag times of all robots). Figure 2 Figure 2

[0200] ​​In 730, at least one processor of the processor-based system provides each robot or each motion controller with a respective motion plan for the robot to move in accordance with the respective motion plan. For example, this can include providing the motion plan to a respective motion controller of each of one or more robots.

[0201] At 732, the method 700 can terminate, for example, until invoked again. While the method 700 is described in sequential flow, in many implementations, various acts or operations will be performed concurrently, or in parallel, and / or can include additional acts and / or omit some acts.

[0202] Thus, for example, a processor-based system can determine, for each of two or more robots, and for each of two or more nominal trajectories of the respective robot of the two or more robots, a respective acceptable lag time for the nominal trajectory that reflects a maximum acceptable delay in timing of poses relative to the nominal trajectory that still guarantees that movement through the sequence of poses specified by the nominal trajectory for the robot remains collision-free relative to movement through the respective sequence of poses specified by the respective nominal trajectory for each of the other robots of the two or more robots, since each of the nominal trajectories for the other robots of the two or more robots is itself delayed by the respective acceptable lag time. For example, the processor-based system can select between the two or more nominal trajectories based at least in part on the respective acceptable lag times for the two or more nominal trajectories; and provide a motion plan for the respective robot based at least in part on the selected one of the nominal trajectories to control operation of the respective robot of the two or more robots. For example, the processor-based system can select the nominal trajectory of the two or more nominal trajectories having the greatest acceptable lag time for the respective robot, thereby generating a motion plan that is more robust than otherwise. For example, the processor-based system can select the nominal trajectory based on a respective cost function that represents the acceptable lag time and, optionally, a risk or probability of collision associated with the respective nominal trajectory of the two or more nominal trajectories for the respective robot. For example, the processor-based system can select the nominal trajectory based on a respective cost function that represents the acceptable lag time, a risk or probability of collision, and, optionally, a severity of collision associated with the respective nominal trajectory of the two or more nominal trajectories for the respective robot. For example, the processor-based system can select the nominal trajectory based on a respective cost function that represents the acceptable lag time, a risk or probability of collision, a severity of collision, and, optionally, at least one of a duration of completion or energy consumption associated with the respective nominal trajectory of the two or more nominal trajectories for the respective robot, where each of the variables in the cost function representing the acceptable lag time, the risk or probability of collision, the severity of collision, and the at least one of the duration of completion or energy consumption is weighted in the respective cost function. For example, the processor-based system can select the respective nominal trajectory for each of the two or more robots such that an overall acceptable lag time for all of the robots of the two or more robots is maximized.For example, the processor-based system can select a respective nominal trajectory for each of two or more robots such that a respective cost function for each robot is optimized over an ensemble of all of the two or more robots, where each cost function represents an acceptable lag time and a risk or probability of at least a collision respectively associated with a respective nominal trajectory of two or more nominal trajectories of the respective robot. For example, the processor-based system can select a respective nominal trajectory for each of two or more robots such that a respective cost function for each robot is optimized over an ensemble of all of the two or more robots, where each cost function represents an acceptable lag time, a risk or probability of at least a collision, and a severity of a collision respectively associated with a respective nominal trajectory of two or more nominal trajectories of the respective robot. For example, the processor-based system can select a respective nominal trajectory for each of two or more robots such that a respective cost function for each robot is optimized over an ensemble of all of the two or more robots, where each cost function represents an acceptable lag time, a risk or probability of at least a collision, a severity of a collision, and at least one of a duration of completion or an energy expenditure respectively associated with a respective nominal trajectory of two or more nominal trajectories of the respective robot, where each variable in the cost function respectively represents the acceptable lag time, the risk or probability of a collision, the severity of a collision, and at least one of the duration of completion or the energy expenditure is weighted in the respective cost function.

[0203] For example, the processor-based system can determine a respective acceptable lag time for a nominal trajectory that reflects a maximum acceptable delay in timing of poses of the nominal trajectory relative to a pose schedule that still guarantees movement of the robot through the sequence of poses specified by the nominal trajectory and maintenance of no self-collisions between the two or more robots in the shared workspace, which can include performing a collision assessment for each of the two or more robots.

[0204] For example, the processor-based system can perform a collision assessment between i) at least a portion of a respective sample trajectory of a respective nominal trajectory of the robot that introduces at least one respective lag time, and ii) at least a portion of a respective sample of each of a respective trajectory of each of the other robots of the two or more robots, where at least one respective lag time is introduced into the respective trajectory of each of the other robots of the two or more robots.

[0205] Figure 8An operational method 800 of a processor-based system for controlling operation of robots operating in a shared workspace based at least in part on an acceptable lag time is shown, in accordance with at least one illustrated embodiment. In accordance with at least one illustrated embodiment, the processor-based system includes one or more processors that monitor an actual lag time as compared to the acceptable lag time, and optionally take one or more remedial measures if the actual lag time exceeds a threshold. The method 800 can be performed, for example, after execution of the method 300 Figure 3 ), the method 400 Figure 4 ), the method 500 Figure 5 ), the method 600 Figure 6 ), and / or the method 700 Figure 7 ). For example, the method 800 can be performed during runtime, i.e., during the time that one or more robots are executing respective motion plans and / or performing tasks. For example, the runtime can follow a configuration or “pre-run” time during which partial or full motion planning can be performed, e.g., to generate nominal trajectories and / or motion plans for each robot.

[0206] The method 800 begins at 802, e.g., in response to startup or power-up of the system or components thereof, receipt of information or data, or invocation or enabling of a routine or program.

[0207] Optionally, in 804, at least one processor of the processor-based system accesses (e.g., receives, retrieves) respective motion plans for each robot. The respective motion plans can be stored and accessed from one or more non-transitory processor-readable media.

[0208] In 806, at least one processor of the processor-based system causes the one or more robots to execute the respective motion plans. For example, the at least one processor can provide instructions to one or more motion controllers of the one or more robots. The one or more motion controllers provide control signals to one or more actuators (e.g., electric motors, solenoids, valves, pumps) that are coupled to drive various links of the robot, thereby causing the robot or portions thereof to move.

[0209] In 808, at least one processor of the processor-based system monitors an amount of lag time (actual lag time) of the respective actual trajectory performed by the respective one robot compared to the respective nominal trajectory. As previously described, the actual trajectory is the actual sequence of poses and timing of those poses performed by the respective robot. While in some cases, the actual trajectory can match the nominal trajectory, in many cases, the actual trajectory will not match the nominal trajectory, typically with at least some of the poses of the actual trajectory of one or more robots being performed with timing that lags the timing of the poses specified by the respective nominal trajectory of the respective robot. The monitoring can be performed in any of a variety of ways. For example, the processor-based system can use input sensed via one or more sensors positioned to sense the position and / or movement and / or pose or configuration or state of the robots in the shared workspace. The sensors can monitor the entire workspace, e.g., the sensors take the form of cameras, video cameras, stereo cameras, motion detectors, etc. that are positioned and have a field of view that covers all or at least a portion of the shared workspace. The sensors can additionally or alternatively monitor the position and / or movement and / or pose or configuration or state of a particular robot, e.g., including any one or more of: cameras, video cameras, motion detectors, position or rotary encoders, Hall effect sensors, and / or Reed switches related to one or more joints, links, and / or actuators of the robot. The one or more processors can perform processing (e.g., machine vision processing) to monitor the actual trajectory. Additionally or alternatively, the processor-based system can use information from the robot control system and / or drive system (e.g., motor controllers, pneumatic controllers, hydraulic controllers, etc.) representing control signals (e.g., PWM motor control signals) used to drive the robot or portions thereof and / or using feedback signals (e.g., back EMF) received from the robot or drive system of the robot. The one or more processors can perform processing (e.g., machine vision processing) to determine the respective actual lag time of the actual trajectory.

[0210] In 810, at least one processor of the processor-based system determines whether the monitored amount of lag time (actual lag time) exceeds a margin or threshold (e.g., a respective determined acceptable lag time; a percentage of a respective determined acceptable lag time) for the respective trajectory of the respective one of the robots operating in the shared workspace.

[0211] In response to determining that the monitored amount of lag time (i.e., actual lag time) exceeds the respective margin or threshold (e.g., determined acceptable lag time; percentage of respective determined acceptable lag time) of the respective trajectory of the respective robot of any of the robots, optionally in 812, at least one processor of the processor-based system selects and / or takes one or more remedial actions.

[0212] For example, to take the at least one remedial action, when the at least one processor of the processor-based system executes the processor-executable instructions, the processor can be caused to perform one or more of the following: stop movement of one or more robots, slow movement of one or more robots, and / or speed up movement of one or more robots. For example, the at least one processor can stop movement of one, two, more, or even all of the robots. Also, for example, the at least one processor can slow movement of one, two, more, or even all of the robots. Also, for example, the at least one processor can speed up movement of one, two, more, or even all of the robots. Also, for example, the at least one processor can stop movement of one or more robots while slowing movement of one or more robots. Also, for example, the at least one processor can stop movement of one or more robots while speeding up movement of one or more robots or causing movement of one or more robots to remain constant. Also, for example, the at least one processor can slow movement of one or more robots while speeding up movement of one or more robots or causing movement of one or more robots to remain constant. Also, for example, the at least one processor can stop movement of one or more robots while slowing movement of one or more robots while speeding up movement of one or more robots or causing movement of one or more robots to remain constant. The term “cause movement to remain constant” means that the nominal trajectory of the robot is not changed, although the speed (i.e., velocity and direction) of movement of the robot can change as dictated by the nominal trajectory. Also, for example, to take the at least one remedial action, when the at least one processor of the processor-based system executes the processor-executable instructions, the processor can be caused to perform one or more of the following: stop movement of one or more robots, cause one or more robots to proceed along the respective trajectory specified by the respective nominal trajectory, after which movement of the robots can be restarted. Restarting movement typically involves resuming movement from where movement stopped, although in some cases can include returning to the start of the respective trajectory and resuming from there.

[0213] Optionally, in 814, at least one processor of the processor-based system monitors an amount of lag time (actual lag time) of the respective actual trajectory performed by the respective robot compared to the respective nominal trajectory.

[0214] Optionally, in 816, for each robot operating in the shared workspace, at least one processor of the processor-based system determines whether the monitored amount of lag time (i.e., actual lag time) no longer exceeds the respective margin or threshold (e.g., determined acceptable lag time; percentage of respective determined acceptable lag time) for the respective trajectory of the respective robot. The system can enter a wait loop, continuing to monitor the amount of lag time (i.e., actual lag time) until it no longer exceeds the respective margin or threshold (e.g., determined acceptable lag time; percentage of respective determined acceptable lag time) for all robots. Once this condition is reached, control passes to 818.

[0215] Optionally, in 818, at least one processor of the processor-based system causes the one or more robots to continue executing (e.g., resume moving) the respective motion plan.

[0216] Method 800 terminates at 820, e.g., until invoked again. While method 800 is described in an ordered flow, in many embodiments, various acts or operations will be performed simultaneously or in parallel, and / or can include additional acts and / or omit some acts.

[0217] The above detailed description has set forth various embodiments of the devices and / or processes via the use of block diagrams, flowcharts, and examples. Insofar as such block diagrams, flowcharts, and examples contain one or more functions and / or operations, it will be understood by those within the art that each function and / or operation within such block diagrams, flowcharts, or examples can be implemented, individually and / or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof. In one embodiment, the subject matter described herein can be implemented via a Boolean circuit, an application specific integrated circuit (ASIC), and / or a FPGA. However, those skilled in the art will recognize the embodiments disclosed herein, in whole or in part, can be implemented in a variety of different ways, as an application specific integrated circuit (ASIC) and / or a FPGA, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more controllers (e.g., microcontrollers), as one or more programs running on one or more processors (e.g., microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and / or writing the code for the software and / or firmware would be well within the skill of one of skill in the art, in light of this disclosure.

[0218] Those skilled in the art will recognize that many of the methods described herein can employ additional acts, can omit some acts, and / or can perform the acts in a different order than specified.

[0219] Furthermore, those skilled in the art will appreciate that the mechanisms taught herein can be implemented in hardware, for example, in one or more FPGAs or ASICs.

[0220] The various embodiments described above can be combined to provide further embodiments. All commonly assigned U.S. patent application publications, U.S. patent applications, foreign patents, and foreign patent applications referred to in this specification, including but not limited to International Patent Application No. PCT / US2017 / 036880, filed June 9, 2017; International Patent Application No. WO2016 / 122840, filed January 5, 2016; U.S. Patent Application No. 62 / 616783, filed January 12, 2018; U.S. Patent Application No. 62 / 626939, filed February 6, 2018; U.S. Patent Application No. 62 / 856548, filed June 3, 2019; U.S. Patent Application No. 62 / 865,431, filed June 24, 2019; U.S. Patent Application No. 62 / 964405, filed January 22, 2020; U.S. Patent Application No. 63 / 327917, filed April 6, 2022; International Patent Application PCT / US2021 / 013610 (Publication No. WO2021150439A1), all of which are incorporated by reference herein. These and other changes can be made to the embodiments in light of the above- detailed description. In general, the selected terminology is used in the selected manner and specific embodiments are described in the knowledge that each term can encompass multiple embodiments and / or multiple meanings as is appreciated by those skilled in the art. The applicants herewith state any Federal Constitutional requirements in the U.S. The claims are not to be limited to the embodiments and / or examples described above, but include modifications within the scope of the present disclosure.

Claims

1. A method for facilitating the operation of multiple robots in a processor-based system, the method being used in a multi-robot operation environment in which multiple robots will operate, the method comprising: For each of two or more robots, For each of one or more nominal trajectories of a corresponding robot among the two or more robots, the nominal trajectory specifies the corresponding pose sequence and pose timing of the corresponding robot. Determine the appropriate acceptable lag time for the nominal trajectory, which reflects the maximum acceptable delay in attitude timing relative to the nominal trajectory, and the maximum acceptable delay still ensures that the movement of the robot through the attitude sequence specified by the nominal trajectory remains collision-free relative to the movement through the corresponding attitude sequence specified by the corresponding nominal trajectory of each of the two or more other robots. as well as The acceptable lag time is provided to at least one processor to control the operation of each of the two or more robots; as well as For each of the two or more robots For each of one or more actual trajectories executed by the corresponding robot Monitor the hysteresis between the nominal trajectory and the corresponding actual trajectory executed by the corresponding robot; Determine whether the detected lag exceeds the corresponding acceptable lag time of the corresponding actual trajectory of the robot; as well as In response to the detected hysteresis exceeding the corresponding determined acceptable hysteresis time of the corresponding trajectory of the corresponding robot, at least one remedial measure shall be taken.

2. The method according to claim 1, wherein, Determining the appropriate acceptable lag time for the nominal trajectory includes: For each of the two or more robots For each of the one or more nominal trajectories A sweep volume representation is generated, which represents the volume swept by at least a portion of the robot between a set of pose sequences specified by the nominal trajectory, from at least one time in the nominal trajectory to another time in the nominal trajectory.

3. The method according to claim 2, wherein, Determining the appropriate acceptable lag time for the nominal trajectory also includes: Perform a collision assessment using the generated swept volume.

4. The method according to claim 3, wherein, Performing a collision assessment using the generated sweep volume includes: for each of the two or more robots, performing a collision assessment between i) and ii), where i) represents at least a portion of a corresponding sample trajectory of the robot's corresponding nominal trajectory with at least one corresponding lag time introduced, and ii) at least a portion of a corresponding sample of each trajectory of each of the other two or more robots, wherein at least one corresponding lag time is introduced into the corresponding trajectory of each of the other two or more robots. In response to determining that a collision will occur between the robot and at least one of the other two or more robots, for one or more nominal trajectories of the robot, a corresponding lag time less than the corresponding lag time that would cause a collision to occur is identified as the corresponding acceptable lag time, wherein the acceptable lag time reflects the maximum acceptable delay in attitude timing relative to the nominal trajectory, the maximum acceptable delay still ensuring that the movement of the robot through the attitude sequence specified by the nominal trajectory remains collision-free relative to the movement of the other two or more robots.

5. The method according to claim 4, wherein, Determining the appropriate acceptable lag time for the nominal trajectory also includes: In response to determining that a collision will occur between the robot and at least one of the other two or more robots, for one or more nominal trajectories of the other two or more robots where a collision has been determined to occur, a corresponding lag time less than the corresponding lag time that caused the collision to occur is identified as the corresponding acceptable lag time, wherein the acceptable lag time reflects the maximum acceptable delay in attitude timing relative to the nominal trajectory, the maximum acceptable delay still ensuring that the movement of the robot through the attitude sequence specified by the corresponding nominal trajectory remains collision-free relative to the movement of the other two or more robots.

6. The method according to claim 1, wherein, Determining the appropriate acceptable lag time for the nominal trajectory also includes: Iterate through each of the multiple candidate lag times in order from the relatively smaller candidate lag times to the relatively larger candidate lag times, at least until the stopping condition is met. Iterate through each of the multiple time periods covered by the nominal trajectory. Based on the current candidate lag time from the candidate lag times, check for collisions between one of the two or more robots and at least one of the other two or more robots; and In response to determining that a collision will occur between one of the two or more robots and at least one of the other two or more robots, the corresponding acceptable lag time of the nominal trajectory of the robot that is determined to collide with at least one of the other two or more robots is set to the closest previous candidate lag time.

7. The method according to claim 6, wherein, Checking for collisions between the robot and other robots based on the current candidate lag time from the candidate lag times includes: For each of the two or more robots Collision assessment is performed on at least a portion of the nominal trajectory of the robot having the current candidate lag time or another previously determined lag time, relative to at least a portion of the trajectory of each of the other two or more robots having the current candidate lag time.

8. The method according to claim 7, wherein, Performing a collision assessment on at least a portion of the nominal trajectory of the robot having the current candidate lag time or another previously determined lag time relative to at least a portion of the corresponding trajectory of each of the two or more other robots having the current candidate lag time comprises: performing a collision assessment to determine whether at least a portion of the robot collides with at least a portion of any of the other robots while the robot and the other two or more robots are moving along at least a portion of the corresponding nominal trajectory delayed by the candidate lag time.

9. The method according to claim 6, wherein, Checking for collisions between the robot and the other robots based on the current candidate lag time from the candidate lag times includes: For each of the two or more robots Generate a sweep volume representation, which represents the volume swept by at least a portion of the robot between a set of pose sequences specified by a corresponding nominal trajectory, moving from at least one time in the corresponding nominal trajectory to another time in the corresponding nominal trajectory; and For each pair of robots of the two or more robots For attitude timing specified by a nominal trajectory without introducing any lag time and for attitude timing specified by a nominal trajectory that introduces each of a plurality of candidate lag times into the corresponding nominal trajectory, determine whether the sweep volume of one robot in the pair intersects with the sweep volume of the other robot in the pair, at least until an intersection is detected, and In response to determining that the sweep volume of one robot in the pair intersects with the sweep volume of the other robot in the pair and identifying the robot that has collided, a collision indication is generated.

10. The method according to claim 1, wherein, Taking at least one remedial action includes at least one or more of the following: stopping the movement of one or more of the robots, slowing down the movement of one or more of the robots, or speeding up the movement of one or more of the robots.

11. The method according to claim 1, wherein, Taking at least one remedial action includes: stopping the movement of the robot, causing one or more of the robots to proceed along the corresponding trajectory specified by the corresponding nominal trajectory, and then restarting the movement of the robot.

12. The method according to claim 1, wherein, The determination of the corresponding acceptable lag time for the nominal trajectory occurs during the configuration time prior to the movement of at least two robots according to one or more nominal trajectories specified by the respective robots.

13. The method according to claim 12, wherein, The lag between the monitored nominal trajectory and the corresponding actual trajectory executed by the corresponding robot occurs during the runtime of the at least two robots executing a movement specified by one or more nominal trajectories of the corresponding robots, the runtime being after the configuration time.

14. The method according to claim 1, wherein, The lag between the nominal monitored trajectory and the corresponding actual trajectory executed by the respective robot includes a central clock shared by the monitors of one or more robots.

15. The method according to any one of claims 1 to 14 and any combination thereof, further comprising: Receive the corresponding motion plan for each of the robots, each motion plan specifying a corresponding nominal trajectory among one or more nominal trajectories for the corresponding robot, the nominal trajectory representing the corresponding collision-free path.

16. A processor-based system for facilitating the operation of multiple robots in a multi-robot operation environment in which multiple robots will operate, the processor-based system comprising: At least one processor; as well as At least one non-transitory processor-readable medium storing processor-executable instructions, which, when executed by the at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 15.

17. A processor-based system for configuring multiple robots in a multi-robot operating environment, the processor-based system comprising: At least one processor; as well as At least one non-transitory processor-readable medium storing at least one of data and processor-executable instructions, which, when executed by the at least one processor, cause the at least one processor to: For each of two or more robots, For each of one or more nominal trajectories of a corresponding robot among the two or more robots, the nominal trajectory specifies the corresponding pose sequence and pose timing of the corresponding robot. Determine the appropriate acceptable lag time for the nominal trajectory, which reflects the maximum acceptable delay in attitude timing relative to the nominal trajectory, and the maximum acceptable delay still ensures that the movement of the robot through the attitude sequence specified by the nominal trajectory remains collision-free relative to the movement of the corresponding attitude sequence specified by the corresponding nominal trajectory of each of the two or more other robots. as well as The acceptable lag time is provided to at least one processor to control the operation of each of the two or more robots; as well as For each of the two or more robots For each of one or more actual trajectories executed by the corresponding robot Monitor the hysteresis between the nominal trajectory and the corresponding actual trajectory executed by the corresponding robot; Determine whether the detected lag exceeds the corresponding acceptable lag time of the corresponding actual trajectory of the robot; as well as In response to the detected hysteresis exceeding a corresponding determined acceptable hysteresis time of the corresponding trajectory of the corresponding robot, at least one remedial action is performed.

18. The processor-based system according to claim 17, wherein, Determining the appropriate acceptable lag time for the nominal trajectory includes: For each of the two or more robots For each of the one or more nominal trajectories A sweep volume representation is generated, which represents the volume swept by at least a portion of the robot between a set of pose sequences specified by the nominal trajectory, from at least one time in the nominal trajectory to another time in the nominal trajectory.

19. The processor-based system according to claim 18, wherein, Determining the appropriate acceptable lag time for the nominal trajectory also includes: Perform a collision assessment using the generated swept volume.

20. The processor-based system according to claim 19, wherein, In order to perform collision assessment using the generated sweep volume, the processor executable instructions, when executed by the at least one processor, cause the at least one processor to perform collision assessment for each of the two or more robots between i) and ii), where i) represents at least a portion of a corresponding sample trajectory of the robot with at least one corresponding lag time introduced, and ii) at least a portion of a corresponding sample of the trajectory of each of the other two or more robots, wherein at least one corresponding lag time is introduced into the corresponding trajectory of each of the other two or more robots. In response to determining that a collision will occur between the robot and at least one of the other two or more robots, for one or more nominal trajectories of the robot, a corresponding lag time less than the corresponding lag time that would cause a collision to occur is identified as the corresponding acceptable lag time, wherein the acceptable lag time reflects the maximum acceptable delay in attitude timing relative to the nominal trajectory, the maximum acceptable delay still ensuring that the movement of the robot through the attitude sequence specified by the nominal trajectory remains collision-free relative to the movement of the other two or more robots.

21. The processor-based system according to claim 20, wherein, In order to determine the corresponding acceptable lag time of the nominal trajectory, the processor-executable instructions, when executed by the at least one processor, also cause the at least one processor to: In response to determining that a collision will occur between the robot and at least one of the other two or more robots, for one or more nominal trajectories of the other two or more robots where a collision has been determined to occur, a corresponding lag time less than the corresponding lag time that caused the collision to occur is identified as the corresponding acceptable lag time, wherein the acceptable lag time reflects the maximum acceptable delay in attitude timing relative to the nominal trajectory, the maximum acceptable delay still ensuring that the movement of the robot through the attitude sequence specified by the corresponding nominal trajectory remains collision-free relative to the movement of the other two or more robots.

22. The processor-based system according to claim 17, wherein, In order to determine the corresponding acceptable lag time of the nominal trajectory, the processor-executable instructions, when executed by the at least one processor, also cause the at least one processor to: Iterate through each of the multiple candidate lag times in order from the relatively smaller candidate lag times to the relatively larger candidate lag times, at least until the stopping condition is met. Iterate through each of the multiple time periods covered by the nominal trajectory. Based on the current candidate lag time among the candidate lag times, check for collisions between one of the two or more robots and at least one of the other two or more robots; as well as In response to determining that a collision will occur between one of the two or more robots and at least one of the other two or more robots, the corresponding acceptable lag time of the nominal trajectory of the robot that is determined to collide with at least one of the other two or more robots is set to the closest previous candidate lag time.

23. The processor-based system according to claim 22, wherein, In order to check for collisions between the robot and the other robots based on the current candidate lag time from the candidate lag times, the processor may execute instructions, when executed by the at least one processor, such that the processor: For each of the two or more robots Collision assessment is performed on at least a portion of the nominal trajectory of the robot having the current candidate lag time or another previously determined lag time, relative to at least a portion of the trajectory of each of the other two or more robots having the current candidate lag time.

24. The processor-based system according to claim 23, wherein, In order to perform a collision assessment on at least a portion of the nominal trajectory of the robot having the current candidate lag time or another previously determined lag time relative to at least a portion of the trajectory of each of the other two or more robots having the current candidate lag time, the processor is executable with instructions, when executed by the at least one processor, causing the processor to perform a collision assessment to determine whether at least a portion of the robot collides with at least a portion of any of the other robots during movement of the robot and the other two or more robots along at least a portion of the nominal trajectory delayed by the candidate lag time.

25. The processor-based system according to claim 22, wherein, In order to check for collisions between the robot and the other robots based on the current candidate lag time from the candidate lag times, the processor may execute instructions, when executed by the at least one processor, such that the processor: For each of the two or more robots Generate a sweep volume representation, which represents the volume swept by at least a portion of the robot from at least one time in the corresponding nominal trajectory to another time in the corresponding nominal trajectory between a set of pose sequences specified by the corresponding nominal trajectory, wherein the current candidate lag time among the candidate lag times is introduced into the corresponding nominal trajectory. as well as For each pair of robots of the two or more robots For attitude timing specified by a nominal trajectory without introducing any lag time and for attitude timing specified by a nominal trajectory that introduces each of a plurality of candidate lag times into the corresponding nominal trajectory, determine whether the sweep volume of one robot in the pair intersects with the sweep volume of the other robot in the pair, at least until an intersection is detected, and In response to determining that the sweep volume of one robot in the pair intersects with the sweep volume of the other robot in the pair and identifying the robot that has collided, a collision indication is generated.

26. The processor-based system according to claim 17, wherein, In order to take at least one remedial measure, the processor executable instructions, when executed by the at least one processor, cause the processor to perform one or more of the following: stop the movement of one or more of the robots, slow down the movement of one or more of the robots, or speed up the movement of one or more of the robots.

27. The processor-based system according to claim 17, wherein, In order to take at least one remedial action, the processor may execute instructions, when executed by the at least one processor, such that the processor: stops the movement of the robot, causes one or more of the robots to move along the corresponding trajectory specified by the corresponding nominal trajectory, and then restarts the movement of the robot.

28. The processor-based system according to claim 17, wherein, The determination of the corresponding acceptable lag time for the nominal trajectory occurs during the configuration time prior to the movement of at least two robots according to one or more nominal trajectories specified by the respective robots.

29. The processor-based system according to claim 28, wherein, The lag between the monitored nominal trajectory and the corresponding actual trajectory executed by the corresponding robot occurs during the runtime of the at least two robots executing a movement specified by one or more nominal trajectories of the corresponding robots, the runtime being after the configuration time.

30. The processor-based system according to claim 17, wherein, The lag between the nominal monitored trajectory and the corresponding actual trajectory executed by the respective robot includes a central clock shared by the monitors of one or more robots.

31. The processor-based system according to any one of claims 17 to 30 and any combination thereof, wherein the processor-executable instructions, when executed by the at least one processor, further cause the processor to: Receive the corresponding motion plan for each of the robots, each motion plan specifying a corresponding nominal trajectory among one or more nominal trajectories for the corresponding robot, the nominal trajectory representing the corresponding collision-free path.

Images