Method and device for online task planning and robot motion control

Abstract

The disclosed technology embodies beneficial methods of online task and motion planning for robots based on robot task decomposition into subtasks with additional features that indicate, among other things, whether dynamic trajectory correction is allowed during execution of the subtasks. Decomposition and feature marking enable hybrid motion control during task execution, for example, some subtasks allow reactive motion planning to be employed to avoid collisions, while other subtasks do not allow reactive motion to be employed. Collision avoidance of non-reactive subtasks involves, for example, coordinating volume reservation within the robot workcell involved to prevent other robots from interfering with the robots that are performing designated non-reactive subtasks. It is determined which sub-tasks are non-reactive dependent on information contained in the input task specification, and may be based on user analysis or automatic analysis of robotic applications involved.

Description

TECHNICAL FIELD

[0001] The subject matter disclosed herein relates to robotics, and in particular, to online task planning and robot motion control. BACKGROUND

[0002] Robot automation enables a wide variety of applications in industrial settings, such as assembly, welding, palletizing, deburring, and inspection. Implementing robot automation requires planning and scheduling of robot tasks.

[0003] Generally, a robot automation application consists of one or more tasks, where example tasks are picking and placing a part. Each task can include one or more subtasks, where example subtasks can be moving to a pre-pick location or moving from a pre-pick location to a pick location. Tasks and subtasks can have associated properties, goals, and constraints (such as paths and / or velocities) that can or can not be specified (flexible) and can change during execution. For example, an industrial robot used to assemble machine parts needs to pick a part, inspect the part, and then perform an operation such as welding, deburring, or gluing the part. Each of these tasks must be performed in a specific order and can involve multiple independent motions of the robot effector and motors.

[0004] Generally, a user manually specifies the tasks and the sequence of motions involved by the robot to complete the tasks while avoiding collisions with other objects or robots in the environment. While robots can be programmed offline manually, this approach can be inappropriate in some application contexts and can require arduous and tedious manual work to set up, integrate, and validate.

[0005] In particular, manual setup becomes especially challenging when there are sources of variation. Sources of variation can include differences in the sequence of robot tasks, the start and goal positions, the start time, and the duration period. For example, in a deburring application, the time and location at which a part arrives on a conveyor belt can not be consistent, or the size and shape of the part can vary between different periods. Furthermore, an inspection step that can be present before deburring can cause branching in the task sequence, such as a part being discarded or not discarded after inspection. There can be hundreds or even thousands of such variations in an application, making manual task assignment and trajectory programming extremely cumbersome and error-prone.

[0006] Further, many applications require multiple robots. For multiple robots operating in a shared workspace, task assignment and motion planning operations become more complex. For example, a user needs to manually ensure (such as by using interlocking techniques) that the robots avoid collisions with other objects in the environment. The creation, validation, and verification of such manual techniques becomes too difficult and expensive to implement.

[0007] As noted above, the complexity and challenges of task planning and motion planning have spurred growing interest in automated online task planning and robot motion control, with increasing applications. In this context, "task planning" can be understood as the combination and sequencing of primitive robot operations to form a structured plan to meet prescribed goals, given specific robot and environment-related specifications. These primitive robot operations can be further decomposed into one or more discrete robot motions.

[0008] The execution of a task plan involves precise and accurate control of robot motion, which means controlling the robot's position, velocity, and acceleration to ensure that it precisely follows the planned trajectory. Such motion control takes into account various factors, such as the dynamics of the robot, the state of its environment, and any obstacles the robot can encounter during the execution of the task. Thus, online motion planning can be understood as the process of generating, immediately before (or during) the motion, discrete robot motions that satisfy motion constraints and possibly optimize certain aspects of the motion. Typically, these discrete robot motions stem from the decomposition of an intended primitive robot operation. For example, the motion from pre-grasp to grasp position.

[0009] Online task and motion planning provides flexibility for handling changes sources and providing automated collision avoidance functionality in multi-robot environments. Specifically, some conventional methods for automated online task and motion planning use reactive motion control, which enables the robot to react in real-time to changes or variations in the environment. Thus, "reactive" motion planning makes real-time adjustments to the robot's trajectory in response to obstacles, self-circumstances, other robots, or other environmental considerations.

[0010] While existing online motion planning methods bring benefits to robot automation, there are applications where existing online task and motion planning methods are not practical or completely inapplicable. For example, inapplicability occurs where certain sub-tasks in an application have incompatible goals, constraints, or properties, rendering online motion planning infeasible. For example, consider a sub-task that requires the robot to move along a strict straight path at a specified velocity for functional success, such as required in applying a fluid (e.g., a bead of glue).

[0011] In such subtasks, any dynamic trajectory modification from a reactive motion planner is likely to cause application failure in order to ensure collision avoidance. There is a mismatch between the functional and operational objectives of collision avoidance and task / application success. Similar problems arise in a wide variety of practical robotic applications, and can be particularly challenging in multi-robot scenarios. SUMMARY

[0012] The techniques described in this specification allow for the allocation (dynamic or pre-allocation) of one or more tasks or subtasks to one or more robots in a multi-robot application to achieve functionality and / or performance. Some embodiments allow for the bundling of two or more subtasks such that these bundled subtasks can be executed without interruption.

[0013] The disclosed techniques embody beneficial approaches to online task and motion planning for robots that are based on decomposition of a robot task into subtasks with additional features that, among other things, indicate whether dynamic trajectory modification is allowed during execution of a subtask. The decomposition and feature labeling enable hybrid motion control during task execution, e.g., some subtasks allow reactive motion planning to avoid collisions, while other subtasks do not allow reactive motion. Collision avoidance for non-reactive subtasks involves, e.g., coordinating volume reservations within the work cell of the robot involved to prevent other robots from interfering with the robot that is executing a subtask designated as non-reactive. Determining which subtasks are non-reactive relies on information contained in the input task specification, and can be based on user analysis or automatic analysis of the robot application involved.

[0014] One embodiment includes an online task planning and robot motion control method for robot control. The method includes: (a) processing an input task specification that specifies a robot task, where the input task specification decomposes the robot task into subtasks and distinguishes between reactive motion (RM) subtasks that allow dynamic trajectory modification and non-reactive motion (NRM) subtasks that do not allow dynamic trajectory modification; (b) generating subtask commands in sequence according to subtask dependencies determined from the input task specification, where the subtask commands are distinguished between RM subtasks and NRM subtasks; (c) determining robot trajectories to execute the subtask commands according to online motion planning that allows dynamic trajectory modification during execution of RM subtasks and does not allow dynamic trajectory modification during execution of NRM subtasks; and (d) outputting position commands to one or more robot controllers to execute robot motion.

[0015] Related implementations include a computer system for online task planning and robot motion control. The computer system includes interface circuitry and processing circuitry. The processing circuitry is configured to: (a) process an input task specification that specifies a robot task, where the input task specification decomposes the robot task into subtasks and distinguishes between reactive motion (RM) subtasks that allow dynamic trajectory modification and non-reactive motion (NRM) subtasks that do not allow dynamic trajectory modification; (b) generate subtask commands in sequence according to subtask dependencies determined from the input task specification, where the subtask commands are distinguished between RM subtasks and NRM subtasks; (c) determine robot trajectories to execute the subtask commands according to online motion planning that allows dynamic trajectory modification during execution of RM subtasks and does not allow dynamic trajectory modification during execution of NRM subtasks; and (d) output position commands to one or more robot controllers to execute robot motion.

[0016] Of course, the present application is not limited to the features and advantages summarized above. Those of ordinary skill in the art will, upon reading the following detailed description and viewing the accompanying drawings, recognize additional features and advantages. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a block diagram of an online task planning and motion control framework according to one implementation.

[0018] Figure 2 is a block diagram of a computer system implementing the online task planning and motion control framework of Figure 1 , further illustrating an example robot under control.

[0019] Figure 3 is an example block diagram of an effector of a robot, which may, for example, receive commands during robot control.

[0020] Figure 4 is a block diagram of example details of processing circuitry for implementing the computer system shown in Figure 2 .

[0021] Figure 5 is a logic flow diagram of an operational method for online task planning and motion control according to example implementations.

[0022] Figure 6 is an illustration of an example task graph, embodying a set of tasks contained in a robot application.

[0023] Figure 7 is a parameterized illustration of an example individual task, including subtask decomposition and corresponding subtask classification.

[0024] Figure 8 is a diagram illustrating volume reservation from a start position A to a target position B.

[0025] Figure 9 is a diagram of an example robot showing different positions associated with different subtask categories of an example pick-and-place operation.

[0026] Figure 10 is a diagram of an example task specification involving an application of two robots, including decomposition of example subtasks and bundling of subtasks.

[0027] Figure 11 is a diagram of two six-axis robots and illustrates an example scenario in which target regions of the two robots overlap.

[0028] Figure 12 is a logic flow diagram illustrating an example pick-and-place application with corresponding subtask decomposition.

[0029] Figure 13 is a logic flow diagram of an example glue application with corresponding subtask decomposition. DETAILED DESCRIPTION

[0030] Embodiments disclosed herein provide an overall framework that enables functional goals embodied in a robot application to be achieved while also enabling operational goals such as collision avoidance for single-robot workcells and multi-robot workcells to be achieved. The overall framework allows a variety of motion planning techniques to be integrated to execute a robot application, both manual planning techniques and automatic motion planning techniques such as reactive motion planning. Different types of motion planning are allowed at the subtask level, enabling precise and automatic adaptation to application-specific and motion-specific constraints in a wide range of practical applications involving single robots or multiple robots operating in multi-robot workcells.

[0031] In this context, a “task” is a goal-oriented functional module of a larger application, e.g., a pick-and-place operation in a robot assembly. A “subtask” is a basic building block of a larger application. One or more subtasks can be combined to form a task, e.g., discrete robot motions from a pre-position to a placement position (in a placement task).

[0032] Figure 1 An example online task planning and motion control framework 10 (“framework 10” or “system 10”) is shown, in which motion control is based on online motion planning. Framework 10 relies on physical processing circuitry, memory, and interface circuitry to be instantiated, although framework 10 can be implemented in a virtualized computing environment based on underlying computing hardware.

[0033] The control framework includes an online task planner 12 that receives an input task specification 14, for example by reading a data file from a computer file system, and the framework also includes a robot interface subsystem 16 that includes a motion planner and workcell coordination functionality in one or more implementations for a multi-robot environment. The robot interface subsystem 16 converts subtask commands to corresponding position commands that are output to one or more robot controllers 18, where the position commands are based on online motion planning. While shown as separate from the framework 10, the one or more robot controllers 18 can be integrated into the framework 10, for example the overall configuration of the computer system can include processing circuitry and associated interfaces to instantiate the framework 10 and the robot controllers 18 as an integrated system.

[0034] Each robot controller 18 converts the position commands it receives to corresponding robot commands, which can be understood as commands sent to motors or other forms of actuators for causing the robot linkages involved to move according to the commanded positions. Each robot 20 can have one robot controller 18, or one robot controller 18 can interact with and control more than one robot 20.

[0035] The operations shown can continue, for example the task planner 12 issues subtask commands in sequence, the robot interface subsystem 16 issues corresponding position commands, and the robot controllers 18 issue corresponding robot commands to drive and otherwise control the robot(s) 20 involved. For example, there can be commands to control end effectors or other control actions in conjunction with or in addition to commands to move the robots 20, for example to grip an object or release an object or to control an attached tool.

[0036] Figure 2 A computer system 30 is shown that implements the framework 10 and robot controllers 18 described in Figure 1 above according to an example implementation. Example implementations of the robots 20 also appear in Figure 2 and the reader will appreciate from the flexibility described above that Figure 2 only represent example configurations. For example, there can be more than one robot 20, each with a robot controller 18. Also, the robot controllers 18 can be implemented independently of the framework 10, and Figure 2 the computer system 30 shown in

[0037] The computer system 30 includes processing circuitry 32, which can include or be communicatively connected to a memory 34. In one or more embodiments, the memory 34 stores one or more computer programs 36 (shown as "CP" in the figure) and one or more types of data 38. In at least one embodiment, the stored data 38 includes the task specification 14 read by the computer system 30 and / or results of processing the task specification 14.

[0038] The interface circuitry 40 of the computer system 30 provides communicative connection of the processing circuitry 32 with one or more external entities, systems, or devices. For example, the interface circuitry 40 includes first transceiver circuitry (shown as transmitter (TX) 42-1 and receiver (RX) 44-1) for interfacing with a local area network (LAN) or other data network. A non-limiting embodiment is to implement the TX 42-1 and RX 44-1 as an Ethernet-based interface circuit. Of course, such circuitry can be radio circuitry for wireless communication, as an alternative or complementary solution to wired connection.

[0039] The illustrated interface circuitry 40 also includes second transceiver circuitry (shown as transmitter (TX) 42-2 and receiver (RX) 44-2) for directly interacting with the robot 20, or for interacting with the respective robot controller 18. It is also noted that, in some embodiments, a single communication interface (e.g., a data network interface) communicatively connects the computer system 30 for receiving the task specification 14 and for controlling the robot 20 or the robot controller 18 associated therewith.

[0040] The interface circuitry 40 can thus include digital or analog signaling circuitry or both for outputting control commands to the robot controller 18 or the robot 20 and receiving respective feedback (respective measurements) such as for response, dynamic control of the robot 20. While the computer system 30 is shown as separate from the example robot 20, in one or more other embodiments, the computer system is integrated with the robot.

[0041] Figure 2 Example details of one robot embodiment are also illustrated, and it is again noted that more than one robot 20 can be involved, such as two or more robots 20 sharing a work cell. The multi-robot scenario can involve two or more robots of the same type, or different types.

[0042] Figure 2The exemplary configuration of the robot 20 shown includes a base 50 having a first rotational axis through a first rotational joint 52. A second rotational joint 54 connects a first arm 56 to the base 50. The other end of the first arm 56 terminates in a third rotational joint 58 that connects the first arm 56 to a second arm 60, which is divided into two segments 60A and 60B, with a fourth rotational joint 62 allowing segment 60B to rotate relative to segment 60A. A fifth rotational joint 64 connects segment 60B to a third arm 66, and a sixth rotational joint 68 connects an end effector 70 to the end of the third arm 66.

[0043] Referring back to Figure 1 The subtask commands are converted to position commands, and the position commands are converted to robot commands, which include, for example, control signals for the actuators of the robot 20 involved. More specifically, where the subtasks involved are motion subtasks, the generated subtask commands result in one or more position commands, while for non-motion subtasks the generated subtask commands typically do not result in the generation of position commands, but can involve the generation of action commands, e.g., open gripper, close gripper, etc.

[0044] Figure 3 Exemplary signaling between the robot controller 18 and an actuator 80 associated with a particular joint of a particular robot 20 is shown. The actuator 80 includes, for example, a direct current (DC) rotary actuator, although in one or more embodiments the robot can incorporate linear actuators, or a mix of linear and rotary actuators. In any case, the exemplary actuator 80 is responsive to input commands that control, for example, the direction of actuation, the speed of actuation, and the amplitude of actuation. For rotary motion, such commands drive changes in the angle of the joint, and control of the joint angles between / over the links of the robot 20 determines the pose of the robot 20, where "pose" refers to the particular position and orientation of all of the links of the robot 20.

[0045] For example, the actuation of the robot 20 occurs in the course of executing one or more "applications." An application can be embodied, for example, as a computer program, and defines certain actions and end effector operations to be performed by the robot 20 to perform one or more tasks. Exemplary tasks include pick-and-place tasks, in which the robot 20 picks up an item from one location and places it in another location. In one example application, the robot 20 takes items from a conveyor belt and sequentially packs them into trays, boxes, or the like. The tasks described in the input task specification 14 can be understood to define an application.

[0046] Accordingly, the processing circuitry 32 of the computer system 30 is configured to process the input task list 14 and, as described above, generate corresponding subtask commands that are converted into motion and action commands for one or more robots 20. In the example implementation shown as Figure 4 In the example implementation shown, the processing circuitry 32 can be implemented as a microprocessor 90 that executes computer program instructions held in an associated memory 92. In other words, the microprocessor 90 is particularly adapted to operate in accordance with the processing circuitry descriptions provided herein based on executing the computer program instructions.

[0047] More generally, the processing circuitry 32 can be implemented as a fixed circuit, a programmably configured circuit, or a hybrid of the two. Furthermore, two or more microprocessors can be used, or a multi-core microprocessor can be used, rather than a single microprocessor. Additionally or alternatively, the implementation of the processing circuitry 32 can be based substantially on any type of digital processing circuitry, such as one or more field-programmable gate arrays (FPGA), system-on-chip (SoC), application-specific integrated circuit (ASIC), complex programmable logic device (CPLD), or digital signal processor (DSP).

[0048] The memory 34 can include one or more types of computer-readable media, including volatile memory as a working memory for program execution and related data storage, and non-volatile memory for long-term storage of computer programs and configuration data, such as one or more user-configured parameter values. Embodiments include dynamic random access memory (DRAM), static RAM (SRAM), non-volatile RAM (NV-RAM), FLASH or solid-state disk (SSD), electrically erasable programmable read-only memory (EEPROM), and the like.

[0049] Regardless of the specific implementation details, the computer system 30 is configured for online task planning and corresponding robot motion control, and in the example implementation includes interface circuitry 40 configured for receiving data including the input task specification 14 that specifies a robot task, and further including the processing circuitry 32.

[0050] The processing circuitry 32 is configured to process the input task specification 14 that specifies a robot task. In particular, the input task specification decomposes the robot task into subtasks, and distinguishes between reactive motion (RM) subtasks that allow for dynamic trajectory correction, and non-reactive motion (NRM) subtasks that do not allow for dynamic trajectory correction.

[0051] The processing circuitry 32 is further configured to generate subtask commands in sequence according to the subtask dependencies determined from the input task specification 14, wherein the subtask commands are distinguished between RM subtasks and NRM subtasks; determine robot trajectories for executing the subtask commands according to an online motion planning, which allows dynamic trajectory modification during execution of the RM subtasks, but does not allow dynamic trajectory modification during execution of the NRM subtasks; and output position commands to the one or more robot controllers 18 via the interface circuitry 40 for executing robot motions.

[0052] By including a subtask feature, the subtask commands are distinguished between RM subtasks and NRM subtasks. Each subtask listed in the input task specification 14 belongs to a respective subtask category, wherein the computer system 30 identifies a plurality of subtask categories. The identified categories include one or more categories of non-motion subtasks and two or more categories of motion subtasks, including the RM and NRM categories. Accordingly, each subtask command includes a feature indicating the respective subtask category for consideration in the online motion planning.

[0053] In at least one embodiment, an idle category is defined. Idle is a special category of motion subtasks, specifically reactive motion subtasks. An idle subtask requires the robot to maintain a desired pose for a given duration of time, except in the case of a collision avoidance situation that requires a reaction (in which case robot motion is involved); the attribute is that once the reactive collision avoidance is completed, the robot returns to the desired pose.

[0054] The category definitions can further include one or more non-motion action (NMA) categories that do not allow dynamic trajectory modification. The processing circuitry 32 is configured to output action commands to the one or more robot controllers 18, which correspond to NMA tasks represented in the subtask commands. As an example, the NMA tasks can involve robot end effector actions, and the action commands are end effector control commands, such as grasp, release, etc.

[0055] In at least one embodiment, in addition to the RM and NRM categories, the two or more categories of motion categories further include a predictive motion category, in which motions of the robot 20 are repeatedly planned over a specified time horizon, and a manual mode motion category, in which motions of the robot 20 are manually defined according to a plurality of waypoints, and an interpolation algorithm is used to generate a trajectory along the waypoints.

[0056] In one or more embodiments, the online motion planner instantiated via the computer system 30 uses characteristics of motion subtasks to determine corresponding trajectory constraints. Determining the constraints includes determining, at a subtask granularity, whether dynamic trajectory modification is allowed. One of several advantages embodied in the operation of the computer system 30 is the ability to have both reactive and non-reactive motion, where decisions are made at the subtask level, such that reactive and non-reactive motion can be used in a mix to perform a given task. This mix maximizes the flexibility of path planning while still allowing non-reactive motion to be employed when deviating from a nominal or desired trajectory is incompatible with the performance of the subtasks involved.

[0057] In at least one embodiment, the input task specification 14 involves two or more robots operating in a shared workspace. For a particular one of the robots 20 that performs RM subtasks, the online motion planning includes dynamic trajectory modification for collision avoidance with the remaining one or more robots 20. For a particular robot 20 that performs NRM subtasks, the processing circuitry 32 is configured to perform collision avoidance by coordinating volume reservations such that the remaining one or more robots 20 do not interfere with the robot trajectory during performance of the NRM subtasks by the particular robot 20.

[0058] In the same or another embodiment in which the input task specification 14 involves two or more robots operating in a shared workspace, the processing circuitry 32 is configured to identify related tasks that must be performed by the same one of the two or more robots 20 and to assign the subtasks that make up the related tasks to the same robot 20.

[0059] In at least one embodiment, the processing circuitry 32 is configured to identify and process “subtask packs.” That is, the input task specification 14 can indicate one or more subtask packs. Each subtask pack contains two or more subtasks in a corresponding task that must be performed by the robot 20 assigned to that task without interference by other robots 20. Here, the processing circuitry 32 is configured to coordinate operations among the two or more robots 20 to ensure uninterrupted performance of each subtask pack.

[0060] In at least one embodiment involving subtask packs, one or more tasks or subtasks are pre-assigned for performance by a particular one of the two or more robots 20 that are collaborating in a shared workspace, while one or more tasks or subtasks listed in the input task specification 14 are not assigned. For the unassigned tasks or subtasks, the processing circuitry 32 is configured to determine an assignment of the robots. The assignment can be made at a task level, and the assignment can be determined based on availability, load balancing, a state of the robots 20 as candidates for selection, or other considerations.

[0061] Figure 5 A method 500 for online task planning and robot motion control performed by a computer system (e.g., the computer system 30 described above) is shown. The method 500 includes processing (block 502) an input task specification 14 that specifies a robot task. Such processing includes, for example, reading the input task specification 14 as a stored computer file, or receiving signaling indicative of the input task specification 14. In any case, the input task specification 14 decomposes the robot task into subtasks, and distinguishes between reactive motion (RM) subtasks that allow dynamic trajectory modification, and non-reactive motion (NRM) subtasks that do not allow dynamic trajectory modification.

[0062] The method 500 further includes generating (block 504) subtask commands in sequence according to subtask dependencies determined from the input task specification. The subtask commands are distinguished between RM subtasks and NRM subtasks, and the method 500 further includes determining (block 506) robot trajectories to execute the subtask commands according to online motion planning that allows dynamic trajectory modification during execution of RM subtasks, and does not allow dynamic trajectory modification during execution of NRM subtasks. Still further, the method 500 includes outputting (block 508) position commands to one or more robot controllers 18 to execute robot motion. The method 500 can further include any additional or alternative operations described above for the processing circuit 32 to implement the framework 10.

[0063] Figure 6 An example task graph representing a single robot or multi-robot application is shown. The task graph shows that the application includes multiple tasks, and the relationships between the tasks represent task dependencies, e.g., task 1 must be executed before task 2, and so on. A key aspect of the task graph is that it represents a real, real-world application in which the execution of the entire application requires both reactive motion and non-reactive motion. By way of example only, task 2 contains a set of subtasks, some of which involve RM category subtasks, and some of which involve NRM category subtasks. Task 4 contains at least RM or NMA category subtasks, and task 5 contains at least NRM subtasks.

[0064] Unlike conventional online task and motion planning, the framework 10 described herein allows for granular control at the subtask level to determine whether to use reactive motion (which allows dynamic trajectory modification) or non-reactive motion (which does not allow dynamic trajectory modification). This level of granularity provides greater flexibility for dynamic collision avoidance in at least some portions of the motion process, while still allowing for deterministic path tracking for those motions that are not suitable for dynamic path modification.

[0065] The disclosed method associates features with subtasks, each subtask's associated features being the subtask's class, and the motion planner involved is configured to identify which subtask classes allow reactive motion and which do not. Of course, additional information can be specified in the input task specification 14, such as additional attributes and / or constraints at the task or subtask level. Also as mentioned previously, the input task specification 14 can indicate a subtask bundle. Here, bundling is a particular way of constraining tasks / subtasks. The framework 10 recognizes the specification of tasks / subtasks and controls the runtime to automatically assign the tasks / subtasks bundle to one or more specific robots 20, with the generation and sequencing of the subtasks and the overall control coordinated to ensure that the subtask bundle is executed in order and without interruption or override.

[0066] Viewed from another perspective, breaking an application task into subtasks, and identifying and using features at the subtask level, makes it possible to impose task-specific constraints and objectives on the online motion planner in an ad hoc manner; and this advantageous way of control can be generalized to any application, including those involving multiple robots and any variation in nature. Here, "ad hoc" refers to the seamless switching of motion control from reactive to non-reactive, or vice versa, at the granularity of the subtask level.

[0067] Returning again to Figure 1 , the framework 10 can be understood as a robot motion control system comprising several functional components, including an online execution system consisting of at least the task planner 12 and the robot interface subsystem 16. These components can be implemented by one or more computer systems such as the computer system 30 executing one or more computer programs such as the computer program 36 shown in Figure 2 . If multiple computer systems are involved, they can be configured to communicate via a network such as Ethernet, EtherCAT, or Firewire. The robot interface subsystem 16 can also include a programmable logic controller (PLC) for implementing the robot controller 18 shown in Figure 1 .

[0068] The task planner 12 provides a sequence of subtask commands to be executed by the robot interface subsystem 16, which in turn sends the command positions to the robot controller, which drives one or more controlled multi-axis robots 20.

[0069] The robot interface subsystem 16 includes a global path planner that generates feasible collision-free paths for one or more robots 20 based on input from the task planner 12, which may include starting and goal configurations, a desired path with velocity constraints, and multiple sub-goals. The robot interface subsystem 16 also includes a reactive planner based on local optimization that takes feasible collision-free paths as input and generates time-parameterized collision-free trajectories that are sent to the corresponding robot controllers 18 for execution. The robot interface subsystem 16 also includes support modules that enable motion coordination between multiple robots to resolve situations involving goal conflicts and deadlocks. Here, support modules can be understood as logical functions implemented by executing computer program instructions.

[0070] like Figure 2 As shown, robot 20 may be a multi-axis robotic arm having an end effector at the end of the arm. The arm comprises multiple joints, and the end effector may include, for example, a two-finger gripper, a vacuum suction cup, or a dispensing device. The robot's position is defined by the angles of each of the robot's joints. A given position and orientation of the robot may correspond to one or more joint configurations of the robot. During execution, the robot interface subsystem 16 sends a command position to the robot controller 18 involved and receives the corresponding measured position as feedback.

[0071] In one embodiment, prior to executing an operation performed by a system referred to as an online task planning and motion planning framework, a user provides input a task specification 14 that includes a complete application to be executed by a particular robot 20 or by any one or more of a plurality of robots 20. The task specification 14 identifies a sequence of tasks, wherein one or more tasks encapsulate subtasks. A task may contain only one subtask, but typically a task will contain multiple subtasks arranged in an ordered sequence that produce an intermediate result (such as a grasping operation); after which the next operation (such as a placing operation) may or may not be deterministically selected. In a sequence, there may be tasks that are executed by multiple allowed robots. Tasks are viewed herein as goal-oriented functional building blocks of larger applications (e.g., robotic assembly).

[0072] The task specification 14 may be provided as a structured text document (such as XML) or as a logical flow chart. The task planner 12 processes the task specification 14 to construct a corresponding directed graph. The information conveyed in the task specification 14 includes task parameterization, where the task parameters indicate: the completion dependencies of any other tasks in the list; the number and unique identifiers (IDs) of robots allowed to perform the task; an indication of whether subtasks are bundled or unbundled; and a description of the subtask decomposition of each task.

[0073] The directed graph preserves the order of tasks to be executed and follows the task dependency relationships. In Figure 6 In the directed graph shown in FIG. 5, task 5 depends on the completion of tasks 2, 3, and 4. This dependency restricts the task planner 12 from sending a request to the robot interface subsystem 16 to execute task 5 (or any of its subtasks) until tasks 2, 3, and 4 are completed. Depending on the constraints of the application, a variety of dependency relationships can be defined. The graph is traversed using a breadth-first search to find the next task for the robot 20.

[0074] Multiple robots 20 can be allowed to execute tasks, which can be identified by the robot number and ID information included in the task specification 14. Accordingly, the robot interface subsystem 16 automatically assigns tasks to one robot and handles priority issues. Tasks can have any number of task dependencies, and the task planner 12 restricts the execution of a current task unless all of its dependencies have been successfully completed. Completion information can be provided to the task planner 12 as feedback from the robot interface subsystem 16. Furthermore, within a given task, subtasks can be bundled together. Alternatively or additionally, a task can contain unbundled subtasks. In at least one embodiment, the robot interface subsystem 16 is configured to automatically switch unbundled subtasks for a particular robot 20 to manage coordination with other robots 20.

[0075] In one or more embodiments, subtasks are classified into a number of categories based on their own attributes and associated constraints, which can be explicitly specified as part of the task specification 14. Example attributes that a subtask can have include: (1) reactivity, in which robot motion planning can change during execution to react to and avoid other dynamic objects; (2) duration, in which a particular subtask must be completed within a specified duration; (3) kinematic constraints, in which a particular subtask must adhere to given joint motion limits, including position, velocity, acceleration, and jerk, or a range of poses for the robot tool center point; and (4) tool, in which a particular subtask can be associated with a particular end-effector tool, and this tool can be switched from one subtask to another.

[0076] The proposed task specification 14 allows a series of subtasks to be specified with given attributes. For a given task "x" specified in the task specification 14, an example subtask framework appears in Figure 7

[0077] ​For example, among the sub-task categories that can be specified in task specification 14 and effectively recognized and acted upon by framework 10, a first example category is reactive motion. With reactive motion, robot 20 moves from one location to another, e.g., from a start location to a target location, and the robot interface system automatically replans the trajectory to avoid collisions with other objects or robots 20 in the environment. Sub-tasks belonging to the reactive motion (RM) category have a characteristic that indicates that they belong to the RM category, and the corresponding sub-task command can specify a start joint configuration and a target joint configuration.

[0078] In one or more embodiments, framework 10 also allows for specification of a start location and a target location for an end-effector position and orientation. Such motions can be implemented, but are not limited to, trajectory optimization or artificial potential fields.

[0079] A second example category of sub-tasks is non-reactive motion (NRM). For NRM sub-tasks, robot 20 moves from a start location to a given target location along a pre-defined path and / or at a pre-defined velocity, and does not react to other objects or robots in the environment during execution. In at least one embodiment, robot interface subsystem 16 handles collision avoidance automatically by first assigning a priority to the robot 20 executing the non-reactive motion, and then computing the volume swept by the body of that robot 20 during the non-reactive motion, and reserving that volume so that no other object or robot can enter that volume until the robot 20 completes the non-reactive motion. This volume reservation can be referred to as a coordination volume reservation because it involves coordination of control of more than one robot 20.

[0080] Based on the required volume reservation, end-effector "actions" can be classified into two categories: 1) actions that do not require a change in volume reservation, which will be classified as NMA (non-motion action); or 2) actions that require a change in volume reservation, which will be classified as NRM (non-reactive motion).

[0081] Here, "representation" refers to the shape of an object used for the purpose of collision avoidance, and "reservation" refers to the volume protected from intrusion (and this volume is based on the representation and the volume swept during motion). In general, the distinction depends on: a) the size of the end-effector volume reservation (e.g., "compact" or "oversized"), and b) the amount of motion associated with the end-effector action (e.g., no motion / minimal motion vs. significant motion).

[0082] For example, consider:

[0083] - Case A: Vacuum gripper that does not move at start-up, so there is no need to change the required end effector volume reservation, which remains unchanged; therefore, the action involving the vacuum gripper belongs to the NMA category.

[0084] - Case B: Small range motion two-fingered gripper with "oversized" end effector volume reservation, which means that although the fingers of the gripper move, they remain within the oversized volume reservation range; therefore, the action involving the fingered gripper belongs to the NMA category; and

[0085] - Case C: Large range motion two-fingered gripper with "tight" end effector volume reservation, which means that when the fingers of the gripper move, they exceed the original "tight" end effector volume reservation range; therefore, the action involving the fingered gripper belongs to the NRM category.

[0086] These considerations are particularly important because it is not uncommon for robots to be equipped with complex, large range motion end effectors, and the techniques described herein apply to all such cases. Finally, the same is true for any workpiece that is gripped by the end effector, whether all three are covered by the volume reservation of the robot arm, by the volume reservations of the robot arm and end effector together, or by the volume reservations of the robot arm, end effector, and workpiece together, the volume reservation (however composed) must cover all three of the robot arm, end effector, and workpiece, if the workpiece is present. Overall, this approach is based on the mindset that the end effector (and workpiece, if present) is not different from the robot arm, at least for the purposes of task planning and motion planning (e.g., collision avoidance).

[0087] Figure 8 A particular robot 20 is shown moving from position A to position B by a non-reactive motion. The motion sweeps the robot 20 through a particular volume, which is calculated and reserved by the robot interface subsystem 16 through coordination of volume reservations. In implementations where the robot interface subsystem 16 issues all position commands for all involved robots 20, not just for a particular one of the involved robots 20, the coordination can be understood as an internal coordination of when position commands are issued and to which robot they are issued. In other implementations, where one or more of the involved robots 20 respond to position commands issued by another entity, the coordination involves the robot interface subsystem 16 communicating with those other entities.

[0088] NRM category subtasks are specified with a feature, e.g., an ID, to indicate that the subtask belongs to the NRM category. Each NRM subtask can also specify a start joint configuration and a target joint configuration. Alternatively, a non-linear (e.g., circular) path can be specified. Velocity constraints along such a path can also be specified.

[0089] Another example subtask category is the non-motion action (NMA) category. For NMA subtasks, the body of the robot 20 does not move, but the end effector can perform some action such as opening or closing. NMA subtasks are specified with an ID to indicate the NMA category, and each NMA subtask can also specify a duration of end effector grasping, holding, or releasing action. In at least one embodiment, the robot interface subsystem 16 automatically handles collision avoidance issues during NMA subtask execution, e.g., by computing a swept volume of the robot during NMA subtask execution (including end effector motion), and reserving the swept volume so that no other objects or robots 20 enter the volume before the robot 20 completes the NMA subtask.

[0090] Another subtask category is the idle category, which is a special category of motion subtasks. During execution of an idle subtask, the robot remains stationary at a given location but remains reactive. Idle subtasks are parameterized based on a specified duration (i.e., idle duration). Idle subtasks allow the robot 20 to wait at a given location before or after starting a task (due to application-specific time constraints), but allow the robot 20 to remain reactive as needed while waiting.

[0091] Still further, the defined subtask categories can include a predicted motion category. For predicted motion subtasks, the robot motion from a given start point to a target point is repeatedly planned over a finite time horizon while estimating the motion of other dynamic objects in the environment. In contrast to RM category subtasks, which have a time horizon limited to a single time step, predicted motion category subtasks explicitly predict and plan robot motion over a longer time horizon.

[0092] Due to the longer time horizon planning, dynamic changes in the environment can be taken into account and collision avoidance in uncertain environments is ensured. Predicted motion subtasks are identified using a corresponding ID and specify a start joint configuration, a target joint configuration, and a planning time horizon. The framework 10 also allows for specification of a start position and a target position for the end effector position and orientation. Such motion can be implemented through model predictive control, but implementation is not limited to this method.

[0093] In at least one embodiment, another defined subtask category is the manual mode motion category. For the manual mode motion subtask, the robot's motion is specified by the user based on multiple waypoints. These waypoints can be specified in Cartesian space or joint space. In this category, the motion between waypoints can be interpolated using fitting techniques (such as splines, linear motion, or polynomials). This subtask category is specified using an ID indicating the MM (manual mode motion) category, and each MM subtask specifies a starting joint configuration and a target joint configuration and can include multiple intermediate waypoints. The robot speed at these intermediate waypoints can also be specified, and the starting position and target position of the end effector position and orientation can be specified in addition or alternatively.

[0094] Therefore, any given subtask listed in the input task specification 14 can be parameterized by the following information: an ID indicating its subtask category, and additional information indicating one or more of the following: starting and target configurations in joint space or Cartesian space, duration, end-effector operation, duration (only required if the subtask is idle or needs to specify the time to acquire a device such as a gripper), time range (required for predictive motion), intermediate waypoints provided in joint space or Cartesian space (for manual mode motion), or the robot's velocity at intermediate waypoints (for manual mode motion).

[0095] In a more specific example of subtask bundling, consider a scenario where a robot's end effector must move along a straight path toward an object, then grasp the object, and finally return to a given location along the same straight path. These three subtasks must be completed by the same robot, and the sequence of approaching the object, grasping it, and leaving with the grasped object must be strictly adhered to and cannot be interfered with. Interference can occur if a second robot in the environment has priority over an overlapping target area and causes the first robot to interrupt its task sequence. Figure 9 An embodiment of the robot 20 performing such motion is shown.

[0096] The type of inter-robot interference just described above may cause application failure.To prevent such events from occurring, one or more embodiments of the disclosed framework allow for subtask bundling. Figure 10 An embodiment of how a user can write a task specification 14 using subtask bundling in the context of a dual-robot application is shown. The different robots are denoted as robot R1 and robot R2.

[0097] Task specifications include RM, NRM, and NMA subtasks, as well as TS (task start), TE (task end), BS (bundle start), and BE (bundle end) commands. The task specification 14 specifies the correct task sequence, and the task planner 12 identifies the sequence. Each command generated by the task planner 12 is added to an internal queue in the specified order.

[0098] Subtasks bundled with TS and TE are considered as a group of commands of the same task. This recognition of subtask bundling is used to determine the priority of the robots Rl and R2. For example, in this embodiment, if the reactive motion of Rl conflicts with the reactive motion of R2 (such as in the case of overlapping target positions), the reactive motion of Rl will be processed first. Of course, there can be other situations in the same overall application where it is appropriate to give priority to R2.

[0099] Commands bundled together with BS and BE are considered "sequential non-reactive motions." In this case, the framework 10 reserves the entire space occupied by the involved robots for the execution of the bundled subtasks to prevent other robots from interfering with the preprogrammed non-reactive motions. As shown in the swept volume diagram, the shaded reserved area is represented by the voxels occupied during the robot motion. Figure 8

[0100] Figure 11 A scenario is presented that illustrates the necessity for coordinating volume reservations, and this embodiment involves a first robot 20-1 (denoted as Rl) and a second robot 20-2 (denoted as R2). Both robots Rl and R2 operate in a shared workspace that requires volume reservations. For example, Rl is given a volume reservation around a specified target to enable Rl to perform NRM and NMA subtasks without interference from R2. As the task priority shifts to R2, Rl must move to a safe and collision-free position, and a volume will be reserved around the target of R2 to enable R2 to perform NRM and NMA subtasks. During the target reservation and until R2 completes its bundled subtasks, no other robots are allowed to enter the reserved volume. As the process progresses, completed subtask commands are removed from the queue, and the framework 10 changes the active command to the next top-level command of the target robot 20. In this way, the framework 10 automatically handles all robots 20 without any conflicts, provided that the task / subtask information is properly specified.

[0101] Figure 12 ​An example application involving the robot 20 grasping and placing an object is shown. Flowchart 1200 divides the application into separate grasping and placing operations, where both grasping and placing are characterized by comprising multiple distinct operations. The operation begins with opening the robot 20's gripper, which is the NMA subtask (block 1202). This operation is followed by the RM subtask (block 1204), in which the robot 20 reactively moves from a given starting position to a desired target position near the grasping point, where it grasps the object.

[0102] The third operation is the NRM subtask (box 1206), in which the robot 20 is constrained to move along a given path at a specific speed limit in order to approach the grasping point at a specific orientation. This operation cannot be reactive in nature because reactive motion may cause collisions with the workpiece or other objects in the environment due to interference from other robots. The fourth operation is the movement of the end effector (e.g., a gripper) (box 1208), in which the object is grasped and the gripper is closed. Depending on whether the volume reservation needs to be changed, the gripper movement can be performed as a non-motion action or a non-reactive motion action. After the gripper action, the robot 20 moves to the pre-grip position (box 1210) by executing the NRM subtask, and this movement action marks the end of the grasping operation.

[0103] The placement operation then begins: first, the RM subtask (block 1212) is executed, during which the robot 20 moves toward the pre-grip point; then, the NRM subtask (block 1214) is executed, during which the robot 20 moves toward the part placement location. At the part release location, the robot performs an end-effector motion (block 1216) to release the part. These types of motions can be non-motion subtasks if no volume reserve is required for the motion change; if so, they can be non-reactive motion motions. The robot 20 then performs the NRM subtask (block 1218), during which it returns to the pre-grip position.

[0104] Many industries require multi-axis robots to perform coating operations. For example, in the automotive industry, multi-axis robots are used to apply a variety of materials, including adhesives, lubricants, and sealants. Window and door manufacturers also use multi-axis robots to apply adhesives and install weatherstripping.

[0105] Figure 13 Flowchart 1300 illustrates an example glue application using the advantageous task decomposition and bundling techniques. The application involves a multi-axis robotic manipulator, a fixed-position glue dispenser, a workpiece, an inspection area, and an input / output conveyor for picking up and placing the workpiece. The application is divided into sequential tasks, which are then decomposed into subtasks, which are categorized according to the classifications described herein and then bundled.

[0106] The application relates to:

[0107] Task 1 (Grab): The robot moves from an initial position to a defined pre- grab position, grabs the part from the conveyor belt along a path with speed limited manner, and subsequently leaves along a limited path with limited speed, returning to the pre- grab position. For subtask classification, see blocks 1302, 1304, 1306, and 1308.

[0108] Task 2 (Approach Feed Device): The robot holding the workpiece moves from the pre- grab position above the conveyor belt to a given position, and subsequently approaches the position of the glue dispenser along a path with speed limited manner. The subtasks involved are seen in blocks 1310 and 1312.

[0109] Task 3 (Glue Application): The robot moves the workpiece in a path and speed limited manner, enabling the glue dispenser to apply a bead of glue on the part. Note that typically this type of motion is parameterized in Cartesian space. During the bead application, this task can contain multiple non-reactive motions. The subtasks involved are seen in blocks 1314 and 1316.

[0110] Task 4 (Inspection): The robot moves the finished part to an inspection area. The subtasks involved are seen in blocks 1318, 1320, and 1322.

[0111] Task 5a (Accept Object): In response to the quality being acceptable (‘Yes’ in block 1324), the robot moves from the inspection position to a placement position and releases the object, and then returns to the designated position. The subtasks involved are seen in blocks 1334, 1336, 1338, and 1340.

[0112] Task 5b (Reject Object): In response to the quality being unacceptable (‘No’ in block 1324), the robot moves the part from the inspection position to a scrap bin. The subtasks involved are seen in blocks 1326, 1328, 1330, and 1332.

[0113] In view of the above example details, the disclosed subject matter embodies techniques for implementing online task planning and robot motion control, even when the application involved requires instances of non-reactive motion by the robot, while also providing flexibility for reactive motion in other cases. These techniques include decomposing a robot application into tasks and subtasks, where the subtasks decomposed from a task have associated attributes, objectives, constraints, and characteristics. For example, attributes can change from one task to another. Example objectives include shortest time operation for a given task, while constraint examples include path limitations and speed limitations depending on the task.

[0114] One particular aspect is to associate features with sub-tasks, where these features indicate the class of the sub-task and allow the online motion planner to use different motion control strategies for motion control depending on the class of the sub-task. For example, RM sub-tasks are executed using dynamic trajectory correction, e.g., for collision avoidance, while NRM sub-tasks do not use dynamic trajectory correction to maintain a deterministic path. Other classes include different non-motion classes that allow or disallow reactive behavior of the robot to different non-motion actions.

[0115] Notably, modifications and other implementations are within the scope of the disclosed implementations and are apparent to those of ordinary skill in the art, based on the teachings herein and associated knowledge in the art. Therefore, the specific implementations discussed above are not to be taken as limiting in nature, and the scope of the implementations is defined by the appended claims.

Claims

1. A method of online task planning and robot motion control, the method comprising: processing an input task specification that specifies a robot task, wherein the input task specification decomposes the robot task into subtasks and distinguishes between reactive motion (RM) subtasks that allow dynamic trajectory modification and non-reactive motion (NRM) subtasks that do not allow dynamic trajectory modification; generating subtask commands in sequence according to subtask dependencies determined from the input task specification, wherein the subtask commands are distinguished between reactive motion subtasks and non-reactive motion subtasks; determining robot trajectories to execute the subtask commands according to online motion planning that allows dynamic trajectory modification during execution of reactive motion subtasks and does not allow dynamic trajectory modification during execution of non-reactive motion subtasks; and outputting position commands to one or more robot controllers to execute robot motion.

2. The method of claim 1, wherein, The subtask commands are distinguished between reactive motion subtasks and non-reactive motion subtasks by including a subtask characteristic.

3. The method of claim 1, wherein, Each subtask listed in the input task specification belongs to a respective subtask category, and wherein a plurality of subtask categories are identified by the processing, the plurality of subtask categories including one or more categories of non-motion subtasks and two or more categories of motion subtasks, the two or more categories of motion subtasks including a reactive motion category and a non-reactive motion category, wherein each subtask command includes a characteristic that indicates the respective subtask category for consideration in online motion planning.

4. The method of claim 3, wherein, The two or more categories of motion subtasks include an idle category that allows dynamic trajectory modification, and the one or more categories of non-motion tasks include a non-motion action (NMA) category that does not allow dynamic trajectory modification.

5. The method of claim 4, wherein, The method includes outputting action commands to the one or more robot controllers, the action commands corresponding to non-motion action tasks represented in the subtask commands.

6. The method of claim 5, wherein, Non-motion action tasks involve robot end effector actions, and the action commands are end effector control commands.

7. The method of claim 3, wherein, The two or more categories of motion categories include, in addition to the reactive motion category and the non-reactive motion category, a predictive motion category in which motion of the robot is repeatedly planned over a specified time horizon, and a manual mode motion category in which motion of the robot is manually defined according to a plurality of waypoints and a trajectory along the waypoints is generated using an interpolation algorithm.

8. The method of claim 3, wherein, The online motion planner uses the characteristics of motion subtasks to determine respective trajectory constraints, including determining whether dynamic trajectory modification is allowed at a subtask granularity.

9. The method of claim 1, wherein, The input task specification involves two or more robots operating in a shared workspace, and for a specific one of the robots performing a reactive motion subtask, the online motion planning includes dynamic trajectory correction for avoiding collision with the remaining one or more robots, and wherein, for the specific robot performing a non-reactive motion subtask, collision avoidance is based on performing coordination volume reservation so that the remaining one or more robots do not interfere with the robot trajectory during the specific robot's performance of the non-reactive motion subtask.

10. The method of claim 1, wherein, The input task specification relates to two or more robots operating in a shared workspace, and wherein the method comprises: identifying related tasks that must be performed by the same robot among the two or more robots; and assigning subtasks constituting the related tasks to the same robot.

11. The method of claim 10, wherein, The input specification indicates one or more subtask packages, each subtask package containing two or more subtasks in a corresponding task, and the two or more subtasks must be performed by the robot assigned to the corresponding task without being interfered with by one or more other robots, and wherein the method also includes coordinating the operations of the two or more robots to ensure that each subtask package is executed uninterruptedly.

12. The method of claim 1, wherein, One or more tasks or subtasks are pre-assigned to specific robots among two or more robots collaborating in a shared workspace for execution, and wherein one or more tasks or subtasks are unassigned, and the method further includes determining an assignment of robots for the unassigned tasks or subtasks.

13. A computer system for online task planning and robot motion control, the computer system comprising: an interface circuit configured to receive data including an input task specification specifying a robot task; as well as a processing circuit, the processing circuit being configured to: processing an input task specification specifying a robotic task, wherein the input task specification decomposes the robotic task into subtasks and distinguishes between reactive motion (RM) subtasks that allow dynamic trajectory correction and non-reactive motion (NRM) subtasks that do not allow dynamic trajectory correction; sequentially generating subtask commands based on subtask dependencies determined from the input task specification, wherein the subtask commands are differentiated between reactive motion subtasks and non-reactive motion subtasks; determining a robot trajectory for executing the subtask command according to an online motion plan, wherein the online motion plan allows dynamic trajectory correction during execution of the reactive motion subtask but does not allow dynamic trajectory correction during execution of the non-reactive motion subtask; and Position commands are output to one or more robot controllers via the interface circuit to execute robot motions.

14. The computer system of claim 13, wherein, The subtask commands are differentiated between reactive motion subtasks and non-reactive motion subtasks by including subtask features.

15. The computer system of claim 13, wherein, Each subtask listed in the input task specification belongs to a respective subtask category, and wherein a plurality of subtask categories are identified by the computer system, the plurality of subtask categories including one or more categories of non-motion subtasks and two or more categories of motion subtasks, the two or more categories of motion subtasks including a reactive motion category and a non-reactive motion category, wherein each subtask command contains a feature indicating the respective subtask category for consideration in online motion planning.

16. The computer system of claim 15, the two or more classes of motion subtasks including an idle class that allows for dynamic trajectory corrections, and wherein, The one or more categories of non-motion subtasks include a non-motion action (NMA) category that does not allow dynamic trajectory modification.

17. The computer system of claim 16, wherein, The processing circuitry is configured to output motion commands to the one or more robot controllers, the motion commands corresponding to non-motion action tasks represented in the subtask commands.

18. The computer system of claim 17, wherein, The non-motion action tasks involve robot end effector actions, and the motion commands are end effector control commands.

19. The computer system of claim 15, wherein, The two or more categories of motion categories include, in addition to the reactive motion and non-reactive motion categories, a predictive motion category in which motion of the robot is repeatedly planned over a specified time horizon, and a manual mode motion category in which motion of the robot is manually defined in terms of a plurality of waypoints, and a trajectory along the waypoints is generated using an interpolation algorithm.

20. The computer system of claim 15, wherein, An online motion planner instantiated by the computer system uses the features of motion subtasks to determine respective trajectory constraints, including determining whether dynamic trajectory modification is allowed at a subtask granularity.

21. The computer system of claim 13, wherein, The input task specification involves two or more robots operating in a shared workspace, and for a particular one of the robots that performs reactive motion subtasks, the online motion planning includes dynamic trajectory modification for collision avoidance with the remaining one or more robots, and wherein for a particular robot that performs non-reactive motion subtasks, the processing circuitry is configured to perform collision avoidance by coordinated volume reservation such that the remaining one or more robots do not interfere with the robot trajectory during performance of the non-reactive motion subtasks by the particular robot.

22. The computer system of claim 13, wherein, The input task specification involves two or more robots operating in a shared workspace, and wherein the processing circuitry is configured to identify related tasks that must be performed by a same robot of the two or more robots, and to assign subtasks that make up the related tasks to the same robot.

23. The computer system of claim 22, wherein, The input specification indicates one or more subtask packs, each subtask pack containing two or more subtasks in a corresponding task that must be performed by a robot assigned the corresponding task without interference by one or more other robots, and wherein the processing circuitry is further configured to coordinate operation of the two or more robots to ensure that each subtask pack is performed without interruption.

24. The computer system of claim 13, wherein, One or more tasks or sub-tasks are pre-assigned for execution by a particular robot of two or more robots that are collaborating in a shared workspace, and wherein one or more tasks or sub-tasks are unassigned, and for the unassigned tasks or sub-tasks, the processing circuit is further configured to determine a robot assignment.

25. A method of online task planning and robot motion control, the method comprising: processing an input task specification pertaining to two or more robots operating in a shared workspace, the input task specification indicating a plurality of tasks to be executed; identifying related tasks that must be executed by a same robot of the two or more robots; and assigning sub-tasks that make up the related tasks to the same robot.

26. The method of claim 25, wherein, the input specification indicates one or more sub-task packs, each sub-task pack containing two or more sub-tasks of a corresponding task that must be executed by a robot assigned the corresponding task without interference by one or more other robots, and wherein the method further comprises coordinating operation of the two or more robots to ensure that each sub-task pack is executed without interruption.

Images