Hierarchical planning of autonomous machines for various tasks

Abstract

The invention relates to hierarchical planning of autonomous machines for various tasks. A computer-implemented method is provided. The method comprises: receiving from a user a task and associated task parameter values for a machine; dividing the task into one or more base movement sub-tasks and one or more material removal sub-tasks using a high-level task planner module; generating a set of one or more waypoints between successive pairs of route points to guide the machine between the route points using a base movement sub-task planner; determining motion primitives and associated parameter values for each sub-task area using a material removal sub-task planner; using the motion primitives, associated parameters, machine parameters and perception module data as inputs to a joint movement primitives module to generate joint trajectories and using the waypoints, machine parameters and perception module data as inputs to a base movement primitives module to generate base trajectories; providing the joint trajectories and base trajectories to a joint controller module and a base controller module respectively to generate controller commands to control the machine's devices.

Description

[0001] Cross-reference to related applications

[0002] This patent application relates to and claims priority to U.S. Patent Application No. 63 / 233,194 (File No. 28888-2535P), filed August 13, 2021, entitled “HIERARCHICAL PLANNING FOR AUTONOMOUS EXCAVATOR ON MATERIAL LOADING TASKS”, which lists Liyang Wang, Zhixian Ye, and Liangjun Zhang as inventors, and is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] This disclosure generally relates to systems and methods for computer learning, which can provide improved computer performance, features, and uses. More particularly, this disclosure relates to autonomous machinery. Background Technology

[0004] The past few years have witnessed a dramatic increase in research and development related to autonomous machinery. Autonomous machinery has been studied for a wide range of applications, including autonomous cars, autonomous trucks, autonomous robots, autonomous drones, and autonomous construction vehicles. The rationale for researching and developing autonomous vehicles varies depending on the application. For example, self-driving cars can assist commuters and drivers. Self-driving trucks reduce costs associated with transporting goods. Autonomous heavy equipment is useful for reducing costs and minimizing the need for human intervention in hazardous conditions.

[0005] One such vehicle that has seen autonomous research is the excavator. Autonomous excavators can be widely used in many different applications, such as moving soil, rock, or other materials. As one of the most versatile heavy equipment, it has a vast market worldwide. However, skilled excavator operators require extensive training. Furthermore, many work sites are located in remote areas with poor infrastructure. Moreover, hazardous working environments can affect the health and safety of on-site operators.

[0006] Autonomous excavators offer advantages in addressing these challenges and improving overall working conditions. In recent years, researchers from academia and industry have increasingly devoted themselves to the development of autonomous excavators.

[0007] A major challenge in developing autonomous excavators is designing an overall planning architecture that is suitable for a wide range of real-world tasks, such as material loading, trench excavation, truck loading, etc. In most of the literature, authors focus on developing key individual components for autonomous excavators, including high-level task planner design, excavation trajectory generation algorithms, and control modules. While these advances are made, very few studies focus on the planning architecture that connects the high-level task planner, subtask planners, and autonomous excavator motion planning.

[0008] Thus, there is a need for systems and methods that provide hierarchical planning for autonomous machines. SUMMARY

[0009] According to a first aspect of embodiments of the present disclosure, there is provided a computer-implemented method comprising: receiving, from a user, a task for an excavator and associated task parameter values; given the task, the associated task parameter values, at least some of the perception-related data received from a perception module, dividing the task into one or more base movement subtasks and one or more material removal subtasks using a high-level task planner module comprising one or more rule-based models, wherein the one or more base movement subtasks comprise a sequence of waypoints and the one or more material removal subtasks comprise a sequence of subtask regions and a goal for each subtask region; given the sequence of waypoints and state information, wherein at least some of the state information is obtained from the perception module, generating a set of one or more waypoints between consecutive pairs of waypoints to guide the excavator between the waypoints using a base movement subtask planner comprising one or more models; given the sequence of subtask regions, associated goals, and state information, wherein at least some of the state information is obtained from the perception module, determining motion primitives and associated parameter values for each subtask region to achieve the goal for that subtask region using a material removal subtask planner comprising one or more planner models; using the motion primitives, associated parameters, excavator parameters, and perception module data as inputs to a joint movement primitive module to generate joint trajectories and using the waypoints, excavator parameters, and perception module data as inputs to a base movement primitive module to generate base trajectories; and providing the joint trajectories and base trajectories to joint controller module(s) and base controller module(s), respectively, to generate controller commands to control the devices of the excavator.

[0010] In some embodiments, the perception module comprises or receives data from one or more sensors comprising: a camera for image data; a LiDAR sensor for generating point cloud data; a real-time motion sensor for localization; and one or more inclinometers.

[0011] In some embodiments, the computer-implemented method further comprises determining a local base area using a rectangular definition or a sector-based definition and a set of parameter values including a block center swing angle, a block angle range or width, a near end distance, and a far end distance.

[0012] In some embodiments, the sub-task areas are determined by the high-level planner by performing steps comprising, given the local task area for the excavator and a block on which the task is to be performed, determining a number of local task areas that cover at least the block.

[0013] In some embodiments, given the sequence of sub-task areas, associated goals, and state information obtained from the perception module, the step of determining motion primitives and associated parameter values for each sub-task area to achieve the goal of that sub-task area using a material removal sub-task planner comprising one or more models comprises, for a sub-task, for a sub-task area, implementing a set of one or more action planners at the sub-task area according to the material removal sub-task planner model associated with the sub-task until the sub-task goal is achieved, and in response to the sub-task goal for the sub-task area being achieved and in response to the sub-task area not being the last sub-task area in the sequence of sub-task areas, processing the next sub-task area in the sequence of sub-task areas.

[0014] In some embodiments, the material removal sub-task planner comprises: a point of attack (POA) planner; a digging motion planner; and a dumping planner.

[0015] In some embodiments, parameters utilized by the digging motion planner include:

[0016] x b ,y b ,z b , which represent a point of attack (POA) location in the base frame;

[0017] p d , which represents a penetration depth;

[0018] p l , which represents a penetration length;

[0019] d l , which represents a drag length, and

[0020] δ, which represents an overall digging motion tilt angle relative to the horizontal plane.

[0021] In some embodiments, the state information for the base movement sub-task planner includes one or more of an excavator joint state of the excavator, an excavator base pose, and an environmental state of the work area; and the state information for the base movement sub-task planner includes an excavator base pose and an environmental state of the work area.

[0022] In some embodiments, the end condition of a sub-task objective can be defined according to different levels of strictness requirements.

[0023] According to a second aspect of embodiments of the present disclosure, a system is provided, comprising: one or more processors; and one or more non-transitory computer-readable media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, cause performance of the computer-implemented method according to the first aspect of embodiments of the present disclosure.

[0024] According to a third aspect of embodiments of the present disclosure, a non-transitory computer-readable medium is provided, comprising one or more sequences of instructions which, when executed by at least one processor, cause performance of the computer-implemented method according to the first aspect of embodiments of the present disclosure.

[0025] According to a fourth aspect of embodiments of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the computer-implemented method according to the first aspect of embodiments of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0026] Reference will be made to embodiments of the present disclosure, examples of which can be illustrated in the accompanying drawings. These drawings are intended to be illustrative and not limiting of the scope of the present disclosure. While the present disclosure is generally described in the context of these embodiments, it should be understood that it is not intended to limit the scope of the present disclosure to these particular embodiments. Items in the figures can not be to scale.

[0027] Figure 1 A planner architecture according to embodiments of the present disclosure is depicted.

[0028] Figure 2 An example method of using a planner system according to embodiments of the present disclosure is depicted.

[0029] Figure 3 A sensor assembly of an excavator according to embodiments of the present disclosure is depicted, which can be provided to a planner via one or more perception modules.

[0030] Figure 4 An example of a local task area definition according to embodiments of the present disclosure is depicted.

[0031] Figure 5 An example of a trenching task high-level plan is depicted in accordance with an embodiment of the present disclosure.

[0032] Figure 6 An example of a pile removal task high-level plan is depicted in accordance with an embodiment of the present disclosure.

[0033] Figure 7 An example of a large-scale pile removal subtask area decomposition is depicted in accordance with an embodiment of the present disclosure.

[0034] Figure 8 An example of a large-scale pile removal subtask base movement plan is depicted in accordance with an embodiment of the present disclosure.

[0035] Figure 9 An alternative method for developing plan information is depicted in accordance with an embodiment of the present disclosure.

[0036] Figure 10 A method of determining points of attack (POAs) is depicted in accordance with an embodiment of the present disclosure.

[0037] Figure 11 An example excavation motion parameter definition is depicted in accordance with an embodiment of the present disclosure.

[0038] Figure 12 An example wheelie model for a excavator is depicted in graphical fashion in accordance with an embodiment of the present disclosure.

[0039] Figure 13 A trenching experiment is depicted in accordance with an embodiment of the present disclosure.

[0040] Figure 14 A pile removal experiment setup is depicted in accordance with an embodiment of the present disclosure.

[0041] Figure 15 A pile removal experiment process is depicted in accordance with an embodiment of the present disclosure.

[0042] Figure 16A And 16B A large-scale pile removal experiment simulation is depicted in accordance with an embodiment of the present disclosure.

[0043] Figure 17 A simplified block diagram of a computing device / information handling system is depicted in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] In the following description, for the purposes of explanation, specific details are set forth in order to provide an understanding of the present disclosure. It will be apparent, however, to one skilled in the art that the present disclosure can be practiced without

[0045] The components or modules shown in the figures are illustrative of exemplary embodiments of the present disclosure and are intended to avoid obscuring the present disclosure. It should be understood that throughout the discussion, components can be described as separate functional units that can include sub-units, but those skilled in the art will recognize that various components or portions thereof can be divided into separate components or can be integrated together, including, for example, in a single system or component. It should be noted that the functions or operations discussed herein can be implemented as components. The components can be implemented in software, hardware, or a combination thereof.

[0046] Moreover, connections between components or systems in the figures are not intended to be limited to direct connections. Rather, data between these components can be modified, reformatted, or otherwise changed by intermediary components. Also, additional or fewer connections can be used. It should also be noted that the terms “coupled,” “connected,” “communicatively coupled,” “joined,” “interface,” or any variant thereof, should be understood as including direct connections between components, indirect connections between components through one or more intermediary components, and wireless connections. It should also be noted that any communication, such as a signal, response, reply, acknowledgement, message, query, etc., can include one or more exchanges of information.

[0047] References in the specification to “one or more embodiments,” “preferred embodiments,” “an embodiment,” “embodiment,” etc., mean that a particular feature, structure, characteristic, or function described in connection with the embodiment is included in at least one embodiment of the disclosure and can be included in more than one embodiment. Also, the appearance of the phrases “in one or more embodiments” or “in at least one embodiment” at various places in the specification are not necessarily all referring to the same one or more embodiments.

[0048] The use of certain terms in various places in the specification is for illustration and should not be construed as limiting. A service, function, or resource is not limited to a single service, function, or resource; use of these terms can refer to a group of related services, functions, or resources, which can be distributed or aggregated. The terms“comprise,”“comprising,” or any variation thereof should be understood to be open terms and any listed items are examples of items that can be included, but not limited to, the listed items. A“layer” can include one or more operations. The words“optimal,”“optimize,”“optimization,” and the like refer to an improvement in a result or process, and do not require that the specified result or process has been reached a“best” or peak state. Use of memory, databases, repositories, tables, hardware, caches, and the like can be used herein to refer to one or more system components that can input or otherwise record information.

[0049] In one or more embodiments, the stopping condition can include: (1) a set number of iterations have been performed; (2) a certain amount of processing time has been reached; (3) convergence (e.g., a difference between successive iterations is less than a first threshold); (4) divergence (e.g., a performance deterioration); (5) an acceptable result has been reached; and (6) all data has been processed.

[0050] Those skilled in the art will recognize that: (1) certain steps can be optionally performed; (2) steps can not be limited to the specific order set forth herein; (3) certain steps can be performed in different orders; and (4) certain steps can be performed concurrently.

[0051] Any headings provided herein are for organizational purposes only and shall not be used to construe the scope of the description or claims. Each reference / document mentioned in this patent file is incorporated by reference in its entirety.

[0052] It should be noted that any experiments and results provided herein are for illustrative purposes only and were conducted under specific conditions using one or more particular embodiments; thus, the experiments and results should not be used to limit the scope of the disclosure of this patent document.

[0053] It should also be noted that while the embodiments described herein can be in the context of a shovel or heavy equipment, aspects of the disclosure are not so limited. Thus, aspects of the disclosure can be applied to or adapted for use with other machinery or objects and used in other contexts.

[0054] Autonomous excavators have seen rapid development in recent years due to labor shortages and dangerous working conditions for operating excavators. Presented herein are embodiments of a novel hierarchical planning system for autonomous machines such as excavators. In one or more embodiments, the overall planning system includes a high-level task planner for task partitioning and base movement planning, and a general subtask planner with motion primitives that include both arm movement and base movement in the case of an excavator. Using embodiments of the system architecture, experiments were performed on ditching and pile removal tasks in real-world and large-scale material loading tasks in simulated environments. Results show that the system architecture embodiments and planner method embodiments generate effective task and motion plans that perform well in autonomous excavation.

[0055] A. GENERAL OVERVIEW

[0056] As previously mentioned, a major challenge in developing autonomous excavators is designing an overall planning architecture that is suitable for a wide range of real-world tasks, such as material loading, trenching, and truck loading. In most of the literature, authors focus on developing key individual components for autonomous excavators, including high-level task planner design, excavation trajectory generation algorithms, and control modules.

[0057] For the design of advanced task planners, some research has covered autonomous excavation or base movement route planning in task partitioning. Seo et al. (Jongwon Seo, Seungsoo Lee, Jeonghwan Kim, and Sung-Keun Kim. Task Planner Design for an Automated Excavation System. Automation in Construction, 20(7):954-966, 2011, which is incorporated herein by reference in its entirety) designed a task planner to create optimized work commands for an automated excavator. Some important components of the excavation task planner can be modules for work area segmentation and excavation path generation. Kim et al., 2012 (Sung-Keun Kim, Jongwon Seo, and Jeffrey S Russell. Intelligent Navigation Strategies for an Automated Earthwork System. Automation in Construction, 21:132-147, 2012, which is incorporated herein by reference in its entirety) proposed intelligent navigation strategies, which can be useful for an automated earthwork system to effectively perform excavation. Kim et al., 2020 (Jeonghwan Kim, Dong-eun Lee, and Jongwon Seo. Task Planning Strategy and Path Similarity Analysis for an Autonomous Excavator. Automation in Construction, 112:103108, 2020, which is incorporated herein by reference in its entirety) allegedly proposed a complete coverage path planning (CCPP) algorithm by considering the characteristics of earthwork. Also, a path similarity analysis method was proposed to test the effectiveness of the CCPP algorithm.

[0058] Other studies have contributed to the development of excavator arm motion generation and controller design. Jud et al. (Dominic Jud, Gabriel Hottiger, Philipp Leemann, and Marco Hutter. Planning and Control for Autonomous Excavation. IEEE Robotics and Automation Letters, 2(4):2151-2158, 2017, which is incorporated herein by reference in its entirety) proposed a planning and control method for autonomous excavation, where a single digging cycle is defined with respect to the end-effector force-torque trajectory. Its advantage over the position trajectory of the bucket motion is that it can overcome the limitations of the soil interaction forces, which dominate and are difficult to predict or estimate. Son et al. (Bukun Son, Chang U Kim, Changmuk Kim, and Dongjun Lee. Expert-emulating excavation trajectory planning for autonomous robotic industrial excavator. 2020 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2656-2662. IEEE, 2020, which is incorporated herein by reference in its entirety) proposed an excavation trajectory planning framework for an industrial autonomous robotic excavator that emulates human operator strategies to optimize digging for different types of soil while maintaining robustness and safety in practice.

[0059] While these advances have been made, there has been little research focusing on a planning architecture that connects high-level task planners, sub-task planners, and motion planning for autonomous machines such as excavators. Elezaby et al. (Ahmed Adel Elezaby, Mohamed Abdelaziz, and Sabri Cetinkunt. Operator model for construction equipment. 2008 IEEE / ASME International Conference on Mechtronic and Embedded Systems and Applications, pages 582-585. IEEE, 2008, which is incorporated herein by reference in its entirety) proposed an event-based finite state machine strategy model that selects different motion primitives for a wheel loader task cycle. While it can provide a promising system architecture design, the motion primitive definitions are specifically based on a wheel loader and do not generalize well (i.e., are not well suited for an excavator). Furthermore, there are no related sub-task definitions in the published work where the architecture of the task planner is generic. Most of the research mentioned above focuses on either high-level task planning or the design of motion primitives. Overall, there is no top-down planning system architecture design for autonomous machines such as excavators.

[0060] To address these limitations, in the present patent document, embodiments of a novel hierarchical planning architecture for autonomous machine systems are presented. It should be noted that, for purposes of illustration, applications to excavators are presented, but aspects of the embodiments can be applied generally to other autonomous machines.

[0061] In one or more embodiments, the problem of a large number of excavating task types is addressed first, and an embodiment of a high-level task planner for excavating task partitioning and base movement planning is designed. Then, two sub-task planners with good portability and generality are abstracted, including a material removal sub-task planner and a base movement sub-task planner. Next, embodiments encapsulate motion primitives and provide them with appropriate controllers for joint control and base control. Finally, the embodiments are implemented and further validated in real-world experiments with a series of excavators and dynamic simulation environments. The results show that the system architecture embodiments and planner method embodiments are able to generate effective work plans that can be fed into autonomous excavating systems in a general manner.

[0062] This patent document is organized as follows. In Section B, an embodiment of a novel overall planner system architecture is presented. In Section C, embodiments of a high-level task planner are presented in terms of excavation task partitioning and base movement planning. First, a definition of a local task region is given. Then, three common tasks and their planning solutions are detailed. Section D focuses on the implementation of sub-task planners, including arm movement and base movement. There, their associated motion primitives are also covered. In Section E, both real-world experimental results and simulation results are presented. In Section F, some conclusions or observations are drawn. In Section G, some computing system embodiments are presented.

[0063] B. Planner Architecture Embodiments

[0064] Presented herein are embodiments of a hierarchical planner architecture for general applications. Figure 1 An example overall architecture of a planner 110 for an excavator according to embodiments of the present disclosure is depicted. Note that the main components are drawn within dashed boxes. Figure 2 An example method of using a planner system 100 according to embodiments of the present disclosure is depicted.

[0065] In the depicted architecture, there are two levels of task planners plus one level of motion primitives. From top to bottom, they are the high-level task planner layer or module 120, the sub-task planner layer(s) (e.g., material removal sub-task region planner 135 and base movement sub-task planner 140), and the motion primitive layer(s) (e.g., joint controller 175 and base controller 180). In most scenarios, an excavator alternates between the motion of its arm to perform excavation work and moving the base to a desired location. Based on this feature, in one or more embodiments, the planner can separate material removal and base movement into two planning pipelines.

[0066] To help determine the planning and primitives in real-time, the various layers can receive inputs from one or more perception modules 185, which can include raw sensor data or processed sensor data. In one or more embodiments, one or more rule-based and / or machine learning-based models can be used to pre-process / process the sensor data before it is provided to the components of the planner 110. Just to illustrate, Figure 3Some examples of sensors that can be used in conjunction with the planner 110 for a shovel are depicted. As depicted, an autonomous machine, such as a shovel 305, can be equipped with sensors that can include one or more cameras 310, a LiDAR sensor 315 (which provides LiDAR point cloud data), a real-time kinematic (RTK) 320 (which provides high accuracy / high precision differential positioning, which can also use signals from a global positioning system (GPS) / global navigation satellite system (GNSS)), one or more pressure sensors 325, and one or more tilt sensors 330.

[0067] In one or more embodiments, there are multiple user-defined tasks (or user-definable tasks) that a user can select via the user interface 105. By allowing the user to define tasks, embodiments allow for highly customized advanced task planners.

[0068] In one or more embodiments, two types of subtasks are defined, although it should be noted that there can be more or fewer subtasks for different machines or different applications. In one or more embodiments, the advanced task planner embodiment closest to the user interface 105 takes a user task as input (205), and then divides the task into these types of subtasks, i.e., material removal subtasks and base movement subtasks. In one or more embodiments, the user can also input (or it can be pre-loaded) one or more associated task parameter values, such as the size and location of a trench to be dug or the location and depth of a pile of dirt to be removed. In one or more embodiments, the planner embodiment acts to determine which locations the shovel should move to and which areas of material the shovel should dig out. In one or more embodiments, for each digging task, the advanced planner determines (210) a sequence of shovel waypoints, and at each shovel waypoint, determines (210) a sequence of dig areas (subtask areas). Once all of the subtasks have been completed in the generated sequence, the user-assigned task can be considered complete. In one or more embodiments, for an arm movement task, the advanced task planner embodiment assumes that the shovel base is stationary, and it computes the subtask areas (material areas) that the shovel’s arm can reach at that fixed location. For a base movement task, the planner embodiment can require the shovel to reach a desired waypoint at a given heading angle.

[0069] In one or more embodiments, given the results of this high-level task planner and its subtasks, one or more subtask planners provide solutions to achieve short-term goals. With waypoint planning, it guides the shovel to reach the target route waypoints. At the same time, it helps the shovel to efficiently and accurately complete sub-area excavation at a fixed base location. For simplicity, embodiments of the material removal subtask planner can be referred to as MRSP, and embodiments of the base movement subtask planner can be referred to as BMSP. In one or more embodiments, the MRSP receives a subtask area from the high-level task planner and decides (220) motion primitives and calculates related parameters. In one or more embodiments, the BMSP receives shovel route waypoints from the high-level task planner and outputs (215) waypoints and related constraints between two route waypoints.

[0070] Motion primitives can be abstracted as a result of the subtask planner, rather than direct motion control, primarily because there are many repetitive, modular operational actions in a mining operation, which simplifies the complexity of the subtask planner. Currently, embodiments include a digging motion primitive, a dumping motion primitive, and a base movement motion primitive— but it should be noted that different and / or additional primitives can exist, especially for different types of equipment. In addition to the task planning results, these primitives can use several external mining parameters (such as from the perception module 185) and parameter values associated with the shovel to generate (225) feasible trajectories (e.g., joint trajectories 165 and base trajectories 170). Examples of mining parameters can include the length of each shovel link, the bucket volume, the material removal subtask area size, the desired digging depth, and the bucket tooth position constraints. In one or more embodiments, the generated trajectories can be the output of the entire planning system, but inputs for the controllers (e.g., joint controllers 175 and base controllers 180) to follow. The controllers can use the trajectories to generate controller commands to control the equipment of the shovel (e.g., motors, hydraulics, etc.).

[0071] In summary, embodiments of the high-level task planner compute global docking route waypoints, local task areas, and local task goals for each docking station. And, in one or more embodiments, the subtask planners MRSP and BMSP handle these subtasks by invoking encapsulated motion modules, which are motion primitives designed herein. In one or more embodiments, the motion primitives ultimately generate trajectories for the controllers.

[0072] As previously mentioned, while some embodiments are provided in the context of excavators, it should be noted that the general architecture and methods can be applied to other contexts. For example, in one or more embodiments, a general planning method can include receiving as input a task to be performed by an autonomous machine and a set of parameters related to the task. A high-level planner can be used to generate a set of subtask configurations for the task, the high-level planner including one or more models for the task and using at least one parameter from the set of parameters related to the task and one or more parameters related to the autonomous machine to generate the set of subtask configurations, where the set of subtask configurations has an order and where each subtask configuration can have an associated type. In one or more embodiments, for each subtask configuration, a primitive parameter value is generated using a subtask planner corresponding to the associated type for that subtask configuration, where the subtask planner uses the subtask configuration from the high-level planner, state information (which can be obtained from one or more sensors or perception modules), and one or more subtask models. In one or more embodiments, for a primitive, control reference data can be generated using a primitive model, one or more primitive parameters from the subtask planner corresponding to the primitive, and one or more sensor inputs corresponding to the primitive. Finally, in one or more embodiments, the control reference data can be used to control one or more components of the autonomous machine.

[0073] C. High-level task planner embodiments

[0074] In one or more embodiments, the high-level task planner takes as input a user-defined task and partitions the task to subtask planners. Thus, in this section, this partitioning is covered in terms of local task regions. Also, to better understand the use of the high-level task planner embodiments, three common tasks in construction are developed, which are the trenching task, the pile removal task, and the large-scale pile removal task, and are provided only as illustrations.

[0075] Here, two different types of excavation tasks are defined based on the moving direction and the requirement of material residue: forward moving task and backward moving task. For the forward moving task, the target material is above the ground surface, which means the target height of material removal is the same as the ground surface height. After the material in front of the excavator is removed, the excavator can move forward to continue. On the other hand, for the backward moving task, the target height of material is below the ground surface, and after the closer material is removed, the excavator usually moves backward to continue. Thus, the trenching task is a backward moving task, while the pile removal and large-scale pile removal are forward moving tasks.

[0076] For a given excavator kinematic model, the local safety reachable range is predetermined. The maximum reachable distance can be denoted as r maxThe minimum reachable distance can be expressed as r min To fully cover the global task area, in one or more embodiments, the local task area overlap area, expressed as r o .

[0077] While specific example tasks are provided below with respect to the excavator example, it can be seen that based on the task and the parameter values associated with the task (e.g., the dimensions of the trench, the parameter values of the excavator defining its local area, etc.), the high-level planner generates a task model using which the task is divided into one or more base movement subtasks and one or more material removal subtasks, where the one or more base movement subtasks include a sequence of route points and the one or more material removal subtasks include a sequence of subtask areas and a goal (e.g., dig to a depth d).

[0078] 1. Local Task Area Embodiments

[0079] Before introducing three specified common tasks, this section provides the definition of the local task area for the MRSP. The task area of the MRSP can be defined locally in the base coordinate frame of the excavator. The local task area can be defined using four parameters: 1) block center swing angle a; 2) block angular range or width b; 3) proximal distance r min ; and 4) distal distance r max Two types of local task areas are provided. The high-level task planner embodiment can choose either a sector block or a rectangular block as the local task area. Thus, the parameter b has two definitions. The angular range definition corresponds to the sector block, while the width definition corresponds to the rectangular block. Figure 4 Two local task area definitions using the parameters are depicted in accordance with embodiments of the present disclosure.

[0080] With respect to the local task area, two points need to be noted. First, the selection of r min and r max should be based on the excavator dimensions to ensure that the bucket can reach the local task area. Second, during the dig-and-dump cycle, the excavator base pose will have a slight change compared to the initial base pose. If the local task area is not updated, the global task area in the map coordinate frame will change. To address this issue, in one or more embodiments, the MRSP initially records the task area center in the map coordinate frame. Before each dig-dump cycle, the MRSP can check the current excavator base pose in the map coordinate frame using data from the perception module and can adjust the local task area parameters to ensure that the local task area does not shift in the global task area.

[0081] 2. Trenching Task Embodiments

[0082] Let the desired trench length be l and width be w. The initial route point of the excavator is along the trench direction, located r max meters away from the trench start point. After each subtask is completed, the excavator base moves backward to perform the next subtask. The backward distance can be defined as d = r max min o The total number of material removal subtasks performed n is To satisfy the trench length requirement, the proximal boundary distance of the last subtask can be defined as r min + n - d - l.

[0083] In general, the desired trench width w is relatively narrow. A rectangle is chosen as the subtask area shape. Figure 5 A high-level plan for a trenching task according to an embodiment of the disclosure is shown. Figure 5 The top portion of the figure represents the high-level plan for the trenching task, and the bottom portion represents the legend.

[0084] 3. Pile removal task embodiment

[0085] Pile removal tasks and trenching tasks share similar definitions and have the same notation to describe the planner. The difference is that a trenching task is a backward moving task, while a pile removal task is a forward moving task. Similar to a trenching task, let the desired pile to be removed have a length l and a width w. The initial route point of the excavator is along the pile length direction, located r max meters away from the pile task area start point. In one or more embodiments, after each subtask is completed, the excavator base moves forward to perform the next subtask. The backward distance d = r max min o The total number of material removal subtasks performed n is To satisfy the pile length requirement, the distal boundary distance of the last subtask is r min + l - (n - 1) - d.

[0086] The pile task area has a relatively wide width w. A sector is chosen as the subtask area shape. If a rectangular task area is applied, the excavator can not be able to reach the upper left corner or the upper right corner. Figure 6 A high-level plan for a pile removal task according to an embodiment of the disclosure is shown. Figure 6 The left portion of the figure represents the high-level plan for the pile removal task, and the right portion represents the legend.

[0087] 4. Large scale pile removal task embodiment

[0088] ​​​​Task area definition for a large-scale pile removal task is similar to that of a pile removal task. However, the task area can be larger than that of a pile removal task, so the excavator route path cannot be a straight line to complete the task. In one or more embodiments, the maximum width of a pile removal task can be denoted as q. If the width w of the task area is larger than q, then the task is a large-scale case. Otherwise, the task is considered a pile removal task, which is described in Section C.3. In one or more embodiments, the following formula can be used:

[0089]

[0090] Assuming the task area width w is larger than q, in one or more embodiments, the task area can be divided into columns. For each column, a 180° sector can be used to cover the area. Considering the limitation of LiDAR horizontal field of view in practical applications, each 180° sector can be further decomposed into smaller angle range sectors. In one or more embodiments, sectors close to each other have overlapping areas to ensure complete coverage. In some applications where the material can be thick, multiple layers can be used for subtask partitioning.

[0091] Figure 7 An example of large-scale pile removal subtask area decomposition is depicted in accordance with an embodiment of the present disclosure. At each route point, a 180° sector 705 can be decomposed into 6 parts, and 4 layers 710 can be used to cover the entire material - but it should be noted that the number of parts and layers can be different. In one or more embodiments, the material removal subtask sequence for this route point is set to complete the first layer, and the sub-areas from part 1 to 6. Then, move to the next layer until all are completed.

[0092] In one or more embodiments, between two columns, a semi-circular arc can be used to design the connection path. Since the excavator work area usually has soft ground, using a small turning radius will cause the excavator to sink into the ground. Figure 8 An example excavator base path planning for a large-scale pile removal task is shown in accordance with an embodiment of the present disclosure. The initial route point 805 is located in the lower right corner. The excavator moves in a straight line to the top to complete the first column. Then, the excavator moves to the route point marked by the hollow circle 810, which represents a route point that is not associated with a material removal subtask. After that, the excavator makes a U-turn along the semi-circular arc 815 to the second column. Finally, the excavator moves in a straight line down to the end point 820 to complete the second column.

[0093] D. Subtask Planner and Motion Primitive Embodiments

[0094] As mentioned in Section B, embodiments can include two or more subtask planners. In the illustrated embodiment, embodiments of a material removal subtask planner plan to excavate material to reach a desired amount and dump them to a given location, while embodiments of a base movement subtask planner navigate the excavator to reach a desired location and heading angle. Thus, in the following subsections, design details of these subtask planners will be presented. After that, motion primitives upon which these subtask planners are based are presented.

[0095] 1. Material Removal Subtask Planner Embodiments

[0096] In one or more embodiments, the input to the MRSP can include two parts: (1) a subtask configuration from the high-level task planner; and (2) state information for online / real-time planning. The subtask configuration can include a local task area, a target material height, and a dumping pattern. Excavator states (such as joint states and base pose of the excavator) and environmental states (e.g., LiDAR point cloud) can be used as state representations. In one or more embodiments, the output of the MRSP is values for motion parameters of motion primitives.

[0097] Method 1 illustrates an embodiment of the process of a material removal subtask module. In one or more embodiments, the MRSP first loads a subtask configuration, which can include subtask area information, a subtask target, and a subtask pattern. Then, the MRSP plans using a corresponding planning model, in this case a dig-and-dump cycle, until the material height in the local task area reaches the target height. The dumping location can be decided according to the dumping pattern.

[0098] Method 1: Material Removal Subtask

[0099]

[0100] Figure 9 An alternative method for developing a MRSP embodiment for a subtask planning information is depicted in accordance with an embodiment of the present disclosure. In one or more embodiments, the MRSP receives (905) a subtask configuration, which can include a subtask area and a subtask target, but can include additional information, which can be user specified or subtask specific. The MRSP can then execute a set of one or more planning actions using a material removal subtask planner model. For example, the model can include lines 6-12 in Method 1. The set of one or more planning actions can be executed until the subtask target is reached (915). In one or more embodiments, if the target is reached, then the current subtask is considered complete and proceeds to the next subtask, which can be of the same type or a different type, according to commands from the high-level planner.

[0101] a) Attack Point Planning Embodiments

[0102] In one or more embodiments, the MRSP finds the point of attack (POA) based on the LiDAR point cloud within the local task zone. In this approach, embodiments can determine the POA based on the highest point with an offset in front. Figure 10 A method of determining the POA according to embodiments of the disclosure is shown.

[0103] In one or more embodiments, the MRSP first finds the highest point and average height of the local task zone. Then, starting from the base link origin, a straight line connecting the highest point is determined. As shown in the top view, a constant distance offset can be added to this line and the (x, y) of the POA is determined. As shown in the side view, the z of the POA can be set equal to the average height.

[0104] b) Excavation motion parameter embodiments

[0105] In one or more embodiments, the excavation motion planner first obtains the POA for the bucket tooth tip to contact. Then, the excavation motion can be divided into three steps. First, the bucket penetrates into the material with a certain depth and distance closer to the base origin. Second, the bucket drags back to the base origin to collect the material into the bucket. Finally, the bucket closes until the bucket joint is flattened with the tooth plane, which prevents material leakage in subsequent motions. Based on the above discussion, the excavation motion can be defined by 7 parameters. In one or more embodiments, the excavation motion parameters can be represented as D = [x b ,y b ,z b ,p d ,p l ,d l ,δ], where x b ,y b ,z b represents the POA in the base frame, p d represents the penetration depth, p l represents the penetration length, d l represents the drag length, and δ represents the entire excavation motion tilt angle relative to the horizontal plane. Figure 11 An example excavation motion parameter definition according to embodiments of the disclosure is shown.

[0106] In one or more embodiments, these parameters can be determined according to the terrain shape that can be obtained from the perception module to optimize the collected material volume to match the bucket volume. The final excavation motion trajectory can be further adjusted based on the excavator base pose to handle the case where the roll angle and pitch angle of the base are not zero.

[0107] c) Dumping mode embodiments

[0108] In one or more embodiments, for generality, the MRSP provides three different dumping modes. The first mode uses a fixed dump point in the base frame of the excavator. The MRSP dumps to the same point for each cycle of excavation-dump. The second mode uses a floating dump point in the base frame. This floating dump point is passed as output for the high-level mission planner for the MRSP, which can be used in dynamic work environments. The third mode is the truck dump mode. In this mode, the MRSP will subscribe to the truck pose through the perception module and calculate the appropriate dump point on its own. In one or more embodiments, to have the truck load material evenly, the MRSP finds a dump point to dump at where the material height is low inside the truck bed.

[0109] d) End Condition Embodiments

[0110] For the end condition, the MRSP can be designed to have three stop conditions to favor different task excavation strictness requirements, i.e., “strict,” “regular,” and “loose.”

[0111] The target height can be denoted as g, the current highest point in the sub-task zone can be denoted as h, and the current average height of the sub-task zone can be denoted as a. For the “strict” mode, the completion trigger condition can be h < g. For the “regular” mode, the trigger condition can be a < g and h < g + b, where b is a positive value denoting a height margin for the highest point. For the “loose” mode, the completion trigger condition can be a < g.

[0112] 2. Base Movement Sub-Task Planner Embodiments

[0113] In one or more embodiments, the base movement sub-task planner navigates the excavator movement to the target pose assigned by the high-level mission planner. Similar to the MRSP, the inputs to the base movement sub-task planner include both the target waypoints from the high-level mission planner and the state information for online planning. The state information can include the excavator base pose and LiDAR point cloud, while the output of the BMSP can be the waypoints between the consecutive waypoints (i.e., the current waypoint and the next waypoint) and the associated walking constraints.

[0114] The BMSP embodiments are developed based on the 2D (two-dimensional) assumption. The 2D target pose can be denoted as B = [x m ,y m , Θ], where (x m ,y m ) is the target position in the map frame and Θ is the target heading angle in the map frame. However, it should be noted that the BMSP can be easily adapted or applied to 3D.

[0115] As for global path planning, path planners such as the Hybrid A* path planner method (Karl Kurzer. Path planning in unstructured environments: A Real-Time Hybrid A* Implementation For Fast And Deterministic Path Generation For The KTH Research Concept Vehicle. Master’s thesis, KTH, Integrated Transport Research Lab, ITRL, 2016, which is incorporated herein by reference in its entirety) can be applied in the BMSP. With the occupancy map generated from the LiDAR point cloud, the current base pose, and the target base pose, a smooth collision-free path of waypoints can be generated. In one or more embodiments, a unicycle model such as in Alessandro De Luca, Giuseppe Oriolo, and Marilena Vendittelli. Control Of Wheeled Mobile Robots: An Experimental Overview. Ramsete, pp. 181-226, 2001, which is incorporated herein by reference in its entirety, can be used as the excavator dynamics model for the MPC (Model Predictive Control) controller. Figure 12 An example unicycle model for an excavator according to embodiments of the disclosure is graphically depicted.

[0116] 3. Motion primitive embodiments

[0117] In one or more embodiments, motion primitives are encapsulated from repeated excavator actions such as digging, dumping, and moving the base. Thus, a digging motion primitive, a dumping motion primitive, and a base moving motion primitive are designed for the embodiments of the framework— although joint moving primitives including digging and dumping use motion parameters or task configurations as described in Sections D.1.b and D.1.c above. These joint moving primitives compute joint trajectories for a proportional-integral-derivative (PID) controller to follow and eventually derive joint velocity control commands.

[0118] On the other hand, the base moving primitive takes as input a path of waypoints generated by the BMSP. And this primitive acts as an MPC controller to generate base control commands.

[0119] For low-level control, both velocity and base control commands can use lookup tables to match the current inputs of the excavator’s hydraulic system.

[0120] E. Experimental results

[0121] It should be noted that these experiments and results are provided by way of illustration and were performed under specific conditions using one or more particular embodiments; as such, neither the experiments nor their results should be used to limit the scope of the disclosure of this patent document.

[0122] The trenching task and the pile removal task were performed in a real excavator, and the large-scale pile removal task was performed in a simulation. In the actual test, the excavator used was an XCMG 490DK, which has a maximum digging radius of 10.47 m, a maximum digging height of 10.12 m, a maximum digging depth of 6.93 m, and a work weight of 36.6 tons. In the simulation, the excavator used was a CAT 365CL, which has a maximum digging radius of 11.44 m, a maximum digging height of 11.90 m, a maximum digging depth of 7.17 m, and a work weight of 67.0 tons. AGX Dynamics was used as the simulation environment.

[0123] 1. Trenching Experiment

[0124] The target trench length in the experiment was 10.0 m, the width was 1.5 m, and the depth was 2.0 m. Using an embodiment of the architecture and planner presented herein, the task was completed using 20 minutes. Based on the trench dimensions, the embodiment automatically set 5 waypoints and 5 material removal subtasks. Figure 13 A field trenching experiment according to an embodiment of the disclosure is shown. The image on the left is before the trenching task started. The image on the right is after the trenching task was completed.

[0125] 2. Pile Removal Experiment

[0126] The target pile length was 8.0 m, the width was 5.6 m, and the height was 0.5 m. Using the planner embodiment, the task was completed in 16 minutes. Figure 14 A field pile removal experiment setup according to an embodiment of the disclosure is shown.

[0127] Figure 15 Four time instances during the removal experiment setup are shown, with the base at different waypoints. Based on the pile dimensions, the embodiment automatically set 4 waypoints and 4 material removal subtasks.

[0128] 3. Large-Scale Pile Removal Experiment

[0129] A large-scale pile removal experiment was performed in a simulation. The target pile length was 36.0 m, the width was 22.5 m, and the height was 0.5 m. In the AGX simulator, the pile dimensions were set to a length equal to 20.0 m, a width equal to 12.0 m, and a height of 0.5 m. Figure 16A and 16BSimulation results are shown according to embodiments of the present disclosure. The images on the left (1605 and 1615) show the initial simulation configuration, and the images on the right (1610 and 1620) show the environment at the completion of the simulation.

[0130] Initially, the excavator is located in the lower right of the environment. It first moves to the first planned route point and starts working on the first column. Once the first column is complete, it performs a U-turn and then works on the second column. In the RVIZ view ( Figure 16B ), the outer box shows the task area assigned by the user, and the inner box shows the actual stakes generated in the AGX simulator. The white marker points represent route points that have not been assigned a material removal subtask, while the darker marker points represent route points that have been assigned a material removal subtask. The color shown in RVIZ is based on the soil height. In the initial configuration, the color inside the inner box is quite different from the color outside. When the task is complete, the color is uniform, which means that the task is well completed. Since the soil is dumped directly on the ground, there are some small stakes on the excavator base movement trajectory.

[0131] F. Some conclusions or observations

[0132] This patent document presents embodiments of novel autonomous excavator planning system designs. In one or more embodiments, the high-level task planning includes task partitioning and base movement planning. Embodiments of a general subtask planner designed with motion primitives that consider both arm movement and base movement are also presented herein. Based on experimental results, embodiments of a general autonomous excavator planning system architecture using a material removal subtask planner and a base movement subtask planner are able to adapt to different excavation tasks.

[0133] In addition to excavation tasks, excavators are used for tasks such as grading and other tasks. It should be noted that the architecture embodiments can be easily extended to other new types of tasks. Also, the base movement planning is 2D; however, in some work environments, the excavator can travel on a rugged terrain surface. Thus, those skilled in the art should recognize that embodiments can be easily extended to 3D base movement planning, taking into account the excavator’s balance.

[0134] G. Computing system embodiments

[0135] In one or more embodiments, aspects of the present patent document can be directed to, can include, or can be implemented on one or more information handling systems (or computing systems). An information handling system / computing system can include any instrumentality or aggregate of instrumentalities operable to compute, estimate, determine, classify, process, transmit, receive, retrieve, originate, route, switch, store, display, communicate, manifest, detect, record, reproduce, handle, or utilize any form of information, intelligence, or data. For example, a computing system can be or include a personal computer (e.g., a laptop computer), a tablet computer, a mobile device (e.g., a personal digital assistant (PDA), a smartphone, a phablet, a tablet, etc.), a smart watch, a server (e.g., a blade server or a rack server), a network storage device, a camera, or any other suitable device, and can vary in size, shape, performance, functionality, and price, depending on the requirements of an end-user or an organization. A computing system can include a random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, read only memory (ROM), and / or other types of memory. Additional components of a computing system can include one or more mass storage devices (e.g., hard disk drives, solid state drives, or both), one or more network ports typically to communicate with external devices, and various input and output (I / O) devices, such as a keyboard, a mouse, a touchscreen, a stylus, a microphone, a camera, a trackpad, a display, etc. A computing system can also include one or more buses operable to transfer communications among the various hardware components.

[0136] Figure 17 A simplified block diagram of an information handling system (or computing system) in accordance with embodiments of the present disclosure is depicted. It will be appreciated that the functionality shown for system 1700 can operate to support various embodiments of a computing system— but it should be appreciated that a computing system can be configured differently and include different components, including fewer or more components than those depicted in Figure 17

[0137] As shown in Figure 17 the computing system 1700 includes one or more CPUs 1701 that provide computing resources and control the computer. The CPU 1701 can be implemented with a microprocessor and can also include one or more graphics processing units (GPUs) 1702 and / or floating point coprocessors for mathematical computations. In one or more embodiments, the one or more GPUs 1702 can be incorporated within a display controller 1709, such as part of one or more graphics cards. The system 1700 can also include a system memory 1719, which can include RAM, ROM, or both.

[0138] A number of controllers and peripherals can also be provided, as Figure 17 ​Input controller 1703 represents an interface to various input devices 1704. The computing system 1700 can also include a storage controller 1707 for interfacing with one or more storage devices 1708 that include storage media such as magnetic tape or disks, or optical media, which can be used for recording programs of instructions, utilities, and applications, which can include embodiments of programs implementing various aspects of the present disclosure. The storage device(s) 1708 can also be used to store data that has been processed or is to be processed according to the present disclosure. The system 1700 can also include a display controller 1709 for providing an interface to a display device 1711, which can be a cathode ray tube (CRT), a thin film transistor (TFT), an organic light emitting diode, an electroluminescent panel, a plasma panel, or any other type of display. The computing system 1700 can also include one or more peripheral controllers or interfaces 1705 for one or more peripheral devices 1706. Examples of peripheral devices can include one or more printers, scanners, input devices, output devices, sensors, etc. A communications controller 1714 can interface with one or more communication devices 1715, which enables the system 1700 to connect to remote devices through any of a variety of networks, including the Internet, cloud resources (e.g., Ethernet cloud, Fibre Channel over Ethernet (FCoE) / Data Center Bridging (DCB) cloud, etc.), local area networks (LANs), wide area networks (WANs), storage area networks (SANs), or through any suitable electromagnetic carrier signals, including infrared signals. As shown in the depicted embodiment, the computing system 1700 includes one or more fans or fan trays 1718 and one or more cooling subsystem controllers 1717 that monitor the thermal temperature(s) of the system 1700 (or components thereof) and operate the fans / fan trays 1718 to help regulate the temperature.

[0139] In the illustrated system, all major system components can be connected to bus 1716, which can represent more than one physical bus. However, various system components can be in physical proximity, or can not be in physical proximity. For example, input data and / or output data can be remotely transmitted from one physical location to another. Moreover, programs implementing various aspects of the present disclosure can be accessed from remote locations (e.g., servers) over a network. Such data and / or programs can be conveyed by any of a variety of machine-readable media, including, for example: magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as compact disks (CDs) and holographic devices; magnetic-optical media; and hardware devices specially configured to store or to store and execute program code, such as application specific integrated circuits (ASICs), programmable logic devices (PLDs), flash devices, other non-volatile memory (NVM) devices (such as 3D XPoint-based devices), and ROM and RAM devices.

[0140] Aspects of the present disclosure can be encoded upon one or more non-transitory computer readable media having instructions for one or more processors or processing units to cause steps to be performed. It should be noted that the one or more non-transitory computer readable media should include volatile and / or non-volatile memory. It should be noted that alternative implementations are possible, including hardware implementations or software / hardware implementations. Hardware implemented functions can be implemented using ASIC(s), programmable arrays, digital signal processing circuitry, etc. As such, the "means" term in any claim is intended to cover software and hardware implementations. Similarly, the term "one or more computer readable media" as used herein includes software and / or hardware having instructions programmed thereon, or a combination thereof. In view of these implementation alternatives, it should be understood that the figures and accompanying description provide functional information necessary to enable a person skilled in the art to write program code (i.e., software) and / or manufacture circuitry (i.e., hardware) to perform the processes described.

[0141] It should be noted that embodiments of the present disclosure can also relate to computer products with non-transitory, tangible computer-readable media storing computer code thereon for execution by the processors of the computer. The media and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind known or available to those having skill in the relevant arts. Examples of tangible computer-readable media include, but are not limited to: magnetic media, such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and holographic devices; magneto-optical media; and hardware devices that are specially configured to store and perform program code, such as ASICs, PLDs, flash devices, other nonvolatile memory devices, such as 3DXPoint devices, and ROM and RAM devices. Examples of computer code include machine code, such as produced by a compiler, and files containing higher-level code that are executed by a computer using an interpreter. Embodiments of the present disclosure can be implemented in whole or in part as machine-executable instructions that can be in program modules that are executed by the processing devices. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In distributed computing environments, program modules can be physically located in settings that are local, remote, or both.

[0142] Those skilled in the art will realize that no computing system or programming language is critical to the practice of the present disclosure. Those skilled in the art will further appreciate that the foregoing examples and embodiments are exemplary and non-limiting of the scope of the disclosure. All permutations, enhancements, equivalents, combinations, and improvements that are evident to those skilled in the art in view of the foregoing description and associated drawings are included within the true spirit and scope of the present disclosure. It should also be noted that any element of any claim can be expressed differently, including having multiple dependencies, configurations, and combinations.

[0143] Those skilled in the art will appreciate that the foregoing examples and embodiments are exemplary and do not limit the scope of the disclosure. All permutations, enhancements, equivalents, combinations, and improvements that are evident to those skilled in the art in view of the foregoing description and associated drawings are included within the true spirit and scope of the present disclosure. It should also be noted that any element of any claim can be arranged differently, including having multiple dependencies, configurations, and combinations.

Claims

1. A computer-implemented method comprising: receiving, from a user, a task for a excavator and associated task parameter values; given the task, the associated task parameter values, at least some of the perception related data received from a perception module, partitioning the task into one or more base movement subtasks and one or more material removal subtasks using an advanced task planner module comprising one or more rule-based models, wherein the one or more base movement subtasks comprise a sequence of waypoints and the one or more material removal subtasks comprise a sequence of subtask regions and a goal for each subtask region; given the sequence of waypoints and state information, wherein at least some of the state information is obtained from the perception module, generating a set of one or more waypoints between successive pairs of waypoints to guide the excavator between waypoints using a base movement subtask planner comprising one or more models; given the sequence of subtask regions, associated goals, and state information, wherein at least some of the state information is obtained from the perception module, determining motion primitives and associated parameter values for each subtask region to achieve the goal for that subtask region using a material removal subtask planner comprising one or more planner models; using the motion primitives, associated parameters, excavator parameters, and perception module data as inputs to a joint movement primitive module to generate joint trajectories and using the waypoints, excavator parameters, and perception module data as inputs to a base movement primitive module to generate base trajectories; and providing the joint trajectories and base trajectories to one or more joint controller modules and one or more base controller modules, respectively, to generate controller commands to control the devices of the excavator.

2. The computer-implemented method of claim 1, wherein the perception module comprises or receives data from one or more sensors comprising: a camera for image data; a LiDAR sensor for generating point cloud data; a real-time motion sensor for localization; and one or more tilt sensors.

3. The computer-implemented method of claim 1 or 2, further comprising determining local base regions using a rectangular definition or a sector-based definition and a set of parameter values comprising a block center swing angle, a block angle range or width, a near end distance, and a far end distance.

4. The computer-implemented method of claim 3, wherein the subtask regions are determined by the advanced task planner module by performing steps comprising: given a local task region for the excavator and a block on which the task is to be performed, determining a number of local task regions that cover at least the block.

5. The computer-implemented method of claim 1 or 2, wherein the step of determining motion primitives and associated parameter values for each subtask region to achieve the goal of that subtask region using a material removal subtask planner comprising one or more models, given the sequence of subtask regions, associated goals, and state information obtained from the perception module, includes: for a subtask region, implementing a set of one or more action planners at the subtask region according to the material removal subtask planner models associated with the subtask until the subtask goal is achieved; and in response to the subtask goal of the subtask region being achieved and in response to the subtask region not being the last subtask region in the sequence of subtask regions, processing the next subtask region in the sequence of subtask regions.

6. The computer-implemented method of claim 1 or 2, wherein the material removal subtask planner comprises: a point of attack (POA) planner; a digging motion planner; and a dumping planner.

7. The computer-implemented method of claim 6, wherein parameters utilized by the digging motion planner include:

8. The computer-implemented method of claim 1 or 2, wherein: which represents the position of the point of attack (POA) in the base frame; which represents the depth of penetration; which represents the penetration length; which represents the length of the tow, and which represents the whole excavation movement inclination angle with respect to the horizontal plane. the state information for the base movement subtask planner includes excavator joint states, excavator base pose, and one or more environmental states of the work area of the excavator; and the state information for the base movement subtask planner includes excavator base pose and one or more environmental states of the work area.

9. The computer-implemented method of claim 1 or 2, wherein end conditions for subtask goals can be defined according to different levels of strictness requirements.

10. A system comprising: one or more processors; and one or more non-transitory computer-readable media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, cause performance of the computer-implemented method of any of claims 1-9.

11. A non-transitory computer-readable medium comprising one or more sequences of instructions which, when executed by at least one processor, cause performance of the computer-implemented method of any of claims 1-9.

12. A computer program product comprising a computer program which, when executed by a processor, implements the computer-implemented method of any of claims 1-9. ​ ​

Images