Autonomous excavation based on a control policy trained with reinforcement learning under depth constraints

Abstract

The invention is notably directed to a computer-implemented method of controlling a bucket (110) of an excavator (1) for autonomous excavation, wherein the method comprises accessing (S5) a specification of a shape to be excavated in a terrain (150), and executing (S30) a control policy (206) including a constraint generator (2061) and a control model (2062) trained with reinforcement learning (preferably in simulation), for the control policy (206) to repeatedly perform (S30) algorithmic cycles. Each cycle of the algorithmic cycles comprises: updating (S32, S33) a state of the bucket (110) and a perception of the terrain (150) based on signals received from a set of sensors (410, 430), which preferably are on board the excavator (1); based on the accessed specification, the updated state of the bucket (110), and the updated perception, generating (S34) one or more maximum-depth constraints (164) through the constraint generator (2061), and inferring (S35), through the control model (2062), control data meeting the one or more maximum-depth constraints (164) generated; and instructing (S38) to control the bucket (110) based on actuator control signals formed (S36) according to the control data inferred, to excavate (S40) the terrain (150) in accordance with said specification. The invention is further directed to related systems and computer program products.

Description

TECHNICAL FIELD

[0001] The invention relates in general to computer-implemented methods for controlling a bucket of an excavator for autonomous excavation, as well as related systems and computer program products. In particular, it is directed to a method that generates maximum-depth constraints based on the specification of an excavation shape and accordingly infers control data that meet the generated constraints, thanks to a control model trained with reinforcement learning.BACKGROUND

[0002] In the construction industry, one of the largest industry sectors worldwide, automation will become increasingly important in the near future to alleviate the prevalent shortage of trained workforce and thus improve productivity. Additionally, automation on construction sites can help reduce the high number of fatal work injuries. In the longer run, also space applications might become increasingly relevant.

[0003] Of particular importance is the automation of excavation, one of the most common tasks on construction sites. Given a desired state of the terrain, e.g., a pit or a trench represented as a georeferenced map, the mission consists of achieving the goal state with an excavator fully autonomously. A common way of solving this task is to split it into tractable submodules, which are coordinated by some form of state machine. Individual states consist of driving the machine to a sequence of positions so that the entire design can be covered, locating the bucket, moving the bucket to the excavation point, excavating, finding a dump location, dumping the excavated soil, and monitoring the progress.

[0004] Several methods have been proposed to automate excavation. As the present inventors observed, such methods are suboptimal as they typically require reworking the excavated area. As the present inventors further observed, some drawbacks of these methods arise from their lack of adherence to geometric excavation constraints and the fact that they cause unnecessary bucket movements.SUMMARY

[0005] According to a first aspect, the invention is embodied as a computer-implemented method of controlling a bucket of an excavator for autonomous excavation. The method comprises accessing a specification of an excavation shape (i.e., a shape to be excavated in a terrain), and executing a control policy that involves a constraint generator and a control model trained with reinforcement learning (RL). I.e., the control model is a computational model that is trained for control purposes. The control policy repeatedly performs algorithmic cycles, where each cycle of the algorithmic cycles comprises the following steps. First, a state of the bucket and a perception of the terrain are updated based on signals received from a set of sensors, which preferably are on board the excavator. The perception may for instance be obtained in the form of an elevation map. Next, based on the accessed specification, the updated state of the bucket, and the updated perception, one or more maximum-depth constraints are generated through the constraint generator, and control data are inferred through the control model. The inferred control data meet the one or more maximum-depth constraints generated at runtime, because the control model was trained based on similar maximum-depth constraints. At the end of the cycle, the method instructs to control the bucket based on actuator control signals formed according to the control data inferred, to excavate the terrain in accordance with the specification accessed.

[0006] Combining all optimal operation criteria is difficult, even for human operators, and requires many years of practice. Therefore, the present inventors propose an RL-based approach that makes it possible to combine disparate objectives and sensor modalities flexibly. The RL agent discovers an ideal behaviour in a wide range of different conditions during the training. Such conditions integrate constraints, and the trained control model can then suitably be used at runtime, for inference purposes. The above method reflects operations performed at runtime. According to the proposed approach, a user or a process (e.g., using a connected device) provides a desired excavation shape, based on which a constraint generator generates constraints, e.g., in the form of maximal depths. In turn, the control policy adheres to such maximum-depth constraints to generate suitable control data and achieve the intended excavation shape. Of particular advantage is that the commands generated do not cause the bucket to dig more than necessary. This, in turn, makes it possible to reduce or even avoid the reworking as usually required with prior automation approaches. Eventually, the above method permits saving time and power.

[0007] In embodiments, the one or more maximum-depth constraints are generated through the constraint generator as one or more maximal depths, each measured relative to a current position of the bucket, preferably relative to a current position of a tip of the bucket. In each case, the current position is determined in accordance with the updated state of the bucket. Generating maximal depths that are defined with respect to the current position of the bucket (e.g., the tip base) is very convenient in the present case. While the generated depths depend on the actual excavation shape specification at runtime, depth constraints defined with respect to the current bucket positions can (i) be partly decorrelated from terrain shape representations and (ii) reduce the dimensionality of the input to the controller. That is, a first advantage is the independence of map representation. E.g., one typically just need to sample points on the elevation map. In addition, instead of having the whole high-dimensional map as an input to the controller, extracting maximum depths reduces the dimensionality of the input (number of input values). Such advantages are valid both during the runtime (inference) and training phases. Furthermore, this makes it much easier to generate training data and permits to explore a large spectrum of input configurations during the training. Still, as defined above, the generated constraints can readily be used for inferencing purposes, i.e., to generate adequate control data based on the current position of the bucket.

[0008] In embodiments, the one or more maximum-depth constraints generated comprise two or more maximum-depth constraints, which are generated through the constraint generator as an array of two or more maximal depths, respectively. The maximal depths in this array extend from a lateral reference plane delimiting the bucket, or the tip thereof, in a direction of a house of the excavator. I.e., this reference plane delimits the bucket, or its tip, laterally, such that the normal of this plane is directed towards the house, i.e., the rotating platform of the excavator.

[0009] Preferably, the method further comprises, at said each cycle, and prior to generating the two or more maximum-depth constraints, identifying an array of terrain height points in accordance with the updated perception. The terrain height points extend from the lateral reference plane in the direction of the house. The maximal depths are vertically aligned with the terrain height points. I.e., the two or more maximal depths are located vertically from two or more corresponding ones of the terrain height points. This amounts to generating pairs of points, which are vertically aligned. The resulting discretization reduces the dimensionality of the inputs to the control model and facilitates the training, as well as the transfer to real excavators (especially when training in simulation). This discretization compensates for the large spectrum of input configurations that may otherwise be used to train the control model. Each of the maximal depths and the terrain height points may for instance be defined as elevations relative to the current elevation of the tip of the bucket.

[0010] In embodiments, each of the first array and the second array includes five points, spaced at a distance g of between 5 % and 20 % of a width of the bucket. The distance (or gap) g is measured perpendicularly to the lateral reference plane, in the direction of the house. The width of the bucket is measured in a direction that extends in the lateral reference plane, perpendicularly to the direction of the house. For example, for a bucket width of 1.5 m, the points may be spaced at a distance of between 7.5 cm and 30 cm. Such a resolution is sufficient in practice. Using several points (as opposed to a single point) provides the control policy a lookahead of the upcoming terrain shape and corresponding maximum depths, which helps the control policy take suitable action.

[0011] In embodiments, the set of sensors comprises proprioceptive sensors and exteroceptive sensors. In principle, each of the perception and bucket state update processes may rely on any of, or both, exteroception and proprioception. However, a preferred implementation is one in which the perception relies on exteroception (at least primarily), while the bucket state updates are based on proprioception (at least primarily). That is, the perception (e.g., an elevation map) is updated based on signals received from the exteroceptive sensors. Meanwhile, the state of the bucket is updated based on signals received from the proprioceptive sensors. That is, each cycle comprises updating states of actuators of the bucket based on signals received from the proprioceptive sensors, whereby the state of the bucket is updated in accordance with the states of the actuators. Exteroception allows a perception of the excavator's environment, while proprioception makes it possible to simply update the bucket's state. In addition, proprioception can be used to provide feedback as to the current soil conditions.

[0012] The perception may solely be based on exteroception. In variants, it is primarily based on exteroception but subsidiarily uses signals received from proprioceptive sensors. For example, the elevation map is also updated by determining how the bucket moved through the soil, thanks to proprioceptive sensors, e.g., joint angle sensors. Similarly, updating the bucket states may solely rely on proprioception. In variants, it is primarily based on proprioception but subsidiarily uses signals received from exteroceptive sensors (e.g., a Lidar). Still, alternative embodiments can be contemplated, where the bucket states are updated by primarily or exclusively using exteroception (e.g., a camera, a lidar, and / or a total station). For completeness, while preferred embodiments use sensors on board the excavator, at least some of the exteroceptive sensors may, in variants, be external sensors. For example, use can be made of an external Lidar, placed in the vicinity of the excavator.

[0013] In embodiments, the control model is a computational model that has been trained to adapt to current soil conditions, amongst other objectives. The control data are inferred at each cycle in accordance with signals received from one or more of the proprioceptive sensors (e.g., from pressure sensors), where such signals implicitly encode a current soil condition. I.e., the controller does not need to measure the soil parameters explicitly but estimates them implicitly though feedback signals from proprioceptive sensors, e.g., pressure sensors. In that case, the RL agent is assumed to have discovered an ideal behaviour in a wide range of different soil conditions, e.g., by interacting with the system through trial and error during the training.

[0014] Note, the control model may have been trained for a specific (type of) excavator. In that case, specific torque limits may have been taken into account during the training of the control model, whereby such limits are implicitly respected at runtime, such that it is not necessary to provide them as observations at runtime. In variants, the control model is a "universal" (or, at least, a more general) control model, which can be applied to various (types of) excavators. In that case, the control model may use torque limits explicitly computed at runtime. That is, each computational cycle further comprises updating joint torque limits of arm joints based on a real-time configuration of the excavator arm obtained in accordance with signals received from the proprioceptive sensors. I.e., the actuator control signals are arm joint control signals meant to control arm joints of the excavator arm. The control data are then inferred based on the updated joint torque limits, in addition to the updated perception, the updated state, and the one or more maximum-depth constraints generated. That is, the controller respects machine limitations to avoid stalling or lifting the machine off the ground, in addition to adhering to maximum depth constraints (used to excavate a desired terrain shape).

[0015] So, two scenarios can be delineated. Where the control model has been specifically trained for a given (type of) excavator, there is no need to provide torque limit observations. These are taken care of thanks to the training as the simulated data already include the torque limits, which are preferably updated at every control step based on the current configuration of the excavator. For example, the control model may be trained for a specific excavator, whereby the agent learns the torque limits from the interaction with the simulation and, therefore, does no longer need such limits as observations for inferencing purposes. Conversely, where a more general control policy is used, which has been trained for multiple excavators (using respective torque limits as inputs), specific torque limits need to be fed as inputs (i.e., observations) to the trained model at runtime to help the control policy disambiguate different machines at runtime.

[0016] Additional constraints may be taken into account besides maximum depth constraints. For instance, in embodiments, the control data are further inferred (at each cycle) based on a pullup distance constraint, in addition to the updated perception, the updated state, the one or more maximum-depth constraints generated, as well as updated joint torque limits, if necessary. The pullup distance constraint is preferably generated through the constraint generator. The pullup distance constraint may for instance be generated so as to be relative to the excavator's base. This way, the method makes sure not to infer control data that would cause the bucket to collide with the excavator.

[0017] In embodiments, the method further comprises, prior to executing the control policy, training the control model on a training dataset reflecting bucket states, representations of terrains to be excavated and associated shapes of excavation. In addition, states of the excavator (e.g., pertaining to the orientation and velocity of the base) may possibly be taken into account, both during the training and at runtime. The control model is trained within an RL framework, which rewards filling the bucket, and under maximum-depth constraints that are generated in accordance with the training dataset. Additional constraints may be taken into account during the training, such as pullup constraints and, if necessary, joint torque limits, as discussed above. In embodiments, the RL framework rewards filling the bucket in a way that depends on a distance between the bucket and the pullup band, where this distance is estimated in accordance with the updated state of the bucket. As noted above, the control model is preferably trained in simulation, which makes it possible to take a large number of potential configurations into account. Alternatively, actual observations, or a mix of actual and simulated observations, may be used for training purposes.

[0018] In embodiments, at training the control model, the maximum-depth constraints are generated as maximal depths, each measured relative to a current position of the bucket, or a tip thereof. Preferably, the maximal depths are vertically aligned with terrain height points. The maximal depths and extend from a lateral reference plane delimiting (the tip of) the bucket in a direction of the house of the excavator, consistently with maximal depths as generated at runtime.

[0019] In embodiments, training the control model further comprises simulating joint velocities and updating joint torque limits of arm joints of an arm of the excavator for the control model to learn to generate control data in accordance with the updated joint torque limits. As noted above, taking such limits into account will benefit both specific and general control policies.

[0020] According to another aspect, the invention is embodied as a computerized system for controlling a bucket of an excavator for autonomous excavation. The system comprises a physical interface adapted to receive signals from a set of sensors, which preferably are on board the excavator, as well as processing means, which are configured to process data received through the physical interface and implement all the steps of a method according to any of the above embodiments.

[0021] According to a final aspect, the invention is embodied as a computer program product for controlling a bucket of an excavator for autonomous excavation. The computer program product comprises a computer readable storage medium having program instructions embodied therewith. The program instructions are executable by processing means of a controller to cause the latter to perform all the steps of a method according to any of the above embodiments.BRIEF DESCRIPTION OF THE DRAWINGS

[0022] These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. The illustrations are for clarity in facilitating one skilled in the art in understanding the invention in conjunction with the detailed description. In the drawings: FIG. 1A is a 2D cross-sectional view illustrating the operation of an excavator, whereby the excavator's bucket is operated to excavate a terrain in accordance with maximum-depth constraints generated based on a specification of a given shape to be excavated, according to embodiments; FIG. 1B depicts maximum-depth constraints, which are generated as maximal depths, defined with respect to the current position of the tip of the bucket, where the maximal depths are vertically aligned with corresponding terrain height points obtained in accordance with a perception of the terrain excavated, as in embodiments; FIG. 2 is a diagram illustrating components and modules implementing high- and low-level policy models to generate commands and actuate a bucket of an excavator such as shown in FIG. 1A, as in embodiments. Some of the concepts shown (rewards, terminations, truncations) are only used during the training of the high-level policy model; FIG. 3 is a flowchart illustrating high-level steps of a method of controlling a bucket of an excavator for autonomous excavation, according to embodiments; and FIG. 4 schematically represents a high-level architecture of an excavator's control system interfacing with an actuation system of the excavator, as involved in embodiments of the invention.

[0023] The accompanying drawings show simplified representations of devices or parts thereof, as involved in embodiments. Technical features depicted in FIG. 1A are not to scale. Similar or functionally similar elements in the figures have been allocated the same numeral references, unless otherwise indicated.

[0024] Computerized systems, methods, and computer program products embodying the present invention will now be described, by way of non-limiting examples.DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

[0025] The following description is structured as follows. General embodiments and high-level variants are described in section 1. Section 2 addresses particularly preferred embodiments. Section 3 concerns technical implementation details. Note, the present method and its variants are collectively referred to as the "present methods". All references Sn refer to methods steps of the flowchart of FIG. 3, while numeral references pertain to devices, components, and concepts involved in embodiments of the present invention.1. General embodiments and high-level variants

[0026] As noted in the background section, several methods have been proposed to automate excavation. However, such methods typically require reworking the excavated area and, in that sense, are suboptimal. The present inventors came to the conclusion that drawbacks of prior automation methods are due to a lack of adherence to geometric excavation constraints, which causes unnecessary bucket movements. They accordingly devised a novel approach, based on reinforcement learning, where maximum depth constraints are explicitly taken into consideration on inferencing (i.e., at runtime).

[0027] A first aspect of the invention is now described in detail, in reference to FIGS. 1 to 4. This aspect concerns a computer-implemented method of controlling a bucket 110 of an excavator 1, such as shown in FIG. 1A, for autonomous excavation. FIG. 1A illustrates the operation of the excavator 1. In operation of the excavator 1, the bucket 110 is controlled through actuators of the arm 10 of the excavator 1, where such actuators include arm joints. That is, the bucket 110 is controlled by actuating the arm joints, as known per se. The following reflects steps performed by a controller, which implements a control policy 206, referred to as a high-level policy 206 in FIG. 2. This controller is typically meant to be integrated into an excavation planning system for autonomous excavation, as discussed in section 2.

[0028] The method first comprises accessing a specification of a shape to be excavated in a terrain 150, see step S5 in the flow chart of FIG. 3. The method then executes S30 the control policy 206, which involves a constraint generator 2061 and a control model 2062. The control model is a computational model that is trained for control purposes. This computational model typically includes an artificial neural network (ANN), which is assumed to have been trained with reinforcement learning (RL).

[0029] The control policy 206 repeatedly performs S30 algorithmic cycles. As seen in FIG. 3, each algorithmic cycle comprises updating S32 a state of the bucket 110 and updating S33 a perception of the terrain 150. The update steps S32, S33 are based on signals received from a set of sensors 410, 430. Such sensors are preferably on board the excavator 1, as assumed in the accompanying drawings.

[0030] Each cycle further comprises generating S34 one or more maximum-depth constraints 164 and inferring S35 control data, based on the accessed specification, the updated state of the bucket 110, and the updated perception. The maximum-depth constraints 164 are generated S34 through the constraint generator 2061. The control data are inferred S35 through the control model 2062, so as to meet the one or more maximum-depth constraints 164 generated. The model is assumed to have learned to generate control data that meet maximum-depth constraints, such that the inferred control data will normally meet the maximum-depth constraints 164 generated at runtime.

[0031] At the end of each algorithmic cycle, the execution of the control policy causes to instruct S3 8 to control the bucket 110 based on actuator control signals formed S36 according to the control data inferred at step S35. The actuator control signals are typically formed thanks to commands generated by a low-level control policy 208, as opposed to the policy 206, which can be regarded as a high-level control policy. Note, the low-level policy model 208 may be a computational model that relies on machine learning, too. In variants, the low-level model 208 involves an adaptive control method or relies on usual control approaches such as approaches based on PID or model predictive control (MPC). The low-level model may also be a composite computational model involving two or more models, e.g., based on PID, MPC, and / or machine learning. Eventually, the actuator control signals cause to excavate S40 the terrain 150 in accordance with the specification accessed at step S5. The goal is to achieve a target terrain shape 166, see FIG. 1A.

[0032] The method preferably relies on onboard sensors 410, 430, i.e., sensors that are mounted directly on the excavator 1. Such sensors 410, 430 preferably form part of distinct sensory systems, i.e., a proprioceptive system 420 (including proprioceptive sensors 410) and an exteroceptive system 440, i.e., a perception system, which includes one or more exteroceptive sensors 430.

[0033] The exteroceptive sensors 430 make it possible to perceive the surroundings of the excavator 1. Exteroceptive sensors yield sensory information that is used to create a perception model of the excavator's surroundings, starting with the terrain 150. Preferably, use is made of one or more Lidars, stereo cameras, depth sensors, time-of-flight sensors, total stations, and / or ultrasonic sensors, which sensors are known as such. For example, Lidars give rise to point clouds. Stereo cameras make it possible to generate depth images, which can be converted into point clouds, too, if necessary.

[0034] The proprioceptive sensors 410 typically include joint encoders, inertial measurement units (IMUs), and / or pressure sensors. Such sensors typically allow joint positions, velocities, torques and, in turn, states of the excavator arm 10 to be estimated by a state estimator (not shown). In principle, the bucket states can be estimated using exteroception, e.g., using a Lidar. Preferably, though, the bucket state is estimated through proprioception. For instance, the states of the arm joints can serve to estimate the bucket state, which typically aggregates information about the position, linear velocities, orientation, and angular velocities, of the bucket 110, and forces acting on the bucket. In particularly preferred embodiments, the bucket state is defined by an nD pose (position, orientation) and an mD velocity (linear, angular velocities). The arm states are defined by states of joints (e.g., k positions, l velocities, p torques). For example, for a 3-link excavator as shown in FIG. 1A, the bucket-space properties are defined in the plane (x, z). In this example, the bucket's state is defined by a 3-dimensional pose (position, orientation) and a 3-dimensional twist (linear, angular velocities). The arm's state is further defined by a state of joints (e.g., 3-dimensional positions, 3-dimensional velocities) and may additionally be computed based on contact states (e.g., 3-dimensional bucket wrench), themselves derived from torque information, e.g., obtained through 3 joint torques.

[0035] A preferred implementation is one relying on one or more Lidars to perceive the terrain (e.g., as an elevation map, preferably a 2.5-dimensional elevation map), as well as onboard sensors 410 for providing pressure feedback, while joint positions and velocities are obtained from IMUs on each link. The environment of the excavator 1 is continually sensed by the exteroceptive sensors 430, meaning that the sensing operations are either repeatedly and / or continuously performed. For example, a Lidar 430 repeatedly scans the environment of the excavator 1 to continually update the terrain's perception (i.e., its apparent surface). In preferred embodiments, one or more Lidars 430 are placed on the roof of the excavator. If necessary, a gantry element (typically made of metal) is used to place the Lidars further away from the roof of the cabin 111 for a better view, as assumed in FIG. 1A. In variants, external exteroceptive sensors (e.g., Lidars) are relied upon to avoid the shadow of the arm 10.

[0036] According to the proposed approach, a user or an automated process (e.g., using a connected device) provides a desired excavation shape (e.g., in the form of a datafile, high-level specification, or by selecting predetermined shapes), from which a constraint generator 2061 generates constraints (e.g., in the form of maximal depths) and, if necessary, additional constraints such as pullup distances. As a result, the controller adheres to the specification of the shape to be excavated. In particular, the control policy 206 adheres to maximum-depth constraints 164 to achieve the desired shape design, i.e., the intended excavation shape. Of particular advantage is that the commands generated do not cause the bucket 110 to dig more than necessary (they follow the maximum-depth constraints), contrary to what is observed with prior automation methods. Thus, the present approach permits saving operation time and power. Beyond the maximum-depth constraints, additional constraints may be adhered to, e.g., in terms of pullup distances and actual machine limitations, to avoid stalling, as in preferred embodiments.

[0037] All this is now described in detail, in reference to particular embodiments of the invention. To start with, the one or more maximum-depth constraints 164 are preferably generated S34 through the constraint generator 2061 as one or more maximal depths 164, noted D 1 , D 2 , D 3 , D 4 , and D 5 , in FIG. 1B. Each maximal depth is defined and measured with respect to (i.e., relative to) the current position of the bucket, or preferably with respect to the current position of the tip 114 of the bucket, as assumed in the following. The current position of the bucket (or the bucket tip) is determined in accordance with the updated state of the bucket, i.e., in accordance with the state of the bucket as updated last.

[0038] For example, the control model may infer maximum depth values for discrete positions, corresponding to relative coordinates, themselves corresponding to shifts along the x-axis with respect to the lateral reference plane P L that delimits the bucket tip 114, see FIG. 1B. The lateral reference plane P L is represented by a vertical, dotted-dashed line in FIG. 1B. Similarly, each maximal depth point may possibly be defined vertically as an elevation number relative to the current elevation of the bucket 110. In the example of FIG. 1B, each maximal depth is measured vertically, relative to a basal reference plane P B , which vertically delimits the current elevation of the tip 114 of the bucket 110. The lateral reference plane P L is perpendicular to the plane P B . So, the control model infers values corresponding to depths measured with respect to the basal reference plane P B , as represented by the horizontal, dotted-dashed line in FIG. 1B. The elevation numbers of the depths D i (i = 1 to 5) correspond to positive or negative elevations with respect to the plane P B , which is itself typically parallel to a reference horizontal plane (i.e., a plane with zero slope). In other words, at run time, the generator generates maximal depth points for certain relative coordinates along the x-axis, each measured vertically from the tip base of the bucket 110.

[0039] As noted above, the current position of the bucket tip is determined in accordance with the bucket state as updated last. In practice, the algorithm identifies the current x-coordinate of the bucket tip (which can also be regarded as the horizontal distance to the excavator's base along the arm), formulates maximum depths for certain relative coordinates along the x-axis, with respect to the excavator's base, and then feeds the relative z-coordinate(s) of the maximum depth(s) as input to the control model, to predict the next actions.

[0040] That is, the maximum-depth constraints are used as an input to the control policy 206, which uses this input to obtain the next control data to actuate the bucket 110, e.g., through the arm joints. The control policy adheres to such maximum-depth constraints when inferencing new control data to achieve the intended excavation shape. Of particular advantage is that the commands generated do not cause the bucket to dig more than necessary. This, in turn, makes it possible to reduce or even avoid the reworking as usually required with prior automation approaches. Eventually, the present approach permits saving time and power.

[0041] As further seen in FIG. 1A and 1B, the method may further rely on terrain height points 162, noted H i (i = 1 to 5) in FIG. 1B. That is, each cycle may further comprise identifying S32 a first array of terrain height points 162 in accordance with the updated perception (i.e., the perception as updated last). The maximum-depth constraints 164 are generated S34 through the constraint generator 2061 (FIG. 2) as a second array of two or more maximal depths 164. As seen in FIG. 1B, the maximal depths 164 are located vertically from the corresponding terrain height points 162. I.e., each point D i (i = 1 to 5) is located vertically from a corresponding height point H i (i = 1 to 5). As further seen in FIG. 1B, each of the first array 162 and the second array 164 of points H i , D i extends from the lateral reference plane P L in the direction of the house 11 of the excavator 1. Generating an array of several points 164 provides more information about the terrain, which helps the control policy take the correct action. I.e., this provides the control policy a lookahead of the upcoming terrain shape and corresponding maximum depths.

[0042] Note, the terminologies used for excavators are not always unambiguous. The "boom" sometimes denotes the entire arm, including all links. In other cases, it denotes just the first link, as assumed in this document. Conversely, the "arm" is sometimes used to denote just the second link. In other cases, it means the entire arm (with all links, as assumed in this document). The second link can also be called stick, crowd, or dipper (as in this document, see FIG. 1A). The third link is called the shovel or bucket. So, according to the present terminologies, FIG. 1A shows an excavator 1 including a house 11, which is the rotating platform of the excavator 1. The house supports the cabin 111 (also called cab). The arm 10 includes the boom 101 (first link), attached to the house 11 (in fact the cabin 111), a dipper 102 (second link), and the bucket 110 (third link), all connected with three joints between them and the house 11.

[0043] The perception sensors 430 (e.g., Lidars) take into account the real-time terrain elevation to extract soil height points 162, which are used as input to the high-level policy 206, in addition to the maximum-depth points 164, to output control signals. As noted above, the height points 162 and maximum depth points 164 are preferably aligned vertically to ease the processing. For example, each of the first array 162 and the second array 164 may include five points, as assumed in FIGS. 1A and 1B. Such points may for instance be spaced at a distance g, which is preferably proportional to the bucket width (as measured along the y-axis). For instance, the distance g will preferably be less than 50% of the bucket width, and will advantageously correspond to between 5 % to 20 % of the bucket width. The wider the bucket, the lower the resolution required. This distance is typically of between 7.5 cm and 30 cm for a 1.5 m wide bucket. It is preferably between 15 cm and 25 cm, and more preferably equal to 20 cm. In principle, however, other ranges can be contemplated. This distance is measured perpendicularly to the lateral reference plane P L , along the x-axis in the direction of the house 11.

[0044] More generally, the points H i , D i may be defined in the (x, y) plane, as opposed to relative coordinates defined along the x-axis only. That is, defining arrays 162, 164 of points as relative coordinates along the x-axis will typically be sufficient to excavate a trench (once the excavator is in a suitable position). However, excavating more complex shapes may require defining successive 1D arrays 162, 164 of points, which will be distributed in the (x, y) plane.

[0045] As illustrated in the diagram of FIG. 4, the set of sensors 410, 430 typically comprises proprioceptive sensors 410 and exteroceptive sensors 430. In embodiments, the perception is updated S32 based on signals received from the exteroceptive sensors 430, while the state of the bucket 110 is updated S33 based on signals received from the proprioceptive sensors 410. Each algorithmic cycle comprises updating states S33 of actuators of the bucket 110 based on signals received from the proprioceptive sensors 410, such that the state of the bucket 110 is updated in accordance with the states of these actuators as updated last. The actuators typically include or consist of arm joints, which are capable of actuating the bucket 110. Note, the perception updates may solely rely on exteroception, while the bucket's state updates may rely on proprioception only. In variants, however, the bucket's state updates additionally make use of exteroception signals and / or the perception updates exploit proprioception signals. The bucket states may also be updated based on exteroception only, e.g., using Lidar signals, as noted in the summary section.

[0046] Referring to FIG. 2, the control model 2062 is a computational model that may advantageously have been trained to adapt to current soil conditions. In that case, the control data are inferred S35 at each cycle in accordance with signals received from one or more of the proprioceptive sensors 410 (e.g., pressure sensors), where such signals implicitly encode the current soil condition. As a result, the controller can react to encountered soil conditions and adapt the excavation behaviour online without the explicit knowledge of soil properties, e.g., thanks to feedback signals from pressure sensors 410. I.e., the controller does not need to measure the soil parameters explicitly but estimates them implicitly through the feedback signals from the sensors 410.

[0047] In embodiments, the present methods take into account joint torque limits. Such limits may be used during the training only should the control policy be trained for a specific (type of) excavator. Where a more general policy is sought, however, specific limits may be fed to the control model at runtime for reasons discussed earlier. In that case, each cycle comprises updating S33 joint torque limits of the arm joints based on a real-time configuration of the arm obtained in accordance with signals received from the proprioceptive sensors 410. The control data are then inferred S35 based on the updated joint torque limits, in addition to the updated perception, the updated state, and the one or more maximum-depth constraints generated. As a result, the controller respects machine limitations to avoid stalling or lifting the machine off the ground, in addition to adhering to maximum depth constraints.

[0048] Further constraints may be taken into account. E.g., in embodiments, the control data are inferred S35 at each cycle based on a pullup distance constraint, in addition to the updated perception, the updated state, the one or more maximum-depth constraints 164 generated, and the updated joint torque limits. This pullup distance constraint is preferably generated S34 through the constraint generator 2061, like the maximum-depth constraints. This pullup constraint may for instance be relative to the excavator's base. This way, the method makes sure not to infer control data that would cause the bucket to collide with the excavator's base.

[0049] The following more specifically concerns the training of the control model. As illustrated in FIG. 2, the present methods may further comprise steps of training the control model 2062, prior to executing the control policy 206. As explained earlier, the control model 2062 is a computational model 2062 that is trained with RL, based on a training dataset reflecting bucket states, representations of terrains to be excavated and associated shapes of excavation, under maximum-depth 164 and pullup constraints 3 generated in accordance with the training dataset. The model 2062 may further use additional information, such as information related to timing (e.g., how much time has elapsed since the control data was generated during the previous algorithmic cycle) and / or historical data, e.g., a history of generated control data and / or generated constraints. Moreover, the training dataset may also incorporate excavator states, for reasons discussed later.

[0050] The RL framework rewards filling the bucket 110. The maximum-depth constraints are preferably generated S34 as maximal depths 164, e.g., vertically aligned with terrain height points 162, where the maximal depths 164 and terrain height points 162 are defined for relative coordinates (along the x-axis) as elevations with respect to the basal reference plane P B delimiting the current elevation of the tip of the bucket 110, as described earlier. That is, each array 162, 164 preferably extends from a lateral reference plane P L (delimiting the tip of the bucket) in a direction of the house 11 of the excavator 1. In other words, the control policy is trained with RL, so as to adhere to certain constraints. The latter notably contain maximum-depth constraints 164, which are preferably generated in the form of maximal depths, as explained earlier.

[0051] The control model 2062 is preferably trained in simulation. For instance, the training of the control model 2062 may comprise simulating joint velocities and updating joint torque limits of arm joints of the arm 10 of the excavator 1 for the control model to learn to generate control data in accordance with updated joint torque limits. In addition, the constraints may include a pullup band, as noted earlier. The reward typically concerns the filling of the bucket, or, more generally, a successful scoop, using suitable heuristics. A successful scoop is not necessarily a full shovel. For example, in hard soil, the bucket 110 can typically not be filled completely.

[0052] Partly random generation of heights H i and maximum depths D i may be relied on. Still, the generated points H i , D i are subject to certain constraints e.g., the maximum depths are always below the height points, they may for instance consist of five points spaced apart 20 cm along the x-axis, as noted earlier.

[0053] It is worth dwelling on the nature of the information produced and used during training and during inference. In preferred embodiments, the exteroceptive observations used at runtime consist of Lidar point clouds, from which terrain elevations H i are derived. A corresponding array of maximal depth points D i is then generated. Moreover, onboard sensors 410 are used for pressure feedback, while joint positions and velocities are obtained from IMUs on each link. Bucket velocities are obtained through forward kinematics and / or directly measured with a camera, a lidar, and / or a total station. Bucket velocity commands can be converted to joint velocity commands via inverse kinematics.

[0054] During the training, however, both the bucket states and joint velocities can be obtained in simulation. The simulator may internally compute the bucket states from the joint states. In principle, it may be sufficient to rely on the bucket states only or the joint states only, given that the bucket states can be obtained from the joint states. However, in preferred embodiments, both the bucket states and joint states (including machine states) are used as observations, even though this information is redundant, as this happens to speed up the training.

[0055] As noted earlier, the training dataset may further incorporate states of the excavator, i.e., states determined by the orientation of the base of the excavator and the velocity of this orientation. Similarly, such excavator states would be computed and fed as observations to the control model at runtime. For example, the observations fed to the model may further include the excavator's pitch angle and pitch rate in the arm direction. The pitch angle determines the orientation of the base. This quantity turns out to be useful in case the bucket orientations (and thus the bucket states) are not explicitly determined but inferred from the joint angles. In that case, the excavator orientation is needed to fully determine the bucket orientation. The pitch rate (i.e., the velocity of the orientation of the base, or its angular rate) is also useful; one typically does not want the excavator to move during the operation, as this will cause the machine to stall or lift. Moreover, a termination condition may explicitly state that the excavator's base velocity must not exceed a certain limit. Therefore, the pitch rate is preferably added as an observation, too. In variants, the pitch angle and rate are assumed to be equal to zero, as is normally the case under ideal conditions. In other variants, the 3-dimensional base orientation and the 3-dimensional base orientation velocity are used as observations, instead of just the pitch angle and pitch rate.

[0056] In embodiments, the RL framework rewards filling the bucket 110 in a way that depends on the distance between the bucket and the pullup band. E.g., the training rewards the bucket filling, reaching a terminal state (not full) if the bucket reaches the pullup band 3 or full if the bucket is outside the pullup band. For example, distinct filling thresholds can be used. E.g., the algorithm may reward filling the bucket 110 entirely if the bucket is outside of the pullup band 3 as per the updated state of the bucket, or only partially when the current position of the bucket reaches the pullup band 3.

[0057] Practically speaking, the control model may for instance be trained with an actor-critic reinforcement algorithm, i.e., using an actor network that interacts with a critic network for training purposes. However, only the actor network is used at runtime, for inference purposes. The actor network is used to infer a desired action. It may for instance be configured as a multilayer perceptron (MLP) network. The critic network is used for estimating a value function. It may also include an MLP network. The control model is trained by jointly training the actor network and the critic network, for the actor network to learn to generate suitable control data at each algorithmic cycle at runtime. An actor-critic architecture has advantages in terms of sample efficiency, convergence, stability, and flexibility. Other reinforcement learning architectures can be contemplated.

[0058] The control model is preferably trained with a proximal policy optimization (PPO) algorithm. PPO algorithms are known per se. If necessary, conditions can be used for truncation of episodes, e.g., when a timer has run out or undesired states are reached. Preferred is to rely on curriculum learning, whereby the control model is gradually trained by gradually increasing the difficulty of the excavation task in the training environment, e.g., increasing diversity of terrains or increasingly restrictive success states, and / or scaling weights of the computed rewards (reward curriculum). When the low-level policy model 208 is a trainable model, the high- and low-level policies can be trained in an interlaced manner. For example, one may train the low-level policy model 208 for it to learn to generate low-level actuator / motor commands, freeze the low-level policy model, and then train the high-level model using the frozen, low-level model, for the high-level policy model 206 to learn to generate suitable trajectories, and so on.

[0059] Referring to FIG. 4, another aspect of the invention concerns a computerized system 401 for controlling the bucket 110 of an excavator 1 for autonomous excavation. The system 401 notably comprises a physical interface, which is adapted to receive signals from a set of sensors 410, 430, e.g., proprioceptive sensors 410 and exteroceptive sensors 430, as described earlier. The sensors are preferably all on board the excavator 1. The system 401 further comprises processing means 400, which are configured to process data received through the physical interface. So, the processing means interface with the sensors through the physical interface, so as to be able to process signals obtained from such sensors. The processing means 400 are otherwise configured to implement steps as described above. The processing means 400 may for instance comprise one or more processors (e.g., a CPU, as well as one or more GPUs) reading from and writing to the main memory. Some redundancy may be ensured, if necessary. The system 401 may further include a persistent storage, which stores computerized methods as described earlier in the form of software instructions. Such computerized methods can be loaded in the main memory, for the processors to implement method steps as described above.

[0060] The control system 401 may be regarded as including the set of sensors, as assumed in FIG. 4. In this example, the control system 401 includes two sensor systems 420, 440, i.e., a proprioceptive sensor system 420 and an exteroceptive sensor system 440, in addition to the processing system 400. The sensor systems 420, 440 have their own processing systems 425, 445, to pre-process sensor signals coming from the sensors. That is, the signals transmitted by the sensors are pre-processed by respective sensor processing systems 425, 445 before being passed to the processing means 400. Beyond the processing means 400 and sensor systems 420, 440, the control system may also integrate the actuation system 450, which notably comprises a motion controller 202 (FIG. 2) and actuators and / or motors.

[0061] The processing means 400 are configured to repeatedly perform algorithmic cycles, wherein each cycle comprises steps as described in reference to the present methods (first aspect of the invention). Such steps result in generating commands, which are transmitted to the actuation system 450 and, in particular, the motion controller 202. The transmitted commands may notably be dispatched to individual actuators and / or motors, e.g., motors that generate currents to follow the defined commands. This makes it possible to control the bucket according to the generated commands. The transmitted commands may be further processed by the actuation system 450, if necessary, before being dispatched to the actuators and / or motors.

[0062] A final aspect of the invention concerns a computer program product comprising a computer readable storage medium having program instructions embodied therewith. The program instructions are executable by processing means of a controller to cause the latter to perform steps as described herein in reference to the present methods. Section 3 provides additional implementation details related to computer program products and control systems 401.

[0063] The above embodiments have been succinctly described in reference to the accompanying drawings and may accommodate a number of variants. Several combinations of the above features may be contemplated. Examples are given in the next section.2. Particularly preferred embodiments

[0064] Particularly preferred embodiments involve a bucket-filling controller for autonomous excavation, which can react to the encountered soil conditions and adapt the excavation behaviour online without explicit knowledge of soil properties. At the same time, it adheres to a set of specifications required for the integration into a full-fledged autonomous excavation planning system.

[0065] Such an excavation planning system may for instance consist of two main modules: a global planner that computes a sequence of excavator poses to cover the desired geometry and a local planner that controls the excavation of the current local workspace segment. The latter consists of a state machine, where one state consists of the actual soil excavation. In the present context, this state is replaced with an RL-based controller. For example, a georeferenced map of the terrain is prerecorded with a Lidar sensor. The global planner splits the desired design into arc-shaped local workspaces, which can be excavated without driving.

[0066] The controller takes into account the current terrain elevation and respects maximum-depth and pullup constraints to achieve a desired design while respecting machine limitations to avoid stalling and preventing self-collisions. The controller is trained entirely in simulation with RL. A simple analytical soil model based on the Fundamental Equation of Earth-Moving (FEE) is used to simulate ground interactions. To ensure robustness in a wide variety of scenarios, soil parameters, as well as other properties of the environment, are randomized extensively during training. Such a controller was satisfactorily tested and evaluated on a 12-t excavator with conventional two-staged hydraulic actuation in a wide range of different situations. Additionally, the inventors have shown the excavation of a complete trench by integrating the controller into an autonomous excavation planning system. The experiments demonstrate that the controller can robustly adapt the excavation trajectory based on the encountered conditions and shows competitive performance compared to a professional machine operator.

[0067] The following describes particularly preferred embodiments as captured in the accompanying drawings.

[0068] FIG. 1A illustrates an excavator 1, which includes an undercarriage 12, a house 11 with a cabin 111, as well as an arm 10, with a boom 101 and a dipper 102 connecting to a bucket 110 with a bottom plate 114, and a tip 114. The boom is a "mono boom," allowing straight up / down movements only. The house 11 supports the operator's cabin 111 and the machinery's control systems. The undercarriage 12 supports the excavator 1 and allows for mobility across the terrain 150. A support 4 provides stability to the excavator 1.

[0069] The boom 101 serves as the primary support structure for the dipper 102 and the bucket 110, allowing for the transfer of forces during the excavation process. The dipper 102 provides the necessary leverage to the bucket 110, allowing for efficient penetration into the terrain 150. In the literature, the secondary separation plate (SSP) 112 refers to a virtual plane, delimiting the soil material that builds up as the bucket gets filled. A common assumption is that the excavated soil material distributes perfectly along this line. The SSP is used to compute the soil forces during the training. The bucket is operated according to soil height samples (SHS) 162 and maximum-depth samples (MDS) 164, forming vertically aligned 1D arrays. This operation causes forces such as deadload force 131, separation force 133, and penetration force (PF) 132 acting on the bucket 110 during the excavation process.

[0070] The bucket 110 is the primary tool for soil removal. The bucket 110 engages with the terrain 150, creating the swept volume 153 as the bucket 110 moves along the excavation path 160. The soil wedge 152 and the SSP 112 forms as a result of the excavation process. The excavation path 160 delineates the trajectory followed by the bucket 110 as the bucket 110 penetrates the terrain 150. The excavation path 160 is shaped by the movement of the bucket 110 and the resulting forces 131, 132, 133.

[0071] FIG. 2 shows components or modules used to control the bucket 110 of the excavator 1. Such components include a motion controller 202, which causes the bucket to move in response to commands received from the low-level policy 208. The high-level policy 206 produces control data, which the low-level policy 208 turns into low-level commands, themselves transmitted to the motion controller 202. Observations 204 include states of the buckets, the excavator, and one or more representations of the environment, forming a perception (e.g., an elevation map), which is computed based on signals obtained from proprioceptive and exteroceptive sensors. The observations can be simulated for training purposes. When training the high- and low-level policy models, observations are fed to the high-level policy 206, along with rewards 210 and terminations / truncations 212. Rewards will typically concern moving down the bucket into the direction of the soil and filling it, amongst other actions, as well as the power needed and maximal depth tracking. Terminations may notably relate to a filling status (full / partially filled) of the bucket, the bucket velocity, its angle of attack, the base motion, max depth, pullup distance, soil spillage, bucket height above the soil, and / or self-collisions.

[0072] FIG. 3 shows a flowchart of a method for controlling an excavator. The method begins with receiving S5 a specification of the shape to be excavated, followed by sensing cycles at step S10. Exteroceptive sensing occurs at step S12, while proprioceptive sensing takes place at step S14. The method then updates the perception and determines terrain height points at step S32, updates the bucket state at step S33, and generates maximum-depth and pullup distance constraints at step S34. Control data meeting these constraints are inferred by the control model at step S35. Subsequently, the method generates low-level commands at step S36, to accordingly control arm joints and, in turn, the bucket at step S38, which causes to excavate S40 the terrain according to the shape specification.

[0073] FIG. 4 illustrates an embodiment of a control system 401 of the excavator 1. The control system 401 comprises proprioceptive sensors 410, exteroceptive (perception) sensors 430, which form part of respective sensor systems 420, 440. Signals obtained are first processed by respective sensor processing systems 425, 445, which fuse data obtained from their respective sensor systems 420, 440, before passing fused data to the processing system 400 (i.e., a CPU). The processing system 400 may perform further fusion operations, if necessary, and may offload matrix-vector operations to a nearby GPU, so as to accelerate inferences from any machine learning-based model present. The processing system 400 computes commands for the actuation system 450, which notably includes a motion controller 202, as discussed earlier. In variants, each processing system 425, 445 may further use signals from the other sensor systems 440, 420 (or a subset of sensors thereof), and fuse data obtained from the two sensor systems 420, 440, before passing fused data to the processing system 400.3. Technical implementation details

[0074] Computerized devices can be suitably designed for implementing embodiments of the present invention as described herein. In that respect, it can be appreciated that the methods described herein are non-interactive, i.e., automated, although human input may be required in certain cases, e.g., should an anomaly or emergency be detected or during a deployment phase of the excavator. Automated parts of such methods can be implemented in software, hardware, or a combination thereof. In exemplary embodiments, automated parts of the methods described herein are implemented in software, as a service or an executable program (e.g., an application), the latter executed by suitable digital processing devices.

[0075] In the present context, each processing system is preferably mapped onto one or more respective sets of processors or, even, one or more respective computers. In particular, the system 401 may typically involve several processors or computers.

[0076] A suitable computer will typically include at least one processor and a memory (possibly several memory units) coupled to one or memory controllers. Each processor is a hardware device for executing software. The processor, which may in fact comprise one or more processing units (e.g., processor cores), can be any custom made or commercially available processor, possibly subject to some certification.

[0077] The memory typically includes a combination of volatile memory elements (e.g., random access memory) and non-volatile memory elements, e.g., a solid-state device. The software in memory may include one or more separate programs, each of which comprises an ordered listing of executable instructions for implementing logical functions. The software in the memory captures methods described herein in accordance with exemplary embodiments, as well as a suitable operating system (OS). The OS essentially controls the execution of other computer (application) programs and provides scheduling, input-output control, file and data management, memory management, and communication control and related services. It may further control the distribution of tasks to be performed by the processors. The methods described herein shall typically be in the form of executable programs, scripts, or, more generally, any form of executable instructions.

[0078] In exemplary embodiments, each computer further includes a network interface or a transceiver for coupling to a network (not shown). In addition, each computer will typically include one or more input and / or output devices (or peripherals) that are communicatively coupled via a local input / output controller. A system bus interfaces all components. Further, the local interface may include address, control, and / or data connections to enable appropriate communications among the aforementioned components. The I / O controller may have additional elements, which are omitted for simplicity, such as controllers, buffers (caches), drivers, repeaters, and receivers, to allow data communication.

[0079] When a computer is in operation, one or more processing units executes software stored within the memory of the computer to communicate data to and from the memory and / or the storage unit (e.g., a solid-state memory), and to generally control operations pursuant to software instruction. The methods described herein and the OS, in whole or in part are read by the processing elements, typically buffered therein, and then executed. When the methods described herein are implemented in software, the methods can be stored on any computer-readable medium for use by or in connection with any computer-related system or method.

[0080] Computer-readable program instructions described herein can be downloaded to processing elements from a computer-readable storage medium, via a network, for example, the Internet and / or a wireless network. A network adapter card or network interface may receive computer-readable program instructions from the network and forward such instructions for storage in a computer-readable storage medium interfaced with the processing means. All computers and processors involved can be synchronized using any suitable protocol or thanks to timeout messages.

[0081] Aspects of the present invention are described herein notably with reference to a flowchart and a block diagram. It will be understood that each block, or combinations of blocks, of the flowchart and the block diagram can be implemented by computer-readable program instructions.

[0082] These computer-readable program instructions may be provided to one or more processing elements as described above, to produce a machine, such that the instructions, which execute via the one or more processing elements create means for implementing the functions or acts specified in the block or blocks of the flowchart and the block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium.

[0083] The flowchart and the block diagram in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of the computerized systems, methods of operating it, and computer program products according to various embodiments of the present invention. Note that each computer-implemented block in the flowchart or the block diagram may represent a module, or a portion of instructions, which comprises executable instructions for implementing the functions or acts specified therein. In variants, the functions or acts mentioned in the blocks may occur out of the order specified in the figures. For example, two blocks shown in succession may actually be executed in parallel, concurrently, or still in reverse order, depending on the functions involved and the algorithm optimization retained. It is also reminded that each block and combinations thereof can be adequately distributed among special purpose hardware components.

[0084] While the present invention has been described with reference to a limited number of embodiments, variants, and the accompanying drawings, it will be understood by those skilled in the art that various changes may be made, and equivalents may be substituted without departing from the scope of the present invention. In particular, a feature (device-like or method-like) recited in a given embodiment, variant, or shown in a drawing, may be combined with or replace another feature in another embodiment, variant, or drawing, without departing from the scope of the present invention. Various combinations of the features described in respect of any of the above embodiments or variants may accordingly be contemplated, that remain within the scope of the appended claims. In addition, many minor modifications may be made to adapt a particular situation or material to the teachings of the present invention without departing from its scope. Therefore, it is intended that the present invention is not limited to the particular embodiments disclosed, but that the present invention will include all embodiments falling within the scope of the appended claims. In addition, many other variants than explicitly touched on above can be contemplated.

[0085] For example, several architecture variants may be contemplated for the processing system 401, which involves one or more distinct computers. In addition, various other machine learning techniques and computational models may possibly be relied on. In particular, the control model may be trained using model-based, model-free, or offline RL. Besides MLP architectures as discussed herein, the control model may involve other types of networks, such as recurrent networks or attention-based networks (transformers).REFERENCE LIST

[0086] 1Excavator202Motion Controller3Pullup Band204Observations (Perception, Bucket State)4Machine Support10Arm206High-Level Policy11House2061Constraint Generator12Undercarri age2062RL-based Model for Inferencing Control Data101Boom208Low-Level Policy102Dipper210Rewards110Bucket212Terminations / Truncations111Cabin400Processing System112Secondary Separation Plate401Control System114Bucket Tip410Proprioceptive Sensors116Bucket Bottom Plate420Proprioceptive Sensor System131Deadload Force425Proprioceptive Sensor Processing System132Penetration Force133Separation Force430Exteroceptive (Perception) Sensors150Actual Terrain Shape440Exteroceptive Sensor System152Soil Wedge445System Exteroceptive Sensor Processing160Excavation Path162Soil Height Samples450Actuation System (Motion Controller 202)164Maximum-Depth Samples166Desired Terrain Shape

Claims

1. A computer-implemented method of controlling a bucket (110) of an excavator (1) for autonomous excavation, wherein the method comprises accessing (S5) a specification of a shape to be excavated in a terrain (150), and executing (S30) a control policy (206) including a constraint generator (2061) and a control model (2062) trained with reinforcement learning, for the control policy (206) to repeatedly perform (S30) algorithmic cycles, wherein each cycle of the algorithmic cycles comprises: updating (S32, S33) a state of the bucket (110) and a perception of the terrain (150) based on signals received from a set of sensors (410, 430), which preferably are on board the excavator (1); based on the accessed specification, the updated state of the bucket (110), and the updated perception, generating (S34) one or more maximum-depth constraints (164) through the constraint generator (2061), and inferring (S35), through the control model (2062), control data meeting the one or more maximum-depth constraints (164) generated; and instructing (S38) to control the bucket (110) based on actuator control signals formed (S36) according to the control data inferred to excavate (S40) the terrain (150) in accordance with said specification.

2. The method according to claim 1, wherein the one or more maximum-depth constraints (164) are generated (S34) through the constraint generator (2061) as one or more maximal depths (164), each measured relative to a current position of the bucket (110), the current position of the bucket determined in accordance with the updated state of the bucket, and each of the one or more maximal depths (164) is preferably measured relative to a current position of a tip (114) of the bucket, the current position of the tip determined in accordance with the updated state of the bucket.

3. The method according to claim 2, wherein the one or more maximum-depth constraints generated comprise two or more maximum-depth constraints (164), which are generated (S34) through the constraint generator (2061) as an array of two or more maximal depths (164), respectively, the maximal depths in said array of two or more maximal depths extend from a lateral reference plane (PL) delimiting the bucket (110), or the tip thereof, in a direction (x) of a house (11) of the excavator (1), preferably, the method further comprises, at said each cycle, and prior to generating (S34) the two or more maximum-depth constraints (164), identifying (S32) an array of terrain height points (162) in accordance with the updated perception, wherein the terrain height points (162) extend from the lateral reference plane (PL) in the direction (x) of the house (11), and the two or more maximal depths (164) are located vertically from two or more corresponding ones of the terrain height points (162).

4. The method according to claim 3, wherein each of the first array and the second array includes five points, spaced at a distance g of between 5 % and 20 % of a width of the bucket, the distance g is measured perpendicularly to said lateral reference plane (PL), in the direction of the house, and the width of the bucket is measured in a direction (y) that extends in the lateral reference plane (PL), perpendicularly to the direction (x) of the house.

5. The method according to any one of claims 1 to 4, wherein the set of sensors (410, 430) comprises proprioceptive sensors (410) and exteroceptive sensors (430), the perception is updated (S32) based on signals received from the exteroceptive sensors (430), the state of the bucket (110) is updated (S33) based on signals received from the proprioceptive sensors (410), and said each cycle further comprises updating states (S33) of actuators of the bucket (110) based on signals received from the proprioceptive sensors (410), whereby the state of the bucket (110) is updated in accordance with the states of the actuators.

6. The method according to claim 5, wherein the control model (2062) is a computational model that has been trained to adapt to current soil conditions, and the control data are inferred (S35), at said each cycle, in accordance with signals received from one or more of the proprioceptive sensors (410), wherein the signals received from the one or more of the proprioceptive sensors implicitly encode a current soil condition, said one or more of the proprioceptive sensors preferably including pressure sensors.

7. The method according to claim 5 or 6, wherein said each cycle further comprises updating (S33) joint torque limits of the arm joints based on a real-time configuration of the arm obtained in accordance with signals received from the proprioceptive sensors (410), and the control data are inferred (S35) based on the updated joint torque limits, in addition to the updated perception, the updated state, and the one or more maximum-depth constraints generated.

8. The method according to any one of claims 1 to 7, wherein the control data are further inferred (S35), at said each cycle, based on a pullup distance constraint, in addition to the updated perception, the updated state, the one or more maximum-depth constraints (164) generated, and, if necessary, updated joint torque limits, the pullup distance constraint is preferably generated (S34) through the constraint generator (2061), and the pullup distance constraint is preferably generated (S34) so as to be relative to a current position of the bucket (110), preferably relative to a current position of a tip of the bucket, where the current position is determined in accordance with the updated state of the bucket.

9. The method according to any one of claims 1 to 8, wherein the method further comprises, prior to executing the control policy (206), training the control model (2062) on a training dataset reflecting bucket states, representations of terrains to be excavated, and associated shapes of excavation, under maximum-depth constraints (164) generated in accordance with said training dataset, in a reinforcement learning framework that rewards filling the bucket (110), and the training dataset is preferably obtained using simulations.

10. The method according to claim 9, wherein the reinforcement learning framework rewards filling the bucket (110) in a way that depends on a distance between the bucket and a pullup band, the distance estimated in accordance with the updated state of the bucket.

11. The method according to claim 9 or 10, wherein, at training the control model (2062), the maximum-depth constraints (164) are generated (S34) as maximal depths, where the maximal depths are defined with respect to a current position of the tip of the bucket (110) and extend from a lateral reference plane (PB) delimiting the tip of the bucket in a direction of a house (11) of the excavator (1).

12. The method according to any one of claims 9 to 11, wherein training the control model (2062) further comprises simulating joint velocities and updating joint torque limits of arm joints of an arm of the excavator (1) for the control model to learn to generate control data in accordance with the updated joint torque limits.

13. A computerized system for controlling a bucket (110) of an excavator (1) for autonomous excavation, wherein the system comprises: a physical interface adapted to receive signals from a set of sensors (410, 430), which preferably are on board the excavator (1); and processing means configured to process data received through the physical interface and implement all the steps of the method according to any of claims 1 to 12.

14. The system according to claim 13, wherein the set of sensors (410, 430) includes one or more sensors on board the excavator (1).

15. Computer program product for controlling a bucket (110) of an excavator (1) for autonomous excavation, the computer program product comprising a computer-readable storage medium having program instructions embodied therewith, the program instructions executable by processing means of a controller to cause the latter to perform all the steps of the method according to any one of claims 1 to 12.

Images