Excavator autonomous control apparatus and method using q-learning behavior tree enhanced by gradient boosting tree

Abstract

An autonomous excavation control device and a method thereof are disclosed. An autonomous excavation control device for hierarchically controlling autonomous excavation operations in an irregular terrain environment may be characterized by comprising: a first-level control unit that receives information of the irregular terrain environment, calculates action values ​​of a plurality of candidate operations, and establishes a work plan based on the action values; a second-level control unit that calculates excavation control parameters based on the work plan and controls the operation of excavation equipment using the excavation control parameters; and a third-level control unit that generates joint trajectories of an excavator based on the control results of the second-level control unit.

Description

Technology Field

[0001] The present invention relates to control technology for an autonomous excavator, and more specifically, to an autonomous excavation control device and method that perform work planning, motion generation, and trajectory control of an excavator using a Q-learning-based action tree (QL-BT) and reinforcement learning and imitation learning. Background Technology

[0003] Excavation is an essential task in various industrial fields, including construction, mining, and civil engineering; historically, excavation equipment has been operated relying on the experience and judgment of skilled workers. In this environment, excavation involves various variables such as topographical unevenness, changes in soil properties, and uncertainty regarding work paths, requiring continuous judgment and response from the operator.

[0004] Recently, the construction and mining industries have seen increasing interest in automation and intelligent technologies for excavation equipment to improve productivity and address labor shortages. To this end, various automation technologies, such as sensor-based environmental perception, work planning, and robot control, have been researched. In particular, technologies for path generation, work stage determination, and soil removal volume control are being treated as important for efficiently performing excavation tasks based on complex terrain information.

[0005] However, existing automation technologies often rely primarily on rule-based control, single-step control, and simple work planning methods, which presents a problem in that it is difficult to adequately reflect the complex work procedures and diverse working conditions of actual excavation sites. Furthermore, since the working environment for excavation equipment is not standardized and ground conditions or work objectives change frequently, it is known that existing methods have limitations in work optimization or real-time response.

[0006] Against this backdrop, research is continuously being conducted on planning techniques that consider various work stages and terrain changes, control techniques capable of adapting to environmental changes, and intelligent motion generation techniques based on work data for the automation of excavation equipment. However, as these technologies are intricately interconnected, significant research is still required to determine methods that simultaneously satisfy efficient work execution and stable control. Prior art literature

[0008] Korean Patent Publication No. 10-2022-0014744 The problem to be solved

[0009] The present invention has been devised to solve the aforementioned problems. The objective of the present invention is to provide an autonomous excavation control device and a method thereof that can perform work planning, motion generation, and control of an excavator more stably and efficiently by improving upon the problems that existing automation systems could not adequately address due to topographical changes in the actual excavation environment, uncertainty in soil properties, and complexity of work stages.

[0010] In addition, the objective of the present invention is to provide an intelligent decision-making structure capable of adapting to various situational changes occurring during the work process, and to realize a control technology that can flexibly adjust operation according to working conditions by learning the trajectory of an excavator based on actual work data.

[0011] However, the problems of the present invention are not limited to those mentioned above, and other unmentioned problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0013] An autonomous excavation control device for hierarchically controlling autonomous excavation work in an irregular terrain environment according to an embodiment of the present invention for achieving the above objective may be characterized by comprising: a first level control unit that receives information of the irregular terrain environment, calculates action values ​​of a plurality of candidate tasks, and establishes a work plan based on the action values; a second level control unit that calculates excavation control parameters based on the work plan and controls the operation of excavation equipment using the excavation control parameters; and a third level control unit that generates joint trajectories of an excavator based on the control results of the second level control unit.

[0014] The first level control unit may be characterized by converting input information of an unstructured terrain environment into multidimensional state information, calculating behavior values ​​(Q-values) of a plurality of candidate tasks corresponding to the state information based on Q-learning, and deriving the task plan by controlling the decision flow of a behavior tree according to the calculated behavior values.

[0015] The above-mentioned first-level control unit may be characterized by calculating the behavior value (Q-value) by approximating it using a Gradient Boosting Tree (GBT).

[0016] The first level control unit may be characterized by performing Q-learning based on a reward function that includes at least one of a behavior reward proportional to the amount of work performed, a progress reward based on terrain flatness, and a bonus reward based on whether there is a transition of the work stage.

[0017] The first level control unit may be characterized by calculating a current reward from the reward function in response to an action performed in the current state, calculating a Temporal Difference Error based on the current reward, the action value of the current state, and the maximum action value in the next state, and performing the Q-learning by updating the action value so as to minimize the Temporal Difference Error.

[0018] The above-described second-level control unit may be characterized by controlling the operation of the excavation equipment by selectively applying at least one of a reinforcement learning-based policy (Proximal Policy Optimization; PPO) or a rule-based control logic based on the excavation control parameters.

[0019] The above excavation control parameters include at least one of a work position, excavation depth, and work angle, and the second level control unit may be characterized by determining whether the work position or work angle is included in the accessible workspace of the excavation equipment, generating a validity mask according to the determination result, and applying at least one of the reinforcement learning-based policy or rule-based control logic.

[0020] The above-described third-level control unit may be characterized by learning a joint trajectory generation policy through Generative Adversarial Imitation Learning (GAIL) on pre-collected expert excavation trajectory data, generating a joint trajectory of the excavator using the learned joint trajectory generation policy, and generating the joint trajectory of the excavator by performing Adaptive Trajectory Modulation that adjusts the duration of the trajectory according to soil density.

[0021] An autonomous excavation control method for hierarchically controlling autonomous excavation work in an irregular terrain environment, performed in an autonomous excavation control device according to an embodiment of the present invention for achieving the above objective, may be characterized by comprising: a first level control step of receiving information of the irregular terrain environment, calculating action values ​​of a plurality of candidate work, and establishing a work plan based on the action values; a second level control step of calculating excavation control parameters based on the work plan and controlling the operation of excavation equipment using the excavation control parameters; and a third level control step of generating joint trajectories of an excavator based on the control results of the second level control step.

[0022] The above-described first-level control step may be characterized by converting information of an input unstructured terrain environment into multidimensional state information, calculating behavior values ​​(Q-values) of a plurality of candidate tasks corresponding to the state information based on Q-learning, and deriving the task plan by controlling the decision flow of a behavior tree according to the calculated behavior values.

[0023] The above first-level control step may be characterized by calculating the behavior value (Q-value) by approximating it with a Gradient Boosting Tree (GBT).

[0024] The above-mentioned first-level control step may be characterized by performing Q-learning based on a reward function that includes at least one of a behavior reward proportional to the amount of work performed, a progress reward based on terrain flatness, and a bonus reward based on whether there is a transition of the work stage.

[0025] The first level control step may be characterized by calculating a current reward from the reward function in response to an action performed in the current state, calculating a Temporal Difference Error based on the current reward, the action value of the current state, and the maximum action value in the next state, and performing Q-learning by updating the action value so as to minimize the Temporal Difference Error.

[0026] The above second-level control step may be characterized by controlling the operation of the excavation equipment by selectively applying at least one of a reinforcement learning-based policy (Proximal Policy Optimization; PPO) or a rule-based control logic based on the excavation control parameters.

[0027] The above excavation control parameters include at least one of a work position, excavation depth, and work angle, and the second level control step may be characterized by determining whether the work position or work angle is included in the accessible workspace of the excavation equipment, generating a validity mask according to the determination result, and applying at least one of the reinforcement learning-based policy or rule-based control logic.

[0028] The above third-level control step may be characterized by learning a joint trajectory generation policy through Generative Adversarial Imitation Learning (GAIL) on pre-collected expert excavation trajectory data, generating a joint trajectory of the excavator using the learned joint trajectory generation policy, and generating the joint trajectory of the excavator by performing Adaptive Trajectory Modulation that adjusts the duration of the trajectory according to soil density.

[0029] According to one embodiment of the present invention, it may be characterized by including a computer-readable recording medium for executing an autonomous excavation control method. Effects of the invention

[0031] According to one aspect of the present invention described above, even in an environment where there are complex terrain changes and uncertainties in soil properties during the excavation process, it is possible to perform more stable and efficient excavation work by providing an autonomous excavation control structure in which work stage determination, joint trajectory generation, and real-time control are organically linked.

[0032] Furthermore, by updating the decision-making structure in real time based on changing terrain information at the work site and requirements for each work stage, the adaptability and accuracy of the work plan can be effectively improved even in situations involving various ground conditions or work objectives.

[0033] In addition, since the joint trajectory of the excavator can be learned using actual work data and the trajectory duration can be adjusted according to the work environment, more precise and consistent excavation motion can be implemented than with a simple rule-based method.

[0034] In particular, since work planning, trajectory generation, and joint control operate with mutual feedback rather than being separated into stages, it can flexibly respond to terrain changes or fluctuations in working conditions that occur during the operation, and simultaneously ensure the reliability and stability of automated excavation work.

[0035] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description in the claims. Brief explanation of the drawing

[0037] FIG. 1 is a diagram illustrating the schematic configuration of an autonomous excavation system including an autonomous excavation control device according to one embodiment of the present invention. FIG. 2 is a diagram illustrating the schematic configuration of an autonomous excavation control device according to one embodiment of the present invention. FIG. 3 is an exemplary diagram illustrating a hierarchical control structure and data flow between each layer in an autonomous excavation control device according to an embodiment of the present invention. FIG. 4 is an example diagram illustrating the structure of an initial action tree and the process of optimizing it through reinforcement learning in an autonomous excavation control device according to an embodiment of the present invention. FIG. 5 is a flowchart illustrating an autonomous excavation control method according to an embodiment of the present invention. FIGS. 6 and 7 are experimental results for demonstrating the performance of an autonomous excavation control device according to one embodiment of the present invention. Specific details for implementing the invention

[0038] The following detailed description of the invention refers to the accompanying drawings, which illustrate specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that various embodiments of the invention are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in relation to one embodiment. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the following detailed description is not intended to be limiting, and the scope of the invention is limited only by the appended claims, including all equivalents to those claimed therein, provided appropriately described. Similar reference numerals in the drawings refer to the same or similar functions across various aspects.

[0039] The components according to the present invention are defined by functional distinction rather than physical distinction, and can be defined by the functions each performs. Each component may be implemented as hardware or as program code and processing units that perform each function, and the functions of two or more components may be included and implemented in a single component. Therefore, it should be noted that the names assigned to the components in the following embodiments are not intended to physically distinguish each component but are assigned to imply the representative function performed by each component, and that the technical concept of the present invention is not limited by the names of the components.

[0040] Preferred embodiments of the present invention will be described in more detail below with reference to the drawings.

[0041] The autonomous excavation control device (10, hereinafter the device) according to the present embodiment is provided to automatically control an excavator that performs autonomous excavation work in an irregular terrain environment, and is provided to hierarchically perform a first-level control function that establishes a work plan based on information of the irregular terrain environment, a second-level control function that calculates excavation control parameters according to the work plan and controls the operation of the excavator, and a third-level control function that generates a joint trajectory of the excavator based on the result of the second-level control.

[0042] To this end, the device (100) according to the present embodiment may include a first level control unit (110), a second level control unit (120), and a third level control unit (130). In addition, software (application) for performing an autonomous excavation control method may be installed and executed on the device (100), and the first level control unit (110), the second level control unit (120), and the third level control unit (130) may be controlled by the software (application) for performing the autonomous excavation control method.

[0043] At this time, the device (100) may be a separate terminal or a part module of the terminal. Additionally, the configuration of the first level control unit (110), the second level control unit (120), and the third level control unit (130) may be formed as an integrated module or composed of one or more modules. However, conversely, each configuration may be composed of a separate module.

[0044] Additionally, the device (10) may be mobile or fixed. This device (10) may be in the form of a server or an engine and may be referred to by other terms such as device, apparatus, terminal, UE (user equipment), MS (mobile station), wireless device, or handheld device. Furthermore, the device (100) may execute or create various software based on an operating system (OS), that is, a system. Here, the operating system is a system program that enables software to use the device's hardware, and may include all mobile computer operating systems such as Android OS, iOS, Windows Mobile OS, Bada OS, Symbian OS, BlackBerry OS, etc., as well as computer operating systems such as Windows family, Linux family, Unix family, MAC, AIX, HP-UX, etc.

[0045] Hereinafter, the configuration of an autonomous excavation control system (10) according to one embodiment of the present invention will be described in detail with reference to FIG. 1.

[0046] Referring to FIG. 1, the autonomous excavation control system (10) may include an autonomous excavation control device (100) and an excavator (E).

[0047] The autonomous excavation control device (100) can generate a control signal to automatically perform excavation work by linking with the excavator (E). The excavator (E) can receive the control signal from the autonomous excavation control device (100) and automatically perform excavation operations according to the control signal. In addition, the excavator (E) can acquire terrain information, joint angle information, and work status information required by the autonomous excavation control device (100) through sensors, etc., and provide them to the autonomous excavation control device (100).

[0048] First, an excavator (E) according to the present embodiment will be described.

[0049] The excavator (E) may include an excavator body for performing excavation work, a boom, an arm, and a bucket mounted on the body, and a hydraulic drive unit for driving them. The excavator body may be responsible for the movement and power transmission of the excavator and may include an engine, a transmission, a driving device (e.g., tracks or wheels), a steering system, a hydraulic system, a cabin, and various control instruments. The boom, arm, and bucket of the excavator (E) operate independently with respect to their respective axes of rotation and may function as key components that determine the working position, working posture, and excavation depth of the excavation equipment.

[0050] Additionally, the excavator (E) may further include a sensor module for recognizing the working environment. For example, the excavator (E) may include sensors such as LiDAR for acquiring terrain elevation information, a stereo camera, a monocular camera, a Global Positioning System (GPS), a joint angle sensor, a joint velocity sensor, and a torque sensor. These sensors may be used to configure environmental state information in the autonomous excavation control device (100) and may be utilized to accurately determine the current state of the excavator during the learning and control process. However, embodiments of the present invention are not limited thereto.

[0051] In one embodiment of the present invention, at least some of the components of the autonomous excavation control device (100) may be mounted on the main body or upper structure of the excavator (E). For example, a control processor, a communication module, some environmental perception sensors, or a storage device may be mounted on the top of the excavator main body, inside the driver's seat, the front frame, or the rotating upper structure. However, the embodiment of the present invention is not limited to such specific mounting locations, and the configuration of the autonomous excavation control device (100) may be provided in the form of an external device as needed.

[0052] Hereinafter, with reference to FIGS. 2 to 4, the configuration of an autonomous excavation control device (100) for hierarchically controlling autonomous excavation work in an irregular terrain environment according to one embodiment of the present invention will be described in detail.

[0053] An apparatus (100) according to one embodiment of the present invention may be characterized by including: a first level control unit (110) that receives information of the irregular terrain environment, calculates action values ​​of a plurality of candidate tasks, and establishes a work plan based on the action values; a second level control unit (120) that calculates excavation control parameters based on the work plan and controls the operation of excavation equipment using the excavation control parameters; and a third level control unit (130) that generates joint trajectories of an excavator based on the control results of the second level control unit.

[0054] The autonomous excavation control device (100) according to the present embodiment is hierarchically divided into a first level control unit (110), a second level control unit (120), and a third level control unit (130) as described above, thereby allowing the excavation work to be processed by dividing it into a work plan establishment stage, an excavation control parameter calculation and motion control stage, and a joint trajectory generation stage. Accordingly, the first level control unit (110) can focus on a high-level work plan to determine which of a plurality of candidate tasks, such as excavation, loading, embankment, and compaction, to perform based on information of an irregular terrain environment, the second level control unit (120) can focus on low-level control to control the actual operation of the excavation equipment by receiving the work plan as input and calculating excavation control parameters related to the excavation position, excavation depth, and work angle, and the third level control unit (130) can be responsible for motion level control to generate joint trajectories for each joint of the excavator based on the control results of the second level control unit (120).

[0055] Through such hierarchical decomposition, the control units at each level can exchange information through mutually defined interfaces, while also enabling independent design and optimization for each level. That is, even if the work planning logic of the first level control unit (110) is changed or improved, the impact on the detailed control structure of the second level control unit (120) and the third level control unit (130) can be minimized, and conversely, even if the joint trajectory generation algorithm or the adjustment method of the excavation control parameters is changed, the decision-making structure of the higher level can be maintained. Therefore, the hierarchical structure according to the present embodiment facilitates stepwise development and debugging at the module level, and allows for sequential integration and verification of each control layer even when applied to the actual field, thereby improving the stability and reliability of the system.

[0056] The first level control unit (110) may be characterized by converting input information of an unstructured terrain environment into multidimensional state information, calculating behavior values ​​(Q-values) of a plurality of candidate tasks corresponding to the multidimensional state information based on Q-learning, and controlling the decision flow of a behavior tree according to the calculated behavior values ​​to derive the task plan.

[0057] Here, an irregular terrain environment refers to an environment where ground slope, elevation difference, distribution of soil, and terrain changes occurring during the work process exist unpredictably, such as natural terrain or artificially damaged terrain. In such an environment, even at the same location, the terrain shape can rapidly change over time due to soil movement, and it is difficult to derive optimal work decisions using only fixed rule-based control. Therefore, the first level control unit (110) is configured to configure multidimensional state information to reflect such environmental uncertainty and to generate a work plan in a data-based manner.

[0058] The first level control unit (110) can construct multidimensional state information based on information of the irregular terrain environment. For example, the first level control unit (110) can discretize the physical characteristics of the irregular terrain environment into a 7-dimensional state space and express the multidimensional state information as a state vector as shown in Equation 1 below.

[0059] [Mathematical Formula 1]

[0060]

[0061] Here, S represents the discrete state space, and is a state variable representing the mean absolute height of the terrain, and is an indicator for quantifying the overall difference in elevation or height level of the terrain. The value range of can be set in various ways depending on terrain measurement resolution, sensor configuration, work area characteristics, etc., and in one embodiment of the present invention, a discrete interval can be configured with integer values ​​from 0 to 4.

[0062] is a terrain standard deviation state variable that indicates the degree of surface irregularity or roughness level of the terrain. The number of discrete intervals and the range of values ​​can be adjusted according to the control purpose, terrain complexity, data processing method, etc., and in one embodiment of the present invention, they can be set to integer values ​​from 0 to 3.

[0063] is a state variable representing excavatable material volumes, used to quantify the amount of soil present in a specific area or adjacent area. The discrete step can be arbitrarily set according to soil volume distribution characteristics or equipment capacity, etc., and in one embodiment of the present invention, it can be configured with integer values ​​from 0 to 2.

[0064] is a volume balance ratio state variable that indicates the degree of balance between the excavation volume and the embankment volume. It can be expressed as a continuous value or multiple discrete steps, and in one embodiment of the present invention, three discrete values ​​such as -1, 0, and 1 can be used.

[0065] is a state variable indicating which work phase the excavation work corresponds to. Depending on the work flow definition, the number of phases may be expanded or contracted, and in one embodiment of the present invention, the work phases may be expressed as integer values ​​from 0 to 3.

[0066] may be a state variable representing an imbalance type existing in the left-right or front-back directions of the terrain. How the magnitude and direction of the imbalance are expressed can be selected design-wise, and in one embodiment of the present invention, it can be distinguished as discrete values ​​of -1, 0, and 1.

[0067] The value ranges and number of intervals of the above state variables can be adjusted in various ways depending on the system environment, sensor configuration, control policy, learning strategy, etc., and the intervals presented in one embodiment of the present invention are merely examples.

[0068] The first level control unit (110) can learn the action value (Q-value) of candidate tasks by performing Q-learning. Q-learning is a model-free reinforcement learning technique that does not require an environment model, and can learn an optimal policy by repeatedly updating the action value (Q-value), which is a value that predicts the total sum of future rewards when a specific task is performed in a specific state. This Q-learning technique can be effectively used in environments where terrain changes are irregular and results accumulate over time, such as in excavation work.

[0069] Here, the action value (Q-value) is a value representing the cumulative expected value of current and future rewards when work a is performed in state s, and is a key indicator used by the first level control unit (110) when establishing a work plan. For example, if a large amount of soil can be removed from the current area, the Q-value of “Dig” may increase, and if the terrain approaches a balanced state, the Q-value of “Level” may increase.

[0070] The first level control unit (110) may be characterized by performing Q-learning based on a reward function that includes at least one of a behavior reward proportional to the amount of work performed, a progress reward according to terrain flatness, and a bonus reward according to whether there is a transition of the work stage.

[0071] For example, compensation It can be defined as shown in mathematical formula 2 below.

[0072] [Mathematical Formula 2]

[0073]

[0074] Here, represents the next state reached after task a is performed from the current state s, and the state transition may be determined by the results of the excavation work, such as terrain changes, soil movement, and work progress. The configuration method and definition of state transitions can be set in various ways depending on the application environment and control policy.

[0075] Behavioral reward It can be defined as shown in mathematical formula 3 below.

[0076] [Mathematical Formula 3]

[0077]

[0078] Here The basic reward, is the soil volume scaling coefficient, is the volume of treated soil.

[0079] Progress rewards It can be defined as shown in the following mathematical formula 4.

[0080] [Mathematical Formula 4]

[0081]

[0082] Here, and represents the weighting coefficients assigned to the degree of improvement in Mean Absolute Height (MAH) and the degree of improvement in Terrain Standard Deviation (STD), respectively. In addition, and is a value representing the amount of reduction in MAH and STD due to the performance of work, and is an indicator to quantitatively express the degree of contribution to the flattening of the terrain or the alleviation of elevation differences.

[0083] Bonus reward It can be defined as shown in the following mathematical formula 5.

[0084] [Mathematical Formula 5]

[0085]

[0086] Here, inside is a value (coefficient) for determining the magnitude of a reward term granted when specific reward conditions are met, and is a constant for adjusting the scale of the reward according to the results of a state transition or task execution. The specific method of setting the above values ​​may vary depending on the application environment, control purpose, or learning policy.

[0087] The above-mentioned first-level control unit (110) may be characterized by calculating the behavior value (Q-value) by approximating it with a Gradient Boosting Tree (GBT).

[0088] Gradient Boosting Tree (GBT) is a model that progressively improves prediction performance by sequentially combining multiple regression trees, enabling high prediction accuracy in excavation environments with non-linear and multidimensional state information. In particular, GBT has the advantage of effectively reflecting complex irregular terrain information through the continuous representation of input features, even when the state space consists of discrete values, and ensuring stability in the learning process due to a structure that corrects the prediction errors of individual trees stepwise. Furthermore, by using an independent GBT model for each action, decision-making tendencies for each candidate action can be learned individually, thereby providing improved Q-value approximation performance in excavation environments with different action characteristics. Meanwhile, in irregular terrain environments, the number of state-action combinations has a very large dimension, making it virtually impossible to construct Q(s,a) in a table form; accordingly, in one embodiment of the present invention, the model may be configured to predict the action value using function approximation.

[0089] Each task Independent GBT models for Q-value approximation It can be used, and the GBT-based Q-value approximation can be defined as Equation 6 below.

[0090] [Mathematical Formula 6]

[0091]

[0092] Here represents a feature extraction function that generates continuous features from state s, and is a function for converting discrete state information into a continuous representation that the GBT model can learn. Also, represents the k-th regression tree corresponding to task a, and It can represent a coefficient that controls the learning rate.

[0093] The first level control unit (110) may be characterized by calculating a current reward from the reward function in response to an action performed in the current state, calculating a temporal difference error based on the current reward, the action value of the current state, and the maximum action value in the next state, and performing the Q-learning by updating the action value so as to minimize the temporal difference error.

[0094] Here, the time difference error refers to the difference between the current action value and the target action value considering future rewards, and serves as a key criterion for updating the value function in Q-learning.

[0095] The above-mentioned first-level control unit (110) can calculate a temporal-difference target value during the update process of Q-learning, which can be defined as Equation 7 below.

[0096] [Mathematical Formula 7]

[0097]

[0098] Here, r is the reward obtained after performing task a from the current state s, is the next state reached after performing a task, represents the discount factor.

[0099] The aforementioned time-lag target value plays a role in enabling the behavioral value (Q-value) to reflect long-term work efficiency by combining the maximum expected values ​​of current and future rewards.

[0100] The GBT model is trained to minimize the error between the above-mentioned time difference target value and the action value predicted by the current GBT, and the regression tree can be iteratively updated based on the loss function of Equation 8 below.

[0101] [Mathematical Formula 8]

[0102]

[0103] Here, D is the experience dataset (transition samples), represents the Q-value prediction of the current GBT model for task a.

[0104] The above loss function is defined in a way that minimizes the mean squared error between the temporal difference target value and the model prediction value, and This is the Temporal Difference Error.

[0105] The first level control unit (110) can perform learning to gradually reduce the time difference error through the stepwise tree complement structure of the GBT.

[0106] In addition, since GBT is configured with a boosting method in which each tree corrects the residuals of the previous step model, it can provide stable action value approximation performance even in environments with high nonlinearity in the state space and complex task characteristics, such as irregular terrain environments.

[0107] The first level control unit (110) can apply Prioritized Experience Replay for the efficient utilization of training data. Experience Replay Buffer In the state , action , compensation , next state Transition data (transition samples) such as the above are stored, and each transition data has a priority value as shown in Equation 9 below. It can be granted.

[0108] [Mathematical Formula 9]

[0109]

[0110] Here, is a constant that guarantees all samples have a priority of 0 or greater, and is an exponential constant that controls the priority strength, the intensity with which the magnitude of the time difference error is reflected in the priority. represents the temporal-difference error of the corresponding transition data and can be defined as shown in Equation 10 below.

[0111] [Mathematical Formula 10]

[0112]

[0113] Here represents the difference between the current Q-value prediction and the time difference target value (Q-target), and the larger the absolute value of the time difference error, the greater the need for that experience to have a greater impact on learning.

[0114] The first level control unit (110) can calculate the probability that transition data i is selected as in the following mathematical formula 11.

[0115] [Mathematical Formula 11]

[0116]

[0117] Here, the denominator is the priority sum of all transition data, and It adjusts so that high-priority transition data is sampled with a higher probability.

[0118] Additionally, to offset sampling bias, importance weights such as Equation 12 below It can be applied.

[0119] [Mathematical Formula 12]

[0120]

[0121] Here, N is the number of samples in the buffer, is a constant that starts at 0 at the beginning of training and gradually increases to 1, and the importance weight It plays the role of correcting bias that may occur during the priority sampling process.

[0122] FIG. 4 is an example diagram illustrating the structure and optimization process of a behavior tree used in a first-level control unit (110) according to an embodiment of the present invention.

[0123] An initial behavior tree (Initial BT) is shown at the top of Fig. 4. The initial behavior tree is constructed based on domain knowledge and consists of condition nodes (C) and action nodes (action nodes) corresponding to the work phases (Initial Phase, Middle Phase, Final Phase).

[0124] The lower part of FIG. 4 illustrates a procedure for optimizing an initial behavior tree during a reinforcement learning process. The first level control unit (110) can replace existing condition nodes (C) with Q-value-based condition nodes (Q-condition nodes) as it learns state-specific behavior values ​​(Q-values) through Q-learning, and can rearrange the placement order of each task node according to the magnitude of the behavior value.

[0125] Through this, the action value (Q-value) obtained during the reinforcement learning process is directly reflected in the decision flow of the action tree, thereby constructing an optimized action tree (Optimized BT) that can establish a more efficient work plan according to terrain characteristics and work progress.

[0126] In addition, as shown in Fig. 4, action nodes are classified according to work characteristics such as balance adjustment, STD reduction, middle compacting, and residual volume, and work priorities are dynamically reconfigured based on Q-values.

[0127] The above-described first-level control unit (110) can automatically generate condition nodes of a behavior tree using learned behavior values ​​(Q-values). This is a procedure for configuring the execution conditions of candidate tasks based on data, wherein if the Q-value in state s is greater than or equal to a specific threshold value, the state is a set of condition nodes. It can be included in. That is, even if the initial behavior tree is constructed based on domain knowledge, it is a method in which the condition judgment structure is dynamically reconstructed by reflecting the learned Q-value.

[0128] The creation of a condition node can be defined as shown in Equation 13 below.

[0129] [Mathematical Formula 13]

[0130]

[0131] Here can be set to the upper p-percentile of the Q-value distribution, and the p-value is generally selected from the 70% to 90% range. This serves as a criterion for extracting only the states where a specific task exhibits high efficiency as condition nodes. This configuration enables the learned policy to automatically distinguish valid states of the task, allowing the behavior tree to respond adaptively to changes in the environment.

[0132] In addition, actions can be reordered according to the maximum Q-value of each candidate state, which can be defined as Equation 14 below.

[0133] [Mathematical Formula 14]

[0134]

[0135] Here Q represents the execution priority of child nodes within the behavior tree, and tasks with higher Q-values ​​are configured to have higher priorities. Therefore, even in situations where conflicts occur between tasks, the most efficient order of tasks can be automatically determined based on Q-values.

[0136] Through this, the behavior tree can be automatically reconstructed to reflect the situational validity of tasks, enabling adaptive task switching based on the state. In other words, since Q-values ​​learned through reinforcement learning are directly reflected in the structural optimization of the behavior tree, the initial manual-based tree structure can be progressively advanced in a data-driven manner.

[0137] The above second level control unit (120) may be characterized by controlling the operation of the excavation equipment by selectively applying at least one of a reinforcement learning-based policy (Proximal Policy Optimization; PPO) or a rule-based control logic based on excavation control parameters.

[0138] The above excavation control parameters include at least one of a work position, excavation depth, and work angle, and the second level control unit (120) may be characterized by determining whether the work position or work angle is included in the accessible workspace of the excavation equipment, and generating a validity mask according to the determination result to apply at least one of the reinforcement learning-based policy or rule-based control logic.

[0139] The second level control unit (120) receives excavation control parameters and calculates a policy to determine the low-level operation of the excavator (E). In this process, a reinforcement learning technique may be applied. Reinforcement learning is a machine learning method that learns a policy to maximize long-term accumulated rewards by utilizing the reward obtained according to the action selected in the state. The reinforcement learning-based policy applied to the second level control unit (120) may be Proximal Policy Optimization (PPO). PPO is a policy gradient-based algorithm that directly approximates the policy and performs clipped policy updates for stable policy updates. Accordingly, abrupt changes in the policy are suppressed, and control suitable for the operational characteristics of the excavation equipment requiring continuous control signals becomes possible.

[0140] The second level control unit (120) can configure an action space including various operation parameters according to the characteristics of the excavation work. For example, when performing an excavation work, the excavation radius ( ), excavation depth( ), excavation angle( A set of operations such as the following mathematical formula 15 can be constructed using parameters such as ).

[0141] [Mathematical Formula 15]

[0142]

[0143] Here represents the excavation radius (radial distance), and the minimum and maximum radii It can be defined within a range. means extraction depth and It can be set as , . represents the working angle (angular position) and can be set within the range of [0°, 180°].

[0144] In the case of a soil dumping operation, the second level control unit (120) can generate a set of operations according to the following mathematical formula 16.

[0145] [Mathematical Formula 16]

[0146]

[0147] Also, dumping volume It can be received and set from the first level control unit (110), which is a higher control stage, and this It can be determined within the range.

[0148] The second level control unit (120) can perform an environment modeling procedure to maintain the physical reality of the terrain by considering the characteristic that the terrain where the excavation work is performed is continuously deformed in an irregular environment. For example, the height of a specific grid cell (i,j) is a set of neighboring cells A smoothing operation can be applied as shown in Equation 17 below, reflecting the state of.

[0149] [Mathematical Formula 17]

[0150]

[0151] Here, κ is the smoothing coefficient, and represents the neighboring cells surrounding the cell.

[0152] The above-mentioned second-level control unit (120) can generate a validity mask that reflects the constraints of the workspace so that the excavation operation can be physically realized. The validity mask indicates whether a specific combination of work parameters is included in the actual workspace (accessible workspace) of the excavator and can be defined as shown in Equation 18 below.

[0153] [Mathematical Formula 18]

[0154]

[0155] Here, R represents the accessible workspace region that the excavator can actually reach, and is an indicator function that outputs 1 if the condition is true and 0 if it is false. In other words, the validity mask enables stable control by eliminating physically impossible combinations of operations.

[0156] The second level control unit (120) can configure individual compensation functions according to the type of work. For example, in the case of digging work, compensation considering the amount of excavated soil, movement efficiency, and whether the work is completed can be defined as in Equation 19 below.

[0157] [Mathematical Formula 19]

[0158]

[0159] Here is the amount of excavated soil It is a reward proportional to, and is a reward intended to deter unnecessary equipment movement. and is a weighting factor for controlling excavation efficiency and movement costs, etc.

[0160] In the case of soil dumping operations, compensation for minimizing residual volume and improving input efficiency can be defined as shown in Equation 20 below.

[0161] [Mathematical Formula 20]

[0162]

[0163] Here is a coefficient that controls filling efficiency, and is a penalty weighting factor for suppressing residual waste. represents the actual amount of filled soil.

[0164] The second level control unit (120) may optionally apply rule-based control logic in addition to reinforcement learning-based policies. Rule-based control logic is a method of applying predefined control rules in response to specific conditions, and serves to ensure stable operation in situations where immediate and safe response is difficult with only reinforcement learning-based policies, such as when excavation equipment approaches a mechanical limit or when a sensor error is detected.

[0165] The second level control unit (120) may optionally apply rule-based control logic in addition to reinforcement learning-based policies. Rule-based control logic is a method of applying predefined control rules in response to specific conditions, and serves to ensure stable operation in situations where immediate and safe response is difficult with only reinforcement learning-based policies, such as when excavation equipment approaches a mechanical limit or when a sensor error is detected.

[0166] In this way, the second level control unit (120) can perform stable and efficient excavation operations even in complex, irregular terrain environments by selectively mixing reinforcement learning-based policies and rule-based control logic depending on the situation.

[0167] Additionally, the second level control unit (120) can dynamically select a reinforcement learning-based policy and a rule-based control logic depending on the nature of the work. For example, in work stages requiring continuous control optimization, such as excavation, loading, and embankment, a reinforcement learning-based policy (PPO) can be applied preferentially, while conversely, in deterministic tasks where the action result is clearly defined, such as returning a bucket to its initial position, preventing equipment from going outside a limit angle, or simple mechanical repositioning, a rule-based control logic can be applied.

[0168] Furthermore, if the excavation control parameter exceeds the physical access range of the equipment, the second level control unit (120) may be configured to generate a validity mask to remove components of the control signal calculated from the reinforcement learning-based policy that exceed the safe area, and to switch to rule-based control logic if necessary.

[0169] In this way, the second level control unit (120) can perform stable and efficient excavation operations even in complex, irregular terrain environments by selectively mixing reinforcement learning-based policies and rule-based control logic depending on the situation.

[0170] The third level control unit (130) may be characterized by learning a joint trajectory generation policy through Generative Adversarial Imitation Learning (GAIL) on pre-collected expert excavation trajectory data, generating a joint trajectory of the excavator using the learned joint trajectory generation policy, and generating a joint trajectory of the excavator by performing Adaptive Trajectory Modulation that adjusts the duration of the trajectory according to soil density.

[0171] Expert excavation trajectory data refers to time-series data such as joint angles, end-effector paths, and work progress stages obtained during the process of a skilled operator actually operating an excavator. This data includes a series of motion patterns of the excavation operation, soil removal methods, and equipment operation characteristics, and serves as a standard for learning the operation policy of an autonomous excavator.

[0172] The third level control unit (130) can perform Generative Adversarial Imitation Learning (GAIL) based on the excavation trajectory data of the expert. Imitation learning is a method of learning a policy to simulate the distribution of the expert's movements, and is a technique for learning a policy to approximate the expert's movement patterns and statistical characteristics through a structure in which a generator and a discriminator compete with each other. Accordingly, the third level control unit (130) can secure the ability to generate joint trajectories at an expert level without designing an explicit compensation function.

[0173] Additionally, the third level control unit (130) generates the joint trajectory of the excavator based on a learned joint trajectory generation policy, and can perform adaptive trajectory modulation. Adaptive trajectory modulation is a method of dynamically adjusting the temporal progression speed or duration of the joint trajectory by reflecting physical characteristics of the work environment, such as soil density. It is configured to simultaneously improve work efficiency and equipment stability by adjusting the progression of the trajectory to slow down in harder soil or to speed up in relatively soft soil. This adjustment method can be stably applied even to changes in the excavation environment, thereby ensuring the continuity and naturalness of the actual excavation work.

[0174] The above third-level control unit (130) can receive the excavation control parameters calculated by the second-level control unit (120) and convert them into joint space commands using inverse kinematics. For example, work parameters is the target joint angle as shown in the following mathematical formula 21 ( It can be converted into ).

[0175] [Mathematical Formula 21]

[0176]

[0177] Here is the working radius, is excavation depth, represents the working angle, and represents a value that includes the target rotation angle that each joint of the boom, arm, and bucket must reach.

[0178] The above third-level control unit (130) can define a state space such as the following mathematical formula 22 by constructing a state vector including joint target values ​​converted into inverse kinematics and work-related variables.

[0179] [Mathematical Formula 22]

[0180]

[0181] Here, is the current joint angle, is the trajectory progression rate (normalized time), is the target excavation depth, is soil density, is the normalized coordinate of the end-effector location, represents the angle tracking error.

[0182] The above third-level control unit (130) generates a joint velocity-based motion, and the motion can be defined as in Equation 23 below.

[0183] [Mathematical Formula 23]

[0184]

[0185] This refers to the target speed for the three main joints of the excavator.

[0186] In addition, the third-level control unit (130) can learn a trajectory generation policy by performing imitation learning based on expert trajectories. For example, expert trajectory samples It can be augmented with added noise as shown in mathematical formula 24 below.

[0187] [Mathematical Formula 24]

[0188]

[0189] Here, represents the variance of the Gaussian noise per joint.

[0190] The above third-level control unit (130) applies adaptive trajectory modulation, so that soil density ( The duration of the trajectory can be adjusted according to ). For example, if the density of the ground to be excavated increases, the trajectory execution time can be set as shown in Equation 25 below.

[0191] [Mathematical Formula 25]

[0192]

[0193] Here is the basic trajectory time, is the density change sensitivity coefficient, is the minimum value of soil density, represents the maximum value of soil density.

[0194] In addition, the third level control unit (130) can check workspace constraints such as the following mathematical formula 26 to ensure that joint driving does not violate kinematic constraints.

[0195] [Mathematical Formula 26]

[0196]

[0197] Here is the forward kinematics function, means the workspace that the end effector can reach.

[0198] Coordinate normalization can be performed as shown in Equation 27 below.

[0199] [Mathematical Formula 27]

[0200]

[0201] By using such a normalized expression, the change in the position of the end effector of the excavator (E) is expressed on a constant scale, so the third level control unit (130) can predict a stable trajectory and generate motion under various terrain conditions and work objectives. In particular, by applying adaptive trajectory modulation according to soil density, the duration of the trajectory can be adjusted even when the excavation intensity and work load change, thereby realizing a natural work flow.

[0202] As a result, the third level control unit (130) can generate a trajectory that accurately reflects the actual working characteristics of the excavator by combining an imitation learning-based joint trajectory generation policy and a soil density-based adaptive control function, which can significantly contribute to improving work efficiency and stability throughout the entire excavation process. With the above configuration, the autonomous excavation control device of the present invention can automatically perform optimal excavation work even in an irregular terrain environment through a hierarchical control structure in which upper, middle, and lower controls are organically linked.

[0203] FIG. 5 is a flowchart for explaining an autonomous excavation control method according to an embodiment of the present invention. Since the autonomous excavation control method according to an embodiment of the present invention proceeds on substantially the same configuration as the autonomous excavation control device (100) shown in FIG. 2, the same reference numerals are assigned to the same components as those in FIG. 2 (100), and repetitive descriptions are omitted.

[0204] An autonomous excavation control method for hierarchically controlling autonomous excavation work in an irregular terrain environment, performed by an autonomous excavation control device (100), may be characterized by including: a first level control step (S100) of receiving information of the irregular terrain environment, calculating action values ​​of a plurality of candidate work, and establishing a work plan based on the action values; a second level control step (S100) of calculating excavation control parameters based on the work plan and controlling the operation of excavation equipment using the excavation control parameters; and a third level control step (S300) of generating joint trajectories of an excavator based on the control results of the second level control step.

[0205] The above-described first-level control step (S100) may be characterized by converting input information of an unstructured terrain environment into multidimensional state information, calculating behavior values ​​(Q-values) of a plurality of candidate tasks corresponding to the multidimensional state information based on Q-learning, and deriving the task plan by controlling the decision flow of a behavior tree according to the calculated behavior values.

[0206] The above first level control step (S100) may be characterized by calculating the behavior value (Q-value) by approximating it with a Gradient Boosting Tree (GBT).

[0207] The above-mentioned first level control step (S100) may be characterized by performing Q-learning based on a reward function that includes at least one of a behavior reward proportional to the amount of work performed, a progress reward based on terrain flatness, and a bonus reward based on whether there is a transition of the work step.

[0208] The first level control step (S100) may be characterized by calculating a current reward from the reward function in response to an action performed in the current state, calculating a Temporal Difference Error based on the current reward, the action value of the current state, and the maximum action value in the next state, and performing Q-learning by updating the action value so as to minimize the Temporal Difference Error.

[0209] The above second level control step (S200) may be characterized by controlling the operation of the excavation equipment by selectively applying at least one of a reinforcement learning-based policy (Proximal Policy Optimization; PPO) or a rule-based control logic based on the excavation control parameters.

[0210] The above third level control step (S300) may be characterized by learning a joint trajectory generation policy through Generative Adversarial Imitation Learning (GAIL) on pre-collected expert excavation trajectory data, generating a joint trajectory of the excavator using the learned joint trajectory generation policy, and generating the joint trajectory of the excavator by performing Adaptive Trajectory Modulation that adjusts the duration of the trajectory according to soil density.

[0211] The autonomous excavation control method of the present invention, as described above, can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either individually or in combination.

[0212] The program instructions recorded on the above-mentioned computer-readable recording medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software.

[0213] Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, and flash memory.

[0214] Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

[0215] The device (100) according to the present invention provides an autonomous excavation control structure in which work stage determination, joint trajectory generation, and real-time control are organically linked, even in an environment where there are complex terrain changes and uncertainty in soil properties during the excavation process, thereby enabling more stable and efficient excavation work.

[0216] Furthermore, by updating the decision-making structure in real time based on changing terrain information at the work site and requirements for each work stage, the adaptability and accuracy of the work plan can be effectively improved even in situations involving various ground conditions or work objectives.

[0217] In addition, since the joint trajectory of the excavator can be learned using actual work data and the trajectory duration can be adjusted according to the work environment, more precise and consistent excavation motion can be implemented than with a simple rule-based method.

[0218] In particular, since work planning, trajectory generation, and joint control operate with mutual feedback rather than being separated into stages, it can flexibly respond to terrain changes or fluctuations in working conditions that occur during the operation, and simultaneously ensure the reliability and stability of automated excavation work.

[0219] FIGS. 6 and FIGS. 7 are experimental results for demonstrating the performance of a device (100) according to one embodiment of the present invention for demonstrating the effects described above.

[0220] Figure 6 is an example diagram showing the results of a reinforcement learning experiment performed to evaluate learning performance regarding the high-level planning stage.

[0221] The experiment was conducted in a 20×40 cell-based discrete grid excavation environment, with each cell configured to have a height value ranging from -30 to +30. To ensure statistical reliability, training was configured to run 100 episodes for both the fixed-seed and randomized environments. A total of 2,000 training episodes were used in the experiment, and an ε-greedy policy was applied with an exploration rate that gradually decreased from 0.3 to 0.1. Additionally, for the GBT-based model, 50 regression trees were configured at a depth of 3 and used in an ensemble form, with retraining performed every 200 episodes.

[0222] As shown in Figure 6, traditional table-based Q-learning tended to reach a stable level of reward values ​​around 1,200 episodes. On the other hand, the GBT-based Q-value approximation technique showed temporary performance fluctuations in the 1,500–1,800 episode range, but showed a pattern of gradually stabilizing performance as learning continued.

[0223] In addition, the model combining GBT with the replay technique showed a stable reward increase trend throughout the learning process and maintained relatively uniform performance, particularly in sections with a high risk of overfitting. This demonstrates that the replay technique contributes to improving the stability of value propagation and policy learning.

[0224] Differences were also observed in the results of state-space exploration after training was completed. The GBT-based model was found to explore approximately 7–8% more states compared to table-based Q-learning, with the highest average Q-value measured at 148.21 and the highest positive Q-value ratio recorded at 74.2%.

[0225] This result demonstrates that the GBT-based approximation method more effectively propagates the learned value function across a wide state space and exhibits superior generalization ability across various states.

[0226] FIG. 7 shows the experimental results of evaluating the dynamic adaptability of action selection according to the terrain state for the action selection mechanism of the first level control unit according to one embodiment of the present invention.

[0227] This experiment was conducted to analyze which task each learning method prioritizes under conditions where the mean absolute height (MAH) value, corresponding to the roughness or height deviation of the excavation terrain, changes. All models exhibited characteristics of being sensitive to terrain roughness, and particularly under rough terrain conditions where the MAH was 4 or higher, all models showed a tendency to select the dig+dump task nearly 100% of the time.

[0228] However, significant differences were observed between models under intermediate terrain conditions at MAH level 2. The GBT-Replay configuration allocated approximately 60–80% of the total selections to dig work, demonstrating a tendency to perform active excavation even in intermediate conditions. In contrast, other models maintained a dig work ratio of less than 5% under the same conditions, indicating significantly lower responsiveness to intermediate terrain conditions. Furthermore, regarding the selection ratio of compacting work, GBT-Replay adaptively varied across a wide range of 20–90%, whereas other models tended to remain fixed within the 70–90% range, showing a difference in adaptability.

[0229] Meanwhile, the GBT-Replay configuration demonstrated the highest performance in average rewards with 660.74, and reward variance was also = 89.4, which was the lowest, maintaining stable performance. This can be interpreted as proving that the configuration of the present invention can secure high adaptability and stable policy execution capabilities even under various terrain conditions.

[0230] Although various embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention. Explanation of the symbols

[0234] 10: Autonomous Excavation Control System E: Excavator 100: Autonomous Excavation Control Device 110: Level 1 control unit 120: Level 2 control unit 130: Third-level control unit

Claims

Claim 1 An autonomous excavation control device for hierarchically controlling autonomous excavation operations in an irregular terrain environment, comprising: a first-level control unit that receives information of the irregular terrain environment, calculates action values ​​of a plurality of candidate operations, and establishes a work plan based on the action values; a second-level control unit that calculates excavation control parameters based on the work plan and controls the operation of excavation equipment using the excavation control parameters; and a third-level control unit that generates a joint trajectory of an excavator based on the control result of the second-level control unit; wherein the third-level control unit learns a joint trajectory generation policy through Generative Adversarial Imitation Learning (GAIL) of pre-collected expert excavation trajectory data, generates a joint trajectory of an excavator using the learned joint trajectory generation policy, and generates the joint trajectory of the excavator by performing Adaptive Trajectory Modulation that adjusts the duration of the trajectory according to soil density. Claim 2 An autonomous excavation control device according to claim 1, wherein the first level control unit converts input information of an unstructured terrain environment into multidimensional state information, calculates behavior values ​​(Q-values) of a plurality of candidate tasks corresponding to the multidimensional state information based on Q-learning, and derives the work plan by controlling the decision flow of a behavior tree according to the calculated behavior values. Claim 3 An autonomous excavation control device according to claim 2, wherein the first level control unit calculates the action value (Q-value) by approximating it with a Gradient Boosting Tree (GBT). Claim 4 An autonomous excavation control device according to claim 3, wherein the first level control unit performs Q-learning based on a compensation function comprising at least one of an action compensation proportional to the amount of work performed, a progress compensation according to terrain flatness, and a bonus compensation according to whether there is a transition of the work stage. Claim 5 An autonomous excavation control device according to claim 4, wherein the first level control unit calculates a current reward from the reward function in response to an action performed in the current state, calculates a Temporal Difference Error based on the current reward, the action value of the current state, and the maximum action value in the next state, and performs Q-learning by updating the action value so as to minimize the Temporal Difference Error. Claim 6 An autonomous excavation control device according to claim 5, wherein the second level control unit controls the operation of the excavation equipment by selectively applying at least one of a reinforcement learning-based policy (Proximal Policy Optimization; PPO) or a rule-based control logic based on the excavation control parameters. Claim 7 An autonomous excavation control device according to claim 6, wherein the excavation control parameter includes at least one of a work position, an excavation depth, and a work angle, and the second level control unit determines whether the work position or the work angle is included in the accessible workspace of the excavation equipment, generates a validity mask according to the determination result, and applies at least one of the reinforcement learning-based policy or rule-based control logic. Claim 8 delete Claim 9 An autonomous excavation control method for hierarchically controlling autonomous excavation operations in an irregular terrain environment, performed by an autonomous excavation control device, comprising: a first-level control step of receiving information of the irregular terrain environment, calculating action values ​​of a plurality of candidate operations, and establishing a work plan based on the action values; and a second-level control step of calculating excavation control parameters based on the work plan and controlling the operation of excavation equipment using the excavation control parameters. The autonomous excavation control method comprises: a third level control step for generating a joint trajectory of an excavator based on the control result of the second level control step; wherein the third level control step learns a joint trajectory generation policy through Generative Adversarial Imitation Learning (GAIL) of pre-collected expert excavation trajectory data, generates a joint trajectory of an excavator using the learned joint trajectory generation policy, and generates the joint trajectory of the excavator by performing Adaptive Trajectory Modulation that adjusts the duration of the trajectory according to soil density. Claim 10 In claim 9, the first level control step is characterized by converting input information of an unstructured terrain environment into multidimensional state information, calculating behavior values ​​(Q-values) of a plurality of candidate tasks corresponding to the multidimensional state information based on Q-learning, and controlling the decision flow of a behavior tree according to the calculated behavior values ​​to derive the work plan, thereby forming an autonomous excavation control method. Claim 11 An autonomous drilling control method according to claim 10, wherein the first level control step calculates the behavior value (Q-value) by approximating it with a Gradient Boosting Tree (GBT). Claim 12 An autonomous excavation control method according to claim 11, wherein the first level control step is characterized by performing Q-learning based on a compensation function comprising at least one of a behavioral reward proportional to the amount of work performed, a progress reward according to terrain flatness, and a bonus reward according to whether there is a transition of the work step. Claim 13 In claim 12, the first level control step is characterized by calculating a current reward from the reward function in response to an action performed in the current state, calculating a Temporal Difference Error based on the current reward, the action value of the current state, and the maximum action value in the next state, and performing Q-learning by updating the action value so as to minimize the Temporal Difference Error, in an autonomous excavation control method. Claim 14 An autonomous excavation control method according to claim 13, wherein the second level control step controls the operation of the excavation equipment by selectively applying at least one of a reinforcement learning-based policy (Proximal Policy Optimization; PPO) or a rule-based control logic based on the excavation control parameters. Claim 15 An autonomous excavation control method according to claim 14, wherein the excavation control parameter comprises at least one of a work position, an excavation depth, and a work angle, and the second level control step determines whether the work position or the work angle is included in the accessible workspace of the excavation equipment, generates a validity mask according to the determination result, and applies at least one of the reinforcement learning-based policy or rule-based control logic. Claim 16 delete Claim 17 A computer-readable recording medium for implementing the autonomous excavation control method according to paragraph 9.

Images