Autonomous excavation control apparatus and method combining reinforcement-learning-based planning and attention-mechanism-based trajectory generation

Abstract

An autonomous excavation control device and a method thereof are disclosed. An autonomous excavation control device for hierarchically controlling autonomous excavation work in an irregular terrain environment according to one embodiment of the present invention may include: a first-level planning unit that receives information of the irregular terrain environment and derives a work plan including at least one of a target excavation position and a target excavation depth of an excavator; and a second-level trajectory generation unit that calculates trajectory state information based on the work plan and generates a joint trajectory of an excavator by inputting the trajectory state information into a joint trajectory generation model learned based on adversarial learning.

Description

Technology Field

[0001] The present invention relates to a control technology for an excavator that performs autonomous excavation work, and more specifically, to a device and method for controlling the operation of an excavator by combining a reinforcement learning-based work plan and trajectory generation using an attention mechanism. Background Technology

[0003] Excavators used at construction sites are critical equipment that directly impacts the productivity and safety of earthwork operations, playing an essential role in various application environments such as urban construction, large-scale civil engineering projects, and autonomous construction equipment. Recently, there has been active technological development aimed at automating or autonomously controlling excavator operations to reduce reliance on operator skill and increase work efficiency.

[0004] Autonomous excavation technology generally aims to automatically perform a series of work procedures, such as determining the excavation location based on terrain information of the target excavation area and controlling the multi-joint structure of the excavator to dig up soil. To this end, computer vision-based terrain recognition technology, force and torque sensor-based adaptive control technology, and dynamic model-based trajectory generation technology have been studied.

[0005] In addition, previous studies have mainly applied methods such as determining excavatable areas by representing the excavation environment in the form of a grid map or a 3D point cloud, or bringing a bucket to a designated location by applying multi-joint robot control techniques. Along with this, various machine learning and robot control techniques have been proposed to learn excavation motion patterns based on expert excavation demonstration data or to mimic the repetitive movements of the excavation process.

[0006] However, excavation operations involve a combination of complex factors, such as terrain irregularity, variations in soil properties, differences in motion patterns at each stage, and inter-joint interactions. Consequently, stable automation is difficult using only existing methods, and there are limitations in the lack of seamless continuous linkage between work planning and the generation of actual joint movements. In particular, fixed rule-based control or simple robot manipulator control methods struggle to respond to diverse terrain conditions or uncertain environmental changes.

[0007] For this reason, the field of autonomous excavation is continuously researching more sophisticated work planning techniques, modeling methods capable of naturally generating complex joint movements, and control technologies that adapt to environmental changes; however, technical challenges still remain to be addressed in order to implement autonomous excavation systems that can operate stably at actual construction sites. Prior art literature

[0009] Korean Registered Patent Publication No. 10-1324917 The problem to be solved

[0010] The present invention was devised to solve the above-mentioned problems and provides an autonomous excavation control device and a control method thereof that can hierarchically process plan calculations reflecting environmental information and complex joint trajectory generation in order to perform work planning and joint motion generation of an excavator more stably and consistently in actual excavation environments with high uncertainty, such as irregular terrain or soil changes.

[0011] However, the problems of the present invention are not limited to those mentioned above, and other unmentioned problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0013] An autonomous excavation control device for hierarchically controlling autonomous excavation work in an irregular terrain environment according to an embodiment of the present invention for achieving the above objective may include: a first-level planning unit that receives information of the irregular terrain environment and derives a work plan including at least one of a target excavation position and a target excavation depth of an excavator; and a second-level trajectory generation unit that calculates trajectory state information based on the work plan and generates a joint trajectory of an excavator by inputting the trajectory state information into a joint trajectory generation model learned based on adversarial learning.

[0014] The above-mentioned first-level planning unit may be characterized by converting information of the above-mentioned irregular terrain environment into grid map-based environmental state information to perform PPO (Proximal Policy Optimization)-based reinforcement learning, and configuring the action space in the PPO-based reinforcement learning as a polar coordinate-based action space including at least one of a radius, rotation angle, and excavation depth based on the rotation center of the excavator.

[0015] The above autonomous excavation control device may further include a bidirectional cooperative learning unit that feeds back terrain change information resulting from applying the joint trajectory of the excavator generated by the second-level trajectory generation unit to the first-level planning unit to update the environment state information.

[0016] The above trajectory state information may be characterized by including at least one factor among a target excavation depth, a target bucket position, the actual joint angle of the excavator, soil density, the actual joint angle of the excavator, and an error between the target joint angle and the actual joint angle of the excavator according to the above work plan.

[0017] The second level trajectory generation unit may be characterized by inputting the trajectory state information into Multi-Head Attention to generate an embedding vector that reflects the relationship between each factor included in the trajectory state information, and inputting the embedding vector into the joint trajectory generation model.

[0018] The above joint trajectory generation model may be characterized as a model based on Generative Adversarial Imitation Learning (GAIL) that imitates the expert's excavation trajectory by adversarially updating a discriminator that distinguishes between the expert's excavation trajectory data collected in advance and the joint trajectory of the excavator generated by the above joint trajectory generation model, and a generator that generates the joint trajectory of the excavator based on the discrimination result of the discriminator.

[0019] The above-mentioned pre-collected expert excavation trajectory data may be characterized by being classified into penetration, scraping, and lifting stages according to the excavation stage.

[0020] An autonomous excavation control method for hierarchically controlling an autonomous excavation operation in an irregular terrain environment, performed by an autonomous excavation control device according to an embodiment of the present invention, may include: a first-level planning step of receiving information of the irregular terrain environment and deriving a work plan including at least one of a target excavation position and a target excavation depth of an excavator; and a second-level trajectory generation step of calculating trajectory state information based on the work plan and inputting the trajectory state information into a joint trajectory generation model learned based on adversarial learning to generate a joint trajectory of an excavator.

[0021] The above-mentioned first-level planning step may be characterized by converting information of the above-mentioned irregular terrain environment into grid map-based environmental state information to perform PPO (Proximal Policy Optimization)-based reinforcement learning, and configuring the action space in the PPO-based reinforcement learning as a polar coordinate-based action space including at least one of a radius, rotation angle, and excavation depth based on the rotation center of the excavator.

[0022] The above autonomous excavation control method may further include a bidirectional cooperative learning step in which terrain change information resulting from applying the joint trajectory of the excavator generated in the second-level trajectory generation step is fed back to the first-level planning step to update the environment state information.

[0023] The above trajectory state information may be characterized by including at least one factor among a target excavation depth, a target bucket position, the actual joint angle of the excavator, soil density, the actual joint angle of the excavator, and an error between the target joint angle and the actual joint angle of the excavator according to the above work plan.

[0024] The second level trajectory generation step may be characterized by inputting the trajectory state information into Multi-Head Attention to generate an embedding vector that reflects the relationship between each factor included in the trajectory state information, and inputting the embedding vector into the joint trajectory generation model.

[0025] The above joint trajectory generation model may be characterized as a model based on Generative Adversarial Imitation Learning (GAIL) that imitates the expert's excavation trajectory by adversarially updating a discriminator that distinguishes between the expert's excavation trajectory data collected in advance and the joint trajectory of the excavator generated by the above joint trajectory generation model, and a generator that generates the joint trajectory of the excavator based on the discrimination result of the discriminator.

[0026] The above-mentioned pre-collected expert excavation trajectory data may be characterized by being classified into penetration, scraping, and lifting stages according to the excavation stage.

[0027] According to one embodiment of the present invention, it may be characterized by including a computer-readable recording medium for executing an autonomous excavation control method. Effects of the invention

[0029] According to one aspect of the present invention described above, by hierarchically separating high-level planning of excavation work and low-level joint trajectory generation and linking them to a learning-based control structure, the operation of an excavator can be stably performed even in environments with irregular terrain or changes in soil properties.

[0030] Furthermore, since the task plan is generated by a policy learned based on environmental state information and a trajectory generation process reflecting trajectory state information is applied during the execution phase, the operational instability or discrepancy between plan and execution problems that occurred in existing rule-based control or single-stage control methods can be effectively mitigated.

[0031] In addition, since embedding processing considering the correlation between various state factors is performed during the process of generating joint motions of the excavator, motion patterns that change according to the excavation stage can be reflected more precisely, and natural connectivity and spatiotemporal consistency of the actual joint trajectory can be secured through patterns learned from expert demonstration data.

[0032] In addition, the model trained in the execution unit can flexibly adapt to variations in the working environment, such as changes in soil density or excavation paths, thereby maintaining stable excavation performance even in situations involving deviations during work iterations or unexpected terrain changes.

[0033] In particular, unlike conventional simple manipulator control or rule-based methods, applying a learning-based trajectory generation model has the advantage of reproducing complex excavation movements more naturally and reliably.

[0034] The effects of the present invention are not limited to those mentioned above, and various other effects will be clearly understood by those skilled in the art from the claims and the entire description of this specification. Brief explanation of the drawing

[0036] FIG. 1 is a diagram illustrating the schematic configuration of an autonomous excavation system including an autonomous excavation control device according to one embodiment of the present invention. FIG. 2 is a diagram illustrating the schematic configuration of an autonomous excavation control device according to one embodiment of the present invention. FIG. 3 is an illustrative diagram for explaining the process of generating a work plan and a joint trajectory in an autonomous excavation control device according to an embodiment of the present invention. FIG. 4 is an example diagram illustrating an excavation work space and a grid map representation for performing a reinforcement learning-based work plan in an autonomous excavation control device according to an embodiment of the present invention. FIG. 5 is a flowchart illustrating an autonomous excavation control method according to an embodiment of the present invention. FIGS. 6 to 8 are experimental results for demonstrating the performance of an autonomous excavation control device according to an embodiment of the present invention. Specific details for implementing the invention

[0037] The following detailed description of the invention refers to the accompanying drawings, which illustrate specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that various embodiments of the invention are different but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in relation to one embodiment. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the following detailed description is not intended to be limiting, and the scope of the invention is limited only by the appended claims, including all equivalents to those claimed therein, provided appropriately described. Similar reference numerals in the drawings refer to the same or similar functions across various aspects.

[0038] The components according to the present invention are defined by functional distinction rather than physical distinction, and can be defined by the functions each performs. Each component may be implemented as hardware or as program code and processing units that perform each function, and the functions of two or more components may be included and implemented in a single component. Therefore, it should be noted that the names assigned to the components in the following embodiments are not intended to physically distinguish each component but are assigned to imply the representative function performed by each component, and that the technical concept of the present invention is not limited by the names of the components.

[0039] Preferred embodiments of the present invention will be described in more detail below with reference to the drawings.

[0040] The autonomous excavation control device (10, hereinafter the device) according to the present embodiment is provided to automatically control an excavator performing excavation work in an irregular terrain environment, and to derive a work plan based on excavation environment information and to generate a joint trajectory of the excavator according to the derived work plan.

[0041] To this end, the device (100) according to the present embodiment may include a first-level planning unit (110), a second-level trajectory generation unit (120), and a bidirectional cooperative learning unit (130). In addition, software (application) for performing an autonomous excavation control method may be installed and executed on the device (100), and the first-level planning unit (110), the second-level trajectory generation unit (120), and the bidirectional cooperative learning unit (130) may be controlled by the software (application) for performing an autonomous excavation control method.

[0042] At this time, the device (100) may be a separate terminal or a part module of the terminal. Additionally, the configuration of the first level planning unit (110), the second level trajectory generation unit (120), and the bidirectional cooperative learning unit (130) may be formed as an integrated module or composed of one or more modules. However, conversely, each configuration may be composed of a separate module.

[0043] Additionally, the device (100) may be mobile or fixed. This device (100) may be in the form of a server or an engine and may be referred to by other terms such as device, apparatus, terminal, UE (user equipment), MS (mobile station), wireless device, or handheld device. Furthermore, the device (100) may execute or create various software based on an operating system (OS), that is, a system. Here, the operating system is a system program that enables software to use the device's hardware, and may include all mobile computer operating systems such as Android OS, iOS, Windows Mobile OS, Bada OS, Symbian OS, BlackBerry OS, etc., as well as computer operating systems such as Windows family, Linux family, Unix family, MAC, AIX, HP-UX, etc.

[0044] Hereinafter, the configuration of an autonomous excavation control system (10) according to one embodiment of the present invention will be described in detail with reference to FIG. 1.

[0045] Referring to FIG. 1, the autonomous excavation control system (10) may include an autonomous excavation control device (100) and an excavator (E).

[0046] The autonomous excavation control device (100) can generate a control signal to automatically perform excavation work by linking with the excavator (E). The excavator (E) can receive the control signal from the autonomous excavation control device (100) and automatically perform excavation operations according to the control signal. In addition, the excavator (E) can acquire terrain information, joint angle information, etc. required by the autonomous excavation control device (100) and provide them to the autonomous excavation control device (100).

[0047] First, an excavator (E) according to the present embodiment will be described.

[0048] The excavator (E) may include an excavator body for performing excavation work, a boom, an arm, and a bucket mounted on the body, and a hydraulic drive unit for driving them. The excavator body may be responsible for the movement and power transmission of the excavator and may include an engine, a transmission, a driving device (e.g., tracks or wheels), a steering system, a hydraulic system, a cabin, and various control instruments. The boom, arm, and bucket of the excavator (E) operate independently with respect to their respective axes of rotation and may function as key components that determine the working position, working posture, and excavation depth of the excavation equipment.

[0049] Additionally, the excavator (E) may further include a sensor module for recognizing the working environment. For example, the excavator (E) may include sensors such as LiDAR for acquiring terrain elevation information, a stereo camera, a monocular camera, a Global Positioning System (GPS), a joint angle sensor, a joint velocity sensor, and a torque sensor. These sensors may be used to configure environmental state information in the autonomous excavation control device (100) and may be utilized to accurately determine the current state of the excavator during the learning and control process. However, embodiments of the present invention are not limited thereto.

[0050] In one embodiment of the present invention, at least some of the components of the autonomous excavation control device (100) may be mounted on the main body or upper structure of the excavator (E). For example, a control processor, a communication module, some environmental perception sensors, or a storage device may be mounted on the top of the excavator main body, inside the driver's seat, the front frame, or the rotating upper structure. However, the embodiment of the present invention is not limited to such specific mounting locations, and the configuration of the autonomous excavation control device (100) may be provided in the form of an external device as needed.

[0051] Hereinafter, with reference to FIGS. 2 and 3, the configuration of an autonomous excavation control device (100) for hierarchically controlling autonomous excavation work in an irregular terrain environment according to one embodiment of the present invention will be described in detail.

[0052] The first level planning unit (110) can receive information on the irregular terrain environment and derive a work plan including at least one of the target excavation location and target excavation depth of the excavator.

[0053] The first level planning unit (110) may be characterized by converting information of an irregular terrain environment into grid map-based environment state information to perform PPO (Proximal Policy Optimization)-based reinforcement learning, and configuring the action space in PPO-based reinforcement learning into a polar coordinate-based action space including at least one of a radius, rotation angle, and excavation depth based on the rotation center of the excavator.

[0054] Here, an irregular terrain environment refers to an environment where the slope of the ground, elevation difference, distribution of soil, and terrain changes occurring during the work process exist unpredictably, such as natural terrain or artificially damaged terrain. In such irregular terrain, the shape of the terrain can change abnormally over time and due to the movement of soil by the excavator, even at the same location, and there is a problem in that it is difficult to derive an optimal work path using fixed rule-based control. Accordingly, the first level planning unit (110) of the present invention adopts a reinforcement learning-based approach, thereby enabling the calculation of an excavation plan that simultaneously considers work efficiency and work safety while adapting to terrain changes in real time.

[0055] The first level planning unit (110) can convert information about the irregular terrain environment into environmental state information in the form of a grid map. The grid map refers to a 2-dimensional or 2.5-dimensional data structure that divides space into grid cells of a certain size and records state variables such as the terrain height value, excavation progress, and soil removal amount at each grid cell. That is, the grid map is an environmental representation method that reflects both the spatial shape of the terrain and changes in excavation over time, enabling a reinforcement learning-based autonomous excavation planning algorithm to quantitatively identify the terrain and derive an optimal plan.

[0056] Such environmental state information can be defined by the following mathematical formula 1.

[0057] [Mathematical Formula 1]

[0058]

[0059] In the above mathematical formula 1 represents a grid map containing terrain conditions at time t, and each cell may contain height before and after excavation, terrain change amount, work boundary information, etc.

[0060] The first level planning unit (110) performs reinforcement learning based on Proximal Policy Optimization (PPO) using the above environment state information as input. PPO is a reinforcement learning algorithm that improves learning stability and sample efficiency compared to the existing policy gradient method, and uses a clipping method that limits the range of policy updates to mitigate the problem of performance instability caused by rapid changes in the policy. Accordingly, PPO enables stable policy learning even in environments with frequent state changes and high uncertainty, such as irregular terrain.

[0061] In reinforcement learning, the action of the first-level planning unit (110) can be selected from a polar coordinate action space configured with respect to the rotation center of the excavator. According to one embodiment of the present invention, the action is as shown in Equation 2 below, with radius r and rotation angle It may include at least one of , excavation depth d.

[0062] [Mathematical Formula 2]

[0063]

[0064] Here, r is the radius distance from the rotation center of the excavator to the excavation start point, d is the target excavation depth, represents the swing rotation angle of the excavator's bucket.

[0065] Since such a polar coordinate-based action space naturally corresponds to the excavator's actual kinematic structure (rotation combinations of boom, arm, and bucket), it enables more efficient and intuitive policy learning than existing Cartesian coordinate-based methods.

[0066] In addition, the first level planning unit (110) designs a reward by considering the accuracy of the excavation work, the amount of soil removed, and the minimization of unnecessary movement. The rewards for each stage can be defined as in the following mathematical formulas 3 to 6.

[0067] [Mathematical Formula 3]

[0068]

[0069] [Mathematical Formula 4]

[0070]

[0071] [Mathematical Formula 5]

[0072]

[0073] [Mathematical Formula 6]

[0074]

[0075] Here, is compensation for the error between the current terrain height and the target height, Compensation based on the actual excavated soil volume, It is a movement cost-based compensation designed to prevent the excavator end from moving unnecessarily large.

[0076] Here, , , are each , and It can refer to weighting parameters used to adjust the relative importance of. Also represents the set of grid cells affected by the excavation work at time t, and and can represent the terrain height before and after excavation at the corresponding grid cell (i,j), respectively. represents the area of ​​each grid cell and can be used as a constant for calculating the volume of soil. Also is excavation accuracy, is material removal efficiency, Each can reflect efficient excavation sequencing through the suppression of unnecessary movement.

[0077] This compensation configuration has the effect of simultaneously enhancing work efficiency (increased soil removal volume), accuracy (meeting target height), and stability (suppression of unnecessary movements).

[0078] When the task is finished, a termination reward may be applied as shown in Equation 7 below.

[0079] [Mathematical Formula 7]

[0080]

[0081] The termination reward in the above mathematical formula 7 is the success reward when approaching the target height. , insufficient compensation in cases of excessively low sales , overcompensation in case of selling excessively beyond the allowed amount By providing [this], the learned policy can be guided to derive a safe and balanced excavation plan.

[0082] At this time, represents the height tolerance threshold allowed for the target height, and represents the maximum allowable excavation volume during excavation operations. Also represents the average terrain height across the entire work area, and represents the total amount of soil actually excavated. This termination compensation design can contribute to maintaining work quality by preventing over-digging or under-digging by the excavator.

[0083] In this way, the first level planning unit (110) can generate a stable and efficient autonomous excavation work plan even under various terrain conditions by converting irregular terrain into a grid map-based state and determining the target excavation location and target excavation depth of the excavator through PPO-based reinforcement learning.

[0084] The second level trajectory generation unit (120) can generate trajectory state information based on the above work plan and input the trajectory state information into a joint trajectory generation model learned based on adversarial learning to generate the joint trajectory of the excavator.

[0085] The above trajectory state information may include at least one factor among the target excavation depth, target bucket position, actual joint angle of the excavator, soil density, actual joint angle of the excavator, and the error between the target joint angle and the actual joint angle of the excavator according to the above work plan.

[0086] The adversarial learning applied in this invention refers to a structure in which a generative model and a discriminative model learn simultaneously while competing with each other. This adversarial learning method induces the generative model to generate behaviors or patterns nearly identical to real data, while the discriminative model determines whether the generated results match expert data or real data, thereby providing a mechanism for the two models to mutually enhance each other. Adversarial learning is widely utilized in various fields, such as image generation, time-series data generation, robot control policy learning, domain difference reduction between simulators and real environments, and reward signal correction in reinforcement learning; in particular, it provides excellent performance in control problems that require mimicking expert behavior.

[0087] GAIL, an adversarial learning-based imitation learning technique, interprets imitation learning as a data distribution matching problem. By having a discriminator evaluate the distribution difference between expert trajectories and generative trajectories and providing this to a generative model in the form of surrogate rewards, it enables the model to mimic expert policies. Existing behavior cloning methods are vulnerable to distribution shifts, which can lead to significant execution failures if small errors accumulate. Inverse reinforcement learning methods, on the other hand, have limitations in that they require iterative reward function estimation, resulting in very high computational loads. While GAIL can overcome these limitations, standard GAIL suffers from a lack of ability to coordinate operations between phases and low learning stability in multi-stage excavation operations with complex temporal structures.

[0088] In the present invention, A-GAIL (Attention-Enhanced Generative Adversarial Imitation Learning), which extends the general GAIL structure, can be applied. A-GAIL consists of a discriminator that compares the generated excavation trajectory with expert demonstration data, specifically expert excavation trajectory data, and a generator that generates joint trajectories of the boom, arm, and bucket based on compensation signals provided by the discriminator; it can be designed to model temporal dependencies that occur during the multi-stage excavation process. Furthermore, A-GAIL can enhance correction control capabilities to prevent error accumulation by including joint angle errors and bucket tip position information in the state representation. Consequently, the generative model can more accurately mimic the multi-stage operation patterns of an actual excavator, such as excavation approach, penetration, and soil handling operations performed by experts.

[0089] The trajectory state information used in the second level trajectory generation unit (120) of the present invention can be defined as a multidimensional vector that reflects the operating state of the excavator and excavation environment information.

[0090] For example, trajectory state information including factors such as the joint angle of the excavator, target excavation depth, soil density, work progress, bucket tip position, and joint angle error can be represented in a vector form as shown in Equation 8 below.

[0091] [Mathematical Formula 8]

[0092]

[0093] In the above mathematical formula 8, is the joint angle of the excavator's boom, arm, and bucket, t indicates the progress of the excavation cycle, and d represents the target excavation depth, represents soil density. Additionally, the bucket tip coordinates [x,z] can be used to reflect the difference between the actual excavation location and the target location, and represents the error between the actual joint angle and the target joint angle. This trajectory state information includes both contextual information of the excavation process and immediate correction information, thereby improving the accuracy of the trajectory generation model.

[0094] Here, each of the above variables can be defined within a range that is substantially variable depending on the physical operation characteristics of the excavator, and this range information can be used as a reference value for normalization of neural network inputs and ensuring learning stability.

[0095] The joint angle error can be defined as shown in the following mathematical formula 9.

[0096] [Mathematical Formula 9]

[0097]

[0098] Here is the planned target joint angle, is the actual joint angle of the excavator, and the joint error is used to correct the accumulation of deviations occurring during excavation work in real time.

[0099] Such joint angle errors can also be calculated within a definable physical range based on the structural characteristics of the excavation equipment and can be used as a criterion for input constraints to prevent the model from generating unrealistic behavior.

[0100] The second level trajectory generation unit (120) receives the trajectory state information as input and generates a sequence of joint angular velocities for the boom, arm, and bucket of the excavator. The joint trajectory of the excavator can be defined as a change in angular velocity over time for each joint axis and can be represented as shown in Equation 10 below.

[0101] [Mathematical Formula 10]

[0102]

[0103] The above angular velocity It corresponds to the boom, arm, and bucket joints of the excavator, respectively, and limit ranges can be set by reflecting expert excavation trajectory data and the hydraulic drive limits of the excavator. For example, the boom joint angular velocity can be selected within a range such as [-1.2, 1.2] / step, the arm joint angular velocity within [-2.5, 2.0] / step, and the bucket joint angular velocity within [-10.0, 2.0] / step. These ranges reflect motion patterns observed in actual expert work, which can guide the model to generate a realistic and safe trajectory.

[0104] The above joint angular velocity can also be limited within the speed range that the equipment can actually implement, and this can serve to prevent motion bursts during the learning process and help learn a stable policy.

[0105] The second level trajectory generation unit (120) of the present invention can generate joint trajectories similar to those of an expert even in complex excavation work environments by utilizing the aforementioned adversarial learning structure, error-based state definition, and policy learning based on expert excavation trajectory data, and can secure temporal continuity of the multi-stage excavation process and precise motion control capabilities.

[0106] The second level trajectory generation unit (120) may be characterized by inputting trajectory state information into Multi-Head Attention to generate an embedding vector that reflects the relationship between each factor included in the trajectory state information, and inputting the embedding vector into the joint trajectory generation model.

[0107] Multi-Head Attention refers to a neural network structure that processes input information in parallel across multiple independent attention heads to learn the interrelationships between input elements from various perspectives. Attention techniques are mechanisms that calculate the importance or association between a specific input element and other elements; by performing these calculations simultaneously across multiple heads, Multi-Head Attention can more richly represent complex patterns, non-linear interactions, and spatiotemporal correlations that are difficult to capture with a single head.

[0108] In the present invention, the multi-head attention applied to the second-level trajectory generation unit (120) can be configured to separate trajectory state information into individual tokens, perform operations as shown in Equations 11 to 13 below, and concatenate them as shown in Equation 14 below to generate an embedding vector.

[0109] [Mathematical Formula 11]

[0110]

[0111] [Mathematical Formula 12]

[0112]

[0113] [Mathematical Formula 13]

[0114]

[0115] [Mathematical Formula 14]

[0116]

[0117] Here, represents the embedding vector in the l-th layer of the multi-head attention block, and represents the embedding vector output from the previous layer. represents a Multi-Head Attention operation on an input vector, and represents the Feed Forward Network operation. Also, LayerNorm refers to the operation that performs layer normalization after residual connections.

[0118] In the above mathematical formula 13, represents the output value of the i-th attention head, and , , represents the Query, Key, and Value vectors at the i-th head, respectively. d represents the total embedding dimension, and h represents the number of attention heads. is a scaling factor used as the denominator of scaled dot-product attention, and is a term used to ensure numerical stability of the attention weights.

[0119] In mathematical formula 14, represents an operation that concatenates the outputs of each attention head (i = 1~4), and represents the weight matrix of the linear projection applied to the combined attention output. This is a process for integrating features learned from multiple heads into a single embedding representation.

[0120] The embedding vector is a fixed-length vector representation obtained by converting input trajectory state information into a state factor with different physical meanings, and is the result of encoding state factors with different physical meanings into a numerical representation suitable for processing by a neural network. The embedding vector is converted into a representation that reflects the correlation, importance, and spatiotemporal dependency between each factor through multi-head attention, thereby enabling the joint trajectory generation model to estimate the actual motion pattern of the excavator more precisely. Furthermore, this embedding process may include a method of processing kinematic, spatial, temporal, and environmental factors into different feature embeddings by distinguishing them, as presented in the A-GAIL structure. Moreover, in the present invention, weighted pooling may be applied to reflect different importance for each type of state factor, which can be expressed as shown in Equation 15 below.

[0121] [Mathematical Formula 15]

[0122]

[0123] Here, represents the finally generated embedding vector, and refers to four characteristic groups representing kinematic, spatial, temporal, and environmental characteristics, respectively. Also is a weighting coefficient for learning the relative importance of each feature group, and It normalizes the contribution of each feature to a value between 0 and 1 as a result of applying a sigmoid function to the above weights. represents the set of state factors belonging to the corresponding characteristic group, and is that number, represents the embedding representation of the corresponding feature factor calculated in the L-th transformer layer.

[0124] In addition, to prevent duplicate pattern learning of multiple attention heads and to ensure that each head learns different features, a diversity regularization term as shown in Equation 16 below may be included.

[0125] [Mathematical Formula 16]

[0126]

[0127] Here, represents a diversity normalization term that suppresses similarity between attention heads to induce each head to learn different information patterns, and is a coefficient for controlling the influence of the above normalization term. and represents the attention maps calculated at the i-th and j-th attention heads, respectively.

[0128] The embedding vector generated in this way is provided as input to a joint trajectory generation model, and a joint trajectory reflecting complex spatiotemporal patterns included in the trajectory state information can be calculated.

[0129] The joint trajectory generation model may be characterized as a model based on Generative Adversarial Imitation Learning (GAIL) that imitates the expert's excavation trajectory by adversarially updating a discriminator that distinguishes between the expert's excavation trajectory data collected in advance and the joint trajectory of the excavator generated by the joint trajectory generation model, and a generator that generates the joint trajectory of the excavator based on the discrimination result of the discriminator.

[0130] The excavation trajectory data collected in advance by experts may be characterized by being classified into penetration, scraping, and lifting stages according to the excavation stage.

[0131] Adversarial learning is a structure designed so that the generator and the discriminator become more sophisticated competitively with each other. The generator attempts to generate a trajectory similar to the expert's excavation trajectory, and the discriminator is trained to distinguish between the generated trajectory and the expert's excavation trajectory. Additionally, the adversarial learning process can be performed based on an objective function including the following Equations 17 and 18.

[0132] [Mathematical Formula 17]

[0133]

[0134] [Mathematical Formula 18]

[0135]

[0136] Here, is the discriminator's loss function, consisting of state-action pairs (s,a) observed in the expert's drill trajectory data and the generator policy It is an objective function for training a discriminator to distinguish state-action pairs generated by.

[0137] is the loss function of the generator, and is an objective function to train the generated joint trajectory of the excavator to look similar to the expert's excavation trajectory from the perspective of the discriminator.

[0138] is a parameter As the output of a discriminator, it represents the probability or similarity that a given state-action pair (s,a) belongs to expert data.

[0139] represents the state-behavior distribution extracted from expert excavation trajectory data.

[0140] is a parameter As a policy of a generator having, it means a model that generates the joint trajectory of an excavator. (s,a) means a sample composed of a state and an action, and in a device (100) according to one embodiment of the present invention, the state may be composed of an 11-dimensional token structure and the action may be composed of angular velocities for three joints.

[0141] The expert's excavation trajectory data collected in advance is actual or simulation-based demonstration data of excavation work performed by an expert, and may include various terrain conditions, various excavation depths, different soil densities, different bucket access positions, etc. In addition, the expert's excavation trajectory data may be classified based on multiple excavation work stages, for example, into a penetration stage, a scraping stage, and a lifting stage. Furthermore, the excavation stages may be classified as shown in the conditional equation of Equation 19 below, but this is one exemplary classification method and the present invention is not limited thereto.

[0142] [Mathematical Formula 19]

[0143]

[0144] Here, is a function representing the excavator's current work stage, and represents soil density, d represents excavation depth, and z represents the height coordinate of the bucket tip, and these are important factors in classifying the excavation stages.

[0145] is a depth threshold value for defining the boundary between the penetration phase and the scraping phase, is the target excavation height, is the allowable error range used when determining the scraping stage, is a soil density reference value for determining whether it transitions to the lifting phase.

[0146] The conditional equation in the above mathematical formula 19 used for the stage classification is merely an example to explain the operational characteristics of the excavation work, and the criteria for stage classification may vary depending on the structural specifications of the excavator, the physical properties of the soil, the working environment, or the operating method of each operator.

[0147] In addition, the joint trajectory generation model can perform supervised learning-based Behavior Cloning (BC) pre-training to directly simulate the excavation trajectory of an expert during the initial training phase, and this process can be carried out by minimizing the loss function as shown in Equation 20 below.

[0148] Here, supervised learning refers to a learning method in which input values ​​and correct answers (labels) are provided together, and it refers to a procedure for adjusting weights so that the model mimics the given correct answers. Additionally, behavior cloning refers to a technique that uses state-action pairs performed by an expert as correct answers to train a policy network to output behavior similar to that of the expert when the same state is given, and it can be usefully applied to rapidly initialize the work patterns of an expert.

[0149] [Mathematical Formula 20]

[0150]

[0151] Here, is the action replication (BC) loss function, which indicates how accurately the generator's output reproduces expert action based on state-action pairs included in the expert's excavation trajectory data.

[0152] represents the sample distribution of states and actions extracted from expert excavation trajectory data.

[0153] Action replication-based pre-training can play a role in improving the stability and convergence of subsequent adversarial imitation learning by enabling the generator to quickly acquire basic excavation behavior patterns.

[0154] A device (100) according to one embodiment of the present invention may further include a bidirectional cooperative learning unit (130) that feeds back terrain change information resulting from applying the joint trajectory of the excavator generated in the second level trajectory generation unit to the first level planning unit to update the environment state information.

[0155] The bidirectional cooperative learning unit (130) can perform the function of periodically updating the environment state information by applying the joint trajectory of the excavator generated by the second-level trajectory generation unit (120) to a real or simulation environment, detecting the terrain change information that resulted therefrom, and providing it back to the first-level planning unit (110). That is, as the excavation work progresses, deformation of the terrain occurs, such as the height, slope, excavation volume, and bucket approach position of the work ground, and the bidirectional cooperative learning unit (130) reflects these terrain changes in the environment state information to support the first-level planning unit (110) in deriving a more accurate target excavation location and target excavation depth in the next excavation cycle.

[0156] Unlike the existing unidirectional structure in which the excavation planning stage and the trajectory generation stage operate independently of each other, this configuration provides a closed-loop learning structure that reflects environmental changes occurring during the excavation process in real time to the first-level planning unit (110). Through this, the first-level planning unit (110) can continuously correct the learning based on the actual performance results of the excavator and produce a planning policy that adapts to various excavation conditions (changes in soil strength, changes in target ground shape, excavation progress, etc.).

[0157] Additionally, the bidirectional cooperative learning unit (130) may be configured to automatically update environmental state information based on simulation maps or actual terrain data generated by the repeated application of joint trajectories of the excavator. At this time, terrain change information may be converted into a grid map-based information form and may include various elements such as the height change amount of each grid cell, cumulative excavation amount, and non-excavation area information. This information is then input into the first level planning unit (110) so that the reinforcement learning-based excavation location selection policy can be retrained or recalculated to suit the latest environment.

[0158] As a result, the bidirectional cooperative learning unit (130) can function as a key component that improves the stability, precision, and adaptability of autonomous excavation work by mediating the interaction between the first-level planning unit (110) and the second-level trajectory generation unit (120) to establish a cyclic structure of excavation planning-trajectory generation-terrain change-environment update.

[0159] FIG. 5 is a flowchart for explaining an autonomous excavation control method according to an embodiment of the present invention. Since the autonomous excavation control method according to an embodiment of the present invention proceeds on substantially the same configuration as the device (100) shown in FIG. 2, the same reference numerals are assigned to the same components as those in FIG. 2 (100), and repetitive descriptions are omitted.

[0160] An autonomous excavation control method for hierarchically controlling autonomous excavation work in an irregular terrain environment, performed by an autonomous excavation control device (100), may include: a first-level planning step (S100) for receiving information of the irregular terrain environment and deriving a work plan including at least one of a target excavation position and a target excavation depth of an excavator; and a second-level trajectory generation step (S200) for calculating trajectory state information based on the work plan and generating a joint trajectory of an excavator by inputting the trajectory state information into a joint trajectory generation model learned based on adversarial learning.

[0161] The above-mentioned first-level planning step (S100) may be characterized by converting information of the above-mentioned irregular terrain environment into grid map-based environment state information to perform PPO (Proximal Policy Optimization)-based reinforcement learning, and configuring the action space in the PPO-based reinforcement learning into a polar coordinate-based action space including at least one of a radius, rotation angle, and excavation depth based on the rotation center of the excavator.

[0162] An autonomous excavation control method according to one embodiment of the present invention may further include a bidirectional cooperative learning step in which terrain change information resulting from applying the joint trajectory of the excavator generated in the second level trajectory generation step (S200) is fed back to the first level planning step (S100) to update the environment state information.

[0163] The above trajectory state information may be characterized by including at least one factor among a target excavation depth, a target bucket position, the actual joint angle of the excavator, soil density, the actual joint angle of the excavator, and an error between the target joint angle and the actual joint angle of the excavator according to the above work plan.

[0164] The second level trajectory generation step (S200) may be characterized by inputting the trajectory state information into Multi-Head Attention to generate an embedding vector that reflects the relationship between each factor included in the trajectory state information, and inputting the embedding vector into the joint trajectory generation model.

[0165] The above joint trajectory generation model may be characterized as a model based on Generative Adversarial Imitation Learning (GAIL) that imitates the expert's excavation trajectory by adversarially updating a discriminator that distinguishes between the expert's excavation trajectory data collected in advance and the joint trajectory of the excavator generated by the above joint trajectory generation model, and a generator that generates the joint trajectory of the excavator based on the discrimination result of the discriminator.

[0166] The above-mentioned pre-collected expert excavation trajectory data may be characterized by being classified into penetration, scraping, and lifting stages according to the excavation stage.

[0167] The autonomous excavation control method of the present invention, as described above, can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., either individually or in combination.

[0168] The program instructions recorded on the above-mentioned computer-readable recording medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software.

[0169] Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, and flash memory.

[0170] Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

[0171] The device (100) according to the present invention hierarchically separates high-level planning of excavation work and low-level joint trajectory generation and links them to a learning-based control structure, thereby enabling stable operation of the excavator even in an environment where irregular terrain or changes in soil properties exist.

[0172] Furthermore, since the task plan is generated by a policy learned based on environmental information and a trajectory generation process reflecting it is applied during the execution phase, the operational instability or discrepancy between plan and execution problems that occurred in existing rule-based control or single-stage control methods can be effectively mitigated.

[0173] In addition, since embedding processing considering the correlation between various state factors is performed during the process of generating joint motions of the excavator, motion patterns that change according to the excavation stage can be reflected more precisely, and natural connectivity and spatiotemporal consistency of the actual joint trajectory can be secured through patterns learned from expert demonstration data.

[0174] In addition, the model trained in the execution unit can flexibly adapt to variations in the working environment, such as changes in soil density or excavation paths, thereby maintaining stable excavation performance even in situations involving deviations during work iterations or unexpected terrain changes.

[0175] In particular, unlike conventional simple manipulator control or rule-based methods, applying a learning-based trajectory generation model has the advantage of reproducing complex excavation movements more naturally and reliably.

[0176] FIGS. 6 to 8 are experimental results for demonstrating the performance of a device (100) according to an embodiment of the present invention for demonstrating the effects described above.

[0177] Figure 6 shows the results of an indicator-based experiment performed to quantitatively evaluate the performance of an A-GAIL-based joint trajectory generation model.

[0178] The above experiment was performed on a total of 2,000 excavation trajectories (1,000 training samples and 1,000 unused test samples), and the performance of different trajectory generation methods, including A-GAIL, conventional GAIL, and BC (Behavior Cloning), was compared under the same conditions.

[0179] Experimental results showed that A-GAIL exhibited an error approximately 4.5 times lower than BC in terms of Mean Squared Error (MSE) and demonstrated an accuracy improvement of about 17.3% compared to the existing GAIL, proving that the attention-based structure precisely captures the step-by-step characteristics of the excavation operation. In terms of the Dynamic Time Alignment (DTW) metric, A-GAIL also recorded the lowest value of 32.3, indicating improved temporal agreement compared to the reference method. Furthermore, the Prechet distance (Fr A-GAIL also showed the best values ​​in terms of chet distance and operation efficiency indicators, confirming that it generates a continuous and stable joint trajectory required for actual excavator control.

[0180] Figure 7 shows the results of comparing the model's adaptive performance to changes in the excavation environment by conducting experiments to generate joint trajectories of the excavator under different soil conditions.

[0181] The above experiment was conducted on three representative scenarios (soft ground, medium density ground, and high density ground), and each scenario was configured to reflect changes in soil density and excavation depth that may occur at actual construction sites.

[0182] Experimental results showed that A-GAIL generated a stable spatial trajectory, maintaining a deviation of less than 100 mm relative to the expert trajectory across all scenarios. In contrast, the BC-based method exhibited large deviations of over 500 mm, particularly under conditions of high density or increased excavation resistance, while the conventional GAIL method demonstrated moderate performance but showed reduced stability under extreme conditions. Furthermore, A-GAIL displayed a continuous joint control pattern with clear phase transitions in each scenario, confirming its excellent adaptability unaffected by changes in soil density.

[0183] Figure 8 shows the results of an environmental adaptability experiment conducted to evaluate adaptability performance according to changes in excavation depth and soil density.

[0184] The above experiment was conducted by comparing the error and dynamic agreement of the joint trajectories generated by the model with respect to changes in excavation depth (various depth ranges) and soil density (range of 20 to 90).

[0185] Experimental results showed that A-GAIL consistently maintained a low error range across all excavation depths and demonstrated stable performance compared to existing methods even in environments where planning interference increases, such as in shallow excavation sections (400-600 mm). In soil density variation experiments, A-GAIL maintained a DTW error of 40 or less across the entire range (20–90), with almost no performance degradation observed due to environmental changes. In contrast, the existing GAIL showed a tendency for performance instability, with the error increasing sharply to over 50 in the high-density range (ρ > 80). The fact that A-GAIL maintains a narrow confidence interval implies that it has learned a stable control pattern even in atypical environments such as changes in soil resistance, demonstrating that it can automatically adapt to excavation conditions without separate parameter adjustments for specific soil conditions.

[0186] Although various embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention. Explanation of the symbols

[0189] 10: Autonomous Excavation Control System E: Excavator 100: Autonomous Excavation Control Device 110: Level 1 Planning Department 120: Level 2 Trajectory Generator 130: Interactive Collaborative Learning Department

Claims

Claim 1 An autonomous excavation control device for hierarchically controlling autonomous excavation work in an irregular terrain environment, comprising: a first-level planning unit that receives information of the irregular terrain environment and derives a work plan including at least one of a target excavation position and a target excavation depth of an excavator; and a second-level trajectory generation unit that calculates trajectory state information based on the work plan and inputs the trajectory state information into a joint trajectory generation model learned based on adversarial learning to generate a joint trajectory of an excavator; wherein the joint trajectory generation model is a model based on Generative Adversarial Imitation Learning (GAIL) that imitates an expert's excavation trajectory by adversarially updating a discriminator that distinguishes between pre-collected expert excavation trajectory data and the joint trajectory of an excavator generated by the joint trajectory generation model, and a generator that generates the joint trajectory of an excavator based on the discrimination result of the discriminator. Claim 2 An autonomous excavation control device according to claim 1, wherein the first level planning unit converts information of the irregular terrain environment into grid map-based environment state information and performs PPO (Proximal Policy Optimization)-based reinforcement learning, and configures the action space in the PPO-based reinforcement learning as a polar coordinate-based action space including at least one of a radius, rotation angle, and excavation depth based on the rotation center of the excavator. Claim 3 The autonomous excavation control device according to claim 2 further comprises a bidirectional cooperative learning unit that feeds back terrain change information resulting from applying the joint trajectory of the excavator generated by the second-level trajectory generation unit to the first-level planning unit to update the environment state information. Claim 4 An autonomous excavation control device according to claim 1, wherein the trajectory state information comprises at least one factor among a target excavation depth, a target bucket position, an actual joint angle of the excavator, soil density, an actual joint angle of the excavator, and an error between the target joint angle and the actual joint angle of the excavator according to the work plan. Claim 5 An autonomous excavation control device according to claim 4, wherein the second-level trajectory generation unit inputs the trajectory state information into Multi-Head Attention to generate an embedding vector reflecting the relationship between each factor included in the trajectory state information, and inputs the embedding vector into the joint trajectory generation model. Claim 6 delete Claim 7 An autonomous excavation control device according to claim 1, characterized in that the pre-collected expert excavation trajectory data is classified into penetration, scraping, and lifting stages according to the excavation stage. Claim 8 An autonomous excavation control method for hierarchically controlling autonomous excavation work in an irregular terrain environment, performed by an autonomous excavation control device, comprising: a first level planning step of receiving information of the irregular terrain environment and deriving a work plan including at least one of a target excavation position and a target excavation depth of an excavator; Autonomous excavation control method comprising: a second-level trajectory generation step of generating trajectory state information based on the above work plan and generating a joint trajectory of an excavator by inputting the trajectory state information into a joint trajectory generation model learned based on adversarial learning; wherein the joint trajectory generation model is a model based on Generative Adversarial Imitation Learning (GAIL) that imitates an expert's excavation trajectory by adversarially updating a discriminator that distinguishes between pre-collected expert excavation trajectory data and the joint trajectory of an excavator generated by the joint trajectory generation model, and a generator that generates the joint trajectory of an excavator based on the discrimination result of the discriminator. Claim 9 An autonomous excavation control method according to claim 8, wherein the first level planning step converts information of the irregular terrain environment into grid map-based environmental state information to perform PPO (Proximal Policy Optimization)-based reinforcement learning, and configures the action space in the PPO-based reinforcement learning as a polar coordinate-based action space including at least one of a radius, rotation angle, and excavation depth based on the rotation center of the excavator. Claim 10 In claim 9, the autonomous excavation control method further comprises a bidirectional cooperative learning step in which terrain change information resulting from applying the joint trajectory of the excavator generated in the second-level trajectory generation step is fed back to the first-level planning step to update the environment state information. Claim 11 An autonomous excavation control method according to claim 8, wherein the trajectory state information comprises at least one factor among a target excavation depth, a target bucket position, an actual joint angle of the excavator, soil density, an actual joint angle of the excavator, and an error between the target joint angle and the actual joint angle of the excavator according to the work plan. Claim 12 In claim 11, the second level trajectory generation step is characterized by inputting the trajectory state information into Multi-Head Attention to generate an embedding vector that reflects the relationship between each factor included in the trajectory state information, and inputting the embedding vector into the joint trajectory generation model, thereby forming an autonomous excavation control method. Claim 13 delete Claim 14 An autonomous excavation control method according to claim 8, characterized in that the excavation trajectory data of the expert collected in advance is classified into penetration, scraping, and lifting stages according to the excavation stage. Claim 15 A computer-readable recording medium for implementing the autonomous excavation control method according to paragraph 8.

Images