A method for predicting a change in a shape of a material pile

Abstract

The present application relates to the technical field of computer simulation, and specifically provides a material pile shape change prediction method, which first establishes a simulation system of a material pile to be operated, obtains a material pile height map before shovel excavation and corresponding shovel posture, and inputs the same to a pre-trained double-agent cooperation model to predict the material pile height after shovel excavation, the double-agent cooperation model comprising a first agent and a second agent, the first agent inputting the material pile height map before shovel excavation and the shovel posture, and outputting a global probability mask; the second agent inputting the material pile height map before shovel excavation, the shovel posture and the global probability mask, outputting a global height change map, and further generating a material pile height map after shovel excavation to update the simulation material pile. The present application adopts a double-agent cooperation architecture model to replace the traditional DEM method, thereby greatly reducing the calculation amount while ensuring the accuracy of the result.

Description

Technical Field

[0001] This invention belongs to the field of computer simulation technology, and in particular relates to a method for predicting changes in the shape of a material pile. Background Technology

[0002] Loaders, as one of the most widely used pieces of equipment in the construction machinery field, undertake a large number of material handling and loading tasks in scenarios such as mining, sand and gravel yards, port loading and unloading, and construction projects. These operations are typically characterized by high frequency and strong repetition, and the operating environment is complex with many hazards, such as severe dust pollution, limited visibility, drastic terrain changes, and significant impact from heavy loads. Traditional manual operation of loaders relies heavily on the operator's experience and skills. Prolonged high-load operation can easily lead to fatigue, misoperation, or even safety accidents, and it is also difficult to guarantee the stability of work quality and efficiency.

[0003] With the advancement of industrial intelligence and smart mining construction, unmanned and intelligent operations are gradually becoming an industry trend. By integrating technologies such as autonomous driving, intelligent sensing, path planning, and mechanical control into loaders, they can autonomously complete the entire process of loading, driving, and unloading. This not only significantly improves operational safety but also increases work efficiency, reduces labor costs, and achieves stable, continuous, and high-intensity production capacity. Especially in extreme working conditions such as mines, plateaus, and harsh climates, unmanned loader systems have irreplaceable application value.

[0004] While autonomous driving technology has made significant progress in road scenarios, directly transferring it to the operating environment of a loader presents extremely high challenges. Loader work sites are often unstructured environments with undulating terrain and irregular material pile shapes. After each loader loads material with its bucket, the material pile may flow, resulting in dynamic changes in material distribution. Furthermore, loaders exhibit significant rigid-flexible coupling characteristics during the loading process; the contact mechanics between the bucket and the soil are complex and highly nonlinear, making it difficult to model using traditional control strategies alone. In addition, issues such as vehicle vibration, tire-ground interaction, and power system response under heavy loads further challenge the stability and reliability of unmanned control.

[0005] To accelerate the development of unmanned technologies while ensuring safety and cost control, building a high-fidelity simulation environment has become an inevitable choice. Simulation platforms can simulate the dynamic characteristics, environmental perception, material pile changes, and sensor noise of loaders without the need for real equipment. These platforms are used for algorithm design, parameter evaluation, and system verification, and have become an indispensable part of the intelligent engineering machinery R&D workflow. In traditional unmanned loader simulation, the Discrete Element Method (DEM) is generally used as the mainstream technology for simulating particulate materials. It can simulate the accumulation, flow, and breakage of materials from the perspective of microscopic particle interactions, and is an important tool for simulating bulk media such as sand, minerals, and soil. In engineering machinery operation research, especially in applications involving large deformations and particle flow such as loader loading and excavator digging, DEM is widely used due to its detailed representation of particle-level dynamic behavior. However, the biggest limitation of DEM is its extremely high computational cost. Because this method requires dynamic integration, contact detection, and collision response calculations for each particle, the computational load increases exponentially when the amount of material reaches the scale of actual engineering projects. Taking a typical sand and soil pile as an example, the actual particle size is usually in the millimeter or sub-millimeter range, and the soil in a loader bucket often contains tens of millions or even hundreds of millions of particles. Even with techniques such as coarse-graining to reduce the number of particles, the simulation may still require millions of particles to maintain basic flow characteristics. Such a massive computational scale makes the DEM run extremely long on a single machine, often requiring a high-performance computing platform to complete a simulation within a reasonable time.

[0006] More importantly, in unmanned loader research, simulation must not only pursue physical accuracy but also serve the rapid iteration of environmental perception, path planning, control algorithms, and closed-loop strategies. However, the high computational cost of DEM makes it difficult to use for real-time system training or online control verification, failing to meet the time requirements of "perception-decision-execution" closed-loop simulation. When batch working condition scanning, strategy training, or repeated verification is required, the computational bottleneck of DEM becomes particularly prominent, often preventing many experiments from being completed within an acceptable timeframe. Therefore, although DEM has unique advantages in simulating particulate material behavior, its low computational efficiency makes it more suitable as a high-precision offline analysis tool than for unmanned loader simulation research with high real-time requirements. Summary of the Invention

[0007] In view of this, the present invention aims to provide a method for predicting changes in the shape of a material pile. It adopts a dual-agent collaborative architecture model to replace the traditional DEM method. By training with real data, the model learns the deformation process of the material pile before and after excavation. While ensuring the accuracy of the results, it significantly improves the computational efficiency, makes the update of the simulation environment smoother and more natural, and avoids the huge computational cost required by the traditional DEM method for detailed description of particle-level dynamic behavior.

[0008] To achieve the above objectives, the technical solution created by this invention is implemented as follows:

[0009] This invention provides a method for predicting changes in the shape of a material pile, comprising:

[0010] Establish a simulation system for the material pile to be worked on, and obtain the height diagram of the material pile before digging and the corresponding bucket posture through the simulation system;

[0011] The material pile height diagram and bucket posture before excavation are input into the pre-trained dual-agent collaborative model to predict the material pile height after excavation.

[0012] The dual-agent collaborative model includes a first agent and a second agent. The first agent is input with the material pile height map and bucket posture before digging, and predicts and outputs a global probability mask. The global probability mask is used to identify the probability of each pixel in the material pile height map undergoing shape change due to the bucket digging. The second agent is input with the material pile height map, bucket posture and global probability mask before digging, and predicts and outputs a global height change map. The global height change map is used to identify the height change value of each pixel in the material pile height map caused by the bucket digging.

[0013] By merging the global height change map and the material pile height map before excavation, a material pile height map after excavation is predicted and generated.

[0014] The simulation system for the material pile to be worked on is updated using the height map of the material pile after excavation.

[0015] Preferably, the simulation system for the material pile to be operated uses the Unity simulation engine program. The simulation system includes: material pile model, environment model, loader model and sensor model.

[0016] Preferably, the first intelligent agent includes: an attitude information processing module, a first fusion module, and a first U-net network. The attitude information processing module converts the bucket attitude into an attitude feature vector through an MLP attitude feature encoder, and broadcasts the attitude feature vector along the spatial dimension to generate an attitude feature map; the attitude feature map is then averaged along the channel dimension to generate a single-channel attitude map.

[0017] The first fusion module is used to stitch together the single-channel attitude diagram and the material pile height diagram before digging as the first input vector of the first U-net network;

[0018] The first U-net network includes a first encoder and a first decoder. The first encoder is used to perform multiple convolution and max pooling operations on the first input vector to obtain the first encoded vector.

[0019] The first decoder is used to perform a skip connection between the first encoded vector and the first input vector, and the skip connection adopts an attention mechanism. The first encoded vector after the skip connection and the first input vector are subjected to multiple convolution and upsampling operations, and finally output a global probability mask with the same size as the material stack height map.

[0020] Preferably, the second intelligent agent includes: a second fusion module and a second U-net network, wherein the second fusion module is used to concatenate the material pile height map before digging, the single-channel attitude map and the global probability mask as the second input vector of the second U-net network;

[0021] The second U-net network includes a second encoder and a second decoder. The second encoder is used to perform multiple convolution and max pooling operations on the second input vector to obtain the second encoded vector.

[0022] The second decoder is used to perform a skip connection between the second encoded vector and the second input vector. The skip connection adopts an attention mechanism. The second encoded vector and the second input vector after the skip connection are subjected to multiple convolution and upsampling operations, and finally output a global height change map with the same size as the material stack height map.

[0023] Preferably, the material pile height diagram is in PNG format; the bucket posture is in JSON format.

[0024] Preferably, during the training process of the dual-agent collaborative model, the training samples are historical digging data in real-world scenarios. Each training sample includes: a real material pile height map before actual digging, a real material pile height map after actual digging, and the actual digging bucket posture.

[0025] The preprocessing procedure for training samples is as follows:

[0026] The point clouds of the actual material pile before and after actual excavation are acquired by sensors on the actual loader. The point clouds of the actual material pile before and after actual excavation are sampled to the same pixel size as the preset height map of the material pile. The height information of the sampled point cloud is mapped to the pixel value of the corresponding pixel, thus obtaining the height map of the actual material pile before and after actual excavation.

[0027] Preferably, the loss function of the first agent for:

[0028] ;

[0029] in, This represents the binary cross-entropy loss function. For global probability mask, A mask representing the actual change area of ​​the actual material pile height map after excavation compared to the actual material pile height map before excavation. Represents similarity loss, used to describe and The similarity.

[0030] Preferably, the loss function of the second agent for:

[0031] ;

[0032] in, For absolute error loss, This is a global height change map. This is a graph showing the actual height change of the material pile after excavation compared to the actual material pile height before excavation. This represents the weighting coefficient.

[0033] Preferably, the total training loss function of the two-agent cooperative model for:

[0034] ;

[0035] in, These are the weighting coefficients.

[0036] Preferably, the simulation system for the material pile to be processed is updated using a transitional simulation algorithm. The update process is as follows:

[0037] ;

[0038] in, A diagram showing the height of the material pile after excavation. A diagram showing the height of the material pile before excavation. For physical flow terms, In order to excavate the target attraction, The coefficient of variation, The target for excavation of the material pile.

[0039] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0040] This invention predicts the deformation of the material pile caused by digging using a dual-agent collaborative network architecture model. The first agent predicts the changes in the material pile morphology, first identifying the area of ​​change and generating a global probability mask. Then, using this global probability mask as an auxiliary channel, the second agent determines the height change of each pixel in the material pile height map caused by the bucket digging, based on the material pile height map before digging and the bucket's posture. This replaces the traditional discrete element method (DEM), avoiding the need for iterative solving of massive inter-particle interaction forces in each simulation frame, thus achieving an order-of-magnitude increase in simulation frame rate while maintaining visual accuracy. This method significantly reduces computational overhead and increases the rate of material pile updates when constructing engineering operation simulation environments. Compared to the DEM, which requires determining a large number of complex environmental parameters, this invention only requires fine-tuning a very small number of hyperparameters to achieve the desired effect. The model inference speed is much faster than physical calculations, enabling the simulation environment to update the material pile morphology in real-time and smoothly, meeting the stringent real-time requirements of excavation planning during unmanned loader operations. Furthermore, for material piles with different properties, the original model can be replaced simply by retraining the model with a new dataset, demonstrating high universality and strong scalability.

[0041] For model training, this invention employs a combination of probability guidance from the first agent and optimization of the total loss function, thereby improving the overall fault tolerance of the system's predictions.

[0042] Furthermore, during the material pile morphology update process, target guidance and local material flow simulation based on critical slope angle are integrated, so that the material pile update process not only converges to the prediction results, but also presents natural collapse, sliding and accumulation visual effects, which greatly enhances the realism of the simulation effect. Attached Figure Description

[0043] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0044] Figure 1 This is a flowchart of a material pile morphology change prediction method provided in an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram illustrating the application of the material pile morphology change prediction method provided by the embodiment of the present invention in a simulation environment;

[0046] Figure 3 This is a basic architecture diagram of the first and second intelligent agents provided according to embodiments of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and do not constitute a limitation thereof. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of the invention. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the invention are not shown or described in the specification. This is to avoid obscuring the core parts of the invention with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; the relevant operations can be fully understood based on the description in the specification and general technical knowledge in the art.

[0048] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined to form various implementations. Furthermore, the order of the steps or actions in the method description can be changed or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various orders in the specification and drawings are merely for the clear description of a particular embodiment and do not imply a mandatory order, unless otherwise stated that a particular order must be followed.

[0049] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0050] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0051] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0052] Please see Figure 1 In one embodiment of the present invention, a method for predicting material pile morphology changes is provided and applied to ship hold clearing operations. By constructing a simulation system to simulate and predict actual material operations, this method replaces the traditional discrete element method (DEM) for predicting material pile deformation after clearing, thus solving the drawbacks of the traditional DEM method, such as huge computational load, low computational efficiency, and inability to achieve real-time interactive operation. The method for predicting material pile morphology changes specifically includes the following steps:

[0053] S1: Establish a simulation system for the material pile to be processed;

[0054] S2: Obtain the material pile height diagram and corresponding bucket posture before digging through the simulation system;

[0055] S3: Input the material pile height map and bucket posture before digging into the pre-trained dual-agent collaborative model to predict the material pile height after digging. The dual-agent collaborative model includes a first agent and a second agent. The first agent inputs the material pile height map and bucket posture before digging and predicts and outputs a global probability mask. The global probability mask is used to identify the probability of each pixel in the material pile height map undergoing shape change due to the bucket digging. The second agent inputs the material pile height map before digging, bucket posture, and global probability mask and predicts and outputs a global height change map. The global height change map is used to identify the height change value of each pixel in the material pile height map caused by the bucket digging. The global height change map and the material pile height map before digging are fused to predict and generate the material pile height map after digging.

[0056] S4: Update the simulation system of the material pile to be worked on using the height map of the material pile after excavation.

[0057] In step S1, as follows Figure 2The simulation system for the material pile to be processed is first established and started using the Unity simulation engine. Within the Unity environment, the material pile, the working environment, the loader, and the sensors mounted on the loader are modeled to obtain material pile models, environment models, loader models, and sensor models. After modeling, the simulation system is initialized based on the actual equipment and environmental parameters. Since this embodiment specifically uses a ship cabin clearing operation environment as an example, the cabin walls and ground are initialized first. The ship cabin is approximated as a cuboid, and to maintain similarity to a real ship cabin, its length and width are both set to 30 meters. Then, the terrain system is used to create the material pile to be processed. Alternatively, several real ship cabin height maps can be pre-built into the simulation system. The environment model is automatically generated based on these real ship cabin height maps, and the terrain system randomly calls a ship cabin height map containing the material pile to be processed, using the height information of the material pile to be processed to complete the initialization of the material pile. After initialization, the materials should be distributed close to the cabin walls, with the center of the ship cabin as an open space. Furthermore, a pre-modeled loader model is placed in the center of the cabin, and the loader model has several sensors to capture and output the bucket posture.

[0058] Finally, a communication connection is established between the simulation system and the dual-agent collaborative model deployed on an external computer. The simulation system provides simulation environment data to the external computer, and the external computer feeds back the model prediction results to the simulation system. At this point, the environment initialization is complete.

[0059] In step S2, the material pile height map and corresponding bucket posture before digging are obtained based on the simulation system. It is important to note that both the material pile height map and bucket posture are simulation data generated by the simulation system. The material pile height map has a fixed, preset size to facilitate input into the dual-agent collaborative model for global height change prediction. Each pixel value in the material pile height map represents the material height at the corresponding location, thus using a two-dimensional height map to reflect the material pile shape in the three-dimensional actual space. The bucket posture is generated at any point during the simulation operation based on the operating commands issued to the loader model. These operating commands typically mimic the following two actual command issuance methods:

[0060] The first method: operation signals transmitted through manual operating devices such as keyboards and gamepads;

[0061] The second method involves receiving automated operation signals from an external end-to-end automated driving model actually mounted on the loader.

[0062] The loader's bucket can be controlled to complete a digging action using any of the above methods. When the bucket contacts the material pile, a digging judgment is triggered. The simulation system records the material pile state of the current frame and generates a material pile height map before digging based on the scenario simulation. The material pile height map is recorded in PNG format, and the bucket posture is recorded in JSON format.

[0063] In step S3, the pre-trained dual-agent collaborative model, which inputs the material pile height map before excavation and the bucket posture, is used to predict the material pile height after excavation. Specifically, the dual-agent collaborative model includes a first agent and a second agent. The first and second agents work together to process the material pile excavation change prediction process. The first agent predicts and outputs a global probability mask based on the input material pile height map before excavation and the bucket posture. The second agent predicts and outputs a global height change map based on the global probability mask output by the first agent. Specifically, both the first and second agents adopt the U-net architecture. The input of the first agent is the material pile height map before excavation and the bucket posture. Structurally, it includes a posture information processing module, a first fusion module, and a first U-net network. The posture information processing module uses an MLP posture feature encoder to convert the input JSON format bucket posture into a high-dimensional posture feature vector, and further broadcasts the posture feature vector along the spatial dimension to generate a posture feature map. Then, mean pooling is performed on the pose feature map along the channel dimension, and the channel mean is taken as the single-channel pose map.

[0064] The first fusion module stitches together the single-channel attitude map and the material pile height map before digging, and uses it as the first input vector of the first U-net network. The material pile height map is used as the main channel, and the single-channel attitude map is used as an additional channel. After stitching, each pixel in the material pile height map can obtain the loading of the single-channel attitude map, and the influence of the digging bucket attitude on the height of the corresponding material for each pixel.

[0065] like Figure 3As shown, the first U-net network includes a first encoder and a first decoder. The first encoder includes multiple encoding layers, and the first decoder includes multiple decoding layers. Each encoding layer corresponds to one decoding layer, and an attention module is provided between the encoding and decoding layers in the same layer, allowing the first agent to pay more attention to regions of height variation. For ease of description, the different encoding layers are specifically denoted as: encoding layer 1, encoding layer 2, ..., encoding layer n; and the different decoding layers are denoted as: decoding layer 1, decoding layer 2, ..., decoding layer n. In each encoding layer, convolution and max pooling operations are performed on the output of the previous encoding layer. The input of encoding layer 1 is the first input vector. Each convolution and max pooling operation doubles the number of channels of the vector and halves its width and height. After the first input vector undergoes multiple convolution and max pooling operations by the encoder, the first encoded vector is obtained. The first encoded vector output by the last encoding layer n reaches the bottleneck layer and is input to the decoder. In the decoder, each encoding layer performs convolution and max pooling operations on the output of the previous encoding layer. Before decoding the next encoding layer from the output of the previous decoding layer, a skip connection is made between the decoded vector output from the previous decoder and the corresponding encoding vector of the encoder in the same layer. The input to encoding layer n is the first encoding vector input to the bottleneck layer. This skip connection is implemented through an in-layer attention module, which concatenates and fuses the decoded vector output from the decoder with the corresponding encoding vector of the encoder in the same layer based on an attention mechanism. This skip connection allows the first agent to focus more on regions of height variation in the material pile height map during decoding. After vector concatenation and fusion, a convolution and upsampling operation is performed in each decoding layer. Each time, the dimension of the input vector is halved, while the width and height are doubled. After multiple convolutions and max pooling operations across multiple decoding layers, the final first input vector is processed into a global probability mask output. The size of the global probability mask is the same as the size of the material pile height map, and each pixel in the global probability mask records the probability of height variation at that location.

[0066] The global probability mask output by the first intelligent agent, along with the material pile height map before digging and the single-channel attitude, are compared. Figure 1 The first agent is used as the input for the second agent, which is guided by the global probability mask output by the first agent. This guides the second agent to predict the specific height changes at each point where the material pile height changes. The first and second agents work together in parallel, avoiding excessive computational pressure on a single agent. Parallel processing also greatly improves data processing efficiency, making it easier to deploy on actual loaders and enabling real-time interactive control decisions for excavation.

[0067] The second agent has a similar topology to the first agent, but its inputs are the material pile height map before excavation, the single-channel attitude map, and the global probability mask output by the first agent. Structurally, it includes a second fusion module and a second U-net network.

[0068] The second fusion module stitches together the single-channel attitude map, the material pile height map before digging, and the global probability mask as the second input vector of the second U-net network. The material pile height map serves as the main channel, while the single-channel attitude map and the global probability mask serve as additional channels. After stitching, each pixel in the material pile height map can obtain the loading of the single-channel attitude map and the global probability mask, the influence of the digging bucket attitude on the height of the corresponding material for each pixel, and the probability that the height of each pixel may change due to digging.

[0069] The second U-net network includes a second encoder and a second decoder. The second encoder includes multiple encoding layers, and the second decoder includes multiple decoding layers. Each encoding layer corresponds to one decoding layer, and an attention module is placed between the encoding and decoding layers within the same layer, allowing the first agent to focus more on regions of height variation. For ease of description, the different encoding layers are specifically denoted as: encoding layer 1, encoding layer 2, ..., encoding layer n; and the different decoding layers are denoted as: decoding layer 1, decoding layer 2, ..., decoding layer n. In each encoding layer, convolution and max-pooling operations are performed on the output of the previous encoding layer. The input of encoding layer 1 is the second input vector. Each convolution and max-pooling operation doubles the number of channels in the vector, while halving the width and height. After multiple convolution and max-pooling operations by the encoder, the second input vector is obtained. The second encoded vector output from the last encoding layer n reaches the bottleneck layer and is input to the decoder. In the decoder, each encoding layer performs convolution and max-pooling operations on the output of the previous encoding layer. Before decoding the next encoding layer from the output of the previous decoding layer, a skip connection is made between the decoded vector output from the previous decoder and the corresponding encoding vector of the encoder in the same layer. The input of encoding layer n is the second encoding vector input to the bottleneck layer. This skip connection is implemented through an in-layer attention module, which concatenates and fuses the decoded vector output from the decoder with the corresponding encoding vector of the encoder in the same layer based on an attention mechanism. This skip connection allows the second agent to focus more on regions of height variation in the material stack height map during decoding. After vector concatenation and fusion, a convolution and upsampling operation is performed in each decoding layer. Each time, the dimension of the input vector is halved, while the width and height are doubled. After multiple convolutions and max pooling operations across multiple decoding layers, the final second input vector is processed into a global height variation map. The output is a global height change map with the same size as the material stack height map. Each pixel in the global height change map records the change in material stack height at that location.

[0070] The output layer of the second agent further integrates the global height change map and the material pile height map before excavation to form the final material pile height map after excavation.

[0071] In step S4, the height map of the excavated material pile output by the second agent is sent to the simulation system of the material pile to be worked on, and the simulation system of the material pile to be worked on is updated using the height map of the excavated material pile. Since the dual-agent cooperative model of this invention directly predicts and outputs the height map of the excavated material pile based on the height map of the material pile before excavation and the corresponding bucket posture, it represents a direct state change and cannot describe the material flow process in the material pile to be worked on during excavation. Therefore, this embodiment of the invention innovatively designs the simulation system update; after the simulation system receives the height map of the excavated material pile, it uses a transitional simulation algorithm to update the simulation system of the material pile to be worked on. Specific assumptions... This indicates the target state of the material pile being excavated. Through multiple excavations, the material pile to be worked on is excavated from its initial state to the target state. Each mining operation corresponds to one frame of image, and the material pile is updated to the target state within several frames using a transition simulation algorithm. The update process is as follows:

[0072] ;

[0073] in, A diagram showing the height of the material pile after excavation. This diagram shows the height of the material pile before excavation. Note that "before" and "after" refer to a single excavation or a pre-set excavation cycle. For physical flow terms, In order to excavate the target attraction, This is the coefficient of variation.

[0074] The above update process decomposes the evolution of the material pile height field before and after excavation into a physical flow term and an excavation target attraction term, wherein the physical flow term... This is used to check if each pixel has an "excessive slope" (height difference exceeding the critical slope angle), and to spread excess material to neighbors, automatically creating a "natural flow, collapse" visual effect. It also digs up the target attraction item. Used to gradually pull the current height field toward the target state. It only provides low-frequency convergence without interfering with the local "physical collapse sensation". Specifically, when... When the value is greater than 0, as long as the simulation duration is sufficient, the material pile height diagram before excavation will be used. To the target state This is a mathematically inevitable result. During the material pile update process and for a period of time after the update is completed, the digging judgment will be disabled to prevent erroneous updates of the material pile.

[0075] Through the above process, operation simulation can be carried out for ship hold clearing operations, predict the changes in the shape of the material pile during actual operations, and facilitate automated guidance and control of the loader. With the support of the dual-agent collaborative model and the autonomous driving model, the loader can achieve fully automated material excavation operations without human intervention.

[0076] As an optional implementation, the training dataset used in the dual-agent collaborative model is collected in real-world scenarios. The dataset includes a sufficient number of training and test samples. Typically, the samples in the dataset are divided into training and test sample sets in an 8:2 ratio.

[0077] Both training and testing samples include: real-world material pile height maps before and after each actual excavation, as well as the actual excavation bucket posture; this data can be collected using actual sensors on the loader. In a data stream of an actual excavation operation, each excavation operation is located, and material pile point clouds are collected before and after excavation. Then, the point clouds of the real material pile before and after excavation are downsampled to the same pixel size as the preset material pile height map. The height information of the sampled point cloud is mapped to the corresponding pixel values, thus obtaining the real material pile height maps before and after excavation. Both the before and after excavation height maps are saved in PNG format. The bucket posture data corresponding to the excavation frame is recorded by reading sensors, and the bucket posture data is modeled into a JSON file. This completes the preparation of one data sample, which includes: the real material pile height map before and after each actual excavation, and the actual excavation bucket posture.

[0078] During the training of the first agent, the actual material pile height map after excavation is processed into a real change region mask. This mask is a binary mask used to represent whether the height of each pixel changes before and after excavation. This real change region mask serves as the ground truth during the training of the first agent. During training, the loss function of the first agent is defined. for:

[0079] ;

[0080] in, This represents the binary cross-entropy loss function. For global probability mask, This represents the actual change area mask of the actual material pile height map after actual excavation compared to the actual material pile height map before actual excavation. It is derived from the area where the material pile height changed before and after excavation and is obtained by subtracting the two. Represents similarity loss, used to describe and The similarity.

[0081] During the training of the second agent, the actual material pile height map after excavation is processed into a true height change map. Each pixel in the true height change map represents the change in material height in the corresponding area before and after the actual excavation. This true height change map serves as the ground truth during the training of the second agent. The loss function of the second agent during training... for:

[0082] ;

[0083] in, For absolute error loss, This is a global height change map. This is a graph showing the actual height change of the material pile after excavation compared to the actual material pile height before excavation. This represents the weighting coefficient.

[0084] During training, the overall training loss function of the two-agent cooperative model is further defined. for:

[0085] ;

[0086] in, These are the weighting coefficients.

[0087] The first agent is trained both independently and through the loss function of the second agent during the training process. The network architecture is robust because it is implicitly tuned through backpropagation. Even if the first agent makes a major error, it will not cause a serious error in the second agent. During the training process, the second agent is guided by the global probability mask output by the first agent, and its prediction accuracy will not deviate significantly.

[0088] In summary, the above description is merely a preferred embodiment of this specification and is not intended to limit the scope of protection of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

[0089] The systems, apparatuses, modules, or units described in one or more of the above embodiments may be implemented by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer. Specifically, a computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0090] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0091] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A method for predicting changes in the shape of a material pile, characterized in that, include: Establish a simulation system for the material pile to be worked on, and obtain the height diagram of the material pile before digging and the corresponding bucket posture through the simulation system; The material pile height diagram and bucket posture before excavation are input into the pre-trained dual-agent collaborative model to predict the material pile height after excavation. The dual-agent collaborative model includes a first agent and a second agent. The first agent is input with a material pile height map and bucket posture before digging, and predicts and outputs a global probability mask. The global probability mask is used to identify the probability of each pixel in the material pile height map undergoing shape change due to bucket digging. The second agent is input with the material pile height map, bucket posture, and the global probability mask before digging, and predicts and outputs a global height change map. The global height change map is used to identify the height change value of each pixel in the material pile height map caused by bucket digging. The first intelligent agent includes: an attitude information processing module, a first fusion module, and a first U-net network. The attitude information processing module converts the bucket attitude into an attitude feature vector through an MLP attitude feature encoder, and broadcasts the attitude feature vector along the spatial dimension to generate an attitude feature map; the attitude feature map is then subjected to mean pooling along the channel dimension to generate a single-channel attitude map. The first fusion module is used to stitch together the single-channel attitude diagram and the material pile height diagram before digging as the first input vector of the first U-net network; The first U-net network includes a first encoder and a first decoder. The first encoder is used to perform multiple convolution and max pooling operations on the first input vector to obtain a first encoded vector. The first decoder is used to perform a skip connection between the first encoded vector and the first input vector, and the skip connection adopts an attention mechanism; the first encoded vector after the skip connection and the first input vector are subjected to multiple convolution and upsampling operations, and finally outputs the global probability mask with the same size as the material stack height map. The loss function of the second agent for: ； in, For absolute error loss, This is the global height change map. This is a graph showing the actual height change of the material pile after excavation compared to the actual material pile height before excavation. The global probability mask, Indicates the weighting coefficient; By fusing the global height change map and the material pile height map before excavation, a material pile height map after excavation is predicted and generated. The simulation system for the material pile to be worked on is updated using the height map of the material pile after excavation.

2. The method for predicting changes in the shape of a material pile according to claim 1, characterized in that, The simulation system for the material pile to be operated uses the Unity simulation engine program. The simulation system includes: material pile model, environment model, loader model and sensor model.

3. The method for predicting changes in the shape of a material pile according to claim 1, characterized in that, The second intelligent agent includes: a second fusion module and a second U-net network, wherein the second fusion module is used to concatenate the material pile height map before digging, the single-channel attitude map and the global probability mask as the second input vector of the second U-net network; The second U-net network includes a second encoder and a second decoder. The second encoder is used to perform multiple convolution and max pooling operations on the second input vector to obtain the second encoded vector. The second decoder is used to perform a skip connection between the second encoded vector and the second input vector, and the skip connection adopts an attention mechanism; the skip-connected second encoded vector and the second input vector are subjected to multiple convolution and upsampling operations, and finally output the global height change map with the same size as the material stack height map.

4. The method for predicting changes in the shape of a material pile according to claim 1, characterized in that, The material pile height diagram is in PNG format; the bucket posture is in JSON format.

5. The method for predicting changes in the shape of a material pile according to claim 1, characterized in that, During the training process of the dual-agent collaborative model, the training samples are historical digging data in real-world scenarios. Each training sample includes: a real material pile height map before actual digging, a real material pile height map after actual digging, and the actual digging bucket posture. The preprocessing procedure for training samples is as follows: The point clouds of the actual material pile before and after actual excavation are acquired by sensors on the actual loader. The point clouds of the actual material pile before and after actual excavation are sampled to the same pixel size as the preset height map of the material pile. The height information of the sampled point cloud is mapped to the pixel value of the corresponding pixel, thus obtaining the height map of the actual material pile before and after actual excavation.

6. The method for predicting changes in the shape of a material pile according to claim 5, characterized in that, The loss function of the first agent for: ； in, This represents the binary cross-entropy loss function. The global probability mask, A mask representing the actual change area of ​​the actual material pile height map after excavation compared to the actual material pile height map before excavation; Represents similarity loss, used to describe and The similarity.

7. The method for predicting changes in the shape of a material pile according to claim 1, characterized in that, The total training loss function of the dual-agent cooperative model for: ； in, These are the weighting coefficients.

8. The method for predicting changes in the shape of a material pile according to claim 1, characterized in that, The simulation system for the material pile to be processed is updated using a transitional simulation algorithm. The update process is as follows: ； in, A diagram showing the height of the material pile after excavation. A diagram showing the height of the material pile before excavation. For physical flow terms, In order to excavate the target attraction, The coefficient of variation, The target for excavation of the material pile.

Images