Track planning method and device for automatic loading of excavator, electronic equipment and medium

Abstract

The invention provides a trajectory planning method and device for automatic loading of an excavator, electronic equipment and a medium. The method comprises the following steps: acquiring environment sensing data of a target excavator and real-time joint position information of a plurality of target parts for executing a loading task, and processing the environment sensing data and the real-time joint position information through an end-to-end trajectory planning model; target trajectory point position information of joints of the multiple target parts in a preset number of time steps in the future is predicted, so that target trajectory information of the target excavator is determined, and the end-to-end trajectory planning model is trained based on a training data set constructed by real task data of loading tasks through manual operation in different scenes; the whole environment sensing information is transmitted in a lossless manner in the whole planning process, so that the global optimization of the track planning of automatic loading is realized, and the earth cutting strategy is more reasonable and personified.

Description

Technical Field

[0001] The present application relates to the field of excavators, and more particularly, to a trajectory planning method, device, electronic device, and medium for automatic loading of excavators. Background Art

[0002] With the popularization of intelligence and the improvement of AI capabilities, in the field of excavators, some automated or semi-automated tasks have gradually begun. A common task is the loading task, because most of the work of excavators is in the behavior of excavating and loading. Taking the excavator loading task as an example, currently, most of the intelligent work in the industry is to enable the excavator operator to remotely control the excavator to perform the loading task through a set of remote solutions, but the participation of the excavator operator is still required. If it is necessary to reduce the input of manpower, it is undoubtedly very necessary to be able to perform automatic loading without human intervention. Currently, the common method of unmanned automatic loading is to complete it through the mutual cooperation of modules, generally including a perception module and a planning module. The perception module extracts and perceives information from images or videos, and then transmits the extracted information to the planning module to perform trajectory planning using traditional planning algorithms or planning algorithms based on deep learning. In this process, the perceived information may be lost during transmission, and the planned trajectory may not be globally optimal. At the same time, the excavation points planned according to the extracted information are not necessarily optimal, or in line with the excavation strategy of the excavator operator, and it is difficult to learn some excavation actions, which is relatively mechanical. Summary of the Invention

[0003] In view of this, the purpose of the present application is to provide a trajectory planning method, device, electronic device, and medium for automatic loading of excavators, which can realize end-to-end planning from environmental perception information to the trajectory of automatic loading, making the excavation strategy more reasonable.

[0004] In a first aspect, a trajectory planning method for automatic loading of an excavator provided by an embodiment of the present application includes:

[0005] Obtain the environmental perception data of the target excavator and the target real-time joint position information matching the environmental perception data; the target real-time joint position information includes the real-time joint position information of multiple target parts performing the loading task;

[0006] Input the environmental perception data and the target real-time joint position information of the target excavator into a pre-trained end-to-end trajectory planning model; the end-to-end trajectory planning model is trained based on a training data set constructed from task data for performing loading tasks in different scenarios;

[0007] Process the environmental perception data and the target real-time joint position information of the target excavator through the end-to-end trajectory planning model to predict the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps;

[0008] Based on the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps, determine the target trajectory information of the target excavator.

[0009] In a second aspect, an embodiment of the present application provides a trajectory planning device for automatic loading of an excavator. The device includes:

[0010] An acquisition module, configured to acquire the environmental perception data of the target excavator and the target real-time joint position information matching the environmental perception data; the target real-time joint position information includes the real-time joint position information of multiple target parts performing the loading task;

[0011] An input module, configured to input the environmental perception data and the target real-time joint position information of the target excavator into a pre-trained end-to-end trajectory planning model; the end-to-end trajectory planning model is trained based on a training data set constructed from task data for performing loading tasks in different scenarios;

[0012] A processing module, configured to process the environmental perception data and the target real-time joint position information of the target excavator through the end-to-end trajectory planning model to predict the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps;

[0013] A determination module, configured to determine the target trajectory information of the target excavator based on the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the memory through the bus. When the machine-readable instructions are executed by the processor, the steps of the trajectory planning method for automatic loading of an excavator are performed.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is run by a processor, the steps of the trajectory planning method for automatic loading of an excavator are performed.

[0016] An embodiment of the present application provides a trajectory planning method, device, electronic device, and medium for automatic loading of an excavator; the method obtains environmental perception data of a target excavator and target real-time joint position information of multiple target parts performing a loading task, processes the environmental perception data and the target real-time joint position information of the target excavator through the end-to-end trajectory planning model, predicts the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps, thereby determining the target trajectory information of the target excavator. The entire environmental perception information is transmitted without loss during the entire planning process, thereby realizing the global optimization of the trajectory planning for automatic loading, taking into account the planning of how the bucket digs the soil, making the soil excavation strategy more reasonable at the global level; the end-to-end trajectory planning model is trained based on a training data set constructed from real task data of performing a loading task through manual operation in different scenarios. The model trained based on a large amount of human data makes the behavior of automatic loading more anthropomorphic and the soil excavation strategy more reasonable, and can also handle more scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and thus should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other relevant drawings can also be obtained based on these drawings.

[0018] Figure 1 The module schematic diagram showing the solution of the automatic loading task in the prior art;

[0019] Figure 2 The flowchart showing the trajectory planning method for automatic loading of the excavator according to the embodiment of the present application;

[0020] Figure 3 The schematic diagram showing the main perspective view and the elevation map according to the embodiment of the present application;

[0021] Figure 4 The flowchart showing the method for constructing the training data set according to the embodiment of the present application;

[0022] Figure 5 The flowchart showing the method for determining the target trajectory information of the target excavator according to the embodiment of the present application;

[0023] Figure 6 The schematic diagram showing the process of the time aggregation operation according to the embodiment of the present application;

[0024] Figure 7 The structural schematic diagram showing the trajectory planning device for automatic loading of the excavator according to the embodiment of the present application;

[0025] Figure 8 The structural schematic diagram of the electronic device according to the embodiment of the present application is shown. Detailed implementation manners

[0026] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. It should be understood that the accompanying drawings in the present application are only for the purposes of illustration and description, and are not used to limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual scale. The flowcharts used in the present application show the operations implemented according to some embodiments of the present application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and the steps without logical context may be reversed or implemented simultaneously. In addition, those skilled in the art may add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of the present application.

[0027] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and illustrated in the accompanying drawings here may be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but merely represents the selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts fall within the protection scope of the present application.

[0028] It should be noted that the term "including" will be used in the embodiments of the present application to indicate the existence of the subsequently stated features, but does not exclude the addition of other features.

[0029] With the popularization of intelligence and the improvement of AI capabilities, in the field of excavators, some automated or semi-automated tasks have gradually begun. The most common one is the loading task, because most of the work of excavators is in the behavior of digging and loading. Taking the excavator loading task as an example, at present, most of the intelligent work in the industry is to enable the excavator operator to remotely control the excavator to perform the loading task through a set of remote solutions, but the participation of the excavator operator is still required. If we want to reduce the input of manpower, it is undoubtedly necessary to be able to load automatically without human intervention. At present, the most common way of unmanned automatic loading is to complete it through the mutual cooperation of modules, generally including a perception module and a planning module. The perception module extracts and perceives information from images or videos, and then transmits the extracted information to the planning module to perform trajectory planning using traditional planning algorithms or planning algorithms based on deep learning. In this process, the perceived information may be lost during transmission, and the planned trajectory may not be globally optimal. At the same time, the excavation points planned according to the extracted information are not necessarily the best, or in line with the excavation strategy of the excavator operator, and it is difficult to learn some excavation actions, which is relatively mechanical.

[0030] At present, the most common way of intelligent loading tasks is to use a remote control system to allow the excavator operator to load remotely, and the whole process is controlled by the excavator operator. Some simple automated functions will be provided on this basis, such as one-key soil loading. The excavator operator needs to specify the excavation point and the dumping point, and then click one-key excavation. The model will automatically plan a path according to the excavation point and the dumping point and then perform a single round of digging, turning and unloading. The excavation position and the dumping position are both determined manually, and the planned path can be achieved through traditional planning algorithms.

[0031] This method of automatic loading has relatively little automation and still requires the participation of the excavator operator, only reducing some of the workload of the excavator operator.

[0032] In addition, there are also a small number of solutions for automatic loading tasks, which usually consist of multiple modules, such as a perception module, a planning module, a control module, etc., as shown below Figure 1 as follows.

[0033] These modules need to be designed and optimized separately, and then combined in sequence to complete the automatic loading task. The perception module perceives the state of the soil and the state of the soil in the truck according to the acquired image information, and then makes a decision to determine the dumping point and the unloading point. The excavation point and the dumping point information are transmitted to the planning module for trajectory planning to complete the movement of the excavator from the excavation point to the unloading point. In addition, an additional bucket planning module is needed to plan the excavation behavior, and different bucket planning schemes need to be designed for different soil states. Finally, the control module performs position control or speed control according to the position information or speed information given by the planning module.

[0034] In the solution for automatically loading trucks based on the module combination method, first, information loss may occur during the transmission of the perceived information, and the planned trajectory is not necessarily globally optimal. Second, since a large amount of loading data from excavator operators is not used for training, many traditional planning methods are used for trajectory planning from the excavation point to the dumping point, or based on specific excavation points and dumping points selected, the resulting automatic loading behavior is relatively mechanical and not human-like. At the same time, if the planning module for how the bucket digs the soil is based on certain rules, it is not applicable to different soil conditions, resulting in a low full-bucket rate for soil excavation.

[0035] Based on this, the embodiments of the present application provide a trajectory planning method, device, electronic device, and medium for automatic loading of an excavator. The method obtains environmental perception data of a target excavator and target real-time joint position information of multiple target parts for performing the loading task, and processes the environmental perception data and the target real-time joint position information of the target excavator through the end-to-end trajectory planning model to predict the target trajectory point position information of the joints of the multiple target parts in a preset number of future time steps, thereby determining the target trajectory information of the target excavator. The entire environmental perception information is transmitted without loss during the entire planning process, thereby achieving global optimization of the trajectory planning for automatic loading, taking into account the planning of how the bucket digs the soil, and making the soil excavation strategy more reasonable at the global level. The end-to-end trajectory planning model is trained based on a training data set constructed from real task data of performing the loading task through manual operation in different scenarios. The model trained based on a large amount of human data makes the automatic loading behavior more anthropomorphic, the soil excavation strategy more reasonable, and can also handle more scenarios.

[0036] Please refer to Figure 2 , Figure 2 which shows the flowchart of the trajectory planning method for automatic loading of an excavator according to the embodiments of the present application. As Figure 2 shown, the trajectory planning method for automatic loading of an excavator includes the following steps S201 - S204:

[0037] S201. Obtain the environmental perception data of the target excavator and the target real-time joint position information matching the environmental perception data. The target real-time joint position information includes the real-time joint position information of multiple target parts for performing the loading task.

[0038] S202. Input the environmental perception data and the target real-time joint position information of the target excavator into a pre-trained end-to-end trajectory planning model. The end-to-end trajectory planning model is trained based on a training data set constructed from task data of performing the loading task in different scenarios.

[0039] S203. Process the environmental perception data and the target real-time joint position information of the target excavator through the end-to-end trajectory planning model, and predict the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps;

[0040] S204. Determine the target trajectory information of the target excavator based on the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps.

[0041] In an alternative embodiment, the trajectory planning method for automatic loading of the excavator can be implemented in the processor of the excavator or in a server. When the trajectory planning method for automatic loading of the excavator runs on the server, the method can be implemented and executed based on a cloud interaction system, where the cloud interaction system includes the server and the processor of the excavator.

[0042] In an alternative embodiment, when the trajectory planning method for automatic loading of the excavator is implemented and executed based on a cloud interaction system, the processor of the excavator acquires the environmental perception data and the real-time position information of the target excavator, and sends the acquired environmental perception data and real-time position information of the target excavator to the server. The server inputs the received environmental perception data and real-time position information into a pre-trained end-to-end trajectory planning model, and processes the environmental perception data and the real-time position information of the target excavator through the end-to-end trajectory planning model; the server determines the planning information based on the real-time position information and sends the planning information to the processor of the excavator.

[0043] In some embodiments, when the trajectory planning method for automatic loading of the excavator is executed based on the processor of the excavator, the processor of the excavator acquires the environmental perception data and the real-time position information of the target excavator, and processes the environmental perception data and the real-time position information of the target excavator through a pre-deployed end-to-end trajectory planning model; the server determines the planning information based on the real-time position information.

[0044] In the step S101, acquire the environmental perception data of the target excavator and the target real-time joint position information matching the environmental perception data; the target real-time joint position information includes the real-time joint position information of multiple target parts performing the loading task.

[0045] The target excavator refers to a specific excavator performing the automatic loading task.

[0046] The environmental perception data refers to the surrounding environment information collected by the excavator through its sensors (such as cameras, radars, lidars, etc.).

[0047] For the automatic loading task, the environmental perception data includes the task objects of the automatic loading task, such as soil piles, material piles, and so on.

[0048] The target real-time joint position information represents the current position and attitude information of multiple target parts of the target excavator performing the loading task.

[0049] For the automatic loading task, the target real-time joint position information includes the real-time joint position information of multiple target parts performing the loading task.

[0050] The multiple target parts performing the loading task refer to the main working components involved when the excavator performs the loading task. Exemplarily, the target parts include the cab, boom, arm, and bucket, etc.

[0051] Each of the target parts corresponds to a joint, and the position and attitude of the target part are controlled by rotating the joint of the target part. Therefore, the real-time joint position information of the target part is the real-time joint position information of the joint of the target part.

[0052] Based on this, the real-time joint position information includes the angle information of the joint.

[0053] In the embodiments of the present application, the obtaining of the real-time position information matching the target excavator and the environmental perception data includes:

[0054] Obtaining the initial joint position information of the joint of the cab of the target excavator when it is powered on;

[0055] Based on the time information of the environmental perception data of the target excavator, obtaining the relative joint position information of the joint of the cab of the target excavator relative to the initial joint position information, and obtaining the absolute joint position information of the joints of the boom, arm, and bucket;

[0056] Determining the relative joint position information of the joint of the cab, and the absolute joint position information of the joints of the boom, arm, and bucket as the target real-time joint position information of the target excavator.

[0057] That is to say, the real-time joint position information of the multiple target parts includes the real-time joint position information of the cab, boom, arm, and bucket. Among them, the real-time joint position information of the cab is relative position information, and the real-time joint position information of the boom, arm, and bucket is absolute position information.

[0058] The environmental perception data of the target excavator and the real-time joint position information are matched in time. For example, if one frame of environmental perception data is collected every 0.1 s, then correspondingly, one frame of the real-time position information is collected every 0.1 s.

[0059] For the scenario of an excavator, since the joint positions of the excavator's cabin change every time it is powered on, there is a zero point. Each time the position directly in front of the cabin at startup is used as the zero point. Then, if the cabin position changes when the machine is powered on each time, the joint position values will be different. Even in the same situation in a large amount of collected training data, the cabin positions are different, so training cannot be carried out. Therefore, in the embodiments of the present application, the automatic loading task is set to start from when the bucket is above the truck and end when a truck is full. The position information of the joints of the cabin is not absolute position information and is all expressed as relative positions relative to the starting position, but the position information of the boom, arm, and bucket is still absolute position information.

[0060] Exemplarily, if the position information of the boom, arm, cabin, and bucket in the initial state is J_0 = {j_boom_0, j_arm_0, j_swing_0, j_bucket_0}, then the position information at the subsequent t-th moment is expressed as J_i = {j_boom_i, j_arm_i, j_swing_i - j_swing_0, j_bucket_i}.

[0061] In an optional embodiment, the obtaining of the environmental perception data of the target excavator includes:

[0062] Obtaining multiple frames of image information captured by a camera of the target excavator and multiple frames of point cloud data obtained by radar scanning;

[0063] Processing the multiple frames of point cloud data to determine multiple frames of elevation maps corresponding to the point cloud data.

[0064] On the target excavator performing a loading task in scenarios such as mines, cameras, lidars, and sensors are installed to obtain the main perspective view, panoramic view, point cloud information obtained by the lidar, and the position information and speed information of each joint (boom, arm, cabin, bucket) of the excavator during the loading task process.

[0065] In some embodiments, the image information captured by the camera is the main perspective view. The main perspective view is the view captured from the perspective of the excavator operator. Therefore, it provides the most direct and real visual feedback for the operator during the loading task. Through the main perspective view, the position and state of the bucket, materials, and loading vehicle of the excavator from the operator's perspective can be clearly seen, enabling more accurate and anthropomorphic trajectory planning and prediction.

[0066] The multiple frames of point cloud data obtained by radar scanning. Specifically, the lidar measures information such as the distance, shape, and position of the target object by emitting laser beams and receiving their reflected signals; these measurement data are distributed in a three-dimensional space in the form of points to form point cloud data.

[0067] In complex loading tasks, excessive visual information can cause data redundancy, increase the computational burden on the model, and may lead to redundancy or confusion in the model when processing data, reducing the prediction efficiency. By only using the main perspective view, unnecessary visual interference can be reduced, and the efficiency and rationality of trajectory planning can be improved.

[0068] Please refer to Figure 3 , Figure 3 which shows a schematic diagram of the main perspective view and the elevation map described in the embodiments of the present application.

[0069] The elevation map is a map representing the height changes of the ground or the surface of an object; based on this, in the embodiments of the present application, the elevation map corresponding to the point cloud data characterizes the height, shape, and position of the material pile in the automatic loading task, and is used to assist the end-to-end trajectory planning model in learning the height, shape, and position of the material pile.

[0070] In step S102, the environmental perception data and the target real-time joint position information of the target excavator are input into a pre-trained end-to-end trajectory planning model; the end-to-end trajectory planning model is trained based on a training data set constructed from task data for loading tasks in different scenarios.

[0071] The data of the loading tasks in different scenarios includes: the positions and speeds of the joints of multiple target parts of the excavator during the loading task by manual operation, the main perspective view in the camera installed on the excavator, and the point cloud data of the lidar.

[0072] Specifically, a large amount of data on the loading tasks of excavator operators in different scenarios is collected and cleaned as the training data set for model training.

[0073] In some embodiments, please refer to Figure 4 , the training data set is constructed based on the following method:

[0074] S401. Obtain sample task data for loading tasks in different scenarios; the sample task data includes: sample environmental perception information and target sample joint position information matching the sample environmental perception information; the sample environmental perception information includes sample image information captured by a camera and sample point cloud data obtained by radar scanning; the target sample joint position information includes: sample joint information of the joints of the multiple target parts; the sample joint information of the joints of the target parts includes sample joint positions and sample joint speeds;

[0075] S402. Verify and clean the sample task data based on the sample joint information of each target part, the sample image information, and the matching relationship between the sample image information and the sample joint information;

[0076] S403. Construct a training data set based on the cleaned sample task data.

[0077] In some embodiments, the verifying and cleaning of the sample task data based on the sample joint information, sample image information, and the matching relationship between the sample image information and the sample joint information of each target part joint includes:

[0078] Perform a primary screening on the sample task data according to the pre-configured joint position range and joint speed range of each target part during normal operation to obtain the sample task data after the primary screening;

[0079] Perform a secondary screening on the sample task data after the primary screening based on the sample image information to obtain the sample task data after the secondary screening;

[0080] Perform visualization processing on the sample task data after the secondary screening, sample and check whether the sample image information after the secondary screening matches the sample joint information, and select the matching sample task data.

[0081] Specifically, on an excavator performing a loading task in diverse scenarios such as mines, cameras, lidar, and sensors are installed to obtain the main perspective view during the loading task process, the point cloud information obtained by the lidar, and the position information and speed information of each joint (boom, arm, cab, bucket) of the excavator.

[0082] While the excavator operator is performing the daily loading task, a data collection script is started (manually started and stopped by the operator, and a set of data is collected for each full truckload), and a large amount of real data will be collected. The data is stored in the hard disk of the excavator body and the data is recovered regularly.

[0083] Since the collected data is real loading data, there are many dirty data that need to be cleaned.

[0084] When cleaning the data, first determine the accurate range of the position and speed of each joint according to the values of each joint during normal operation, which can initially screen out the data of the normal operation of the excavator operator and exclude the dirty data caused by sensor abnormalities or abnormal behaviors of the excavator operator; secondly, perform numerical screening on the image information to exclude phenomena such as green screens, black screens, and flower screens of the camera caused by accidental damage; finally, the collected data will be visualized, and sample and check whether the image information corresponds to the joint information.

[0085] Visualize the collected data, and sample and check whether the image information corresponds to the joint information as follows: A set of data is when a truck is fully loaded. Visualize the set of data, and the visualization result is the loading video of the main perspective view, the video of the elevation map, and the curve graph of the position information of the four joints; the content of the loading video is the behavior of fully loading a truck, and then check whether the curve changes of the joints in the curve graph of the position information of the four joints are normal corresponding to the video.

[0086] Based on the cleaned data training dataset, the number of data in the training dataset is constantly increasing.

[0087] In some alternative embodiments, the end-to-end trajectory planning model is constructed based on a generative policy learning method of a conditional denoising diffusion process.

[0088] The conditions of the end-to-end trajectory planning model are the environmental perception information and the joint position information of the target part.

[0089] Based on this, when training the constructed end-to-end trajectory planning model, the sample environmental perception information and the target sample joint position information matching the sample environmental perception information are used as conditions.

[0090] In some alternative embodiments, the core algorithm of the model adopts the Diffusion Policy algorithm, which is a generative artificial intelligence method based on diffusion policy. It is more suitable for robot scenarios with multi-dimensional action spaces and can predict multi-step actions at one time, which is beneficial to completing the automatic loading task.

[0091] In the embodiments of the present application, the environmental perception information (visual information) and the real-time position information (joint information) of the target part are used as conditions for the denoising diffusion process of the action, and finally the action sequence required for decision-making is obtained to complete the automatic loading task.

[0092] In the step S103, the environmental perception data and the target real-time joint position information of the target excavator are processed by the end-to-end trajectory planning model to predict the target trajectory point position information of the joints of the multiple target parts at the future preset number of time steps.

[0093] The target trajectory point position information of the joints of the multiple target parts at the future preset number of time steps, that is, the target trajectory point position information within the future preset time period.

[0094] The target trajectory point position information of the joints of the target part is the change amount of the joints of the target part relative to the input real-time joint position information.

[0095] Processing the environmental perception data and the target real-time joint position information of the target excavator through the end-to-end trajectory planning model to predict the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps, including:

[0096] The end-to-end trajectory planning model uses the environmental perception data and the target real-time joint position information of the target excavator as conditions to denoise and diffuse the actions, and predicts the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps.

[0097] The output of the end-to-end trajectory planning model described in the embodiments of the present application does not directly predict the absolute position information of each joint in the future, which will increase the learning difficulty. The model output is the relative position of each joint relative to the position of each joint in the input state; if the current is time i and the model input is {O_(i - 7), O_(i - 6),... O_i, J_(i - 7), J_(i - 6),…, J_i}, then the output of the model of the present invention is {ΔJ_(i + 1), ΔJ_(i + 2),…, ΔJ_(i + 16)} = {J_(i + 1) - J_i, J_(i + 2) - J_i,…, J_(i + 16) - J_i}, that is, the change amount of each joint relative to the current position; then the position information of each joint in the future is obtained through conversion and sent to the position controller for execution.

[0098] In step S104, based on the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps, determine the target trajectory information of the target excavator.

[0099] That is to say, the target trajectory information includes the target trajectory points of the joints of the multiple target parts at a preset number of future time steps.

[0100] The determining the target trajectory information of the target excavator based on the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps includes:

[0101] Based on the input real-time joint position information of the multiple target parts and the change amount of the multiple target parts relative to the input real-time joint position information at a preset number of future time steps, determine the target trajectory information of the joints of the multiple target parts at a preset number of future time steps.

[0102] The target trajectory points of the joints of the multiple target parts at a preset number of future time steps, that is, the target trajectory point sequence within a preset future time period.

[0103] The target trajectory points for the preset number of future time steps, that is, the trajectory points that the joints of the target part should reach at the corresponding time step, or the expected joint positions to be reached.

[0104] Task modeling is performed for the automatic loading task to obtain an end-to-end trajectory planning model; the inputs of the model: the elevation map corresponding to the point cloud data of the radar, the main perspective image and the cockpit, and the joint positions of the boom, arm, and bucket; the outputs of the model: the predicted joint positions of the boom, arm, cockpit, and bucket, so as to directly implement an end-to-end trajectory planning scheme from the input of perception information to the output of joint position planning.

[0105] Among them, the input elevation map, main perspective image, and joint positions are all multi-frame data that are time-matched. The multi-frame joint position information is to input the position information within a period of time, which can be used to express the speed information of the joints.

[0106] For the output of the model, it is set as the position differences of the boom, arm, cockpit, and bucket within a future period of time, that is, the change amount relative to the current position. Then, based on the current position, the position information of each joint planned within a future period of time can be obtained.

[0107] In some alternative embodiments, after determining the target trajectory information of the target excavator, the method further includes:

[0108] Sending the target trajectory information of the target excavator to the controller of the target excavator, so that the controller controls the joints of multiple target parts of the target excavator to act at the corresponding time step based on the target trajectory information.

[0109] Sending the position information of multiple target parts to the position controller can guide the controller to execute commands to reach the planned positions.

[0110] Exemplarily, in the embodiments of the present application, the inputs of the model are multi-frame visual information and real-time joint position information: {O_(i - 7), O_(i - 6),... O_i, J_(i - 7), J_(i - 6), …, J_i}, where O_i represents the main perspective and elevation map images at time i, and J_i represents the real-time joint position information of the boom, arm, cockpit, and bucket at time i. The input is stacked for 8 frames in total, representing the state information in the past 0.8 seconds (assuming perception is performed at a frequency of 10Hz). The output of this algorithm is {J_(i + 1), J_(i + 2), …, J_(i + 16)}, representing the position information of the boom, arm, cockpit, and bucket in the future 1.6s.

[0111] In some alternative embodiments, the end-to-end trajectory planning model is trained based on the following method:

[0112] The end-to-end trajectory planning model is trained based on the following method:

[0113] For the constructed end-to-end trajectory planning model, determine the main task for the loading task;

[0114] Determine the subtasks for image processing; the subtasks include: image reconstruction task and / or image segmentation task;

[0115] While training the end-to-end trajectory planning model based on the main task, after the feature output of the network in the end-to-end trajectory planning model for processing images, perform the subtask training to obtain the trained end-to-end trajectory planning model.

[0116] The concept of subtasks is added to the basic Diffusion Policy algorithm, mainly to increase the capabilities of the neural network, which helps improve the network's perception ability of images. The present invention mainly designs two subtasks for selection. One is the image reconstruction task, which does not require an additional dataset. The other is the image segmentation task, which requires using a segmentation algorithm to first segment and label the images in the dataset. The idea of this subtask is general, and more subtasks can be constructed according to one's own needs.

[0117] The subtask is to, while performing the normal training task, after the feature output of the network for processing images in the previous part, perform other task training, and the loss of this task will also be backpropagated to update the parameters of the network for processing images. That is, the gradients of training the loading task and the gradients of training the image reconstruction task will both be backpropagated to update the parameters of the image processing network, but the weights of the gradient backpropagation of the two can be adjusted. During the training phase, the main task and the subtask are trained simultaneously, so that the network in the image processing part has the ability of the subtask while completing the main task. During the test deployment, the perception of the image by this part of the network is better than that of the network that has not undergone subtask training.

[0118] In some embodiments, please refer to Figure 5 , determining the target trajectory information of the target excavator based on the position information of the target trajectory points of the joints of the multiple target parts at the future preset number of time steps includes the following steps S501 - S503:

[0119] S501. Obtain the position information of the target trajectory points of the joints of the target parts at the future preset number of time steps output by the end-to-end trajectory planning model in multiple rounds of prediction; wherein, the time interval between adjacent rounds of prediction is less than the total duration of the future preset number of time steps;

[0120] S502. Aggregate the position information of the target trajectory points of the joints of the target parts at the same time step in multiple rounds of prediction to obtain the updated position information of the target trajectory points of the joints of the target parts;

[0121] S503: Determine the target trajectory information of the target excavator based on the updated target trajectory point position information of the joint of the target part.

[0122] In some embodiments, the aggregating the target trajectory point position information of the joints of the target part at the same time step in multiple rounds of prediction to obtain the updated target trajectory point position information of the joints of the target part includes:

[0123] Determine the aggregation weights of different rounds of predictions based on the degree of influence of different rounds of predictions on the target trajectory point position information of the joints of the target parts at the same time step;

[0124] The target trajectory point position information of the joints of the target part at the same time step in multiple rounds of predictions is aggregated based on the aggregation weights of different rounds of predictions to obtain updated target trajectory point position information of the joints of the target part.

[0125] Specifically, the time aggregation operation takes into account the model's reasoning time and data acquisition time, and combines the predictions of the previous rounds with the predictions of the current round; please refer to Figure 6 , Figure 6 A schematic diagram of the process of the time aggregation operation described in the embodiment of the present application is shown; Figure 6 As shown, for example, the time of each step is defined as 0.1s, that is, the frequency of acquiring the state is 10Hz. The first row is the result of the model reasoning output at time t0 {J1_0, J2_0, ..., J16_0}. The subscript 0 here is to distinguish the position information of the same step number inferred from the following steps, which is the predicted position information of the arm, arm, cabin and bucket in the next 16 steps. Since it takes time to obtain state information in actual applications, it is shown in the figure that it takes 0.4s to acquire data. At this moment, the output of the second reasoning of the model is {J5_1, J6_1, ..., J20_1}, but model reasoning also takes time. If it is 0.2s in the figure, then only 14 steps of future position information {J7, J8, ..., J20} will be sent to the controller at this time. Similarly, the future position information sent to the controller at the next moment is {J11, J12, ..., J24}.

[0126] And each time the model sends the controller the joint position information for the next few steps, it needs to integrate the information of the current step model reasoning result and the previous reasoning connection (i.e. vertically). Figure 6 As shown in the dark dashed box, J10 is calculated based on J10_0 and J0_1, and since J20 was not predicted in the previous step, J20 = J20_1.

[0127] The specific calculation formula is as follows: Assume that currently, Ji needs to be sent to the controller, and Ji has given a total of n predictions (vertically), that is, there exist {Ji^0, Ji^1, … Ji^m …, Ji^(n - 1)}. Then each value will be given a weight wi^m, that is, {wi^0, wi^1, … wi^m …, wi^n}. Then Ji = Σm(wi^m * Ji^m) / Σj(wi^m), where wi^m = exp(-0.01 * j).

[0128] Here, Ji represents the position information of the i-th step of the prediction; wi^m represents the position information of the i-th step of the m-th prediction.

[0129] In some embodiments, Ji can be represented as J i , Ji^m can be represented as J i m , the weight wi^m can be represented as w i m , then

[0130] For example, the J10 (dark dashed box) sent by the model for the second time to the outside is composed of J10_0 and

[0131] J0_1 two predictions, that is, n is 2, and it needs to be sent to the outside:

[0132] J10 = (w10^0 * J10^0 + w10^1 * J10^1) / (w10^0 + w10^1) = (w10^0 * J10_0 + w10^1 * J10_1) / (w10^0 + w10^1) = (J10_0 + 0.99 * J10_1) / 1.99; w10^0 = exp(-0.01 * 0) = 1, w10^1 = exp(-0.01 * 1) = 0.99; the J20 (dark dashed box) sent by the model for the second time to the outside only has J20_1, then J20 = (w20^0 * w20^0) / w20^0 = w20^0 = w20_1.

[0133] Looking at the light dashed-line box again, this is the third time for model inference. At this time, J12 corresponds to three predictions, that is, n = 3. At this time, the externally sent J12 = (w12^0 * J12^0 + w12^1 * J12^1 + w12^2 * J12^2) / (w12^0 + w12^1 + w12^2) = (w12^0 * J12_0 + w12^1 * J12_1 + w12^2 * J12_2) / (w12^0 + w12^1 + w12^2) = (J12_0 + 0.99 * J12_1 + 0.98 * J12_2) / 2.97; at this time, J18 corresponds to two predictions, that is, n = 2. At this time, the externally sent J18 = (w18^0 * j18^0 + w18^1 * j18^1) / (w18^0 + w18^1) = (w18^0 * j18_1 + w18^1 * j18_2) / (w18^0 + w18^1) = (J18_1 + 0.99 * J18_2) / 1.99.

[0134] In some embodiments, in the trajectory planning method for automatic loading of an excavator, the end-to-end trajectory planning model is trained offline, and the trained end-to-end trajectory planning model is deployed to the target excavator.

[0135] The model is trained in an offline training manner, that is, after collecting a large amount of training data, the model is trained on a local high-performance device, and then the model is deployed to the excavator after completion. It has less dependence on the excavator body, as long as it can perform model inference. The problems such as delay caused by the on-site network or the performance of the excavator body have been solved by the above-mentioned time aggregation method. After the deployment of the model, place the bucket of the excavator above the truck, and then execute the model operation to perform the automatic loading task. The overall performance looks similar to the loading task performed manually.

[0136] Based on the same inventive concept, an embodiment of the present application also provides a trajectory planning device for automatic loading of an excavator corresponding to the trajectory planning method for automatic loading of an excavator. Since the principle of solving problems by the device in the embodiment of the present application is similar to the above-mentioned trajectory planning method for automatic loading of an excavator in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0137] Please refer to Figure 7 , Figure 7 which shows a schematic structural diagram of the trajectory planning device for automatic loading of an excavator according to an embodiment of the present application; as Figure 7 shown, the trajectory planning device for automatic loading of an excavator includes:

[0138] An acquisition module 701 is configured to acquire environmental perception data of a target excavator and target real-time joint position information that matches the environmental perception data; the target real-time joint position information includes real-time joint position information of multiple target parts performing a loading task.

[0139] An input module 702 is configured to input the environmental perception data and the target real-time joint position information of the target excavator into a pre-trained end-to-end trajectory planning model; the end-to-end trajectory planning model is trained based on a training dataset constructed from task data for performing loading tasks in different scenarios.

[0140] A processing module 703 is configured to process the environmental perception data and the target real-time joint position information of the target excavator through the end-to-end trajectory planning model, and predict target trajectory point position information of joints of the multiple target parts at a preset number of future time steps.

[0141] A determination module 704 is configured to determine target trajectory information of the target excavator based on the target trajectory point position information of joints of the multiple target parts at a preset number of future time steps.

[0142] In some embodiments, the trajectory planning device for automatic loading of an excavator further includes:

[0143] A sending module is configured to, after determining the target trajectory information of the target excavator, send the target trajectory information of the target excavator to a controller of the target excavator, so that the controller controls joints of multiple target parts of the target excavator to act at corresponding time steps based on the target trajectory information.

[0144] In some embodiments, in the trajectory planning device for automatic loading of an excavator, the target trajectory point position information of the joint of the target part is the change amount of the joint of the target part relative to the input real-time joint position information.

[0145] In some embodiments, in the trajectory planning device for automatic loading of an excavator, when the determination module determines the target trajectory information of the target excavator based on the target trajectory point position information of joints of the multiple target parts at a preset number of future time steps, it is specifically configured to:

[0146] Based on the input real-time joint position information of the multiple target parts and the change amount of the multiple target parts relative to the input real-time joint position information at a preset number of future time steps, determine the target trajectory information of joints of the multiple target parts at a preset number of future time steps.

[0147] In some embodiments, for the trajectory planning device of automatic loading of an excavator, when the determining module determines the target trajectory information of the target excavator based on the position information of the target trajectory points of the joints of the plurality of target parts at a preset number of future time steps, it specifically is used for:

[0148] Obtain the position information of the target trajectory points of the joints of the target part output by the end-to-end trajectory planning model in multiple rounds of prediction at a preset number of future time steps; wherein, the time interval between adjacent rounds of prediction is less than the total duration of the preset number of future time steps;

[0149] Aggregate the position information of the target trajectory points of the joints of the target part at the same time step in multiple rounds of prediction to obtain the updated position information of the target trajectory points of the joints of the target part;

[0150] Based on the updated position information of the target trajectory points of the joints of the target part, determine the target trajectory information of the target excavator.

[0151] In some embodiments, for the trajectory planning device of automatic loading of an excavator, when the determining module aggregates the position information of the target trajectory points of the joints of the target part at the same time step in multiple rounds of prediction to obtain the updated position information of the target trajectory points of the joints of the target part, it specifically is used for:

[0152] Based on the influence degree of different rounds of prediction on the position information of the target trajectory points of the joints of the target part at the same time step, determine the aggregation weights of different rounds of prediction;

[0153] Aggregate the position information of the target trajectory points of the joints of the target part at the same time step in multiple rounds of prediction based on the aggregation weights of different rounds of prediction to obtain the updated position information of the target trajectory points of the joints of the target part.

[0154] In some embodiments, for the trajectory planning device of automatic loading of an excavator, when the obtaining module obtains the environmental perception data of the target excavator, it specifically is used for:

[0155] Obtain multiple frames of image information captured by the camera of the target excavator and multiple frames of point cloud data obtained by radar scanning;

[0156] Process the multiple frames of point cloud data to determine the multiple frames of elevation maps corresponding to the point cloud data.

[0157] In some embodiments, for the trajectory planning device of automatic loading of an excavator, when the obtaining module obtains the target real-time joint position information matching the target excavator and the environmental perception data, it specifically is used for:

[0158] Obtain the initial joint position information of the joints of the cockpit of the target excavator when it is powered on;

[0159] Based on the time information of the environmental perception data of the target excavator, obtain the relative joint position information of the joints of the cockpit of the target excavator relative to the initial joint position information, and obtain the absolute joint position information of the joints of the boom, arm, and bucket.

[0160] Determine the relative joint position information of the joints of the cockpit, and the absolute joint position information of the joints of the boom, arm, and bucket as the target real-time joint position information of the target excavator.

[0161] In some embodiments, the trajectory planning device for automatic loading of the excavator, the end-to-end trajectory planning model is constructed based on a generative policy learning method of a conditional denoising diffusion process; when training the constructed end-to-end trajectory planning model, the sample environmental perception information and the target sample joint position information matching the sample environmental perception information are used as conditions.

[0162] In some embodiments, the trajectory planning device for automatic loading of the excavator further includes:

[0163] A training module for training the end-to-end trajectory planning model based on the following steps:

[0164] For the constructed end-to-end trajectory planning model, determine the main task for the loading task.

[0165] Determine the sub-tasks for image processing; the sub-tasks include: image reconstruction tasks and / or image segmentation tasks.

[0166] While training the end-to-end trajectory planning model based on the main task, after the feature output of the network for processing images in the end-to-end trajectory planning model, perform the sub-task training to obtain the trained end-to-end trajectory planning model.

[0167] In some embodiments, the trajectory planning device for automatic loading of the excavator further includes:

[0168] A construction module for constructing the training data set based on the following steps:

[0169] Obtain sample task data for loading tasks in different scenarios; the sample task data includes: sample environmental perception information, and target sample joint position information matching the sample environmental perception information; the sample environmental perception information includes sample image information captured by a camera, and sample point cloud data obtained by radar scanning; the target sample joint position information includes: sample joint information of the joints of the multiple target parts; the sample joint information of the joints of the target parts includes sample joint positions and sample joint speeds.

[0170] Verify and clean the sample task data based on the sample joint information of the joints of each target part, the sample image information, and the matching relationship between the sample image information and the sample joint information;

[0171] Construct a training data set based on the cleaned sample task data.

[0172] In some embodiments, in the trajectory planning device for automatic loading of an excavator, when the construction module verifies and cleans the sample task data based on the sample joint information of the joints of each target part, the sample image information, and the matching relationship between the sample image information and the sample joint information, it is specifically used for:

[0173] Perform a primary screening on the sample task data according to the pre-configured joint position range and joint speed range of each target part during normal operation to obtain the sample task data after the primary screening;

[0174] Perform a secondary screening on the sample task data after the primary screening based on the sample image information to obtain the sample task data after the secondary screening;

[0175] Perform visualization processing on the sample task data after the secondary screening, sample and check whether the sample image information after the secondary screening matches the sample joint information, and select the matching sample task data.

[0176] In some embodiments, in the trajectory planning device for automatic loading of an excavator, the end-to-end trajectory planning model is trained offline, and the trained end-to-end trajectory planning model is deployed to the target excavator.

[0177] In some embodiments, in the trajectory planning device for automatic loading of an excavator, when the processing module processes the environmental perception data and the target real-time joint position information of the target excavator through the end-to-end trajectory planning model to predict the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps, it is specifically used for:

[0178] The end-to-end trajectory planning model uses the environmental perception data and the target real-time joint position information of the target excavator as conditions to denoise and diffuse the actions, and predicts the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps.

[0179] Based on the same inventive concept, embodiments of the present application also provide an electronic device corresponding to the trajectory planning method for automatic loading of an excavator. Since the principle of solving problems by the electronic device in the embodiments of the present application is similar to the above-mentioned trajectory planning method for automatic loading of an excavator in the embodiments of the present application, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be described again.

[0180] Please refer to Figure 8 , Figure 8 which shows a schematic structural diagram of the electronic device described in the embodiments of the present application; as Figure 8 shown, the electronic device 800 includes: a processor 801, a memory 802, and a bus. The memory 802 stores machine-readable instructions executable by the processor 801. When the electronic device 800 runs, the processor 801 communicates with the memory 802 through the bus. When the machine-readable instructions are executed by the processor 801, the steps of the trajectory planning method for automatic loading of the excavator are executed, specifically as follows:

[0181] Obtain the environmental perception data of the target excavator and the target real-time joint position information matching the environmental perception data; the target real-time joint position information includes the real-time joint position information of multiple target parts performing the loading task;

[0182] Input the environmental perception data and the target real-time joint position information of the target excavator into a pre-trained end-to-end trajectory planning model; the end-to-end trajectory planning model is trained based on a training dataset constructed from task data for performing the loading task in different scenarios;

[0183] Process the environmental perception data and the target real-time joint position information of the target excavator through the end-to-end trajectory planning model, and predict the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps;

[0184] Based on the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps, determine the target trajectory information of the target excavator. The target trajectory information includes the target trajectory point positions of the multiple target parts at a preset number of future time steps.

[0185] In some embodiments, the processor further executes the following steps:

[0186] After determining the target trajectory information of the target excavator, send the target trajectory information of the target excavator to the controller of the target excavator, so that the controller controls the joints of the multiple target parts of the target excavator to act at the corresponding time steps based on the target trajectory information.

[0187] In some embodiments, the target trajectory point position information of the joint of the target part is the change amount of the joint of the target part relative to the input real-time joint position information.

[0188] In some embodiments, when the processor executes the step of determining the target trajectory information of the target excavator based on the position information of the target trajectory points of the joints of the multiple target parts in the preset number of future time steps, the following steps are specifically executed:

[0189] Based on the real-time joint position information of the multiple input target parts and the change amount of the multiple target parts relative to the input real-time joint position information in the preset number of future time steps, determine the target trajectory information of the joints of the multiple target parts in the preset number of future time steps.

[0190] In some embodiments, when the processor executes the step of determining the target trajectory information of the target excavator based on the position information of the target trajectory points of the joints of the multiple target parts in the preset number of future time steps, the following steps are specifically executed:

[0191] Obtain the position information of the target trajectory points of the joints of the target parts in the preset number of future time steps output by the end-to-end trajectory planning model in multiple rounds of prediction; wherein, the time interval between adjacent rounds of prediction is less than the total duration of the preset number of future time steps;

[0192] Aggregate the position information of the target trajectory points of the joints of the target parts at the same time step in multiple rounds of prediction to obtain the updated position information of the target trajectory points of the joints of the target parts.

[0193] Based on the updated position information of the target trajectory points of the joints of the target parts, determine the target trajectory information of the target excavator.

[0194] In some embodiments, when the processor executes the step of aggregating the position information of the target trajectory points of the joints of the target parts at the same time step in multiple rounds of prediction to obtain the updated position information of the target trajectory points of the joints of the target parts, the following steps are specifically executed:

[0195] Based on the influence degree of different rounds of prediction on the position information of the target trajectory points of the joints of the target parts at the same time step, determine the aggregation weights of different rounds of prediction;

[0196] Aggregate the position information of the target trajectory points of the joints of the target parts at the same time step in multiple rounds of prediction based on the aggregation weights of different rounds of prediction to obtain the updated position information of the target trajectory points of the joints of the target parts.

[0197] In some embodiments, when the processor executes the step of obtaining the environmental perception data of the target excavator, the following steps are specifically executed:

[0198] Obtain multiple frame image information captured by the camera of the target excavator and multiple frame point cloud data obtained by radar scanning;

[0199] Process the multi-frame point cloud data to determine the multi-frame elevation maps corresponding to the point cloud data.

[0200] In some embodiments, when the processor executes the step of obtaining the target real-time joint position information matching the target excavator and the environmental perception data, the following steps are specifically executed:

[0201] Obtain the initial joint position information of the joints of the target excavator's powered-on cockpit;

[0202] Based on the time information of the environmental perception data of the target excavator, obtain the relative joint position information of the joints of the target excavator's cockpit relative to the initial joint position information, and obtain the absolute joint position information of the joints of the boom, arm, and bucket;

[0203] Determine the relative joint position information of the joints of the cockpit, and the absolute joint position information of the joints of the boom, arm, and bucket as the target real-time joint position information of the target excavator.

[0204] In some embodiments, the end-to-end trajectory planning model is constructed based on the generative strategy learning method of the conditional denoising diffusion process; when training the constructed end-to-end trajectory planning model, the sample environmental perception information and the target sample joint position information matching the sample environmental perception information are used as conditions.

[0205] In some embodiments, the processor further executes the following steps:

[0206] For the constructed end-to-end trajectory planning model, determine the main task for the loading task;

[0207] Determine the subtasks for image processing; the subtasks include: image reconstruction tasks and / or image segmentation tasks;

[0208] While training the end-to-end trajectory planning model based on the main task, after the feature output of the network for processing images in the end-to-end trajectory planning model, perform the subtask training to obtain the trained end-to-end trajectory planning model.

[0209] In some embodiments, the processor further executes the following steps:

[0210] Obtain the sample task data for the loading task in different scenarios; the sample task data includes: sample environmental perception information, and target sample joint position information matching the sample environmental perception information; the sample environmental perception information includes sample image information captured by a camera, and sample point cloud data obtained by radar scanning; the target sample joint position information includes: sample joint information of the joints of the multiple target parts; the sample joint information of the joints of the target parts includes sample joint positions and sample joint velocities;

[0211] Based on the sample joint information, sample image information of the joints of each target part, and the matching relationship between the sample image information and the sample joint information, verify and clean the sample task data;

[0212] Based on the cleaned sample task data, construct a training data set.

[0213] In some embodiments, when the processor executes the step of verifying and cleaning the sample task data based on the sample joint information, sample image information of the joints of each target part, and the matching relationship between the sample image information and the sample joint information, the following steps are specifically executed:

[0214] Perform a first screening on the sample task data according to the pre-configured joint position range and joint speed range of each target part during normal operation to obtain the sample task data after the first screening;

[0215] Perform a second screening on the sample task data after the first screening based on the sample image information to obtain the sample task data after the second screening;

[0216] Perform visualization processing on the sample task data after the second screening, sample and check whether the sample image information after the second screening matches the sample joint information, and select the matching sample task data.

[0217] In some embodiments, the end-to-end trajectory planning model is trained offline, and the trained end-to-end trajectory planning model is deployed to the target excavator.

[0218] In some embodiments, when the processor executes the step of predicting the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps by processing the environmental perception data and the target real-time joint position information of the target excavator through the end-to-end trajectory planning model, the following steps are specifically executed:

[0219] The end-to-end trajectory planning model uses the environmental perception data and the target real-time joint position information of the target excavator as conditions to denoise and diffuse the actions, and predicts the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps.

[0220] Based on the same inventive concept, a computer-readable storage medium corresponding to the trajectory planning method for automatic loading of an excavator is also provided in the embodiments of the present application. Since the principle of solving problems by the computer-readable storage medium in the embodiments of the present application is similar to the above-mentioned trajectory planning method for automatic loading of an excavator in the embodiments of the present application, the implementation of the computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be described again.

[0221] A computer-readable storage medium stores a computer program thereon. When the computer program is run by a processor, the processor executes the steps of the trajectory planning method for automatic loading of an excavator as follows:

[0222] Obtain the environmental perception data of the target excavator and the target real-time joint position information matching the environmental perception data; the target real-time joint position information includes the real-time joint position information of multiple target parts performing the loading task;

[0223] Input the environmental perception data and the target real-time joint position information of the target excavator into a pre-trained end-to-end trajectory planning model; the end-to-end trajectory planning model is trained based on a training dataset constructed from task data for performing the loading task in different scenarios;

[0224] Process the environmental perception data and the target real-time joint position information of the target excavator through the end-to-end trajectory planning model to predict the target trajectory point position information of the joints of the multiple target parts in a preset number of future time steps;

[0225] Based on the target trajectory point position information of the joints of the multiple target parts in a preset number of future time steps, determine the target trajectory information of the target excavator. The target trajectory information includes the target trajectory point positions of the multiple target parts in a preset number of future time steps.

[0226] In some embodiments, the processor further performs the following steps:

[0227] After determining the target trajectory information of the target excavator, send the target trajectory information of the target excavator to the controller of the target excavator, so that the controller controls the joints of the multiple target parts of the target excavator to act at the corresponding time steps based on the target trajectory information.

[0228] In some embodiments, the target trajectory point position information of the joint of the target part is the change amount of the joint of the target part relative to the input real-time joint position information.

[0229] In some embodiments, when the processor executes the step of determining the target trajectory information of the target excavator based on the target trajectory point position information of the joints of the multiple target parts in a preset number of future time steps, it specifically executes the following steps:

[0230] Determine the target trajectory information of the joints of the multiple target parts at a preset number of future time steps based on the real-time joint position information of the multiple target parts in the input and the change amount of the multiple target parts relative to the real-time joint position information of the input at a preset number of future time steps.

[0231] In some embodiments, when the processor executes the step of determining the target trajectory information of the target excavator based on the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps, the following steps are specifically executed:

[0232] Obtain the target trajectory point position information of the joints of the target parts at a preset number of future time steps output by the end-to-end trajectory planning model in multiple rounds of prediction; wherein, the time interval between adjacent rounds of prediction is less than the total duration of a preset number of future time steps;

[0233] Aggregate the target trajectory point position information of the joints of the target parts at the same time step in multiple rounds of prediction to obtain the updated target trajectory point position information of the joints of the target parts;

[0234] Determine the target trajectory information of the target excavator based on the updated target trajectory point position information of the joints of the target parts.

[0235] In some embodiments, when the processor executes the step of aggregating the target trajectory point position information of the joints of the target parts at the same time step in multiple rounds of prediction to obtain the updated target trajectory point position information of the joints of the target parts, the following steps are specifically executed:

[0236] Determine the aggregation weights of different rounds of prediction based on the influence degree of different rounds of prediction on the target trajectory point position information of the joints of the target parts at the same time step;

[0237] Aggregate the target trajectory point position information of the joints of the target parts at the same time step in multiple rounds of prediction based on the aggregation weights of different rounds of prediction to obtain the updated target trajectory point position information of the joints of the target parts.

[0238] In some embodiments, when the processor executes the step of obtaining the environmental perception data of the target excavator, the following steps are specifically executed:

[0239] Obtain multiple frame image information captured by the camera of the target excavator and multiple frame point cloud data obtained by radar scanning;

[0240] Process the multiple frame point cloud data to determine the multiple frame elevation maps corresponding to the point cloud data.

[0241] In some embodiments, when the processor executes the step of obtaining the target real-time joint position information that matches the target excavator and the environmental perception data, the following steps are specifically executed:

[0242] Obtain the initial joint position information of the joints of the target excavator's powered-on cockpit;

[0243] Based on the time information of the environmental perception data of the target excavator, obtain the relative joint position information of the joints of the target excavator's cockpit relative to the initial joint position information, and obtain the absolute joint position information of the joints of the boom, arm, and bucket;

[0244] Determine the relative joint position information of the joints of the cockpit, the absolute joint position information of the joints of the boom, arm, and bucket as the target real-time joint position information of the target excavator.

[0245] In some embodiments, the end-to-end trajectory planning model is constructed based on a generative policy learning method of a conditional denoising diffusion process; when training the constructed end-to-end trajectory planning model, the sample environmental perception information and the target sample joint position information that matches the sample environmental perception information are used as conditions.

[0246] In some embodiments, the processor further executes the following steps:

[0247] For the constructed end-to-end trajectory planning model, determine the main task for the loading task;

[0248] Determine the sub-tasks for image processing; the sub-tasks include: image reconstruction tasks and / or image segmentation tasks;

[0249] While training the end-to-end trajectory planning model based on the main task, after the feature output of the network for processing images in the end-to-end trajectory planning model, perform the sub-task training to obtain the trained end-to-end trajectory planning model.

[0250] In some embodiments, the processor further executes the following steps:

[0251] Obtain sample task data for the loading task in different scenarios; the sample task data includes: sample environmental perception information, and target sample joint position information that matches the sample environmental perception information; the sample environmental perception information includes sample image information captured by a camera, and sample point cloud data obtained by radar scanning; the target sample joint position information includes: sample joint information of the joints of the multiple target parts; the sample joint information of the joints of the target parts includes sample joint positions and sample joint speeds;

[0252] Verify and clean the sample task data based on the sample joint information, sample image information, and the matching relationship between the sample image information and the sample joint information of the joints of each target part;

[0253] Construct a training data set based on the cleaned sample task data.

[0254] In some embodiments, when the processor executes the step of verifying and cleaning the sample task data based on the sample joint information, sample image information, and the matching relationship between the sample image information and the sample joint information of the joints of each target part, the following steps are specifically executed:

[0255] Perform a first screening on the sample task data according to the pre-configured joint position range and joint speed range of each target part during normal operation to obtain the sample task data after the first screening;

[0256] Perform a second screening on the sample task data after the first screening based on the sample image information to obtain the sample task data after the second screening;

[0257] Perform visualization processing on the sample task data after the second screening, sample and check whether the sample image information after the second screening matches the sample joint information, and select the matching sample task data.

[0258] In some embodiments, the end-to-end trajectory planning model is trained offline, and the trained end-to-end trajectory planning model is deployed to the target excavator.

[0259] In some embodiments, when the processor executes the step of processing the environment perception data and the target real-time joint position information of the target excavator through the end-to-end trajectory planning model to predict the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps, the following steps are specifically executed:

[0260] The end-to-end trajectory planning model uses the environment perception data and the target real-time joint position information of the target excavator as conditions to perform denoising diffusion on the action, and predicts the target trajectory point position information of the joints of the multiple target parts at a preset number of future time steps.

[0261] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the method embodiments, and will not be elaborated herein. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there can be other division methods in actual implementation. For another example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some communication interfaces. The indirect couplings or communication connections of the devices or modules can be in electrical, mechanical, or other forms.

[0262] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0263] In addition, in each embodiment of this application, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0264] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of this application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs, etc., which can store program codes.

[0265] The above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application, and all should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A trajectory planning method for automatic loading of an excavator, characterized in that: The method comprises: Acquire environmental perception data of a target excavator and target real-time joint position information that matches the environmental perception data; the target real-time joint position information includes real-time joint position information of multiple target parts that perform a loading task; Inputting the environmental perception data of the target excavator and the target real-time joint position information into a pre-trained end-to-end trajectory planning model; the end-to-end trajectory planning model is trained based on a training data set constructed based on task data of loading tasks in different scenarios; Processing the environmental perception data of the target excavator and the target real-time joint position information through the end-to-end trajectory planning model, predicting the target trajectory point position information of the joints of the multiple target parts in a preset number of time steps in the future; The target trajectory information of the target excavator is determined based on the target trajectory point position information of the joints of the multiple target parts in a preset number of time steps in the future.

2. The trajectory planning method for automatic loading of an excavator according to claim 1, characterized in that: After determining the target trajectory information of the target excavator, the method further includes: The target trajectory information of the target excavator is sent to a controller of the target excavator, so that the controller controls the joints of multiple target parts of the target excavator to move at corresponding time steps based on the target trajectory information.

3. The trajectory planning method for automatic loading of an excavator according to claim 1, characterized in that: The target trajectory point position information of the joint of the target part is the change amount of the joint of the target part relative to the input real-time joint position information.

4. The trajectory planning method for automatic loading of an excavator according to claim 3 is characterized in that: The determining of the target trajectory information of the target excavator based on the target trajectory point position information of the joints of the multiple target parts in a preset number of time steps in the future includes: Based on the input real-time joint position information of the multiple target parts and the change of the multiple target parts relative to the input real-time joint position information in a preset number of time steps in the future, the target trajectory information of the joints of the multiple target parts in a preset number of time steps in the future is determined.

5. The trajectory planning method for automatic loading of an excavator according to claim 1 or 4, characterized in that: The determining of the target trajectory information of the target excavator based on the target trajectory point position information of the joints of the multiple target parts in a preset number of time steps in the future includes: Obtaining target trajectory point position information of a preset number of future time steps of a joint of a target part output by the end-to-end trajectory planning model in multiple rounds of prediction; wherein the time interval between adjacent rounds of prediction is less than the total duration of the preset number of future time steps; Aggregate the target trajectory point position information of the joints of the target part at the same time step in multiple rounds of prediction to obtain the updated target trajectory point position information of the joints of the target part; Based on the updated target trajectory point position information of the joints of the target part, the target trajectory information of the target excavator is determined.

6. The trajectory planning method for automatic loading of an excavator according to claim 5, characterized in that: The target trajectory point position information of the joints of the target part at the same time step in multiple rounds of prediction is aggregated to obtain the updated target trajectory point position information of the joints of the target part, including: Determine the aggregation weights of different rounds of predictions based on the degree of influence of different rounds of predictions on the target trajectory point position information of the joints of the target parts at the same time step; The target trajectory point position information of the joints of the target part at the same time step in multiple rounds of predictions is aggregated based on the aggregation weights of different rounds of predictions to obtain updated target trajectory point position information of the joints of the target part.

7. The trajectory planning method for automatic loading of an excavator according to claim 1, characterized in that: The step of obtaining the environmental perception data of the target excavator includes: Acquire multi-frame image information captured by a camera of the target excavator and multi-frame point cloud data obtained by radar scanning; The multi-frame point cloud data is processed to determine a multi-frame elevation map corresponding to the point cloud data.

8. The trajectory planning method for automatic loading of an excavator according to claim 1, characterized in that: The method of obtaining target real-time joint position information matching the target excavator and the environmental perception data comprises: Acquire initial joint position information of the joints of the target excavator start-up cabin; Based on the time information of the environmental perception data of the target excavator, relative joint position information of the joints of the cockpit of the target excavator relative to the initial joint position information is obtained, and absolute joint position information of the joints of the boom, the forearm and the bucket is obtained; The relative joint position information of the joints of the cockpit and the absolute joint position information of the joints of the boom, the forearm and the bucket are determined as the target real-time joint position information of the target excavator.

9. The trajectory planning method for automatic loading of an excavator according to claim 1, characterized in that: The end-to-end trajectory planning model is constructed based on a generative strategy learning method of a conditional denoising diffusion process; when training the constructed end-to-end trajectory planning model, sample environment perception information and target sample joint position information matching the sample environment perception information are used as conditions.

10. The trajectory planning method for automatic loading of an excavator according to claim 1 or 9, characterized in that: The end-to-end trajectory planning model is trained based on the following method: Based on the constructed end-to-end trajectory planning model, determine the main tasks for the loading task; Determine subtasks for image processing; the subtasks include: image reconstruction tasks and / or image segmentation tasks; While the end-to-end trajectory planning model is trained based on the main task, the sub-task training is performed after the features of the network used to process the image in the end-to-end trajectory planning model are output to obtain a trained end-to-end trajectory planning model.

11. The trajectory planning method for automatic loading of an excavator according to claim 1 or 9, characterized in that: The training data set is constructed based on the following method: Obtain sample task data for loading tasks in different scenarios; The sample task data includes: sample environment perception information, and target sample joint position information matching the sample environment perception information; the sample environment perception information includes sample image information captured by a camera and sample point cloud data obtained by radar scanning; the target sample joint position information includes: sample joint information of the joints of the multiple target parts; the sample joint information of the joints of the target parts includes sample joint positions and sample joint velocities; Based on the sample joint information, sample image information, and the matching relationship between the sample image information and the sample joint information of each target part joint, verifying and cleaning the sample task data; Construct a training data set based on the cleaned sample task data.

12. The trajectory planning method for automatic loading of an excavator according to claim 11, characterized in that: The sample joint information, sample image information, and the matching relationship between the sample image information and the sample joint information of each target part joint are verified and cleaned, including: The sample task data is screened once according to the pre-configured joint position range and joint speed range of each target part during normal operation to obtain the screened sample task data; Performing a secondary screening on the sample task data after the primary screening based on the sample image information to obtain the sample task data after the secondary screening; The sample task data after the secondary screening is visualized, and a sampling check is performed to see whether the sample image information after the secondary screening matches the sample joint information, and the matching sample task data is selected.

13. The trajectory planning method for automatic loading of an excavator according to claim 1 or 9, characterized in that: The end-to-end trajectory planning model is trained offline, and the trained end-to-end trajectory planning model is deployed on the target excavator.

14. The trajectory planning method for automatic loading of an excavator according to claim 1, characterized in that: Processing the environmental perception data of the target excavator and the target real-time joint position information through the end-to-end trajectory planning model, predicting the target trajectory point position information of the joints of the multiple target parts in a preset number of time steps in the future, including: The end-to-end trajectory planning model uses the environmental perception data of the target excavator and the target real-time joint position information as conditions to denoise and diffuse the action, and predicts the target trajectory point position information of the joints of the multiple target parts in a preset number of time steps in the future.

15. A trajectory planning device for automatic loading of an excavator, characterized in that: The device comprises: An acquisition module, used to acquire environmental perception data of a target excavator and target real-time joint position information matching the environmental perception data; the target real-time joint position information includes real-time joint position information of multiple target parts performing a loading task; An input module, used to input the environmental perception data of the target excavator and the target real-time joint position information into a pre-trained end-to-end trajectory planning model; the end-to-end trajectory planning model is trained based on a training data set constructed based on task data of loading tasks in different scenarios; A processing module, used to process the environmental perception data of the target excavator and the target real-time joint position information through the end-to-end trajectory planning model, and predict the target trajectory point position information of the joints of the multiple target parts in a preset number of time steps in the future; The determination module is used to determine the target trajectory information of the target excavator based on the target trajectory point position information of the joints of the multiple target parts in a preset number of time steps in the future.

16. An electronic device, characterized in that include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the trajectory planning method for automatic loading of an excavator are performed as described in any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the trajectory planning method for automatic loading of an excavator as described in any one of claims 1 to 14.

Images