Mining learning for cluttered rigid objects

Abstract

Embodiments of a learning-based mining planning method for mining cluttered rigid objects are disclosed, which is challenging due to the high variability of the objects' geometric and physical properties and the large drag during mining. A convolutional neural network is utilized to predict the probability of successful mining. Embodiments of a sampling-based optimization method are disclosed for planning high-quality mining trajectories by leveraging the learned predictive model. To reduce the simulation-to-reality gap for mining learning, a voxel-based representation of the mining scene is used. Mining experiments are conducted in simulation and the real world to evaluate the learning-based mining planner. Experimental results demonstrate that embodiments of the disclosed method can plan high-quality mining for cluttered rigid objects and significantly outperform baseline methods.

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for computer learning that can provide improved computer performance, features, and uses. More specifically, the present disclosure relates to systems and methods for automated mining learning and planning. Background Art

[0002] Deep neural networks have achieved great success in computer vision, natural language processing, recommendation systems and other fields.

[0003] Excavators are widely used in a variety of applications, including construction, material loading, and mining. Automation of excavator operations has become an active area of ​​research due to its potential to increase safety, reduce costs, and improve productivity. Excavators may need to operate in extreme environments or weather conditions, which can be challenging for operators. Machine-related fatalities and injuries occur annually. Furthermore, operating an excavator requires specialized and expensive training to ensure safe operation of the equipment.

[0004] In developing autonomous excavator systems, there have been many efforts focused on specific aspects, including perception, planning, control, remote operation, and system integration and application. Despite these advances, the automatic excavation of hard or solid materials, especially irregular rigid objects, remains challenging, and there are relatively few studies investigating this issue. Rock excavation is a typical situation at mining operation sites. Compared to granular materials, rocks are hard and often in cluttered piles. Excavation is more challenging, more time-consuming, and more expensive. The excavation of rocks creates a large resistance to the bucket. In addition, unlike granular materials composed of uniform particles, rigid objects often have large variations in geometry (e.g., concavity and convexity), appearance, and physical properties (e.g., mass), which greatly increases the challenges of robot perception and manipulation.

[0005] Therefore, there is a need for systems and methods for automatic mining learning and planning, especially for cluttered rigid objects. Summary of the Invention

[0006] One aspect of the present disclosure provides a computer-implemented method for automatic mining learning, comprising: tiling a mining trajectory represented by a plurality of trajectory parameters in a voxel grid dimension to obtain a tiled trajectory parameter voxel grid; connecting the tiled trajectory parameter voxel grid with a voxel grid representation of a mining scene to generate an input voxel grid defining a mining instance of a pair of mining trajectories and a voxel grid representation of the mining scene; feeding the input voxel grid into a neural network to generate a multidimensional feature vector; and processing the multidimensional feature vector using at least one fully connected layer to predict a probability of mining success.

[0007] Another aspect of the present disclosure provides a system for automatic mining learning and planning, comprising: one or more processors; and a non-transitory computer-readable medium comprising one or more sets of instructions, which, when executed by at least one of the one or more processors, causes the following steps to be performed, including: tiling a mining trajectory represented by a plurality of trajectory parameters in a voxel grid dimension to obtain a tiled trajectory parameter voxel grid; connecting the tiled trajectory parameter voxel grid with a voxel grid representation of a mining scene to generate an input voxel grid defining a mining instance of a pair of mining trajectories and a voxel grid representation of the mining scene; feeding the input voxel grid into a neural network to generate a multidimensional feature vector; and processing the multidimensional feature vector using one or more fully connected layers to predict the probability of mining success.

[0008] Yet another aspect of the present disclosure provides a computer-implemented method for learning-based mining planning, comprising: given a voxel grid representation of a mining scenario, generating a plurality of mining trajectories using a mining planner to initialize a Gaussian distribution of trajectory parameters, each mining trajectory being represented by a plurality of trajectory parameters; updating the Gaussian distribution through mining trajectory sampling and multiple iterations of sampled mining trajectory selection according to a predicted mining success probability of each sampled mining trajectory predicted by a mining prediction model; sampling a subset of mining trajectories from the Gaussian distribution of the last iteration; and selecting a mining trajectory with the highest predicted mining success probability from the subset of mining trajectories as the planned mining trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Reference will be made to embodiments of the present disclosure, examples of which may be shown in the accompanying drawings. These drawings are illustrative and non-restrictive. Although the present disclosure has been generally described in the context of these embodiments, it should be understood that they are not intended to limit the scope of the present disclosure to these specific embodiments. Items in the drawings may not be drawn to scale.

[0010] Figure 1 The mechanics of an excavator and a Franka arm "Excavator" with a three-dimensional (3D) printed bucket according to embodiments of the present disclosure are described.

[0011] Figure 2 A visual mining trajectory representation in a task space according to an embodiment of the present disclosure is described.

[0012] Figure 3 A process for generating a 3D voxel grid representation of an excavation scene is described according to an embodiment of the present disclosure.

[0013] Figure 4 An excavation-voxel-net structure according to an embodiment of the present disclosure is described.

[0014] Figure 5 The process of learning-based mining prediction according to an embodiment of the present invention is described.

[0015] Figure 6 A process for learning-based mining planning according to an embodiment of the present disclosure is described.

[0016] Figure 7 Simulated and real-world RGB-Depth (RGBD) camera setups according to embodiments of the present disclosure are described.

[0017] Figure 8 Simulated and real-world RGB image examples are described according to embodiments of the present disclosure.

[0018] Figure 9 A simulated excavation generated by a CEM-voxel planner according to an embodiment of the present disclosure is described.

[0019] Figure 10 Successful real robotic excavations and their bucket fill volumes generated by the CEM-voxel planner according to an embodiment of the present disclosure are described.

[0020] Figure 11 Three examples of a robot getting stuck due to exceeding a limit force / torque during excavation according to an embodiment of the present disclosure are described.

[0021] Figure 12A Described is a mined volume histogram of CEM-voxel in a simulation experiment according to an embodiment of the present disclosure.

[0022] Figure 12B Mining volume histograms of CEM-RGBD in simulation experiments according to embodiments of the present disclosure are described.

[0023] Figure 12C CEM-voxel mining volume histograms in simulation experiments according to embodiments of the present disclosure are described.

[0024] Figure 12D The mined volume histogram of CEM-RGBD-reg in simulation experiments according to an embodiment of the present disclosure is described.

[0025] Figure 12E Described is a histogram of the excavation volume of CEM-traj in a simulation experiment according to an embodiment of the present disclosure.

[0026] Figure 12F The excavation volume histogram of random-heu in the simulation experiment according to the embodiment of the present disclosure is described.

[0027] Figure 12G A histogram of the excavation volume of the highest-heu in a simulation experiment according to an embodiment of the present disclosure is described.

[0028] Figure 12H Mining volume histograms of training data in simulation experiments according to embodiments of the present disclosure is described.

[0029] Figure 13A The point-of-attack (PoA) distribution of CEM-voxels in simulations according to embodiments of the present disclosure is described.

[0030] Figure 13B The PoA distribution of CEM-RGBD in simulations according to an embodiment of the present invention is described.

[0031] Figure 13C The PoA distribution of CEM-voxel-reg in simulation according to an embodiment of the present disclosure is described.

[0032] Figure 13D The PoA distribution of CEM-RGBD-reg in simulations according to an embodiment of the present invention is described.

[0033] Figure 13E The PoA distribution of CEM-traj in simulation according to an embodiment of the present disclosure is described.

[0034] Figure 13F The PoA distribution of random-heu in simulation according to an embodiment of the present invention is described.

[0035] Figure 13G The PoA distribution of highest-heu in simulation according to an embodiment of the present invention is described.

[0036] Figure 14 The 2D PoA distribution of training data in simulation according to an embodiment of the present invention is described.

[0037] Figure 15 A simplified block diagram of a computing device / information processing system is depicted in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In the following description, for purposes of explanation, specific details are set forth to provide an understanding of the present invention. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these details. Furthermore, those skilled in the art will recognize that the embodiments of the present disclosure described below may be implemented in a variety of ways, such as as a process, apparatus, system, device, or method on a tangible computer-readable medium.

[0039] The components or modules shown in the figures are examples of exemplary embodiments of the present disclosure and are intended to avoid obscuring the present disclosure. It should be understood that throughout the discussion, components may be described as separate functional units, which may include subunits, but those skilled in the art will recognize that various components or portions thereof may be divided into separate components or may be integrated together, for example, included in a single system or component. It should be noted that the functions or operations discussed herein may be implemented as components. Components may be implemented using software, hardware, or a combination thereof.

[0040] In addition, the connection between the components or systems in the accompanying drawings is not limited to direct connection. On the contrary, the data between these components can be modified, reformatted or otherwise changed by an intermediate component. In addition, other or fewer connections can be used. It should also be noted that the terms "connect," "connect," "communicatively connect," "interfacing," "interface" or any derivatives thereof should be understood to include direct connection, indirect connection through one or more intermediate devices, and wireless connection. It should also be noted that any communication such as signal, response, reply, confirmation, message, inquiry can include the exchange of one or more information.

[0041] References in the specification to "one or more embodiments," "preferred embodiments," "an embodiment," "embodiments," etc., mean that a particular feature, structure, characteristic, or function described in conjunction with the embodiment is included in at least one embodiment of the present disclosure and may be included in multiple embodiments. In addition, the appearance of the above phrases in various places in the specification does not necessarily refer to the same one or more embodiments.

[0042] The use of certain terms in various places in this specification is for illustrative purposes and should not be construed as limiting. Services, functions, or resources are not limited to a single service, function, or resource; the use of these terms may refer to groupings of related services, functions, or resources, which may be distributed or aggregated. The terms "include," "including," "comprise," "comprising," or any variations thereof should be understood as open-ended terms, and any list of items that follows is an example item and is not meant to be limited to the listed items. A "layer" may include one or more operations. The words "optimal," "optimize," "optimization," and the like refer to improvements to a result or process, and do not require that the specified result or process have reached an "optimal" or peak state. The use of memory, database, repository, data store, table, hardware, cache, and the like may be used herein to refer to one or more system components into which information may be input or recorded.

[0043] In one or more embodiments, stopping conditions may include: (1) a set number of iterations has been performed; (2) a processing time amount has been reached; (3) convergence (e.g., the difference between consecutive iterations is less than a first threshold); (4) divergence (e.g., performance degrades); (5) an acceptable result has been achieved; and (6) all data has been processed.

[0044] Those skilled in the art will recognize that: (1) certain steps may be performed optionally; (2) the steps may not be limited to the specific order described herein; (3) certain steps may be performed in a different order; and (4) certain steps may be performed simultaneously.

[0045] Any headings used herein are for organizational purposes only and should not be used to limit the scope of the specification or claims. Each reference / document mentioned in this patent document is incorporated herein by reference in its entirety.

[0046] It should be noted that any experiments and results provided herein are provided by way of illustration and are performed under specific conditions using one or more specific embodiments; therefore, these experiments and their results should not be used to limit the scope of the disclosure of this patent document.

[0047] It should also be noted that although the embodiments described herein are in the context of automated excavation, aspects of the present invention are not limited thereto. Accordingly, aspects of the present disclosure may be applied or adapted for use in other automated engineering vehicles or machines.

[0048] A. Overview

[0049] Excavators are widely used in a variety of applications, including construction, material loading, and mining. Automation of excavator operations has become an active area of ​​research due to its potential to increase safety, reduce costs, and improve productivity. Excavators may need to operate in extreme environments or weather conditions, which can be challenging for operators. Machine-related fatalities and injuries occur annually. Furthermore, operating an excavator requires specialized and expensive training to ensure safe operation of the equipment.

[0050] In developing autonomous excavator systems, there have been many efforts focused on specific aspects, including perception, planning, control, remote operation, and system integration and application. Despite these advances, the automatic excavation of hard or solid materials, especially irregular rigid objects, remains challenging, and there are few studies investigating this issue. Rock excavation is a typical situation at mining operation sites. Compared to granular materials, rocks are hard and often piled up in a messy manner. Excavation is more challenging, more time-consuming, and more expensive. The excavation of rocks creates a large resistance to the bucket. In addition, unlike granular materials composed of uniform particles, rigid objects typically have very different geometric shapes (e.g., concave and convex), appearance, and physical properties (e.g., mass), which greatly increases the challenges of robot perception and manipulation. In one or more embodiments, a rigid object is defined as an object that does not deform during the excavation operation.

[0051] In this patent document, an embodiment of mining learning and planning for cluttered irregular rigid objects is disclosed. Various deep learning methods are used to address the challenges of cluttered rigid object mining problems. Given a visual representation of the mining scene, each excavation plans a high-quality trajectory to mine objects with a large total volume. This article provides new RGBD and voxel-based convolutional neural network (CNN) models for predicting mining success. A large number of training mining samples are collected in simulation to train the prediction network. The mining planning is then formulated as an optimization problem using the learned prediction model. Mining experiments are conducted in simulation and the real world to evaluate the embodiments of the learning-based mining method. The results of the mining experiments in simulation and the real world show that the embodiments of the learning-based planner are able to generate excavations with a high success rate. The experimental results also show the advantages of the learning-based mining planner over the two heuristic planners.

[0052] In summary, some of the main contributions of this patent document include but are not limited to:

[0053] (1) We propose two CNN models for the successful prediction of a new task (mining cluttered rigid objects) and solve the mining planning problem as an optimization problem using known models.

[0054] (2) Mining experiments in simulation and the real world demonstrate that embodiments of the disclosed learning-based planner are capable of generating mining trajectories with a high success rate.

[0055] (3) The implementation method represents the mining trajectory in the task space, allowing the transfer of learned mining prediction models across different hardware platforms.

[0056] (4) This patent document shows that for mining learning, voxel grid representation of mining scenes reduces the gap between simulation and reality compared to RGBD image representation.

[0057] (5) Collect and mine datasets for cluttered rigid objects.

[0058] Section B summarizes some related work. Section C defines the mining planning problem for cluttered rigid objects. Section D then outlines various implementations of mining learning and planning. Section E provides a detailed description of simulation and real-world robotics experiments. Section F presents mining data collection, model training, offline validation, further results analysis, and ablation studies. Section G provides some conclusions and observations.

[0059] B. Related Work

[0060] This section summarizes the relevant literature on autonomous excavators, manipulation learning, and voxel-based planning.

[0061] 1. Autonomous excavator

[0062] Existing work on autonomous excavators has primarily focused on earthwork excavation and granular material handling. A prototype system for automated material loading on dump trucks has been proposed. An automated trenching system has been introduced and validated on an actual excavator. A trajectory optimization method for granular material excavation has been proposed. Recently, various prototypes and experiments have been conducted on task planning for large-scale excavation tasks, such as soil pile removal. A novel real-time panoramic telepresence system for construction machinery has been proposed. Others have discussed a new method for interactive operation of working machines. A reinforcement learning approach has also been proposed for autonomous arm control of a hydraulic excavator.

[0063] Different control approaches have been proposed for automated excavation. One proposed a control structure for earthwork excavation using explicit disturbance compensation. Others have proposed a force control approach, and the resulting bucket motion can adapt to different terrains. A linear motion tracking control scheme has been proposed for a hydraulic excavator system. Still others have proposed a model-free extremum finding method using power maximization.

[0064] There is relatively little work related to rigid object excavation. An iterative learning-based admittance control algorithm is proposed for autonomous gravel excavation using a robotic wheel loader. Some have explored an admittance-based automatic loading controller for gravel excavation. Compared to some previous low-level excavation control work, the embodiments of this paper focus on learning-based excavation trajectory planning that takes into account visual scene representations of cluttered rigid objects. Some have combined Gaussian process rock motion models with unscented Kalman filtering for rock excavation. However, this approach only focuses on the excavation of single isolated rocks and uses an OptiTrack motion capture system to track the motion of the rock. In contrast, one or more embodiments of this paper focus on the excavation of cluttered rigid objects using RGBD cameras.

[0065] 2. Deep Learning for Operations

[0066] In recent years, researchers have been focusing on the success of deep learning to improve the operation of machines (including non-grasping operations, grasping operations, and granular material grasping operations). For example, deep learning has been well generalized to previously invisible objects (only partial view visual information is available for grasping). Some people have trained an end-to-end neural network controller for automatic loading of granular media using artificial demonstration data. Some people have used learned statistical models to predict the behavior of earth excavation and established a predictive model for controlling the amount of earth excavated. In one or more embodiments of the present patent document, deep learning is applied to solve the perception and operation challenges of excavation of messy rigid objects and produce high-quality excavation.

[0067] Various planning methods have been developed to utilize deep neural network predictive models. Some have proposed connected deep networks to efficiently evaluate a large number of candidate grasps. Others have developed a highly customized CNN model to learn the dynamics of the granular material excavation task and used the cross entropy method (CEM) with the learned predictive model for excavation planning. Still others have developed grasp planning and solved it as a gradient-based optimization of the grasp configuration using a grasp prediction network. In one or more embodiments, excavation planning is modeled as an optimization problem that maximizes the probability of excavation success predicted by the excavation prediction network and the CEM is used to solve the optimization.

[0068] 3. Voxel-based planning

[0069] Some have proposed a voxel-based object representation and two 3D CNNs for multi-finger grasp learning and planning. Others have modeled environments with voxel grids and proposed faster motion planning methods in such environments by predicting good path initializations. To overcome the gap between simulation and reality, this paper proposes an implementation that mines a 3D voxel grid representation of the scene.

[0070] C. Implementation Methods for Mining Task Definition and Mining Trajectory Representation

[0071] In this section, the mining task and mining trajectory representation for rigid objects are defined.

[0072] 1. Task Overview

[0073] Embodiments of this patent document focus on cluttered rigid object mining. Given a visual representation (i.e., an RGBD image or voxel grid) Z of the current mining scene, the goal is to plan a trajectory T that can mine rigid objects (e.g., stones or wood blocks) with a maximum total volume V. In one or more embodiments, a mining instance / sample is defined as a pair of a visual representation of the scene (an RGBD image or a voxel grid representation of the mining scene) and a mining trajectory (Z, T). In one or more embodiments, the focus is on greedily maximizing the mining object volume of the current excavation without considering future excavations. The mining task can potentially be further extended to maximize the cumulative mining object volume using a fixed number of excavations, or to clear a target area of ​​an object using a minimum number of excavations.

[0074] In an embodiment of this patent document, a Franka Panda 7DoF robotic arm equipped with a 3D printed bucket is used to simulate a standard 4-degree-of-freedom (DoF) excavator model. Figure 1 The mechanical structure of an excavator (left) and a Franka arm "excavator" with a 3D printed bucket (right) according to an embodiment of the present disclosure is shown. The Franka arm "excavator" is shown facing the following site setting, where rigid wooden blocks are placed in an excavation tray and the task is to excavate these blocks and dump them into a dump tray after each excavation. Figure 1 The joints shown for the excavator and Franka are base swing (102 and 112), boom (104 and 114), rod (106 and 116), and bucket joint (108 and 118), respectively.

[0075] 2. Mining Trajectory Representation in Task Space

[0076] like Figure 1 As shown, an excavator arm typically has 4 DoFs, including base swing, boom, rod, and bucket joints. The 4D digging pose of the bucket in task (Cartesian) space consists of a 3D digging position (x, y, z) and a 1D digging angle α. The digging angle determines the bucket orientation, which is also equal to the sum of the joint angles of the last three digging joints. The digging angle is defined as zero degrees when the bucket orientation is horizontal and pointing away from the robot. When the bucket orientation is vertical and pointing downward, the digging angle is -90 degrees. It can be obtained from Figure 2An example of digging angle visualization can be seen in the closed digging angle β in .

[0077] Generally, the excavation trajectory T can be divided into multiple stages. Figure 2 A visual representation of an excavation trajectory in task space according to an embodiment of the present disclosure is shown. The trajectory is divided into five phases: attack 204, penetration 206, drag 208, close 210, and lift 212. In the attack phase 204, the excavator arm moves the bucket from the starting position 202 to its 4D target attack position p = (x, y, z, α). In the penetration phase 206, the bucket penetrates the object at a specified depth along the direction of gravity. Then, in the drag phase 208, the bucket is dragged horizontally in the excavation plane by a given length l toward the excavator base. Dragging allows the excavator arm to push and accumulate more objects into the bucket along the way. In the close phase 210, the excavator arm reduces the angle between the bucket and the horizontal plane to manipulate the object into the bucket and close the bucket. The excavator arm closes the bucket by reducing the wrist joint angle to β degrees. Finally, in the lift phase 212, the excavator arm raises the bucket to a certain height h.

[0078] In one or more embodiments, it is assumed that the attack point is always on the surface of the clutter object. Given the 2D or surface coordinates (x, y) of the attack pose p, the z coordinate value of the attack point on the surface of the clutter object is calculated as the height of the grid / height map of the clutter object at (x, y). More details and examples of the grid map can be found in Section F.3. In one or more embodiments, the lifting height h for lifting the bucket is fixed to the height of the robot base. Therefore, the task space excavation trajectory can be represented using 6 parameters T = (x, y, α, d, l, β). The attack point (x, y) can be learned and planned in the object pallet framework.

[0079] In one or more embodiments, the excavation trajectory is interpolated using the 6D task trajectory parameters, and its corresponding joint space trajectory is generated by applying inverse kinematics (IK) of the excavator arm. The interpolated joint trajectory waypoints are then sent to the position controllers in the simulation and the real world. Although the robotic arm is used for excavation in this patent document, the task trajectory representation and excavator IK can also be directly applied to hydraulically actuated excavator arms.

[0080] D. Implementation of Object Mining Learning and Planning

[0081] In this section, we present an implementation of a deep network model to predict mining success for cluttered rigid objects. We then disclose an implementation of a mining planner that leverages the learned predictive mining model.

[0082] 1. Implementation Methods of Mining Scene Representation

[0083] In one or more embodiments, two visual representations (RGBD images and voxel grids) of an excavation scene are considered. RGB and depth images are captured using an RGBD camera in simulation or the real world. Results show that the RGBD image representation suffers from a large simulation-to-real gap when transferring learned excavation knowledge from simulation to the real world because (1) the simulated excavation environment (e.g., the geometry and color of the excavation tray and the color of the ground) is different from the real-world excavation environment; and (2) the RGBD image depends on both the internal and external features of the camera. Figure 8 Simulated and real-world RGB images are shown respectively.

[0084] To overcome the simulation-to-reality problem, an implementation of a 3D voxel grid representation of the mining scene is proposed. Figure 3 A process for generating a 3D voxel grid representation of an excavation scene according to an embodiment of the present disclosure is described. In step 305, a point cloud (i.e., depth) of the excavation scene obtained from an RGBD camera is first transformed into an object pallet frame. In step 310, the transformed point cloud is filtered according to a specific excavation cube space. In step 315, the filtered point cloud is voxelized to generate a voxel grid representation of the excavation scene. More details of the excavation space specifications for simulation and real-world can be found in subsections F.1 and F.2. For example, the voxel grid has dimensions of 64×64×32 and a resolution of 0.01m. Figure 4 An example of a voxel grid visualization and its source point cloud is shown. The voxel grid dimensions and resolution can be designed empirically to cover the mining space and retain a reasonable level of visual detail.

[0085] Since voxelization is focused only on the designated excavation volume, the voxel grid representation is unaffected by the surrounding environment. Furthermore, the voxel grid representation is unaware of camera intrinsics and extrinsics because voxelization is applied in the tray frame rather than the camera frame. Various experimental results in Section E demonstrate the simulation-to-real benefits of the voxel grid representation over the RGBD representation.

[0086] 2. Implementation Methods of Mining Prediction Models

[0087] In one or more embodiments, excavation prediction is modeled as a binary classification problem. An excavation classifier predicts the probability Y of excavation success (i.e., bucket filling success) as a function of the excavation instance. In this patent document, two CNN models are proposed to predict the excavation success probability, namely, "excavation RGBD network (excavation-RGBD-net)" and "excavation voxel network (excavation-voxel-net)". Each model takes as input an excavation instance including a task trajectory and an RGBD image (or voxel grid) and predicts the probability of excavation success as output.

[0088] Residual networks (ResNet) provide an advanced CNN architecture for various computer vision tasks such as image classification and object detection. In one or more embodiments, ReNet-18 (an 18-layer deep CNN) is used as the backbone network architecture for excavation-RGBD-net and as the backbone network for excavation-voxel-net, which is extended to a 3D CNN. The offline verification results in Section F.5 and the experiments in Section E confirm the effectiveness of the two models, especially the excavation-voxel-net. It should be understood by those skilled in the art that, inspired by this patent document, some other alternative network structures, such as a dig & dump network for 2D CNN RGBD models and voxel configuration networks or a shape completion CNN for 3D voxel models, can also potentially be applied to mining planning. Such optional network structures will be within the scope of this patent disclosure.

[0089] Figure 4 shows an excavation-voxel-net structure using ReNet-18 according to an embodiment of the present disclosure, and Figure 5 Shows the use of Figure 4 4. The process of learning-based mining prediction of the excavation-voxel-net structure shown in FIG. 4. In step 505, the mining instance 410 represented by multiple trajectory parameters is tiled point by point on the voxel grid dimensions (e.g., 64×64×32) to obtain a tiled trajectory parameter voxel grid 415. In step 510, the tiled trajectory parameter voxel grid is then connected with the voxel grid representation 420 of the mining scene to generate the input voxel grid 430 of the mining instance. In one or more embodiments, the input voxel grid has a total of multiple channels corresponding to the dimensions of each voxel. For example, the input voxel grid can have seven channels, one channel for the scene voxel grid and six channels for the spliced ​​trajectory parameters. The scene voxel grid can be obtained by voxelizing the filtered point cloud of the mining scene 420.

[0090] In one or more embodiments, the 2D convolution filters of the original ResNet-18 are replaced with 3D convolution filters to construct the ResNet 3D-18 backbone network. In step 515, the input voxel grid 430 is fed into a 3D CNN (e.g., ResNet3D-18) 435 to generate a multi-dimensional (e.g., 1000-dimensional) feature vector 440. Then, in step 520, the feature vector (e.g., ResNet3D-18 feature vector) is processed using at least one fully connected layer (e.g., 3 layers) 445 to predict the probability of mining success. In one or more embodiments, the three fully connected layers have 512, 256, and 128 rectified linear unit (ReLU) neurons, respectively. The fully connected layers can be empirically tuned during training. In one or more embodiments, at least one fully connected layer is followed by a sigmoid output layer 450 to process the feature vector 440. In one or more embodiments, batch normalization is applied to all fully connected layers except the output layer. In one or more implementations, the mining classifier is trained using a cross entropy loss.

[0091] In one or more embodiments, the excavation-RGBD-net may share a similar structure to the excavation-voxel-net, except that the original ResNet-18 backbone with 2D convolutions is used for the excavation-RGBD-net and the trajectory parameters are tiled in image space instead of voxel grid space.

[0092] In one or more embodiments, mining prediction can also be modeled as a regression problem, rather than a classification problem. By replacing the sigmoid output layer with a fully connected layer, the excavation-RGBD-net and excavation-voxel-net can be adapted as "excavation RGBD regression network (excavation-RGBD-reg-net)" and "excavation voxel regression network (excavation-voxel-reg-net)", respectively. The regression model can be trained using a smooth L1 loss (i.e., Huber loss).

[0093] In one or more embodiments, to demonstrate the importance of scene dependency for mining learning and to provide a data-driven benchmark for experiments, a fully connected mining classification network, "Excavation Trajectory Network (Excavation-Traj-Net)", was developed. The scene-independent Excavation Trajectory Network (Excavation-Traj-Net) can only take task trajectories without visual scene representation as input. It has three fully connected layers with 512, 256, and 128 Reinforced Unit (ReLU) neurons, respectively. Its final sigmoid layer outputs the probability of successful mining.

[0094] In summary, five mining prediction model implementation methods are proposed in this patent document: excavation-RGBD-net, excavation-voxel-net, excavation-traj-net, excavation-RGBD-reg-net and excavation-voxel-reg-net.

[0095] 3. Implementation Methods of Learning-Based Mining Planning

[0096] Given a visual representation of the mining scenario, the goal is to plan a mining trajectory T that maximizes the probability Y of successful mining. In one or more embodiments, mining planning is formulated as an optimization problem:

[0097]

[0098] In Equation 1, f(T, Z, W) defines a neural network classifier with a trained logistic output to predict the probability of mining success as a Bernoulli distribution on Y. The parameter W defines the neural network parameters.

[0099] In one or more embodiments, CEM can be used to solve mining optimization problems using learned mining prediction models. Figure 6The process for learning-based mining planning according to an embodiment of the present disclosure is shown. As a sampling-based optimization method, CEM iteratively samples from the current distribution and selects the top K samples using a scoring function to update the distribution. Given a visual representation of the current mining scene, the Gaussian distribution of the 6D task trajectory parameters is optimized. In step 605, a random heuristic mining planner is used to generate multiple (e.g., 256) trial mining trajectories for the mining scene to initialize the Gaussian distribution. More details of the random heuristic planner can be found in Section F.3. In step 610, the Gaussian distribution is updated through multiple iterations of mining trajectory sampling and multiple iterations of mining trajectory selection based on the predicted mining success probability of each sampled mining trajectory. In this step, multiple (e.g., 5) iterations are used for CEM mining planning to update the Gaussian distribution. In each iteration, multiple (e.g., 256) mining trajectory samples are first sampled from the current distribution. Then, the mining success probability of each mining trajectory is predicted using the learned mining prediction model. Then, a top subset (e.g., 64) of the samples based on the predicted mining success probability is selected to update the Gaussian distribution. In summary, CEM uses the learned prediction model as a quality metric to iteratively improve the distribution of task trajectory parameters through sampling and distribution updating. In step 615, a subset (e.g., 64) of mining trajectories is sampled from the Gaussian distribution of the last iteration for evaluation using the learned mining prediction model. In step 620, the mining trajectory with the highest predicted success probability is selected as the planned task mining trajectory. In one or more embodiments, the selected mining trajectory may need to be a valid IK solution and need to have a valid attack point range.

[0100] E. Experimental Results

[0101] In this section, we first introduce the experimental setup and experimental results in simulation. Then, we present the experimental setup and experimental results in the real world. In simulation and the real world, we compare the proposed learning-based planner implementation with two heuristic planners and a data-driven baseline planner. The experimental results show that the learning-based planner is able to plan high-quality mining and significantly outperforms the baseline method. Data collection, model training, offline validation, more detailed results analysis, and ablation studies are provided in Section F of the following experimental appendix.

[0102] It should be noted that these experiments and results are provided by way of illustration and were performed under specific conditions using one or more specific embodiments; therefore, these experiments and their results should not be used to limit the scope of the disclosure of this patent document.

[0103] 1. Experimental setup in simulation

[0104] Training data was collected and simulations were conducted in PyBullet. A UR5 robot arm was used for excavation data collection. The UR5 arm has a total of 6 degrees of freedom (DoF). The simulation controlled the shoulder translation, shoulder lift, elbow joint, and first wrist joint of the UR5 arm, while disabling the other two wrist joints by fixing their joint angles. A 3D-designed bucket was used as the UR5's end effector in the simulation. The total volume of the bucket was 450 cm. 3 .

[0105] In one or more embodiments, RGB and depth images of each excavation trial may be generated by a built-in simulated camera in PyBullet. Figure 7 The camera setup in the simulation is shown (on the left). Figure 8 An example of an RGB image produced by a simulated camera is seen on the left. More details on the camera and mining scene setup used for the real robotic experiments are discussed in Section F.1.

[0106] In one or more embodiments, for each experimental trial of a particular excavation planner, a joint space trajectory can be calculated using IK and interpolated from the excavation planner planned task trajectory T. The joint space waypoints are then sent to the UR5 arm joint position controller in simulation.

[0107] 2. Experiments in Simulation

[0108] Simulation experiments were conducted with the learning-based planners excavation-voxel-net, excavation-RGBD-net, excavation-voxel-reg-net, excavation-RGBD-reg-net, and excavation-traj-net. These learning-based planners are named "CEM-voxel," "CEM-RGBD," "CEM-voxel-reg," "CEM-RGBD-reg," and "CEM-traj," respectively. CEM-traj serves as a data-driven baseline planner without visual scene representation input. Furthermore, these five learning-based planners were compared with two heuristic planners: random-heu and highest-heu. More details on these two heuristic planners can be found in Section F.3. For each method, multiple mining events, for example, 100 events, were experimented with. Ten mining trials were performed sequentially for each mining event. This provided a total of 1000 mining trials for each method.

[0109] The simulation results of all seven different methods are given in Table I. The mining of each planner is evaluated using three metrics: the volume of mined objects (mining volume), the number of mined objects, and the mining success rate. Same as the model training in Section F.4, a sample is considered successful if the total volume of objects successfully mined is above a predefined threshold (i.e., 134 cm 3 or 30% bucket filling rate), it is counted as a success, otherwise it is a failure. The computation time of each planner is also reported.

[0110] The mean and standard deviation of all metrics except success rate are listed (in parentheses). The mean and standard deviation of each method were calculated over its 1000 experimental mining trials. As shown in Table 1, CEM-voxel achieved the best mining performance in terms of mined volume, number of mined objects, and success rate. CEM-voxel mined an average of 136 cm per mining trial. 3 objects, which is 30.2% of the entire bucket volume (i.e., bucket volume filling rate). On these three mining metrics, CEM-voxel, CEM-RGBD, and CEM-voxel-reg have a large advantage over the two heuristic planners and CEM-traj, which shows the effectiveness of scene-dependent mining learning.

[0111] Table 1: Experimental results of seven mining planners in simulation

[0112] method <![CDATA[Volume (cm 3 )]]> quantity Success rate Time(s) CEM-voxel 136.23(106.14) 7.58(6.05) 51.9％ 10.5(0.77) CEM-RGBD 129.78(101.50) 7.51(6.03) 48.1％ 17.3(0.28) CEM-voxel-reg 127.89(105.22) 7.15(6.09) 47％ 10.44(0.29) CEM-RGBD-reg 107.93(98.18) 6.29(5.73) 35.3％ 17.4(0.27) CEM-traj 97.27(100.73) 5.54(5.87) 32.4％ 3.17(0.28) random-heu 85.81(87.65) 4.73(4.99) 28.4％ 0.2(0.03) highest-heu 67.24(76.43) 3.36(4.19) 19.3％ 0.2(0.02)

[0113] The classification-based CEM-voxel and CEM-RGBD perform better than the regression-based CEM-voxel-reg and CEM-RGBD-reg, respectively. Since classification is about predicting labels and regression is about predicting continuous quantities, it is believed that mining regression is more complex and requires more training data to perform as well as or better than mining classification.

[0114] The fact that the scene-dependent planners CEM-voxel, CEM-RGBD, and CEM-voxel-reg significantly outperform the scene-independent CEM-traj planner suggests that it is important to learn to plan mining trajectories based on visual scene information.

[0115] In terms of mining volume and number of objects, all five learning-based planners have higher standard deviations than the two heuristic planners. CEM-voxel has the highest standard deviation. The experimental results of the heuristic planners mostly show failures in mining with small mining volumes. The learning-based planners, especially CEM-voxel, produce mining with relatively high mining volumes. This makes the mining volumes of the learning-based planners more evenly distributed and have larger standard deviations, which are shown in the Appendix. Figures 12A-12GThe volume histograms of different planners are shown in Figure 4.

[0116] In terms of computational speed, the heuristic planner takes 0.2 seconds to plan a mining trajectory. CEM-voxel, CEM-RGBD, CEM-voxel-reg, and CEM-RGBD-reg take more than 10 seconds to generate a mining trajectory. CEM-traj takes 3 seconds to plan a trajectory. Finally, Figure 9 Visualization of 6 high-quality excavation examples planned by the CEM-voxel planner in simulation.

[0117] 3. Real-world Experimental Setup

[0118] A real robotic excavation experiment was conducted using the Franka Panda robotic arm. The Franka Panda arm has a total of 7 DoFs. The shoulder translation, shoulder lift, elbow lift, and wrist lift joints of the Franka arm are controlled as excavation joints, while the other three joints (i.e., elbow swing and the last two wrist joints) are disabled by fixing their joint angles. The same bucket model used in the simulation was 3D printed as the Franka arm end effector. The Azure Kinect camera generates RGBD images and point clouds of the excavation scene. Figure 7 The right side shows a real-world camera setup. Figure 8 An example of an Azure RGB image showing the mining setup is shown in . More details on the camera and mining scene setup used for the real robotic experiments are presented in Subsection F.2.

[0119] For each experimental trial of a particular planner, the joint space trajectory was calculated from the planned task trajectory T using IK and sent to the Franka arm's built-in joint position controller. The Franka arm can only generate a limited amount of force and torque. For example, Franka's force and torque ranges are (i.e., in the direction of gravity) [-50; 150] N and [-10; 10] Nm, respectively. This makes it difficult for the bucket to penetrate the rigid object given the high resistance of the rigid object. During the penetration process, the robot was automatically commanded to move the bucket back and forth 2 cm alternately at each waypoint, which helped prevent the robot from getting stuck.

[0120] 4. Real Robot Experiment

[0121] A mining model learned in UR5 simulation was transferred to the real-world Franka for rigid object mining experiments. Representing the mining trajectory in task space allows the mining prediction model to be transferred from one hardware platform to another with similar kinematic reachability. The reachabilities of the UR5 and Franka arms are 850 mm and 800 mm, respectively. In addition to the task trajectory representation, the mining pose in the pallet frame is also represented, making mining learning and planning agnostic to the different pallet poses across simulation and the real world.

[0122] Mining experiments were conducted to evaluate the proposed learning-based planner implementations, CEM-voxel and CEM-RGBD, which achieved the best performance in simulation experiments. The learning-based planner implementations were also compared with two heuristic planners (random-heu and highest-heu). Five mining events were tested in the real world for each method. The rigid objects in each mining event were randomly repositioned. Five mining trials were performed for each mining event, resulting in a total of 25 mining trials for each method.

[0123] The details of the two heuristic planners used for simulation are described in Section F.3. In the real world, the random parameter range of the heuristic planner is smaller than that in the simulation. This is because experiments with large heuristic ranges may be unsafe for humans or robots. For example, a relatively long drag length causes collisions with the pallet. In addition, the Franka arm can only generate a limited amount of force and torque, which makes it difficult to penetrate into rigid objects with a depth greater than 5 cm. Specifically, in the real world, the attack digging angle α and the closing angle β are randomly generated in the ranges of [-110, -70] degrees and [-110, -140] degrees, respectively. The penetration depth d and the drag length l are randomly generated in the ranges of [0.02, 0.05] m and [0.02, 0.06] m, respectively. As described in Section F.3, because the trial digging trajectories are generated to initialize the CEM, the trajectory parameter range of the heuristic planner also affects the proposed learning-based planner.

[0124] The real robot experiment results of all four methods are given in Table 2. The mining performance is evaluated based on the volume of the mining object and the mining success rate. The mean and standard deviation (in brackets) are reported for the volume of the mining object. The success threshold of the mining object volume is 134 cm 3 , the same as simulation.

[0125] Table 2: Real robot experimental results of four mining planning methods

[0126]

[0127]

[0128] Table 2 also shows the effectiveness of each planner. A digging trial is considered valid if a trajectory can be successfully planned and executed. Invalid digging trials are primarily caused by exceeding the robot's force / torque limits. High resistance during digging (especially penetration) and collisions with the pallet can lead to exceeding the force / torque limits. Figure 11 An example of a Franka arm stuck due to exceeding the force / torque limit is shown. The force control used for the execution of the excavation trajectory can also be checked to mitigate exceeding the force / torque limit. Excavation trials without a valid trajectory IK are also calculated as invalid.

[0129] As shown in Table 2, the CEM-voxel planner significantly outperforms these other three planners in terms of the volume and success rate of excavated objects in the real world. CEM-voxel excavates an average of 110 cm per excavation. 3 The object is 24.4% of the entire bucket volume. CEM-voxel significantly outperforms both heuristic planners, demonstrating the effectiveness of mining learning in the real world. The fact that CEM-voxel outperforms CEM-RGBD suggests that voxel-based visual representations are better at addressing the simulation-to-reality gap than RGBD representations. The computation time for each planner in the real world is similar to that in simulation.

[0130] The CEM-RGBD planner performs poorly in the real world, worse than random-heu and roughly on par with highest-heu. The attack poses of the trajectories planned by CEM-RGBD are mostly close to the edge of the pallet, which leads to invalid mining trials with collisions. This is because the RGBD image representation suffers from a large simulation-to-real gap when transferring the mining knowledge gained in simulation to the real world. In addition to the poor mining performance, another evidence of the RGBD simulation-to-real gap is that the predicted success probability of CEM-RGBD trajectories in the real world is close to zero. More details on the visual representation of the mining scene are discussed in Section E.1.

[0131] Figure 10 Shown are 6 high-quality excavation examples planned by the CEM-voxel planner on a real robot, with the volume of the excavated object annotated on each example.

[0132] F. Experimental Appendix

[0133] The experimental appendix first introduces the mining scenario settings in simulation and real world, describes the data collection and training of the mining prediction model, and presents an offline evaluation of the learning model results. In addition, mining volume histograms, trajectory analysis, and ablation studies are discussed.

[0134] As defined in Section C.2, the task space mining trajectory T can contain six parameters T = (x, y, α, d, l, β). In one or more embodiments, for ablation studies, the (x, y) and (α, d, l, β) of the task trajectory T are named "points of attack" (PoA) and "geometric trajectory parameters" (GTP), respectively.

[0135] 1. Camera and mining scene settings in simulation

[0136] like Figure 7 As shown on the left side of the figure, the camera is located at (0.5m, 0.8m, 0.91m) in the robot base frame. The robot frame's y-axis points from the robot base to the center of the pallet. The robot frame's z-axis is along the direction of gravity. The transformations between the camera and the robot base frame, as well as the transformations between the robot base frame and the pallet frame, are known. Using these two transformations, the point cloud acquired from the PyBullet RGBD camera can be transformed to the pallet frame to generate a grid map and voxel grid.

[0137] For each mining scene, there are n objects uniformly sampled in the range [200, 400]. Then for the current mining scene, the same number of test objects are scattered into the tray in random poses. As described in Section F.3, the test object mesh is not visible during training. In the tray framework, a 0.38×0.4×0.3m 3 The point cloud of the excavation scene is filtered by a cube extent of , which is then used for grid map and voxel grid generation. In the simulation, this cube extent covers the excavation space of the rigid objects in the tray.

[0138] 2. Real-world camera and mining scene setup

[0139] like Figure 7 As shown on the right, the Azure camera is located at (0.57m, 1m, 1.14m) in the robot frame. The robot frame's y-axis points from the robot base to the center of the pallet. Its z-axis is along the direction of gravity. The transformation between the camera and the robot base frame is manually calibrated using ArUco markers. The pallet frame is manually defined relative to the robot base frame based on the excavation range, which gives the transformation between the robot frame and the pallet frame. The pallet frame has the same orientation as the robot base frame. The origin of the pallet frame is defined as the center of the excavation cube range. Knowing these two transformations, the point cloud obtained from the Azure camera can be transformed into the pallet frame to generate a raster map and voxel grid.

[0140] The real-world robotics experiment used multiple rigid wooden objects of various geometric shapes and colors, including 100 "Melissa & Doug wooden blocks" and 32 "Biubee wood-stone balance blocks." For example, there were both convex and concave objects. The density of the wooden rigid objects was estimated to be 0.5 kg / cm. 3 . All these rigid objects are not visible from training.

[0141] First, a layer of heavy rock is placed in the excavation tray, which stabilizes the tray during the excavation process. Then a layer of mulch is laid on top of the rock. Finally, a rigid wooden structure is placed on the mulch in front of the Franka arm to start excavating. For safety reasons, the relatively deformable mulch is used as the excavation surface. In the tray frame, a 0.4×0.3×0.3m 3 The point cloud of the excavation scene is filtered using a cube, which is used to generate the grid map and voxel grid. This cube covers the excavation space used in the real robot experiments. Only about half of the pallet space near the robot base is used for the excavation experiments.

[0142] Digging events were generated by shaking these 132 rigid objects in a box and then dumping them into the digging area of ​​a tray. For each trial, the robot dumped the excavated objects into a dump tray after digging. Sometimes, a certain amount of cover beneath the rigid objects might be excavated and dumped. Across all experimental trials, the amount of cover excavated was relatively small on average.

[0143] The desired dumping pose of the bucket end-effector is assigned to the center of the dump tray. The robot first moves to the desired dumping pose and then controls the bucket to dump the object vertically downward into the dump tray. For each trial, the object dumped into the tray is weighed using a kitchen scale under the dump tray. Given the mass and density of the excavated object, the volume of the excavated object can be calculated.

[0144] 3. Simulation Data Collection

[0145] An excavation simulation test was conducted using a UR5 arm with a 3D designed bucket. The total bucket volume of the bucket is 450cm 3 . The data collection setup is the same as the simulation experiment setup in Section E.1. The camera and excavation scene setup for data collection is described in Section F.1. Random geometric rigid object meshes for simulated excavation are generated using trimesh. The number of vertices for each object mesh is randomly selected in the range of 10 to 50. For the object mesh, the maximum value of each coordinate is uniformly sampled from 1 cm to 5 cm. The 3D coordinates of all vertices of the object mesh are randomly generated from the range of zero to its maximum coordinate value. The convex hull of the original mesh is calculated and used as the final object mesh. The object density is assumed to be 6 g / cm in the simulation. 3At the same time, 100k training and test candidate object meshes are generated respectively. The training object mesh dataset is used for training data collection in simulation. The test object mesh dataset is used for offline evaluation and experimentation of mining prediction models in simulation.

[0146] For each mining event in data collection, a certain number of object meshes are randomly selected from the training object set. Each selected object is then dumped into the mining tray in a randomly generated pose. The number of objects per scene is randomly and uniformly generated within the range of 50 to 400. Twenty mining trials are performed sequentially for each mining event. One of two heuristic planners is randomly selected to calculate the total volume of objects successfully mined in each trial. For each mining trial, the mined objects are dumped into the dump tray in a random pose. To validate the mining prediction network offline, 50,000 training mining samples and 10,000 test samples are collected.

[0147] Trial Mining Planners: Two trial mining planners are designed for data collection, namely “random-heu” and “highest-heu”. For the random-heu planner, a grid map cell of the mining scene is randomly selected, and the center of the grid map cell is used as the 2D coordinate (x, y) of the attack mining pose. For the highest-heu planner, the center of the grid map cell with the maximum height is generated as the 2D coordinate (x, y) of the attack mining pose. It is assumed that the attack point is on the surface of the cluttered object. Under this assumption, the z coordinate value of the attack mining pose is calculated as the height of the corresponding grid map cell.

[0148] In both simulation and real-world scenarios, a grid map library was used to generate mining grid maps from the mining space's point cloud. The attack mining angle α and closing angle β were randomly generated within the ranges of [-120, -60] degrees and [-180, -120] degrees, respectively. The penetration depth d and drag length l were randomly generated within the ranges of [0.05, 0.2] m and [0.05, 0.4] m, respectively. The same random parameter ranges were used for data collection in simulation and experiments.

[0149] 4. Mining prediction model training

[0150] For mining prediction model training, 50,000 training mining samples were collected in simulation. Among the training mining samples, 45,000 training samples were used for training, while the other 5,000 training samples were used as validation sets. For mining binary classification, if the total volume of the sample's successful mining object is higher than 134 cm 3, then the mining sample is considered successful, otherwise it is a failure. The mining sample without a valid task trajectory IK is marked as a failed mining, which aims to learn to plan a mining trajectory with a valid IK. Among these 50,000 training samples, 4768 (10%) are successful mining.

[0151] All five mining prediction models, including excavation-RGBD-net, excavation-voxel-net, excavation-RGBD-reg-net, excavation-voxel-reg-net, and excavation-traj-net, are trained using the same specifications. In order to overcome class imbalance (i.e., the low percentage of successful mining samples), the successful samples are oversampled so that the number of positive and negative samples is roughly the same for all five models in each training period. The results of training excavation-RGBD-net from scratch and training excavation-RGBD-net from a tuned ResNet-18 are compared. Training from scratch has significantly better performance, which may be because the mining task is significantly different from ResNetImageNet classification. In addition to excavation-RGBD-net, all other four models are trained from scratch.

[0152] All networks were trained for 50 epochs using the Adam optimizer with a mini-batch size of 64. The learning rate started at 0.1 and was reduced by a factor of 10 every 10 epochs. Training of excavation-RGBD-net and excavation-RGBD-reg-net took approximately 810 minutes on an Alienware desktop computer with an Intel i7-6800K processor, 32GB of RAM, and an Nvidia GeForce GTX TITAN Z graphics card. Training of excavation-voxel-net and excavation-voxel-reg-net took approximately 500 minutes on the same machine. Training of excavation-traj-net took 101 minutes on the same machine. All mining prediction networks were implemented in PyTorch.

[0153] 5. Offline evaluation of mining prediction models

[0154] To validate the mining prediction model offline, 10,000 test samples were collected using the test object dataset in simulation. Of these 10,000 test samples, 967 were successfully mined.

[0155] Table 3 shows the accuracy, precision, recall, and F1 score of three different methods. The second, third, and fourth rows show the offline test results of excavation-voxel-net, excavation-RGBD-net, and excavation-traj-net, respectively. The "random-0.5" method in the fifth row refers to a random guess with a probability of 0.5 for a positive prediction. The "random-0.1" method in the last row means a random guess with a probability of 0.1 for a positive prediction. The prediction metrics of random guessing show the classification challenges caused by the low percentage of successfully mined samples. In terms of these offline evaluation metrics, excavation-voxel-net and excavation-RGBD-net perform quite well and significantly outperform random guessing. Excavation-voxel-net achieves the best offline evaluation performance. For offline evaluation, excavation-traj-net performs worse than excavation-voxel-net and excavation-RGBD-net, but significantly outperforms random guessing.

[0156] Table 3: Offline evaluation results of excavation-voxel-net, excavation-RGBD-net, excavation-traj-net and random guessing for successful classification mining on the test set

[0157]

[0158]

[0159] The mining regression models excavation-voxel-reg-net and excavation-RGBD-net are also evaluated offline on the test set using the L1 norm error. The mean and standard deviation of the test L1 norm error of excavation-voxel-reg-net are 31.94 cm 3 and 37.66cm 3 The test L1 norm error of excavation-voxel-reg-net has 35.22cm 3 The average value is 43.71cm 3 Both regression models achieved fairly good test performance.

[0160] 6. Simulation experiment to mine volume histogram

[0161] exist Figures 12A-12GFigure 2 visualizes the histograms of the mined volumes mined by seven planners across 1,000 simulated experiments. In addition to the mean mined volumes and mining rates shown in Table 1, the histograms show that the learning-based planner mines objects with larger volumes than the heuristic planner. The histograms also indicate that the distribution of the mined volumes for the learning-based planner is more uniform and has a larger standard deviation than the heuristic planner. Only the histograms for the mined volumes in the simulated experiments are shown here because the number of mined volumes in the real-world experiments is relatively small.

[0162] exist Figure 12H The mining volume histogram of the training data is plotted in . If the mining volume is larger than 134cm 3 (i.e., 30% bucket filling rate), the excavation is considered successful. Figure 12H The vertical line 1275 shows that the excavation volume is 134 cm 3 Place. 134cm 3 As the mining success threshold, 4768 (10%) of the 50,000 training samples are successful.

[0163] As described in Section F.4, since there are far fewer successful training mining samples than failed training mining samples, we oversample the successful samples to overcome this class imbalance problem in mining training. 3 This may result in fewer successful mining training samples, which will make mining training more difficult due to more severe classification imbalance. On the other hand, if the success threshold is lowered to less than 134cm 3 , then the learning-based planner will be more likely to generate excavation trajectories with bucket filling ratios below 30%. This will harm the excavation performance of the learning-based planner. Therefore, 134 cm is selected. 3 as a reasonable success threshold for mining learning. In addition, in Section E.2, we show that the classification-based CEM-voxel and CEM-RGBD outperform the regression-based CEM-voxel-reg and CEM-RGBD-reg, respectively, which empirically justifies the choice of mining threshold.

[0164] 7. Experimental mining trajectory analysis

[0165] Tables 4 and 5 show the mean and standard deviation of trajectory parameters mined from 1000 simulation trials for each of the seven planners, respectively. Figures 13A to 13G The PoA distribution of simulation results for all 7 planners is shown. In each PoA diagram, the coordinate origin is located at the center of the pallet. The robot base is located at (x=-50, y=0) in the 2D pallet frame.

[0166] Table 4: Average trajectories of different methods in simulation experiments

[0167] method Trajectory average CEM-voxel [0.03,0.04,-1.54,0.12,0.24,-2.56] CEM-RGBD [0.02,0.05,-1.59,0.13,0.24,-2.79] CEM-voxel-reg [0.02,0.04,-1.54,0.13,0.25,-2.51] CEM-RGBD-reg [0.06,0.06,-1.52,0.14,0.26,-2.59] CEM-traj-opt [0.02,0.1,-1.56,0.14,0.23,-2.54] random-heu [0,0,-1.57,0.12,0.22,-2.61] highest-heu [-0.06,0.01,-1.59,0.12,0.23,-2.62]

[0168] The mean and standard deviation of GTP for different planners are generally similar, indicating that learning-based planners produce mining trajectories with large GTP diversity. Due to the randomness of the heuristic planner, the standard deviation of PoA for the learning-based planner is smaller than that of the heuristic planner. In terms of PoA mean, CEM-voxel, CEM-RGBD, and CEM-voxel-reg are similar, and they are relatively different from CEM-RGBD-reg, CEM-traj-opt, random-heu, and highest-heu.

[0169] Table 5: Standard deviation of trajectories of different methods in simulation experiments

[0170] method Trajectory standard deviation CEM-voxel [0.07,0.1,0.34,0.04,0.1,0.36] CEM-RGBD [0.09,0.08,0.33,0.04,0.11,0.29] CEM-voxel-reg [0.07,0.11,0.32,0.04,0.1,0.37] CEM-RGBD-reg [0.07,0.07,0.28,0.04,0.09,0.26] CEM-traj-opt [0.06,0.05,0.36,0.03,0.1,0.37] random-heu [0.11,0.11,0.3,0.04,0.1,0.31] highest-heu [0.12,0.13,0.3,0.04,0.1,0.3]

[0171] From the trajectory averages and Figures 13A-13G The PoA distribution plot in Figure 2 shows that the learning-based planner prefers to generate PoAs in the right (i.e., positive x-direction) and upward (i.e., positive y-direction) regions. Highest-heu generates a large number of PoAs near the left edge of the pallet (i.e., x = -0.19 m). During excavation, objects tend to be pushed toward the left wall of the pallet. Therefore, the highest point is more likely to occur near the left wall of the pallet.

[0172] There are 5000 randomly selected mining training mining samples. The PoAs of successful and failed mining are plotted in Figure 13 respectively. As described in Section F.3, the training data statistically contains half random-heu and half highest-heu mining. As shown in the PoA distribution of successful training samples in Figure 13, there are more successful PoAs in the upper half of the pallet, which explains that the learning-based planner prefers PoAs in the top area. The UR5 excavator swings around the swing center (x=-0.5m, y=0.109m) in the pallet 2D coordinates. When the PoA is close to the bottom of the pallet (i.e., y=-0.2m), the robot will have relatively small space to drag and close due to collision with the pallet. In addition, in the PoA distribution of successful training samples shown in Figure 13, there are more failed mining near the left edge of the pallet, which prompts the learning-based planner to plan the PoA away from the left edge.

[0173] We have reported in Section E.2 that learning-based scenario-dependent planners, such as CEM-voxel, significantly outperform CEM-traj, indicating that it is important to use visual representation learning of mining scenarios to plan scenario-dependent mining trajectories. However, the trajectory and PoA distributions may not reflect the benefits of learning-based scenario-dependent planning.

[0174] 8. Ablation Experiment

[0175] Ablation experiments were conducted to provide insights into how learning-based planners can improve mining of cluttered rigid objects. Since the CEM-voxel planner achieved the best mining performance in both simulation and real-world robotics experiments, the ablation study primarily focused on the CEM-voxel planner. Two ablation experiments were performed by replacing the PoA and GTP of each CEM-voxel trajectory with random parameters, respectively. The random PoA and GTP parameters were uniformly sampled from the same range as for the heuristic planner introduced in Section F.3. Using the same experimental setup and protocol as in Section E.1, 1000 mining trials were performed in simulation for both ablation experiments.

[0176] The mining volume, number of mined objects, and mining success rate of the two ablation experiments are given in Table 6. For the mining volume and number of objects, the mean and standard deviation are listed (in parentheses). The experimental results of the original CEM-voxel in simulation are shown in Table 1. In terms of the three mining metrics, CEM-voxel with random PoA and random GTP performs worse than the original CEM-voxel. This shows that CEM-voxel understands how to generate good PoA and GTP parameters for mining. CEM-voxel with random PoA has worse mining performance than CEM-voxel with random GTP. This means that for CEM-voxel, learning PoA is more important than learning GTP.

[0177] G. Some conclusions

[0178] In this patent document, multiple deep networks are proposed for successful prediction of a new task: cluttered rigid object mining. In one or more embodiments, mining planning is modeled as an optimization problem using a learned prediction model. Mining experiments in simulation and the real world demonstrate that embodiments of a learning-based planner can produce high-quality mining. Experimental results also show that the learning-based mining planner outperforms two heuristic planners and a data-driven, scenario-independent planner.

[0179] In one or more embodiments, the excavation trajectory is greedily planned by maximizing the excavation volume of the current excavation. Although the excavation success rate of a single excavation is used in one or more embodiments, the long-term expected excavation reward of continuous excavation can also be considered for excavation of rigid objects. Potentially, force control can be used instead of position control, making the execution of the excavation trajectory smoother and more robust to large resistance forces. Embodiments of the proposed learning-based planner can be applied from robotic arms to real excavators.

[0180] H. Computing System Implementation

[0181] In one or more embodiments, aspects of this patent document may be directed to, may include, or may be implemented on one or more information processing systems (or computing systems). An information processing system / computing system may include any tool or set of operable tools to calculate, measure, determine, classify, process, send, receive, retrieve, originate, route, exchange, store, display, communicate, visualize, detect, record, reproduce, process, or utilize any form of information, intelligence, or data. For example, a computing system may be or may include a personal computer (e.g., a laptop computer), a tablet computer, a mobile device (e.g., a personal digital assistant (PDA), a smartphone, a tablet phone, a tablet, etc.), a smartwatch, a server (e.g., a blade server or a rack server), a network storage device, a camera, or any other suitable device, and may also vary in size, shape, performance, functionality, and price. A computing system may include random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, read-only memory (ROM), and / or other types of memory. Additional components of a computing system may include one or more drives (e.g., a hard drive, a solid-state drive, or both), one or more network ports for communicating with external devices, and various input and output (I / O) devices such as a keyboard, mouse, touch screen, stylus, microphone, camera, trackpad, display, etc. A computing system may also include one or more buses for transmitting communications between the various hardware components.

[0182] Figure 15 A simplified block diagram of an information processing system (or computing system) according to an embodiment of the present disclosure is depicted. It should be understood that the functionality shown in system 1500 can be used to support various embodiments of computing systems, although it should be understood that computing systems can be configured differently and include different components, including having fewer or more components such as Figure 15 Parts shown.

[0183] like Figure 15 As shown, computing system 1500 includes one or more CPUs 1501 that provide computing resources and control the computer. CPU 1501 can be implemented using a microprocessor or the like and may also include one or more graphics processing units (GPUs) 1502 and / or floating-point coprocessors for mathematical calculations. In one or more embodiments, one or more GPUs 1502 may be incorporated into a display controller 1509, such as a graphics card or part of multiple graphics cards. System 1500 may also include system memory 1519, which may include RAM, ROM, or both.

[0184] like Figure 15 As shown, multiple controllers and peripheral devices may also be provided. Input controller 1503 represents an interface to various input devices 1504. Computing system 1500 may also include a storage controller 1507 for interfacing with one or more storage devices 1508, wherein each storage device 1508 includes a storage medium or optical medium such as a tape or disk that can record instruction programs for operating systems, utilities, and applications, which may include implementations of programs that implement various aspects of the present disclosure. Storage device 1508 may also be used to store processed data or data to be processed according to the present disclosure. System 1500 may also include a display controller 1509 for providing an interface to a display device 1511, which may be a cathode ray tube (CRT) display, a thin film transistor (TFT) display, an organic light emitting diode, an electroluminescent panel, a plasma panel, or any other type of display. Computing system 1500 may also include one or more peripheral device controllers or interfaces 1505 for one or more peripheral devices 1506. Examples of peripheral devices may include one or more printers, scanners, input devices, output devices, sensors, etc. The communication controller 1514 can interface with one or more communication devices 1515, which enable the system 1500 to connect to remote devices via any of a variety of networks, including the Internet, cloud resources (e.g., Ethernet cloud, Fibre Channel over Ethernet (FCoE) / Data Center Bridging (DCB) cloud, etc.), local area networks (LANs), wide area networks (WANs), storage area networks (SANs), or via any suitable electromagnetic carrier signals, including infrared signals. As shown in the depicted embodiment, the computing system 1500 includes one or more fans or fan trays 1518, and one or more cooling system controllers 1517 that monitor the thermal temperature of the system 1500 (or components thereof) and operate the fans / fan trays 1518 to help regulate the temperature.

[0185] In the system shown, all major system components can be connected to bus 1516, which can represent more than one physical bus. However, various system components may or may not be physically close to each other. For example, input data and / or output data can be remotely transmitted from one physical location to another. In addition, programs that implement various aspects of the present disclosure can be accessed from a remote location (e.g., a server) over a network. Such data and / or programs can be transmitted via any of a variety of machine-readable media, including: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as compact disks (CDs) and holographic devices; magneto-optical media; and hardware devices specifically configured to store or execute program code, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, other non-volatile memory (NVM) devices (such as 3D XPoint-based devices), and ROM and RAM devices.

[0186] Aspects of the present disclosure may be encoded on one or more non-transitory computer-readable media having instructions for one or more processors or processing units so that the steps are performed. It should be noted that the one or more non-transitory computer-readable media should include volatile and / or non-volatile memory. It should be noted that alternative implementations are possible, including hardware implementations or software / hardware implementations. Hardware-implemented functions may be implemented using ASICs, programmable arrays, digital signal processing circuits, etc. Therefore, the term "device" in any claim is intended to cover both software and hardware implementations. Similarly, the term "computer-readable medium" as used herein includes software and / or hardware, or a combination thereof, having an instruction program embodied thereon. With these implementation alternatives in mind, it should be understood that the figures and corresponding descriptions provide the functional information required by those skilled in the art to write program code (i.e., software) and / or manufacture circuits (i.e., hardware) to perform the desired processing.

[0187] It should be noted that embodiments of the present disclosure may also relate to computer products having non-transitory, tangible computer-readable media having computer code thereon for performing various computer-implemented operations. The media and computer code may be specially designed and constructed for the purposes of the present disclosure, or they may be of a type known or available to those skilled in the relevant art. Examples of tangible computer-readable media include, for example: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CDs and holographic devices; magneto-optical media; and hardware devices specifically configured to store or execute program code, such as ASICs, PLDs, flash memory devices, other non-volatile storage devices (such as 3D XPoint-based devices), and ROM and RAM devices. Examples of computer code include machine code such as produced by a compiler, and files containing higher-level code executed by a computer using an interpreter. Embodiments of the present disclosure may be implemented in whole or in part as machine-executable instructions in program modules that can be executed by a processing device. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In a distributed computing environment, program modules may be physically located locally, remotely, or both.

[0188] Those skilled in the art will recognize that no computing system or programming language is critical for the practice of the present disclosure. Those skilled in the art will also recognize that the above-mentioned multiple elements can be physically and / or functionally separated into modules and / or submodules or combined together.

[0189] Those skilled in the art will appreciate that the foregoing embodiments and implementations are illustrative and not limiting of the scope of the present disclosure. All permutations, enhancements, equivalents, combinations, and improvements thereof that would be apparent to one skilled in the art upon reading this specification and studying the accompanying drawings are intended to be within the spirit and scope of the present disclosure. It should also be noted that the elements of any claim may be arranged in various ways, including in multiple dependencies, configurations, and combinations.

Claims

1. A computer-implemented method for automatic mining learning, comprising: Tiling the excavation trajectory represented by the plurality of trajectory parameters in a voxel grid dimension to obtain a tiled trajectory parameter voxel grid; concatenating the tiled trajectory parameter voxel grid with a voxel grid representation of an excavation scene to generate an input voxel grid defining an excavation instance of a pair of the excavation trajectory and the voxel grid representation of the excavation scene; Feeding the input voxel grid into a neural network to generate a multidimensional feature vector; as well as Processing the multidimensional feature vector using at least one fully connected layer to predict a mining success probability; The voxel grid representation of the mining scene is generated by the following steps: Converting the point cloud of the excavation scene obtained from the RGB-depth camera to the object pallet framework; filtering the converted point cloud into a filtered point cloud according to a specified mining cube space; The filtered point cloud is voxelized to generate the voxel grid representation of the mining scene.

2. The computer-implemented method of claim 1 , wherein: The plurality of trajectory parameters include coordinates of a point of attack PoA, an attack digging angle, a penetration depth, a closing angle, and a drag length.

3. The computer-implemented method of claim 1 , wherein: The neural network is a 3D convolutional neural network (CNN) comprising a plurality of three-dimensional 3D convolutional filters.

4. The computer-implemented method of claim 1 , wherein: The excavation scene includes one or more rigid objects that are non-deformable during excavation.

5. The computer-implemented method of claim 1 , wherein: The at least one fully connected layer is followed by a sigmoid output layer to process the multi-dimensional feature vector.

6. The computer-implemented method of claim 5, wherein: Batch normalization is applied to the at least one fully connected layer.

7. The computer-implemented method of claim 1 , wherein: The at least one fully connected layer is followed by an additional fully connected layer for mining a predicted regression model.

8. The computer-implemented method of claim 1 , wherein: The mining trajectory is one of a plurality of tentative mining trajectories generated using a mining planner for initializing a Gaussian distribution of trajectory parameters.

9. The computer-implemented method of claim 8, further comprising: updating the Gaussian distribution through multiple iterations of mining trajectory sampling and mining trajectory selection according to the predicted mining success probability of each sampled mining trajectory; A subset of mining trajectories is sampled from the Gaussian distribution of the last iteration for evaluation; as well as A mining trajectory with the highest predicted mining success probability is selected from the subset of mining trajectories as the planned mining trajectory.

10. A system for automatic mining learning and planning, comprising: one or more processors; as well as A non-transitory computer-readable medium comprising one or more sets of instructions that, when executed by at least one of the one or more processors, cause the following steps to be performed, comprising: Tiling the excavation trajectory represented by the plurality of trajectory parameters in a voxel grid dimension to obtain a tiled trajectory parameter voxel grid; concatenating the tiled trajectory parameter voxel grid with a voxel grid representation of an excavation scene to generate an input voxel grid defining an excavation instance of a pair of the excavation trajectory and the voxel grid representation of the excavation scene; Feeding the input voxel grid into a neural network to generate a multi-dimensional feature vector; and Processing the multidimensional feature vector using one or more fully connected layers to predict the probability of successful mining; The voxel grid representation of the mining scene is generated by the following steps: Converting the point cloud of the excavation scene obtained from the RGB-depth camera to the object pallet framework; filtering the converted point cloud into a filtered point cloud according to a specified mining cube space; The filtered point cloud is voxelized to generate the voxel grid representation of the mining scene.

11. The system of claim 10, wherein: The plurality of trajectory parameters include coordinates of a point of attack (PoA), an attack digging angle, a penetration depth, a closing angle, and a drag length.

12. The system of claim 10, wherein: The neural network is a 3D convolutional neural network (CNN) comprising a plurality of three-dimensional 3D convolutional filters.

13. A computer-implemented method for learning-based mining planning, comprising: Given a voxel grid representation of a mining scene, a mining planner is used to generate multiple mining trajectories to initialize a Gaussian distribution of trajectory parameters, where each mining trajectory is represented by multiple trajectory parameters; updating the Gaussian distribution through multiple iterations of mining trajectory sampling and sampled mining trajectory selection according to a predicted mining success probability of each sampled mining trajectory predicted by a mining prediction model; sampling a subset of mining trajectories from the Gaussian distribution of the last iteration; as well as selecting a mining trajectory with the highest predicted mining success probability from the subset of mining trajectories as the planned mining trajectory; The voxel grid representation of the mining scene is generated by the following steps: Converting the point cloud of the excavation scene obtained from the RGB-depth camera to the object pallet framework; filtering the converted point cloud into a filtered point cloud according to a specified mining cube space; The filtered point cloud is voxelized to generate a voxel grid representation of the mining scene.

14. The computer-implemented method of claim 13, wherein: The plurality of trajectory parameters include coordinates of a point of attack (PoA), an attack digging angle, a penetration depth, a closing angle, and a drag length.

15. The computer-implemented method of claim 14, wherein: The mining prediction model predicts the predicted mining success probability of each sampled mining trajectory using the following steps: Tiling each sampled excavation trajectory in the voxel grid dimension to obtain a tiled trajectory parameter voxel grid; concatenating the tiled trajectory parameter voxel grid with the voxel grid representation of the excavation scene to generate an input voxel grid defining an excavation instance of a pair of the sampled excavation trajectory and the voxel grid representation of the excavation scene; Feeding the input voxel grid into a neural network to generate a multidimensional feature vector; as well as The multidimensional feature vector is processed using one or more fully connected layers to output the predicted mining success probability.

16. The computer-implemented method of claim 15, wherein: The neural network is a 3D convolutional neural network (CNN) comprising a plurality of three-dimensional 3D convolutional filters.

17. The computer-implemented method of claim 15, wherein: The one or more fully connected layers are followed by a sigmoid output layer to process the multi-dimensional feature vector.

18. The computer-implemented method of claim 14, wherein: The plurality of excavation trajectories are tentative excavation trajectories.

Images