A laser radar rockfall detection method for an open-pit mine road

Abstract

The application discloses an open-pit mine road multi-line laser radar rockfall detection method based on an improved PointPillars network, aiming at common rockfalls above 30CM in open-pit mine transportation operation, using a 128-line bundle solid-state laser radar to obtain open-pit mine road rockfall point cloud data as a training set, performing data enhancement and body columnization processing on the point cloud data, putting the point cloud data into the improved PointPillars network for training, performing TensorRT acceleration on the trained model, and integrating the model into a ROS project; wherein the open-pit mine road rockfall detection result comprises a rockfall type and a rockfall candidate box; a corresponding system comprises a point cloud data acquisition module, a point cloud data processing module, a model training module, a model acceleration module and a result output module. The application can effectively solve the problems of different sizes of open-pit mine rockfalls and strong light of the open-pit mine, and realize rockfall detection with real-time performance and accuracy.

Description

Technical Field

[0001] This invention belongs to the field of mine safety driving technology, and specifically relates to a multi-line lidar method for detecting falling rocks on open-pit mine roads based on an improved PointPillars network. Background Technology

[0002] Ensuring safety and reducing transportation costs are primary goals in open-pit mining production. As an enclosed environment, open-pit mines have already implemented autonomous driving technology. During transportation, open-pit mine roads often rely on slopes and are narrow, making them prone to overturning of unmanned mining trucks due to large rockfalls, leading to serious accidents. Furthermore, poor road conditions cause tire wear, increasing transportation costs. Additionally, route planning requires assessing the drivability of specific routes in challenging conditions. Therefore, the detection of rockfalls on open-pit mine roads is of paramount importance.

[0003] Currently, most object detection methods are used in urban scenes with structured roads, and are primarily effective for large, regularly shaped, and highly recognizable objects. Therefore, mainstream algorithms are not suitable for the unique scenario of open-pit mines, where fallen rocks are characterized by variable size, irregular shape, and low recognizability. Thus, rockfall detection in open-pit mines represents a new direction.

[0004] As an emerging technology, lidar has been the subject of much debate in the field of autonomous driving in recent years. LiDAR boasts advantages such as short measurement time, large measurement range, high accuracy, and immunity to lighting conditions, remaining unaffected even in strong light environments, while also containing rich 3D information. Compared to image processing, lidar is more suitable for the complex road conditions in open-pit mines.

[0005] It can be observed that many solutions for rockfall detection tend to be based on images, millimeter-wave radar, or traditional clustering-based lidar algorithms. However, in open-pit mines, images are easily affected by strong sunlight and the nighttime operation of unmanned mining trucks. While millimeter-wave radar is less affected by the complex environment of open-pit mines, its short range limits the speed of unmanned mining trucks and the production efficiency of the mine. Traditional clustering methods, although offering good real-time performance, are less than satisfactory in terms of accuracy, making it difficult to accurately guarantee safety in the hazardous workplace of open-pit mines. Furthermore, the rugged terrain of open-pit mines, the varying sizes and irregular shapes of rocks necessitate detectors with long detection ranges for both efficiency and safety. Summary of the Invention

[0006] In order to overcome the shortcomings of the prior art, the present invention aims to provide a multi-line lidar method for detecting falling rocks on open-pit mine roads based on an improved PointPillars network, which aims to solve the common problem of detecting falling rocks larger than 30 cm during unmanned mining truck transportation operations on open-pit mine roads.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A multi-line lidar method for detecting falling rocks on open-pit mine roads based on an improved PointPillars network includes the following steps:

[0009] Step 1: Collect point cloud data of open-pit mine roads under different conditions as a training set, and analyze the characteristics of rockfalls on open-pit mine roads, specifically including:

[0010] 1) Use multi-line lidar to collect data on fallen rocks on dirt roads and rock roads, and use this data as a training set;

[0011] 2) Compare and analyze the point cloud data of rockfalls on different roads to summarize the characteristics of rockfalls on open-pit mine roads;

[0012] Step Two: Train and test the original PointPillars network using the training set. Based on the test results, summarize the shortcomings of the original algorithm in the open-pit mine road environment, improve the algorithm, and train the improved PointPillars network using the training set. Specifically, this includes:

[0013] 1) Data augmentation of point cloud data is performed using a random sampling method, and the number of points in all volumetric pillars is sampled to N points;

[0014] 2) Increase the dimension of all points from 9 to C, perform max pooling on the volume cylinder, and then use the scatter operator to generate a pseudo-image;

[0015] 3) Use FPN (Feature Pyramid Network) as the backbone network to encode and decode features;

[0016] 4) Improve the original detection head with anchor frame to a detection head without anchor frame;

[0017] 5) Train the improved PointPillars network to obtain a trained rockfall detection network model;

[0018] Step 3: Accelerate the trained rockfall detection network model using TensorRT and integrate it into the ROS (Robot Operating System) project. This includes:

[0019] 1) Convert the trained PyTorch rockfall detection model into an ONNX model using ONNX;

[0020] 2) Convert the ONNX model to a TensorRT model using TensorRT;

[0021] 3) Configure the TensorRT model in the ROS project;

[0022] This invention uses point cloud data collected from real open-pit mine roads and an improved PointPillars network that has been accelerated and integrated into the ROS project for testing, and finally outputs the rockfall detection results of open-pit mine roads.

[0023] Compared with the prior art, the beneficial effects of the present invention are:

[0024] This invention can effectively address the problems of varying rock sizes and strong sunlight in open-pit mines, and achieve rockfall detection that is both real-time and accurate.

[0025] To address the issue of varying rock sizes in open-pit mines, the original PointPillars network uses a detection head with fixed anchor frames. Experimental results show that candidate boxes cannot completely select larger rocks, only a portion of them. For smaller rocks, while candidate boxes can select them, they can sometimes be too large. This is because the size of the candidate box is fixed during detection, but the size of the rock is not. To address these issues, this invention proposes a detection head without fixed anchor frames. It generates candidate boxes based on the center point and adjusts their position and size using other attributes to accommodate the varying sizes of rockfalls in open-pit mines. Furthermore, it reduces the non-maximum suppression step in the fixed anchor frame method, effectively decreasing the network's runtime.

[0026] To address the issues of real-time performance and accuracy, most existing rockfall detection algorithms are based on clustering methods from traditional machine learning. However, in 3D data, these methods suffer from high time complexity, making them unsuitable for scenarios with high real-time requirements. Furthermore, while these methods perform well in simple environments, their accuracy drops rapidly in complex environments, often proving inadequate for the complex environments of open-pit mines where safety is paramount. To resolve this, this invention employs a deep learning algorithm that balances speed and accuracy. This algorithm performs volumetric processing on the original point cloud data, reducing the number of points to be processed. It also uses 2D convolution on pseudo-images, significantly reducing runtime compared to 3D convolution and traditional machine learning clustering methods. Moreover, this algorithm demonstrates good performance on the publicly available KITTI complex environment dataset.

[0027] To address the issue of strong sunlight in open-pit mines, where the environment is unobstructed and open, cameras are easily affected by light, thus impacting detection accuracy. This invention utilizes lidar, where strong sunlight has no effect on lidar detection. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the process of this invention.

[0029] Figure 2 This is an architecture diagram of the improved PointPillars network of this invention. Detailed Implementation

[0030] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0031] like Figure 1 As shown, this invention is a multi-line lidar method for detecting falling rocks on open-pit mine roads based on an improved PointPillars network, comprising the following steps:

[0032] Step 1: Using a 128-beam solid-state lidar, collect a large amount of point cloud data on rockfalls along different roads and under different road conditions in open-pit mines; perform data augmentation and columnarization on the collected point cloud data to obtain columnar point cloud maps, see... Figure 2 The Pillar Feature Net part, the specific steps are as follows:

[0033] 1) A 128-beam solid-state LiDAR was mounted on the front of the experimental vehicle and fixed with a bracket to keep the LiDAR tilted downwards at 20°. The industrial control computer (PC host) was installed in the vehicle. Data sets were recorded and acquired on the open-pit mine roads through the ROS platform. A large number of scenes with different road sections and conditions were recorded, and rocks were randomly placed to increase the sample size. The recorded dataset was used as the training set, and a portion of the training set was selected as the test set. The solid-state LiDAR used is not easily affected by strong light, contains richer 3D features, and is more suitable for complex mining environments. Moreover, the solid-state LiDAR only scans at a fixed angle, reducing a large amount of useless point cloud data.

[0034] 2) Increase the number of falling rocks in a frame of point cloud data by random sampling. Specifically, iteratively extract all labeled falling rock data from each frame of point cloud data in the collected dataset and store them in a one-dimensional array. Then, randomly select several falling rock data from the one-dimensional array and repeatedly insert them into each frame of point cloud data. If the falling rock data to be inserted already exists, skip it and continue to the next frame, thereby increasing the number of positive samples (falling rocks). Increasing the number of positive samples is more beneficial for model training.

[0035] 3) Perform volumetric columnarization on the augmented point cloud data. Specifically, for the 3D point cloud data, from the XY top view plane, divide the 3D point cloud data into small 3D cubes (H=440, W=500, D=9) with a step size [0.16, 0.16], resulting in P=16000 volumetric columns. See [link to documentation]. Figure 2 The Point Cloud in the lower left corner represents a single volumetric column. Each point in the P volumetric columns is then encoded into a 9-dimensional vector D (i.e., 9-dimensional features). That is, the original features of each point in the P volumetric columns are the x-coordinate, y-coordinate, z-coordinate, and reflection intensity r. Five additional features are added: xc, yc, zc (the geometric center of the volumetric column), and xp and yp (the distance between the point and the geometric center). See [link to documentation]. Figure 2 The stacked pillars in the lower left corner; then, for each pillar with more than N points, sample up to N points, and fill with 0 for those with less than N points, so that the number of points in each pillar is N; the stacking of point cloud data into pillars does not require manually setting intervals in the vertical direction, thus making it more flexible.

[0036] Step Two: Convert the volume column index value (x, y coordinates corresponding to each volume column) of each point in the columnar point cloud image back to its original columnar point cloud image (in the large cube) from Step One, thereby generating a pseudo-image. This pseudo-image is then fed into a backbone network for feature extraction. The backbone network includes an encoder and a decoder. The encoder captures feature information at different scales from top to bottom to extract features generated by increasingly smaller spatial resolutions, primarily consisting of convolutional layers, normalization layers, and nonlinear layers. The decoder performs upsampling and concatenation from top to bottom to fuse feature information at different scales, mainly achieved through deconvolution, such as... Figure 2 The specific steps for Pillar Feature Net and Backbone components are as follows:

[0037] 1) For all points with 9-dimensional features obtained in the previous step, use convolutional and normalization layers to extract features, increasing the feature dimension of all points from D=9 to C=64; then perform max pooling on P individual pillars to obtain a (C, P)-dimensional feature map (C is the feature dimension after convolution), see... Figure 2 LearnedFeatures in the bottom left corner;

[0038] 2) Generate a pseudo-image using the scatter operator. The scatter operator uses the index values ​​of each volume pillar (the x, y coordinates of each volume pillar in the top view) to transform the (C, P) dimensional feature map generated in the previous step back into the 3D small cube from step one. That is, it places P volume pillars back into the original cube according to the previously stored x, y coordinates to create a pseudo-image of size (C = 64, H = 440, W = 500). See [link to documentation].Figure 2 The Pseudo image in the lower left corner; projected onto the pseudo image, it can utilize 2D convolution to process columnar features, thus greatly increasing the running efficiency on the GPU and better meeting the requirements of real-time performance;

[0039] 3) Backbone Network: The backbone network uses FPN (Feature Pyramid Network), see... Figure 2 In the backbone part of the encoder, the encoder is represented by a series of blocks (S, L, F), each block running with a stride S; each block has L 3×3 2D convolutional layers and F output channels, followed by BatchNorm and ReLU layers to extract features, see... Figure 2 In the first column of the bottom right corner, there are three blocks of different sizes. When the stride is set to 2 and the padding is set to 1, the volume of the block becomes 1 / 4 of its original size. The input pseudo-image size is set to 440*500*64. The convolutional networks corresponding to the three blocks are shown in the table below:

[0040] Layer Input Struct Output 1 440*500*64 3*3 conv, stride = 2, padding = 1 220*250*64 2 220*250*64 3*3 conv, stride = 2, padding = 1 110*125*128 3 110*125*128 3*3 conv, stride = 2, padding = 1 55*62*256

[0041] The decoder upsamples (i.e., deconvolves) the features output by each block using a transposed 2D convolution with F final features. Then, BatchNorm and ReLU are applied to the upsampled features. The final output features are concatenated feature maps derived from different amplitudes, obtained by adding the feature dimensions of the three outputs (C' = 384, H' = 220, W' = 250). See [link to documentation]. Figure 2 The second column in the bottom right corner contains three blocks of the same size. The specific convolutional components are shown in the table below:

[0042] Layer Input Struct Output 1 220*250*64 1*1 conv, stride = 1, padding = 0 220*250*128 2 110*125*128 1*1 conv, stride = 1, padding = 0 220*250*128 3 55*62*256 1*1 conv, stride = 1, padding = 0 220*250*128

[0043] Step 3: Input the features extracted in Step 2 into the anchorless frame detection head to generate the position and category of the target object; the anchorless frame detection head is the main improvement of the original algorithm, and consists of four parts: a key point heatmap head, a local offset head, a z-axis positioning head, and a 3D target size head, see... Figure 2 The above describes the entire training process for the rockfall detection network. After training, the network model and weight file for rockfall detection will be obtained. The specific convolutional part and the anchorless detection head are structured as follows:

[0044] Layer Input Struct Output 1 220*250*384 3*3 conv, stride = 1, padding = 1 220*250*64 2 220*250*64 3*3 conv, stride = 1, padding = 1 220*250*2 3 220*250*64 3*3 conv, stride = 1, padding = 1 220*250*1 4 220*250*64 3*3 conv, stride = 220*250*3

[0045] 1) Keypoint Heatmap Header: For each pixel (x, y) covered by the 2D bounding box of the pseudo-image, its value in the heatmap is set to:

[0046]

[0047] Where d represents the Euclidean distance between the bounding box center and the corresponding pixel in discrete pseudo-image coordinates; M x,y,c =1 indicates the center of the target object, M x,y,c =0 indicates background; the keypoint heatmap header predicts the label score, using a 3×3 convolution kernel and a 2D convolution with a stride of 1. The convolutional layer is shown in the row where Layer=3 in the table above, and the loss function used is Focal Loss.

[0048] 2) Local offset head: Within a square region of radius r of the center pixel of the target object, the farther away from the center of the object, the larger the offset value. It can not only help the heatmap (the value of each pixel in the heatmap in the pseudo image) find a more accurate center of the target object in the pseudo image, but also make up for the discretization error caused by the volume columnarization process. The offset regression head uses a 3×3 convolution kernel and a 2D convolution with a stride of 1. The convolutional layer is shown in the row where Layer=3 in the table above. The loss function uses L1 loss to train the offset.

[0049] 3) Z-axis positioning head: Since the pseudo image is two-dimensional and lacks z-axis information, a 3×3 convolution kernel and a 2D convolution with a stride of 1 are used to regress the height of the z-axis. The convolutional layer is shown in the row where Layer=2 in the table above, and the loss function used is L1 loss.

[0050] 4) 3D target size header: Similarly, a 3×3 convolution kernel and a 2D convolution with a stride of 1 are used to regress the length, width and height of the candidate box. The convolutional layer is shown in the row where Layer=4 in the table above, and the loss function is L1 loss.

[0051] Step 4: Convert the rockfall detection PyTorch model trained in the previous step into the ONNX model middleware; optimize it using ONNX_TensorRT; the optimized rockfall detection network model runs faster, is easier to integrate into embedded systems, and can be integrated into ROS projects to deploy the rockfall detection network model. The specific steps are as follows:

[0052] 1) Once trained, the rockfall detection network model must be accelerated using TensorRT if it is to be ported to an embedded system. This is because deep learning networks typically use 32-bit or 16-bit data during training. TensorRT uses lower precision during network inference to accelerate the inference process. It also restructures the network structure, merging some operations that can be combined, and optimizes for GPU characteristics. After optimization, it is easier to integrate into embedded systems.

[0053] 2) Integrate into the ROS project, use the optimized rockfall detection model from the previous step as a ROS node, make basic configuration file modifications to enable the rockfall detection model node to publish topic messages, and then apply it to autonomous vehicles for testing, thus realizing the deployment of the rockfall detection network model.

[0054] 3) The latest mine point cloud data was selected as the test set, and the deployed rockfall detection network model was used for testing. The identified rockfall information was played in continuous frames to realize the graphical output of the rockfall detection results.

[0055] This invention uses point cloud data of real open-pit mine roads as a training set, and uses an improved PointPillars network that has been accelerated and integrated into the ROS project for testing, and finally outputs the rockfall detection results of open-pit mine roads.

Claims

1. A multi-line lidar method for detecting falling rocks on open-pit mine roads based on an improved PointPillars network, characterized in that, Includes the following steps: Step 1: Use a multi-beam solid-state lidar to collect point cloud data of rockfalls on open-pit mine roads with different road conditions as a training set, and take a portion of the training set as a test set; perform data augmentation and columnarization on the collected point cloud data to obtain columnar point cloud maps. Step 2: After processing the columnar point cloud image, a pseudo image is generated using the scatter operator. The pseudo image is then fed into the backbone network of the improved PointPillars network for feature extraction. The improved PointPillars network is based on the PointPillars network, but its anchor-bound detection head is modified to an anchor-free detection head. The anchor-free detection head consists of four parts: a key point heatmap head, a local offset head, a z-axis positioning head, and a 3D target size head. In this process, feature extraction is performed on all points with 9-dimensional features, and the processed dimension is C. Then, max pooling is performed on P individual cylinders to obtain a feature map with (C, P) dimensions, where C is the feature dimension after convolution. Then, a pseudo-image is generated by the scatter operator, which puts the P cylinders with C-dimensional features back into the large cube according to their original x and y coordinates. The key point heatmap head is used to find the position of the target object's center in the pseudo-image; the local offset head is used to eliminate errors caused by the volumetric scaling process and improve the prediction of the target object's center in the heatmap; the z-axis positioning head is used to determine the target object's xy coordinate position in the pseudo-image and then regress the z-axis coordinate value; the 3D target size head regresses the size of the candidate bounding box of the target object. Step 3: Train the improved PointPillars network to obtain a trained rockfall detection network model. Accelerate the trained rockfall detection network model using TensorRT and integrate it into the robot operating system ROS to deploy the rockfall detection network model. Step 4: Use the latest mine point cloud data as the test set, test it through the ROS platform, output the detection results of continuous frames, and obtain the information on falling rocks on the road.

2. The multi-line lidar method for detecting falling rocks on open-pit mine roads based on an improved PointPillars network according to claim 1, characterized in that, In step one, the training dataset is obtained, and the point cloud data is collected by the lidar tilted downwards at 20°.

3. The multi-line lidar method for detecting falling rocks on open-pit mine roads based on an improved PointPillars network according to claim 1, characterized in that, The data augmentation is achieved by increasing the number of falling rocks in a frame of point cloud data through random sampling.

4. The multi-line lidar method for detecting falling rocks on open-pit mine roads based on an improved PointPillars network according to claim 1, characterized in that, The volumetric processing involves placing the 3D point cloud data within a large cube, dividing the large cube into P smaller cubes (H×W×D) of equal volume (i.e., volumetric columns) of a set length. Then, all points in each of the P volumetric columns are encoded into a vector D with 9-dimensional features: x-coordinate, y-coordinate, z-coordinate, and reflection intensity r. xc, yc, and zc are added as the geometric center of the volumetric column, along with the distances xp and yp between the point and the geometric center. For each volumetric column, if the number of points exceeds N, the number is increased to N; otherwise, points are filled with 0. Therefore, the number of points in each volumetric column is N.

5. The multi-line lidar method for detecting falling rocks on open-pit mine roads based on an improved PointPillars network according to claim 1, characterized in that, The backbone network consists of a top-down network that generates features at increasingly smaller spatial resolutions and performs upsampling and automatic concatenation functions. The backbone network includes an encoder and a decoder. The encoder captures feature information at different scales from top to bottom to extract features at increasingly smaller spatial resolutions, and is mainly composed of convolutional layers, normalization layers, and nonlinear layers. The decoder performs upsampling and concatenation functions from top to bottom to fuse feature information at different scales, and is mainly implemented by deconvolution.

6. The multi-line lidar method for detecting falling rocks on open-pit mine roads based on an improved PointPillars network according to claim 1, characterized in that, In step three, the trained rockfall detection network model is converted from a PyTorc model to an ONNX model middleware; it is then optimized using ONNX_TensorRT, and the optimized rockfall detection network model is used as a node in ROS. Basic configuration file modifications are made to enable the rockfall detection model node to publish topic messages, thus achieving the deployment of the rockfall detection network model.

7. The multi-line lidar method for detecting falling rocks on open-pit mine roads based on an improved PointPillars network according to claim 1, characterized in that, In step four, the latest mine point cloud data is selected as the test set, and the deployed rockfall detection network model is used for testing. The identified rockfall information is played in continuous frames to achieve graphical output of the rockfall detection results.

Images