Vehicle-mounted material pile volume calculation method based on multi-modal information fusion and semantic segmentation

Abstract

The present application belongs to the field of environmental perception and intelligent industrial detection technology, a kind of vehicle load volume calculation method based on multi-modal information fusion and semantic segmentation, comprising the following steps: (1) mining truck operating environment data set acquisition and establishment, (2) semantic segmentation is carried out by deep convolutional neural network, (3) the discrete load surface point cloud is reconstructed into a plane by Poisson reconstruction, (4) the volume of vehicle load is calculated.The present application has the following advantages: first, to solve the problem that traditional point cloud segmentation algorithm is easy to ignore the rich visual information obtained by camera except laser radar sensor, color information is embedded into data layer to improve the performance of real-time point cloud segmentation;Second, the present application is based on three-dimensional surface reconstruction technology, and combined with the geometric feature recognition of three-dimensional point cloud data, the efficient and high-precision calculation of vehicle load volume is realized.

Description

Technical Field

[0001] This invention relates to a method for calculating the volume of a vehicle-mounted material stack based on multimodal information fusion and semantic segmentation, belonging to the field of environmental perception and intelligent industrial detection technology. Background Technology

[0002] Mining electric shovels (such as) Figure 1 (As shown) is a key mining equipment used in open-pit mining for ore stripping and loading operations, typically paired with a mining dump truck (such as... Figure 2 (As shown) These are used in conjunction with other equipment. The large and complex structure of mining electric shovels means that traditional manual operation requires a significant amount of effort, easily leading to reduced digging efficiency. Therefore, intelligentization has become one of the important directions for the development of large mining electric shovels. One pressing issue in the intelligentization of large mining electric shovels is the compatibility of them with mining dump trucks. Currently, in mining applications, a shovel driver is still required to manually determine whether the truck is fully loaded. Therefore, to achieve intelligent unmanned electric shovels, the first priority is a method that can automatically identify and determine the volume of the material pile on the truck, thus laying the groundwork for intelligent planning of subsequent digging operations. Summary of the Invention

[0003] To address the aforementioned issues, this invention aims to provide a method for calculating the volume of a vehicle-mounted material stack using multimodal information fusion data. Addressing the problem that traditional point cloud segmentation algorithms tend to overlook the rich visual information acquired by cameras in addition to LiDAR sensors, this invention embeds color information into the data layer while incorporating an attention mechanism to improve the performance of real-time point cloud segmentation. Furthermore, based on 3D surface reconstruction technology and combined with geometric feature recognition of 3D point cloud data, it achieves efficient and high-precision calculation of the vehicle-mounted material stack volume.

[0004] To achieve the aforementioned objectives, the technical solution adopted by this invention is: a method for calculating the volume of a vehicle-mounted material stack based on multimodal information fusion and semantic segmentation, comprising the following steps:

[0005] Step 1: Collection and establishment of mining truck operating environment dataset. A 3D environmental perception system is built using LiDAR and cameras to collect point cloud and image data of the electric shovel under normal operating conditions. The establishment of the mining truck operating environment dataset includes the following sub-steps:

[0006] (a) Data labeling: The point clouds in the 2100 point cloud data collected in the complex operating environment of mining trucks are divided into 9 categories, namely material pile, ground, vehicle, people, miscellaneous points, wall, stairs, material on the vehicle and electric shovel itself. Each point in the point cloud is labeled with the corresponding category by manual labeling.

[0007] (b) 3D color point cloud fusion: For a real scene, a 3D color laser scanning system simultaneously acquires one frame of 3D laser point cloud data and one corresponding frame of 2D image data of the scene, denoted as p = [x, y, z]. T For a laser point in a frame of point cloud data, its homogeneous coordinates are represented as: The laser point p is projected onto a point q = [u, v] in the pixel coordinate system. T It is described by formula (1).

[0008]

[0009] In the formula, This represents the geometric mapping relationship between the 3D laser coordinate system and the pixel coordinate system, also known as the homography matrix, where A is the camera intrinsic parameter matrix, obtained from camera calibration. and Let p represent the row vectors corresponding to the rows of the homography matrix H, then the coordinate form of the three-dimensional colored laser point is p. c It is described by formula (2).

[0010] p c =[x,y,z,R(u,v),G(u,v),B(u,v)] T (2)

[0011] In the formula, R(u,v), G(u,v) and B(u,v) represent the three-dimensional color information of the projected pixel points, respectively. It is worth noting that as the distance of the laser points increases, the accuracy of data fusion will decrease due to the combined effects of calibration error and measurement errors of the three-dimensional laser and camera. For each laser point in the three-dimensional laser point cloud of the real scene, the homography matrix is ​​used to project one by one to find the corresponding projected pixel point, and the laser points are colored to obtain the three-dimensional color point cloud.

[0012] Step 2: Semantic segmentation using a deep convolutional neural network. This step uses the traditional PointNet semantic segmentation network as a framework, combined with an attention mechanism, to perform semantic segmentation based on the position and color information of the point cloud. This enables the electric shovel to perceive and recognize mining trucks in complex environments and extract point cloud data from the surface of the unloading mining truck and its onboard material pile. Specifically, this includes the following sub-steps:

[0013] (a) Feature enhancement: Color features were added as auxiliary features. Three-dimensional tri-color point cloud data (1×6) containing color information was used at the input to replace the point cloud data (1×3) containing only location information in the original Point-Net network.

[0014] (b) The encoding layer contains 5 convolutional layers. Conv1 has a kernel size of 1×6, a stride of [1, 1], no edge padding, and 64 output channels. Conv2 and Conv3 both have a kernel size of 1×1, a stride of [1, 1], no edge padding, and 64 output channels. Conv4 has a kernel size of 1×1, a stride of [1, 1], no edge padding, and 128 output channels. Conv5 has a kernel size of 1×1, a stride of [1, 1], no edge padding, and 1024 output channels. All convolutional layers are activated using the Leaky ReLU activation function. The pooling layer Maxpool1 has the following parameters: (4096)×1 input points, max pooling, a stride of [2, 2], and no padding. After passing through two fully connected layers, the output is a 128-dimensional feature vector.

[0015] (c) The decoding end first goes through two convolutional layers, Conv6 and Conv7. The parameters of Conv6 are 1×1 kernel size, stride [1, 1], no edge padding, and 512 output channels; the parameters of Conv7 are 1×1 kernel size, stride [1, 1], no edge padding, and 256 output channels. Both are activated using the Leaky_ReLU activation function. Then, to prevent overfitting, a Dropout operation is performed with a ratio of 0.7. Then, a convolutional layer, Conv8, is passed through with 1×1 kernel size, stride [1, 1], no edge padding, and 9 output channels. Finally, the dimension with dimension 1 is removed, and the network parameters are returned.

[0016] (d) An attention mechanism is inserted. An attention module (ECA-Net) is added after each convolutional layer at the encoding end of the network to improve the performance of point cloud segmentation in real time. This attention module first transforms the feature map from a matrix of [h, w, c] into a vector of [1, 1, c] through global average pooling, where h and w are the size of the feature map and c is the number of channels; then, it adaptively determines the size of the convolutional kernel. Then, a one-dimensional convolution is performed to obtain the weights for each channel of the feature map. Finally, after activation by the Sigmoid function, the normalized weights are multiplied with the original input feature map channel by channel to generate a weighted feature map.

[0017] Step 3: Reconstruct the discrete point cloud of the material pile surface into a plane using Poisson reconstruction. Poisson reconstruction is a reconstruction scheme that combines the advantages of global and local matching. The core idea is to construct an implicit surface with watertightness by transforming the discrete sample point information of the object surface into a continuously integrable surface function. This method obtains the implicit surface represented in the form of a triangular mesh by inputting a directed point cloud, and specifically includes the following sub-steps:

[0018] (a) Calculate the point cloud normal vector. The point cloud data collected by the lidar is directionless, so the point cloud normal vector needs to be calculated manually. The method is to fit a local plane with six adjacent points for each discrete point cloud and find the normal vector of the plane. The directed point cloud is obtained by approximating the point cloud normal vector by finding the plane normal vector.

[0019] (b) Reconstructing the surface of the vehicle-mounted material stack using Poisson reconstruction: First, construct an indicator function that is set to 1 within the object and 0 elsewhere. Now, let M be the object region, then the indicator function χ... M It is described by formula (3).

[0020]

[0021] This transforms the problem of reconstructing the object's surface into reconstructing x. M The problem, at the same time χ M The gradient of the point cloud is equal to the normal vector of the point cloud surface, so the problem is transformed into solving for its own gradient. sum vector field The indicator function closest to the point cloud normal vector Applying the Euler-Lagrange formula, the problem is transformed into a standard Poisson problem, described by formula (4).

[0022]

[0023] The indicator function χ is obtained by solving the above equation. M Finally, the indicator function χ is extracted using the MarchingCubes method for extracting isosurfaces. M The part that equals 0 will yield a triangular mesh representing the surface of the object;

[0024] Step 4: Calculate the volume of the truck-mounted material pile. Using the point cloud of the mining truck extracted in Step 2 and the known dimensions of the mining truck, the position of the bottom plane of the truck bed can be accurately determined. Now, taking each triangular mesh as a group, project its three vertices onto the plane where the truck bucket is located to form multiple hexahedrons. Now, divide each hexahedron into two tetrahedrons along the diagonal, and calculate the volume of each tetrahedron as V. i1 V i2 Taking a single tetrahedron as an example: Given that the coordinates of the vertices of the tetrahedron are (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), and (x4, y4, z4), and the three vectors passing through the point (x1, y1, z1) are a = (x2-x1, y2-y1, z2-z1), b = (x3-x1, y3-y1, z3-z1), and c = (x4-x1, y4-y1, z4-z1), then its volume can be obtained by the mixed product of the vectors, and described by formula (5).

[0025] V=|(a×b)·c| / 6 (5)

[0026] The volume V of the resulting polyhedron can be calculated using the above formula. i =V i1 +V i2 Finally, by summing the volumes of these hexahedrons... The volume of the on-board material stack can then be calculated.

[0027] The beneficial effects of this invention are: a method for calculating the volume of a vehicle-mounted material pile based on multimodal information fusion and semantic segmentation, comprising the following steps: (1) collecting and establishing a dataset of mining truck operating environment; (2) performing semantic segmentation through a deep convolutional neural network; (3) reconstructing the discrete point cloud of the material pile surface into a plane through Poisson reconstruction; and (4) calculating the volume of the vehicle-mounted material pile. Compared with existing technologies, this invention has the following advantages: First, addressing the problem that traditional point cloud segmentation algorithms tend to ignore the rich visual information acquired by cameras other than lidar sensors, color information is embedded in the data layer to improve the performance of real-time point cloud segmentation; Second, this invention is based on three-dimensional surface reconstruction technology and combines geometric feature recognition of three-dimensional point cloud data to achieve efficient and high-precision calculation of the volume of the vehicle-mounted material pile. Attached Figure Description

[0028] Figure 1 This is an example image of a large electric shovel used in mining.

[0029] Figure 2 This is an example image of a mining dump truck that is paired with an electric shovel.

[0030] Figure 3 This is a flowchart of the method steps of the present invention.

[0031] Figure 4 It is a diagram of the on-site equipment installation.

[0032] Figure 5 This invention relates to on-site point cloud data maps and on-site image data maps.

[0033] In the figure: (a) is the on-site point cloud data map, and (b) is the on-site image data map.

[0034] Figure 6 This is a three-dimensional color point cloud fusion effect diagram of the present invention.

[0035] Figure 7 This is a comparison chart of the accuracy of point cloud segmentation networks before and after the introduction of the attention mechanism in this invention.

[0036] Figure 8 This is the target point cloud image extracted by point cloud segmentation according to the present invention.

[0037] Figure 9 This is a surface view of the vehicle-mounted material stack after Poisson reconstruction according to the present invention. Detailed Implementation

[0038] To provide a more detailed description of the technical solution of this invention, the following description, in conjunction with the accompanying drawings and simulation experiments conducted at the State Key Laboratory of Mining Equipment and Intelligent Manufacturing of Taiyuan Heavy Industry Co., Ltd. in Taiyuan City, Shanxi Province, further illustrates the invention (as shown in the accompanying drawings of the laboratory site). Figure 4 (As shown).

[0039] like Figure 3 As shown, a method for calculating the volume of a vehicle-mounted material stack based on multimodal information fusion and semantic segmentation includes the following steps:

[0040] Step 1: Collection and establishment of mining truck operating environment dataset. A 3D environmental perception system is built using LiDAR and cameras to collect point cloud and image data of the electric shovel under normal operating conditions. The establishment of the mining truck operating environment dataset includes the following sub-steps:

[0041] (a) Data labeling: The point clouds from 2100 data points collected under the complex operating environment of mining trucks were divided into nine categories: material piles, ground, vehicles, people, miscellaneous points, walls, stairs, materials on the vehicles, and the electric shovel itself. Each point in the point cloud was manually labeled with its corresponding category. Data collected in the laboratory, such as... Figure 5 As shown.

[0042] (b) 3D color point cloud fusion: For a real scene, a 3D color laser scanning system simultaneously acquires one frame of 3D laser point cloud data and one corresponding frame of 2D image data of the scene, denoted as p = [x, y, z]. T For a laser point in a frame of point cloud data, its homogeneous coordinates are represented as: The laser point p is projected onto a point q = [u, v] in the pixel coordinate system. T It is described by formula (1).

[0043]

[0044] In the formula, This represents the geometric mapping relationship between the 3D laser coordinate system and the pixel coordinate system, also known as the homography matrix, where A is the camera intrinsic parameter matrix, obtained from camera calibration. and Let p represent the row vectors corresponding to the rows of the homography matrix H, then the coordinate form of the three-dimensional colored laser point is p. c It is described by formula (2).

[0045] p c =[x,y,z,R(u,v),G(u,v),B(u,v)] T (2)

[0046] In the formula, R(u, v), G(u, v), and B(u, v) represent the 3D color information of the projected pixels, respectively. It is worth noting that as the distance between the laser points increases, the accuracy of data fusion decreases due to the combined effects of calibration errors and measurement errors of the 3D laser and camera. Each laser point in the 3D laser point cloud of the real scene is projected one by one using a homography matrix to find its corresponding projected pixel. The laser points are then colored to obtain a 3D color point cloud. The fusion effect is as follows: Figure 6 As shown.

[0047] Step 2: Semantic segmentation using a deep convolutional neural network. This step uses the traditional PointNet semantic segmentation network as a framework, combined with an attention mechanism, to perform semantic segmentation based on the position and color information of the point cloud. This enables the electric shovel to perceive and recognize mining trucks in complex environments and extract point cloud data from the surface of the unloading mining truck and its onboard material pile. Specifically, this includes the following sub-steps:

[0048] (a) Feature enhancement: Color features were added as auxiliary features. Three-dimensional tri-color point cloud data (1×6) containing color information was used at the input to replace the point cloud data (1×3) containing only location information in the original Point-Net network.

[0049] (b) The encoding layer contains 5 convolutional layers. Conv1 has a kernel size of 1×6, a stride of [1, 1], no edge padding, and 64 output channels. Conv2 and Conv3 both have a kernel size of 1×1, a stride of [1, 1], no edge padding, and 64 output channels. Conv4 has a kernel size of 1×1, a stride of [1, 1], no edge padding, and 128 output channels. Conv5 has a kernel size of 1×1, a stride of [1, 1], no edge padding, and 1024 output channels. All convolutional layers are activated using the Leaky ReLU activation function. The pooling layer Maxpool1 has the following parameters: (4096)×1 input points, max pooling, a stride of [2, 2], and no padding. After passing through two fully connected layers, the output is a 128-dimensional feature vector.

[0050] (c) The decoding end first goes through two convolutional layers, Conv6 and Conv7. The parameters of Conv6 are 1×1 kernel size, stride [1, 1], no edge padding, and 512 output channels; the parameters of Conv7 are 1×1 kernel size, stride [1, 1], no edge padding, and 256 output channels. Both are activated using the Leaky_ReLU activation function. Then, to prevent overfitting, a Dropout operation is performed with a ratio of 0.7. Then, a convolutional layer, Conv8, is passed through with 1×1 kernel size, stride [1, 1], no edge padding, and 9 output channels. Finally, the dimension with dimension 1 is removed, and the network parameters are returned.

[0051] (d) An attention mechanism is inserted. An attention module (ECA-Net) is added after each convolutional layer at the encoding end of the network to improve the performance of point cloud segmentation in real time. This attention module first transforms the feature map from a matrix of [h, w, c] into a vector of [1, 1, c] through global average pooling, where h and w are the size of the feature map and c is the number of channels; then, it adaptively determines the size of the convolutional kernel. Then, a one-dimensional convolution is performed to obtain the weights for each channel of the feature map. Finally, after activation by the Sigmoid function, the normalized weights are multiplied channel-by-channel by the original input feature map to generate a weighted feature map. Figure 7 As can be seen, the accuracy of point cloud segmentation is greatly improved after the introduction of the attention mechanism, with the average accuracy of point cloud segmentation increasing from 60% to 85%.

[0052] Step 3: Reconstruct the discrete point cloud of the material pile surface into a plane using Poisson reconstruction. Poisson reconstruction is a reconstruction scheme that combines the advantages of global and local matching. The core idea is to construct an implicit surface with watertightness by transforming the discrete sample point information of the object surface into a continuously integrable surface function. This method obtains the implicit surface represented in the form of a triangular mesh by inputting a directed point cloud, and specifically includes the following sub-steps:

[0053] (a) Calculate the point cloud normal vector. The point cloud data collected by the lidar is directionless, so the point cloud normal vector needs to be calculated manually. The method is to fit a local plane with six adjacent points for each discrete point cloud and find the normal vector of the plane. The directed point cloud is obtained by approximating the point cloud normal vector by finding the plane normal vector.

[0054] (b) Reconstructing the surface of the vehicle-mounted material stack using Poisson reconstruction: First, construct an indicator function that is set to 1 within the object and 0 elsewhere. Now, let M be the object region, then the indicator function χ... M It is described by formula (3).

[0055]

[0056] This transforms the problem of reconstructing the object's surface into reconstructing x. M The problem, at the same time χ M The gradient of the point cloud is equal to the normal vector of the point cloud surface, so the problem is transformed into solving for its own gradient. sum vector field The indicator function closest to the point cloud normal vector Applying the Euler-Lagrange formula, the problem is transformed into a standard Poisson problem, described by formula (4).

[0057]

[0058] The indicator function χ is obtained by solving the above equation. M Finally, the indicator function χ is extracted using the MarchingCubes method for extracting isosurfaces. M The part equal to 0 yields a triangular mesh representing the object's surface, and the Poisson reconstruction result is as follows: Figure 9 As shown.

[0059] Step 4: Calculate the volume of the truck-mounted material pile. Using the point cloud of the mining truck extracted in Step 2 and the known dimensions of the mining truck, the position of the bottom plane of the truck bed can be accurately determined. Now, taking each triangular mesh as a group, project its three vertices onto the plane where the truck bucket is located to form multiple hexahedrons. Now, divide each hexahedron into two tetrahedrons along the diagonal, and calculate the volume of each tetrahedron as V. i1 V i2 Taking a single tetrahedron as an example: Given that the coordinates of the vertices of the tetrahedron are (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), and (x4, y4, z4), and the three vectors passing through the point (x1, y1, z1) are a = (x2-x1, y2-y1, z2-z1), b = (x3-x1, y3-y1, z3-z1), and c = (x4-x1, y4-y1, z4-z1), then its volume can be obtained by the mixed product of the vectors, and described by formula (5).

[0060] V=|(a×b)·c| / 6 (5)

[0061] The volume V of the resulting polyhedron can be calculated using the above formula. i =V i1 +V i2 Finally, by summing the volumes of these hexahedrons... The volume of the vehicle-mounted material stack can then be calculated. This experiment was conducted using point clouds collected from three different scenarios at the State Key Laboratory of Mining Equipment and Intelligent Manufacturing of Taiyuan Heavy Industry Co., Ltd. The final results showed an error of less than 10%. The experimental calculation results are compared with the actual results, as shown in Table 1.

[0062] Table 1

[0063] Scene <![CDATA[Actual volume / m 3 > <![CDATA[Calculated volume / m 3 > Scene 1 0.6192 0.5760 Scene 2 0.6230 0.5892 Scene 3 0.6200 0.5695

Claims

1. A method for calculating the volume of a vehicle-mounted material stack based on multimodal information fusion and semantic segmentation, characterized in that... Includes the following steps: Step 1: Collection and establishment of mining truck operating environment dataset. A 3D environmental perception system is built using LiDAR and cameras to collect point cloud and image data of the electric shovel under normal operating conditions. The establishment of the mining truck operating environment dataset includes the following sub-steps: (a) Data labeling: The point clouds in the 2100 point cloud data collected in the complex operating environment of mining trucks are divided into 9 categories, namely material pile, ground, vehicle, people, miscellaneous points, wall, stairs, material on the vehicle and electric shovel itself. Each point in the point cloud is labeled with the corresponding category by manual labeling. (b) 3D color point cloud fusion: For a real scene, a 3D color laser scanning system simultaneously acquires one frame of 3D laser point cloud data and one corresponding frame of 2D image data of the scene, denoted as p = [x, y, z]. T For a laser point in a frame of point cloud data, its homogeneous coordinates are represented as: The laser point p is projected onto a point q = [u, v] in the pixel coordinate system. T It is described by formula (1). In the formula, This represents the geometric mapping relationship between the 3D laser coordinate system and the pixel coordinate system, also known as the homography matrix, where A is the camera intrinsic parameter matrix, obtained from camera calibration. and Let p represent the row vectors corresponding to the rows of the homography matrix H, then the coordinate form of the three-dimensional colored laser point is p. c It is described by formula (2). p c =[x,y,z,R(u,v),G(u,v),B(u,v)] T (2) In the formula, R(u,v), G(u,v) and B(u,v) represent the three-dimensional color information of the projected pixel points, respectively. It is worth noting that as the distance of the laser points increases, the accuracy of data fusion will decrease due to the combined effects of calibration error and measurement errors of the three-dimensional laser and camera. For each laser point in the three-dimensional laser point cloud of the real scene, the homography matrix is ​​used to project one by one to find the corresponding projected pixel point, and the laser points are colored to obtain the three-dimensional color point cloud. Step 2: Semantic segmentation using a deep convolutional neural network. This step uses the traditional PointNet semantic segmentation network as a framework, combined with an attention mechanism, to perform semantic segmentation based on the position and color information of the point cloud. This enables the electric shovel to perceive and recognize mining trucks in complex environments and extract point cloud data from the surface of the unloading mining truck and its onboard material pile. Specifically, this includes the following sub-steps: (a) Feature enhancement: Color features were added as auxiliary features. Three-dimensional tri-color point cloud data (1×6) containing color information was used at the input to replace the point cloud data (1×3) containing only location information in the original Point-Net network. (b) The encoding layer contains 5 convolutional layers. Conv1 has a kernel size of 1×6, a stride of [1, 1], no edge padding, and 64 output channels. Conv2 and Conv3 both have a kernel size of 1×1, a stride of [1, 1], no edge padding, and 64 output channels. Conv4 has a kernel size of 1×1, a stride of [1, 1], no edge padding, and 128 output channels. Conv5 has a kernel size of 1×1, a stride of [1, 1], no edge padding, and 1024 output channels. All convolutional layers are activated using the Leaky ReLU activation function. The pooling layer Maxpool1 has the following parameters: (4096)×1 input points, max pooling, a stride of [2, 2], and no padding. After passing through two fully connected layers, the output is a 128-dimensional feature vector. (c) The decoding end first goes through two convolutional layers, Conv6 and Conv7. The parameters of Conv6 are 1×1 kernel size, stride [1, 1], no edge padding, and 512 output channels; the parameters of Conv7 are 1×1 kernel size, stride [1, 1], no edge padding, and 256 output channels. Both are activated using the Leaky_ReLU activation function. Then, to prevent overfitting, a Dropout operation is performed with a ratio of 0.

7. Then, a convolutional layer, Conv8, is passed through with 1×1 kernel size, stride [1, 1], no edge padding, and 9 output channels. Finally, the dimension with dimension 1 is removed, and the network parameters are returned. (d) An attention mechanism is inserted. An attention module (ECA-Net) is added after each convolutional layer at the encoding end of the network to improve the performance of point cloud segmentation in real time. This attention module first transforms the feature map from a matrix of [h, w, c] into a vector of [1, 1, c] through global average pooling, where h and w are the size of the feature map and c is the number of channels; then, it adaptively determines the size of the convolutional kernel. Then, a one-dimensional convolution is performed to obtain the weights for each channel of the feature map. Finally, after activation by the Sigmoid function, the normalized weights are multiplied with the original input feature map channel by channel to generate a weighted feature map. Step 3: Reconstruct the discrete point cloud of the material pile surface into a plane using Poisson reconstruction. Poisson reconstruction is a reconstruction scheme that combines the advantages of global and local matching. The core idea is to construct an implicit surface with watertightness by transforming the discrete sample point information of the object surface into a continuously integrable surface function. This method obtains the implicit surface represented in the form of a triangular mesh by inputting a directed point cloud, and specifically includes the following sub-steps: (a) Calculate the point cloud normal vector. The point cloud data collected by the lidar is directionless, so the point cloud normal vector needs to be calculated manually. The method is to fit a local plane with six adjacent points for each discrete point cloud and find the normal vector of the plane. The directed point cloud is obtained by approximating the point cloud normal vector by finding the plane normal vector. (b) Reconstructing the surface of the vehicle-mounted material stack using Poisson reconstruction: First, construct an indicator function that is set to 1 within the object and 0 elsewhere. Now, let M be the object region, then the indicator function χ... M It is described by formula (3). This transforms the problem of reconstructing the object's surface into reconstructing x. M The problem, at the same time χ M The gradient of the point cloud is equal to the normal vector of the point cloud surface, so the problem is transformed into solving for its own gradient. sum vector field The indicator function closest to the point cloud normal vector Applying the Euler-Lagrange formula, the problem is transformed into a standard Poisson problem, described by formula (4). The indicator function χ is obtained by solving the above equation. M Finally, the indicator function χ is extracted using the MarchingCubes method for extracting isosurfaces. M The part that equals 0 will yield a triangular mesh representing the surface of the object; Step 4: Calculate the volume of the truck-mounted material pile. Using the point cloud of the mining truck extracted in Step 2 and the known dimensions of the mining truck, the position of the bottom plane of the truck bed can be accurately determined. Now, taking each triangular mesh as a group, project its three vertices onto the plane where the truck bucket is located to form multiple hexahedrons. Now, divide each hexahedron into two tetrahedrons along the diagonal, and calculate the volume of each tetrahedron as V. i1 V i2 Taking a single tetrahedron as an example: Given that the coordinates of the vertices of the tetrahedron are (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), and (x4, y4, z4), and the three vectors passing through the point (x1, y1, z1) are a = (x2-x1, y2-y1, z2-z1), b = (x3-x1, y3-y1, z3-z1), and c = (x4-x1, y4-y1, z4-z1), then its volume can be obtained by the mixed product of the vectors, and described by formula (5). V=|(a×b)·c| / 6 (5) The volume V of the polyhedron formed can be calculated from the above formula. i =V i1 +V i2 Finally, by summing the volumes of these hexahedrons... The volume of the on-board material stack can then be calculated.

Images