LiDAR granary volume calculation method and system based on material surface point cloud compensation

Abstract

The present invention discloses a method and system for calculating the volume of a granary based on a laser radar compensation of a material surface point cloud, and relates to the field of laser radar technology and point cloud processing technology, including: data acquisition; data preprocessing; roughly segmenting the preprocessed laser radar point cloud data to extract the target object; further finely segmenting the roughly segmented object by a machine learning or deep learning method to obtain a granary wall and a material surface point cloud; optimizing the granary wall and the material surface point cloud by a material surface point cloud compensation algorithm; and performing volume calculation on the point cloud after the material surface point cloud compensation. The present invention integrates a point cloud segmentation model of a local attention machine system learning method, and then performs a cavity analysis and repairs the cavity based on the extracted target material surface point cloud. The cavity repair algorithm is not a simple repair, but is gradually repaired according to the point cloud in the neighborhood, thereby improving the repair accuracy and the final volume accuracy.

Description

Technical Field

[0001] The present invention relates to the field of laser radar technology and point cloud processing technology, and in particular to a laser radar granary volume calculation method and system based on material surface point cloud compensation. Background Art

[0002] Laser radar technology (LiDAR) is a modern remote sensing technology that measures the distance, speed and other characteristics of a target by emitting a laser beam to the target and receiving the reflected beam. The LiDAR system can quickly scan the surface of the pile and obtain a large amount of data from the target points. After processing, these data can generate accurate three-dimensional stereo images to reveal the shape and volume of the pile. This technology is a non-contact measurement. When measuring the volume of the pile, it does not require direct contact with the measured object, and is suitable for dangerous or inaccessible environments. Existing LiDAR technology has been widely used in measuring the volume of objects. However, the accuracy of traditional LiDAR in measuring the volume of grain silos is only about 95%, which cannot meet the 99% accuracy requirements of some customers. In addition, due to the fluctuation of radar stability and the lack of point cloud data, the point cloud data of the material surface is incomplete, which further affects the accuracy of volume measurement. Summary of the invention

[0003] The purpose of the present invention is to improve the accuracy of laser radar in measuring the volume of a granary, and the present invention provides a laser radar granary volume calculation method based on material surface point cloud compensation.

[0004] The technical solution of the present invention is as follows:

[0005] A method for calculating the volume of a granary using laser radar based on material surface point cloud compensation comprises the following steps:

[0006] S1. Data collection: Use the LiDAR device to scan the target object and obtain the LiDAR point cloud data set;

[0007] S2. Data preprocessing; the data preprocessing includes filtering and denoising, data smoothing and data registration;

[0008] S3. Rough segmentation of point cloud: Roughly segment the preprocessed LiDAR point cloud data to extract the target object;

[0009] S4. Fine segmentation of point cloud: Use machine learning or deep learning methods to further fine segment the roughly segmented objects to obtain point clouds of granary walls and material surfaces;

[0010] S5. Material surface point cloud compensation: optimize the granary wall and material surface point cloud through the material surface point cloud compensation algorithm;

[0011] S6. Calculate the volume of the point cloud after the material surface point cloud is compensated.

[0012] Specifically, the filtering and denoising process in S2 includes:

[0013] S21. Identify and remove isolated points from the collected point cloud data;

[0014] S22. performing Gaussian smoothing on the point cloud data after removing isolated points;

[0015] S23. Perform local downsampling on the point cloud processed in S2.

[0016] Specifically, the point cloud rough segmentation of S3 includes:

[0017] S31. Splitting the point cloud into shapes, where the split shapes include but are not limited to cylinders, cones, and planes;

[0018] S32. Use RANSAC to segment the cylinder and cone. If the cylinder and cone are successfully segmented, combine them to obtain a rough segmentation result;

[0019] S33. If the cylinder and cone are not segmented in S32, the RANSAC method is used to perform point cloud plane segmentation, and the segmented planes are combined to generate a rough point cloud segmentation result.

[0020] Further, the S4 includes:

[0021] Calculate the curvature of the preprocessed point cloud and the normal vector ,in , Indicates the number of point clouds. Each point corresponds to a normal vector and a curvature. The point cloud data, normal vector and curvature form a new feature vector. The new feature vector is expressed as:

[0022] ;

[0023] in, , Respectively indicate points The components of the coordinate values ​​on the x-axis, y-axis, and z-axis, and Respectively indicate points The components of the normal vector on the x-axis, y-axis, and z-axis, Indicate point The curvature of

[0024] The feature vectors of all point clouds are input into the point cloud segmentation model with attention mechanism, and the segmented target point cloud is output.

[0025] Furthermore, the point cloud segmentation model includes an encoding part and a decoding part; the encoding part includes a first downsampling and grouping module, a first feature extraction module, a second downsampling and grouping module, and a second feature extraction module; the decoding part includes a first interpolation module, a third feature extraction module, a second interpolation module, and a fourth feature extraction module; a skip-layer connection is adopted between the first downsampling and grouping module and the fourth feature extraction module, and a skip-layer connection is adopted between the second downsampling and grouping module and the third feature extraction module; in the encoding part, the specific steps of the downsampling and grouping module include:

[0026] (1) Locally downsample the point cloud features according to the curvature of the point cloud. The specific operation of local downsampling is as follows:

[0027] a). Read the curvature value in the point cloud and set the curvature threshold ;

[0028] b). Traverse all points in the point cloud to obtain small curvature point clouds whose curvature values ​​are less than or equal to the curvature threshold and large curvature point clouds whose curvature values ​​are greater than the curvature threshold;

[0029] c). Evenly sample the extracted small curvature point cloud to reduce the density of the point cloud;

[0030] d). Combining the point cloud sampled in step c) with the large curvature point cloud in step b) to obtain a point cloud after local downsampling;

[0031] (2) Taking any point in the point cloud as the initial position, use a spherical area with a radius of R to perform the nearest neighbor search, keep the number of nearest neighbor points fixed, and truncate other points that exceed the fixed number. If the number of nearest neighbor points is less than the fixed number, randomly add all points in the spherical area with a radius of R until the fixed number of points is reached, and treat all the points above the fixed number as a cluster;

[0032] (3) Traverse all points in the point cloud after local downsampling, take each point as the center, and repeat step (2) to obtain Cluster point cloud;

[0033] In the encoding part, the specific steps of the feature extraction module include:

[0034] (1) Extract the curvature and normal vector of each point in each cluster of point clouds, and use the curvature of each point as the curvature weight of each point , the cosine value of the angle between the normal vectors of the nearest neighbor points of each point is used as the normal vector weight of each point ;

[0035] (2) Calculate the mixing weight:

[0036] ;

[0037] Among them, the value range of λ is [0,1];

[0038] (3) Multiply the calculated mixing weight value by the eigenvalue;

[0039] (4) Perform local feature extraction ,in, represents the extracted local features, Represents a multilayer perceptron; a multilayer perceptron consists of three layers, the first layer contains 64 nodes, the second layer contains 64 nodes, and the third layer contains 128 nodes;

[0040] (5) A global feature is extracted from each cluster through the maximum pooling operation, and the global features of all clusters are concatenated to obtain a new feature;

[0041] Furthermore, the specific operation of the decoding part is:

[0042] (1) The first interpolation module takes each point in the low-dimensional feature map as the center, calculates the distance from it to the k points in the neighborhood of the high-dimensional feature map, and uses the inverse of the distance as the weight;

[0043] (2) The third feature extraction module adopts an MLP structure, which consists of two layers. The first layer includes 512 nodes and the second layer includes 1024 nodes.

[0044] (3) After the second interpolation module and the fourth feature extraction module, the probability of each point belonging to the target point is obtained, and the point cloud of the material surface and the silo wall is output.

[0045] Furthermore, the loss function of the point cloud segmentation model is a weighted loss function combining cross entropy and attention mechanism:

[0046] ;

[0047] in, It is a hyperparameter used to adjust the weights of each part; is the cross entropy loss function, is the true label, To predict the results; is the weighted loss function; N represents the number of points in the point cloud; The feature vector representing the center point of the material surface. The center point of the material surface is the point in the current point cloud whose horizontal coordinates are within the rectangular area enclosed by (-0.05, -0.05) and (0.05, 0.05) and whose vertical coordinates are the smallest. Represents the feature vector of any point in the point cloud, Attention() represents the attention function, and softmax represents the normalized exponential function.

[0048] Furthermore, the material surface point cloud compensation in S5 includes the following steps:

[0049] S51. Make all normal vectors point to the outside of the granary. The specific operations are as follows:

[0050] S511. For each point The normal vector is normalized:

[0051] ;

[0052] in, Indicate point The original normal vector of express The European length of After unitization , , , Respectively Vector components in the x, y, and z directions;

[0053] S512. Calculation point The direction vector to the center of mass of the entire point cloud , and then calculate the cosine of the angle between the direction vector and the normal vector:

[0054] ;

[0055] in, Indicate point The angle between the direction vector to the centroid and the normal vector:

[0056] S513. Normal vectors are forced to be consistent, through step S512 The sign of the normal vector determines whether it is necessary to flip the normal vector:

[0057] ;

[0058] S52. Identify the missing area of ​​the material surface, and select different repair methods according to the size of the missing area: if the missing area accounts for less than 10% of the entire material surface area, use the quadratic Bezier curve to perform point cloud interpolation; if the missing area accounts for more than 10% of the entire material surface area, use the constrained Delaunay triangulation method to generate a surface, and then adjust the position of the surface vertices according to the normal vector and curvature of the point cloud surface, and extract the vertices of the surface as a new material surface point cloud;

[0059] S53. The new material surface point cloud is fitted into a Mesh mesh model through a polynomial, and then the distribution of the material surface point cloud on the Mesh is counted within the accuracy range of the radar, and the material surface point cloud is adjusted according to the distribution. The specific process is as follows:

[0060] S531. Based on the Mesh model, Within the range of , count the number of points in the original point cloud on both sides of the Mesh model.

[0061] S532. Adjust the position of the vertex of the model every 0.01 meters according to the number of points on both sides, and update the number of points on both sides at the same time. If the number of points on both sides accounts for less than 1% of the total number of points on both sides, stop adjusting, otherwise continue to adjust the Mesh grid.

[0062] S533. Combining the point clouds of the repaired material surface and the granary wall;

[0063] S55. Generate a convex hull model for the combined point cloud in S54, then perform mesh subdivision on the convex hull model, and gradually fit the subdivided mesh model to the point cloud data until the fitting accuracy no longer changes.

[0064] Specifically, the specific steps of S6 include:

[0065] S61. Get the normal vector of a specified triangle in the convex hull model, denoted as ;

[0066] S62. Move the point cloud along the normal vector of the triangle Perform projection, extract the points projected into the triangle, and calculate the projection distance of each point inside the triangle;

[0067] S63. Subdivide the points inside the triangle into a mesh, and then mesh each vertex of the subdivided triangle along the normal direction Move in the opposite direction, and the moving distance is the projection distance of each point;

[0068] S64. Repeat steps S61 to S63 until all triangles in the convex hull model are traversed, completing the fitting of the convex hull model;

[0069] S65. Split the fitted convex hull model into small tetrahedrons, and obtain the volume of the complete model by calculating the volume of each tetrahedron and summing them up.

[0070] At the same time, the present invention also discloses a laser radar granary volume calculation system based on material surface point cloud compensation, the system is used to calculate the volume of the laser radar granary, including:

[0071] A data acquisition module is used to scan the target object using a laser radar device to obtain a laser radar point cloud data set;

[0072] The data preprocessing module is used to preprocess the laser radar point cloud data, including filtering and denoising, data smoothing and data registration.

[0073] The point cloud coarse segmentation module is used to roughly segment the pre-processed lidar point cloud data and extract the target object;

[0074] The point cloud segmentation module is used to further segment the roughly segmented objects using machine learning or deep learning methods to obtain the point clouds of the granary wall and material surface;

[0075] Material surface point cloud compensation module, used to optimize the grain silo wall and material surface point cloud through material surface point cloud compensation algorithm;

[0076] The volume calculation module is used to calculate the volume of the point cloud after the material surface point cloud is compensated.

[0077] After adopting the above scheme, the beneficial effects of the present invention are:

[0078] (1) The segmentation of the granary in the present invention includes rough segmentation and fine segmentation, thereby improving the segmentation accuracy;

[0079] (2) In the fine segmentation of the present invention, the deep learning method is a point cloud segmentation model that integrates the local attention mechanism. It can make the model focus on the positions that are prone to segmentation errors and improve the segmentation accuracy;

[0080] (3) The present invention adopts a material surface compensation algorithm, which performs cavity analysis and repairs the cavity based on the extracted target material surface point cloud. The cavity repair algorithm is not a simple repair, but a step-by-step repair based on the point cloud in the neighborhood, thereby improving the repair accuracy and the final volume accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is the main flow chart of the present invention;

[0082] Figure 2 It is a point cloud segmentation model diagram of the present invention;

[0083] Figure 3 Point cloud diagram of the missing part of the material surface;

[0084] Figure 4 This is a complete material surface point cloud image after the material surface compensation algorithm is used in this specific implementation mode;

[0085] Figure 5 A point cloud diagram showing that the granary in this specific implementation has an arching tendency;

[0086] Figure 6 This is an interactive interface diagram of a granary belt 3D model rendering in this specific implementation manner;

[0087] Figure 7 It is a schematic diagram of mesh subdivision inside a triangle in a specific implementation manner. DETAILED DESCRIPTION

[0088] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0089] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0090] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0091] like Figure 1 As shown, a method for calculating the volume of a granary using a laser radar based on material surface point cloud compensation includes the following steps:

[0092] S1. Data collection: Use laser radar equipment to perform all-round scanning of the target object to obtain a laser radar point cloud data set; for example, the H13 3D contour scanning laser radar can scan a 3D area covering a large field of view and output information such as the height and volume of the material cargo; the laser radar equipment emits a laser beam through the laser radar and receives the reflected signal, calculates the distance using the time difference or phase difference, and obtains data of a large number of target points;

[0093] S2. Data preprocessing; the data preprocessing includes filtering and denoising, data smoothing and data registration;

[0094] The purpose of denoising is to remove simple isolated noise points and point clouds of other non-grain silo equipment in the granary point cloud data due to material problems, so as to improve data quality. In order to reduce abnormal points and sharp discontinuities, data smoothing is used to make the data more suitable for subsequent analysis; specifically, mean filtering and median filtering can be used. In this specific implementation, the filtering denoising process in S2 includes:

[0095] S21. Identify and remove isolated points from the collected point cloud data;

[0096] S22. Gaussian smoothing is performed on the point cloud data after the isolated points are removed, so that the point cloud is closer to the material surface or wall;

[0097] S23. Perform local downsampling on the point cloud processed in S2. Downsampling can speed up the processing efficiency while retaining the local features of the point cloud.

[0098] S3. Rough segmentation of point cloud: Roughly segment the preprocessed LiDAR point cloud data to extract the target object; segment the LiDAR data using prior knowledge and morphological segmentation methods combined with RANSAC to ensure that the segmentation result contains the point cloud data of the target granary. In this specific implementation, the rough segmentation of point cloud in S3 includes:

[0099] S31. split the shape of the point cloud into simpler shapes, where the split shapes include but are not limited to cylinders, cones, and planes, thereby collecting prior knowledge such as the shape and diameter of the granary;

[0100] S32. Use RANSAC to segment the cylinder and cone. If the cylinder and cone are successfully segmented, combine them to obtain a rough segmentation result;

[0101] S33. If the cylinder and cone are not segmented in S32, the RANSAC method is used to perform point cloud plane segmentation, and the segmented planes are combined to generate a rough point cloud segmentation result;

[0102] S34. Determine whether the point cloud data in S32 or S33 is a granary by combining it with prior knowledge.

[0103] The point cloud segmentation in S4 adopts a deep learning method, and the specific deep learning method is a point cloud segmentation model that integrates a local attention mechanism. The deep learning model needs to train the segmentation model based on the collected point cloud data, and then use the trained model to identify the granary wall and material surface.

[0104] S4. Fine segmentation of point cloud; using machine learning or deep learning methods to further finely segment the roughly segmented objects to obtain point clouds of granary walls and material surfaces; in this specific implementation, S4 includes:

[0105] Calculate the curvature of the preprocessed point cloud and the normal vector ,in , Indicates the number of point clouds. Each point corresponds to a normal vector and a curvature. The point cloud data, normal vector and curvature form a new feature vector. The new feature vector is expressed as:

[0106] ;

[0107] in, , Respectively indicate points The components of the coordinate values ​​on the x-axis, y-axis, and z-axis, and Respectively indicate points The components of the normal vector on the x-axis, y-axis, and z-axis, Indicate point The curvature of

[0108] The feature vectors of all point clouds (dimension N*7) are input into the point cloud segmentation model with attention mechanism, and the segmented target point cloud is output.

[0109] like Figure 2 As shown, the point cloud segmentation model includes an encoding part and a decoding part; the encoding part includes a first downsampling and grouping module, a first feature extraction module, a second downsampling and grouping module, and a second feature extraction module; the decoding part includes a first interpolation module, a third feature extraction module, a second interpolation module, and a fourth feature extraction module; a skip-layer connection is used between the first downsampling and grouping module and the fourth feature extraction module, and a skip-layer connection is used between the second downsampling and grouping module and the third feature extraction module; the function of the skip-layer connection layer is to splice feature information of different levels, which helps to enhance the expression ability of the model;

[0110] In the encoding part, the downsampling and grouping modules (the first downsampling and grouping module, the second downsampling and grouping module) specifically include the following steps:

[0111] (1) Locally downsample the point cloud features according to the curvature of the point cloud so that the local features can be retained while speeding up the training. The specific operation of local downsampling is as follows:

[0112] a). Read the curvature value in the point cloud and set the curvature threshold ;

[0113] b). Traverse all points in the point cloud to obtain small curvature point clouds whose curvature values ​​are less than or equal to the curvature threshold and large curvature point clouds whose curvature values ​​are greater than the curvature threshold;

[0114] c). Evenly sample the extracted small curvature point cloud to reduce the density of the point cloud;

[0115] d). Combining the point cloud sampled in step c) and the point cloud with large curvature in step b) to obtain a point cloud after local downsampling; generally speaking, the greater the curvature, the more features there are in this place. The local downsampling of the present invention better retains more important details and helps to accurately segment; in specific implementation, the number of points of the original point cloud data is fixed to 140,000, and the number of points obtained after the first local sampling is 10,240;

[0116] (2) Taking any point in the point cloud as the initial position, use a spherical area with a radius of R to perform the nearest neighbor search, keep the number of nearest neighbor points fixed, and truncate other points that exceed the fixed number. If the number of nearest neighbor points is less than the fixed number, randomly add all points in the spherical area with a radius of R until the fixed number of points is reached, and treat all the points above the fixed number as a cluster;

[0117] (3) Traverse all points in the point cloud after local downsampling, take each point as the center, and repeat step (2) to obtain Cluster point cloud;

[0118] In the encoding part, the specific steps of the feature extraction module include:

[0119] (1) Extract the curvature and normal vector of each point in each cluster of point clouds, and use the curvature of each point as the curvature weight of each point , the cosine value of the angle between the normal vectors of the nearest neighbor points of each point is used as the normal vector weight of each point ;

[0120] (2) Calculate the mixing weight:

[0121] ;

[0122] The value range of λ is [0,1]. In the specific implementation, if we want the model to pay more attention to the curvature, the value of λ is 0.4.

[0123] 3) Multiply the calculated mixing weight value by the feature value; this process can achieve feature detail amplification;

[0124] (4) Perform local feature extraction ,in, Represents the extracted local features, MLP represents multi-layer perceptron; the multi-layer perceptron consists of three layers, the first layer contains 64 nodes, the second layer contains 64 nodes, and the third layer contains 128 nodes;

[0125] (5) A global feature is extracted from each cluster through the maximum pooling operation, and the global features of all clusters are concatenated to obtain a new feature;

[0126] At the same time, in order to realize feature extraction at multiple scales, the radius is selected as R / 2 and R / 4, and feature extraction operations are performed at the same time to realize and splice features of different scales to achieve feature fusion of different scales;

[0127] In this specific implementation, the number of original point cloud data points is 140,000, and the number of points obtained after local sampling is 10,240. The feature vector size of the input point cloud segmentation model is 10,240x7; assuming that the selected neighborhood R size is 64 (the value of R should be smaller than the radius of the granary), there are 10,240 such clusters after clustering, and each cluster feature size is 64x7. After the initial layer, it becomes a 64x64 feature map, and the weight calculated by the attention mechanism is multiplied by the obtained feature map; the feature dimension of the first feature extraction module is 2560*128; after the second downsampling and grouping module, the feature dimension is 128, and the feature dimension of the second feature extraction module is 640*256; in the second feature extraction module, the first layer contains 128 nodes, the second layer contains 256 nodes, and the third layer contains 256 nodes.

[0128] The specific operations of the decoding part are:

[0129] (1) As the output of the encoding module increases, the dimension of its feature map becomes higher and higher. The first interpolation module takes each point in the low-dimensional feature map as the center, calculates the distance from it to the k points in the neighborhood of the high-dimensional feature map, and uses the inverse of the distance as the weight for weighting. The features of these points are all known, and the weighted average of these known features can be used to obtain the features of the neglected points.

[0130] (2) The third feature extraction module adopts an MLP structure, which consists of two layers. The first layer consists of 512 nodes and the second layer consists of 1024 nodes. It also includes batch normalization and activation functions (such as ReLU).

[0131] (3) After passing through the second interpolation module and the fourth feature extraction module, the point cloud is restored to the size of the original point cloud, the probability of each point belonging to the target point is obtained, and the point clouds of the material surface and the silo wall are output.

[0132] The second interpolation module and the fourth feature extraction module have the same structure as the first interpolation module and the third feature extraction module. After the first interpolation module, the feature dimension is 2560*256, and the feature dimension of the third feature extraction module is 2560*384. After point-by-point convolution, the feature dimension is reduced to 2560*128; after the second interpolation module, the feature dimension is 10240*256, and the dimension after the fourth feature extraction module is 10240*384. After point-by-point convolution, the output dimension is 10240*1024, and then the classification probability is calculated through the fully connected layer.

[0133] The loss function of the point cloud segmentation model is a weighted loss function that combines cross entropy with the attention mechanism. The weighted loss function can enhance the model's ability to learn shape features:

[0134] ;

[0135] in, It is a hyperparameter used to adjust the weights of each part; is the cross entropy loss function, is the true label, To predict the results; is the weighted loss function; N represents the number of points in the point cloud; The feature vector representing the center point of the material surface. The center point of the material surface is the point in the current point cloud whose horizontal coordinates are within the rectangular area enclosed by (-0.05, -0.05) and (0.05, 0.05) and whose vertical coordinates are the smallest. Represents the feature vector of any point in the point cloud, Attention() represents the attention function, and softmax represents the normalized exponential function.

[0136] S5. Material surface point cloud compensation: Optimize the granary wall and material surface point cloud through the material surface point cloud compensation algorithm; Figure 3 As shown, in actual use, dust may cause missing of the scanned material surface and the error of the radar itself may cause the material surface point cloud to appear undulating instead of an accurate curved surface. These factors will lead to a large error between the final calculated point cloud volume and the actual point cloud volume. The present invention first analyzes the point cloud distribution of the material surface obtained by S4, selects the most likely material surface distribution point cloud from the distribution map as the target material surface, and then performs a cavity analysis based on the extracted target material surface point cloud and repairs the cavity. The material surface point cloud compensation of S5 includes the following steps:

[0137] S51. In order to more accurately use the normal vector to repair the missing area of ​​the material surface point cloud, it is necessary to process the direction of the normal vector so that all normal vectors point to the outside of the granary. The specific operations are as follows:

[0138] S511. For each point Normalize the normal vector to ensure that it is a unit vector:

[0139] ;

[0140] in, Indicate point The original normal vector of express The European length of Represents the unitized , , , Respectively Vector components in the x, y, and z directions;

[0141] S512. Calculation point The direction vector to the center of mass of the entire point cloud , and then calculate the cosine of the angle between the direction vector and the normal vector:

[0142] ;

[0143] S513. Normal vectors are forced to be consistent, through step S512 The sign of the normal vector determines whether it is necessary to flip the normal vector:

[0144] ;

[0145] S52. Identify the missing area of ​​the material surface, and select different repair methods according to the size of the missing area: if the missing area accounts for less than 10% of the entire material surface area, use the quadratic Bezier curve to perform point cloud interpolation; if the missing area accounts for more than 10% of the entire material surface area, use the constrained Delaunay triangulation method to generate a surface, and then adjust the position of the surface vertices according to the normal vector and curvature of the point cloud surface, and extract the vertices of the surface as a new material surface point cloud;

[0146] S53. The new material surface point cloud is fitted into a Mesh mesh model through polynomials, and then the distribution of the material surface point cloud on the Mesh is counted within the accuracy range of the radar. The material surface point cloud is adjusted according to the distribution to make it more accurate. The specific process of adjustment is as follows:

[0147] S531. Based on the Mesh model, Within the range (within the accuracy of the radar), count the number of points in the original point cloud on both sides of the Mesh model.

[0148] S532. Adjust the position of the vertex of the model every 0.01 meters according to the number of points on both sides, and update the number of points on both sides at the same time. If the number of points on both sides accounts for less than 1% of the total number of points on both sides, stop adjusting, otherwise continue to adjust the Mesh grid.

[0149] S533. Combining the point clouds of the repaired material surface and the granary wall;

[0150] S55. Generate a convex hull model for the combined point cloud in S54, then perform mesh subdivision on the convex hull model, and gradually fit the subdivided mesh model to the point cloud data until the fitting accuracy no longer changes. Figure 4 shown.

[0151] S6. Calculate the volume of the point cloud after the compensation of the material surface point cloud; create a three-dimensional mesh model of the object through a three-dimensional model reconstruction algorithm, adjust the mesh model, and then calculate its volume. Compared with other surface reconstruction methods, this method has the advantages of good robustness, absolute closure, and no unpredictable errors. The specific steps of S6 include:

[0152] S6. Volume calculation; the specific steps of S6 include:

[0153] S61. The convex hull model is composed of many triangles; get the normal vector of a specified triangle in the convex hull model, denoted as ;

[0154] S62. Move the point cloud along the normal vector of the triangle Perform projection and extract the points projected into the interior of the triangle, such as Figure 6 As shown: and calculate the projection distance of each point inside the triangle;

[0155] S63. Figure 7 As shown, the points inside the triangle are meshed (the large triangle is subdivided into many small triangles according to the points inside), and then each vertex of the subdivided triangle is meshed along the normal direction. Move in the opposite direction, and the moving distance is the projection distance of each point;

[0156] S64. Repeat steps S61 to S63 until all triangles in the convex hull model are traversed, completing the fitting of the convex hull model;

[0157] S65. Split the convex hull model after fitting into small tetrahedrons (composed of four points, a triangle and a point outside the triangle), and obtain the volume of the complete model by calculating the volume of each tetrahedron and accumulating them.

[0158] S7.3D rendering presents the various states of the granary (such as feeding, unloading, empty, full, arching, dust) to users in the form of 3D interactive models and animations. The calculated object volume results are displayed through a graphical interface or other forms to generate a three-dimensional visualization model to show the detailed structure and volume distribution of the object; this is convenient for users to intuitively understand and further analyze. Furthermore, the high-precision material surface generated by algorithm optimization provides a good data basis for judging the state of the granary; through material surface shape analysis, common arching problems (such as Figure 5 and Figure 6 and issue early warning to prevent abnormal operation of the granary.

[0159] The above method can be used in a laser radar granary volume calculation system based on material surface point cloud compensation. Correspondingly, the system includes:

[0160] A data acquisition module is used to scan the target object using a laser radar device to obtain a laser radar point cloud data set;

[0161] The data preprocessing module is used to preprocess the laser radar point cloud data, including filtering and denoising, data smoothing and data registration.

[0162] The point cloud coarse segmentation module is used to roughly segment the pre-processed lidar point cloud data and extract the target object;

[0163] The point cloud segmentation module is used to further segment the roughly segmented objects using machine learning or deep learning methods to obtain the point clouds of the granary wall and material surface;

[0164] Material surface point cloud compensation module, used to optimize the grain silo wall and material surface point cloud through material surface point cloud compensation algorithm;

[0165] The volume calculation module is used to calculate the volume of the point cloud after the material surface point cloud is compensated. Each module in the system corresponds to each step in the method.

[0166] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0167] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for calculating the volume of a granary using laser radar based on surface point cloud compensation, characterized in that: The steps include: S1. Data collection: Use the LiDAR device to scan the target object and obtain the LiDAR point cloud data set; S2. Data preprocessing; the data preprocessing includes filtering and denoising, data smoothing and data registration; S3. Rough segmentation of point cloud: Roughly segment the preprocessed LiDAR point cloud data to extract the target object; S4. Fine segmentation of point cloud: Use machine learning or deep learning methods to further fine segment the roughly segmented objects to obtain point clouds of granary walls and material surfaces; S5. Material surface point cloud compensation: optimize the granary wall and material surface point cloud through the material surface point cloud compensation algorithm; S6. Calculate the volume of the point cloud after the material surface point cloud compensation; The S4 includes: Calculate the curvature of the preprocessed point cloud and the normal vector ,in , k represents the number of point clouds, each point corresponds to a normal vector and a curvature, and the point cloud data, normal vector and curvature form a new feature vector. The new feature vector is expressed as: ； in, , and Respectively indicate points The components of the coordinate values ​​on the x-axis, y-axis, and z-axis, , and Respectively indicate points The components of the normal vector on the x-axis, y-axis, and z-axis, Indicate point The curvature of Input the feature vectors of all point clouds into the point cloud segmentation model with attention mechanism, and output the segmented target point cloud; The point cloud segmentation model includes an encoding part and a decoding part; the encoding part includes a first downsampling and grouping module, a first feature extraction module, a second downsampling and grouping module, and a second feature extraction module; the decoding part includes a first interpolation module, a third feature extraction module, a second interpolation module, and a fourth feature extraction module; the first downsampling and grouping module and the fourth feature extraction module are connected by a skip layer, and the second downsampling and grouping module and the third feature extraction module are connected by a skip layer; in the encoding part, the specific steps of the downsampling and grouping module include: (1) Locally downsample the point cloud features according to the curvature of the point cloud. The specific operation of local downsampling is as follows: a). Read the curvature value in the point cloud and set the curvature threshold ; b). Traverse all points in the point cloud to obtain small curvature point clouds whose curvature values ​​are less than or equal to the curvature threshold and large curvature point clouds whose curvature values ​​are greater than the curvature threshold; c). Evenly sample the extracted small curvature point cloud to reduce the density of the point cloud; d). Combining the point cloud sampled in step c) with the large curvature point cloud in step b) to obtain a point cloud after local downsampling; (2) Taking any point in the point cloud as the initial position, use a spherical area with a radius of R to perform the nearest neighbor search, keep the number of nearest neighbor points fixed, and truncate other points that exceed the fixed number. If the number of nearest neighbor points is less than the fixed number, randomly add all points in the spherical area with a radius of R until the fixed number of points is reached, and treat all points above the fixed number of points as a cluster; (3) Traverse all points in the point cloud after local downsampling, take each point as the center, and repeat step (2) to obtain N clusters of point clouds; In the encoding part, the specific steps of the feature extraction module include: (1) Extract the curvature and normal vector of each point in each cluster of point clouds, and use the curvature of each point as the curvature weight of each point , the cosine value of the angle between the normal vectors of the nearest neighbor of each point is used as the normal vector weight of each point ; (2) Calculate the mixing weight: ； Among them, the value range of λ is [0,1]; (3) Multiply the calculated mixing weight value by the eigenvalue; (4) Perform local feature extraction ,in, Represents the extracted local features, MLP represents multi-layer perceptron; the multi-layer perceptron consists of three layers, the first layer contains 64 nodes, the second layer contains 64 nodes, and the third layer contains 128 nodes; (5) A global feature is extracted from each cluster through the maximum pooling operation, and the global features of all clusters are concatenated to obtain a new feature.

2. The method for calculating the volume of a granary using laser radar based on material surface point cloud compensation according to claim 1 is characterized in that: The filtering and denoising process in S2 includes: S21. Identify and remove isolated points from the collected point cloud data; S22. performing Gaussian smoothing on the point cloud data after removing isolated points; S23. Perform local downsampling on the point cloud processed in S2.

3. The method for calculating the volume of a granary using laser radar based on material surface point cloud compensation according to claim 1 is characterized in that: The point cloud rough segmentation of S3 includes: S31. Splitting the point cloud into shapes, where the split shapes include but are not limited to cylinders, cones, and planes; S32. Use RANSAC to segment the cylinder and cone. If the cylinder and cone are successfully segmented, combine them to obtain a rough segmentation result; S33. If the cylinder and cone are not segmented in S32, the RANSAC method is used to perform point cloud plane segmentation, and the segmented planes are combined to generate a rough point cloud segmentation result.

4. The method for calculating the volume of a granary using laser radar based on material surface point cloud compensation according to claim 1 is characterized in that: The specific operations of the decoding part are: (1) The first interpolation module takes each point in the low-dimensional feature map as the center, calculates the distance from it to the k points in the neighborhood of the high-dimensional feature map, and uses the inverse of the distance as the weight; (2) The third feature extraction module adopts an MLP structure, which consists of two layers. The first layer includes 512 nodes and the second layer includes 1024 nodes. (3) After the second interpolation module and the fourth feature extraction module, the probability of each point belonging to the target point is obtained, and the point cloud of the material surface and the silo wall is output.

5. The method for calculating the volume of a granary using laser radar based on material surface point cloud compensation according to claim 4 is characterized in that: The loss function of the point cloud segmentation model is a weighted loss function that combines cross entropy with the attention mechanism: ； in, It is a hyperparameter used to adjust the weights of each part; is the cross entropy loss function, is the true label, To predict the results; is the weighted loss function; N represents the number of points in the point cloud; The feature vector representing the center point of the material surface. The center point of the material surface is the point in the current point cloud whose horizontal coordinates are within the rectangular area enclosed by (-0.05, -0.05) and (0.05, 0.05) and whose vertical coordinates are the smallest. Represents the feature vector of any point in the point cloud, Attention() represents the attention function, and softmax represents the normalized exponential function.

6. The method for calculating the volume of a granary using laser radar based on material surface point cloud compensation according to claim 1 is characterized in that: The material surface point cloud compensation of S5 includes the following steps: S51. Make all normal vectors point to the outside of the granary. The specific operations are as follows: S511. For each point The normal vector is normalized: ； in, Indicate point The original normal vector of express The European length of Represents the unitized , Respectively Vector components in the x, y, and z directions; S512. Calculation point The direction vector to the center of mass of the entire point cloud , and then calculate the cosine of the angle between the direction vector and the normal vector: ； in, Indicate point The angle between the direction vector to the centroid and the normal vector: S513. Normal vectors are forced to be consistent, through step S512 The sign of the normal vector determines whether it is necessary to flip the normal vector: ； S52. Identify the missing area of ​​the material surface, and select different repair methods according to the size of the missing area: if the missing area accounts for less than 10% of the entire material surface area, use the quadratic Bezier curve to perform point cloud interpolation; if the missing area accounts for more than 10% of the entire material surface area, use the constrained Delaunay triangulation method to generate a surface, and then adjust the position of the surface vertices according to the normal vector and curvature of the point cloud surface, and extract the vertices of the surface as a new material surface point cloud; S53. The new material surface point cloud is fitted into a Mesh mesh model through a polynomial, and then the distribution of the material surface point cloud on the Mesh is counted within the accuracy range of the radar, and the material surface point cloud is adjusted according to the distribution. The specific process is as follows: S531. Based on the Mesh model, Within the range of , count the number of points in the original point cloud on both sides of the Mesh model; S532. Adjust the position of the vertices of the model every 0.01 meters according to the number of points on both sides, and update the number of points on both sides. If the number of points on both sides accounts for less than 1% of the total number of points on both sides, stop adjusting, otherwise continue to adjust the Mesh grid; S533. Combining the point clouds of the repaired material surface and the granary wall; S55. Generate a convex hull model for the combined point cloud in S54, then perform mesh subdivision on the convex hull model, and gradually fit the subdivided mesh model to the point cloud data until the fitting accuracy no longer changes.

7. The method for calculating the volume of a granary using laser radar based on material surface point cloud compensation according to claim 6 is characterized in that: The specific steps of S6 include: S61. Get the normal vector of a specified triangle in the convex hull model, denoted as ; S62. Move the point cloud along the normal vector of the triangle Perform projection, extract the points projected into the triangle, and calculate the projection distance of each point inside the triangle; S63. Subdivide the points inside the triangle into a mesh, and then mesh each vertex of the subdivided triangle along the normal direction Move in the opposite direction, and the moving distance is the projection distance of each point; S64. Repeat steps S61 to S63 until all triangles in the convex hull model are traversed, completing the fitting of the convex hull model; S65. Split the fitted convex hull model into small tetrahedrons, and obtain the volume of the complete model by calculating the volume of each tetrahedron and summing them up.

8. The laser radar granary volume calculation system based on material surface point cloud compensation is characterized by: The system is used to calculate the volume of the laser radar granary, including: A data acquisition module is used to scan the target object using a laser radar device to obtain a laser radar point cloud data set; The data preprocessing module is used to preprocess the laser radar point cloud data, including filtering and denoising, data smoothing and data registration. The point cloud coarse segmentation module is used to roughly segment the pre-processed lidar point cloud data and extract the target object; The point cloud segmentation module is used to further segment the roughly segmented objects using machine learning or deep learning methods to obtain the point clouds of the granary wall and material surface; Material surface point cloud compensation module, used to optimize the grain silo wall and material surface point cloud through material surface point cloud compensation algorithm; Volume calculation module, used to calculate the volume of the point cloud after the material surface point cloud is compensated; The processing steps in the point cloud fine segmentation module include: Calculate the curvature of the preprocessed point cloud and the normal vector ,in , k represents the number of point clouds, each point corresponds to a normal vector and a curvature, and the point cloud data, normal vector and curvature form a new feature vector. The new feature vector is expressed as: ； in, , and Respectively indicate points The components of the coordinate values ​​on the x-axis, y-axis, and z-axis, , and Respectively indicate points The components of the normal vector on the x-axis, y-axis, and z-axis, Indicate point The curvature of Input the feature vectors of all point clouds into the point cloud segmentation model with attention mechanism, and output the segmented target point cloud; The point cloud segmentation model includes an encoding part and a decoding part; the encoding part includes a first downsampling and grouping module, a first feature extraction module, a second downsampling and grouping module, and a second feature extraction module; the decoding part includes a first interpolation module, a third feature extraction module, a second interpolation module, and a fourth feature extraction module; the first downsampling and grouping module and the fourth feature extraction module are connected by a skip layer, and the second downsampling and grouping module and the third feature extraction module are connected by a skip layer; in the encoding part, the specific steps of the downsampling and grouping module include: (1) Locally downsample the point cloud features according to the curvature of the point cloud. The specific operation of local downsampling is as follows: a). Read the curvature value in the point cloud and set the curvature threshold ; b). Traverse all points in the point cloud to obtain small curvature point clouds whose curvature values ​​are less than or equal to the curvature threshold and large curvature point clouds whose curvature values ​​are greater than the curvature threshold; c). Evenly sample the extracted small curvature point cloud to reduce the density of the point cloud; d). Combining the point cloud sampled in step c) with the large curvature point cloud in step b) to obtain a point cloud after local downsampling; (2) Taking any point in the point cloud as the initial position, use a spherical area with a radius of R to perform the nearest neighbor search, keep the number of nearest neighbor points fixed, and truncate other points that exceed the fixed number. If the number of nearest neighbor points is less than the fixed number, randomly add all points in the spherical area with a radius of R until the fixed number of points is reached, and treat all points above the fixed number of points as a cluster; (3) Traverse all points in the point cloud after local downsampling, take each point as the center, and repeat step (2) to obtain N clusters of point clouds; In the encoding part, the specific steps of the feature extraction module include: (1) Extract the curvature and normal vector of each point in each cluster of point clouds, and use the curvature of each point as the curvature weight of each point , the cosine value of the angle between the normal vectors of the nearest neighbor of each point is used as the normal vector weight of each point ; (2) Calculate the mixing weight: ； Among them, the value range of λ is [0,1]; (3) Multiply the calculated mixing weight value by the eigenvalue; (4) Perform local feature extraction ,in, Represents the extracted local features, MLP represents multi-layer perceptron; the multi-layer perceptron consists of three layers, the first layer contains 64 nodes, the second layer contains 64 nodes, and the third layer contains 128 nodes; (5) A global feature is extracted from each cluster through the maximum pooling operation, and the global features of all clusters are concatenated to obtain a new feature.

Images