A volume measurement method based on surface reconstruction and triple integral

Abstract

The present invention discloses a volume measurement method based on surface reconstruction and triple integral, comprising the following steps: 1. using a 3D scanning device and an image capturing device to obtain a point cloud and an image; 2. calibration and registration of the image and the point cloud; 3. fusion of the point cloud and the image; 4. training of a neural network and segmentation of the point cloud; 5. reconstruction and calculation of the point cloud. The measurement method can effectively and accurately measure the volume of various irregular objects, the application of deep learning in point cloud segmentation enables it to adapt to different scenes, the surface reconstruction of the point cloud can restore the original posture of the object to a large extent in the three-dimensional space, and the triple integral of its parameterized equation can quickly and accurately calculate its volume, and has the advantages of high efficiency, high precision, and high degree of automation.

Description

Technical Field

[0001] The invention relates to a measurement method, in particular to a volume measurement method based on surface reconstruction and triple integral. Background Art

[0002] In the field of 3D data processing and analysis, point cloud data processing technology is particularly important. Point cloud is a data set consisting of a series of points in 3D space, usually collected by various scanning devices such as LiDAR or stereo photography systems. These points contain the position information of the surface of the object, and sometimes also include color and intensity information, making point cloud data very useful in many applications, such as environmental perception of autonomous vehicles, digital preservation of cultural anomalies, terrain mapping, and virtual reality.

[0003] Although point cloud data has many advantages in digitally expressing the three-dimensional world, its unstructured nature also brings many processing challenges, especially when performing accurate volume measurement and complex data segmentation. Traditional point cloud processing methods, such as cluster segmentation and grid-based volume estimation, can provide preliminary solutions, but often lack accuracy and efficiency, especially when processing large-scale or high-density point cloud data.

[0004] In addition, with the development of computer vision and machine learning technology, especially the successful application of deep learning in image and video analysis, researchers have begun to explore the possible application of these advanced technologies in point cloud data processing. Point cloud segmentation technology based on deep learning can effectively identify and classify different objects and structures in point clouds, but how to combine these technologies with traditional geometric measurement methods to achieve more accurate and automated volume measurement is still a problem worth studying.

[0005] To overcome these challenges, we propose a new method, "Color point cloud segmentation and volume measurement method based on triple integral". This method combines advanced color point cloud processing technology with the mathematical triple integral theory, and provides a new perspective and tool for the analysis and application of point cloud data by accurately calculating the volume within a closed surface. This method can achieve efficient and accurate processing of complex point cloud data, supporting a variety of application scenarios from cultural heritage protection to precision engineering modeling. Summary of the invention

[0006] The purpose of the present invention is to provide a volume measurement method based on surface reconstruction and triple integral, which solves the problems of low precision and low efficiency of traditional methods.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] A volume measurement method based on surface reconstruction and triple integral, characterized in that the method comprises the following steps:

[0009] Step 1: Collect point cloud information and image information of the target object at different angles through multiple sets of laser radars and cameras;

[0010] Step 2: Use the data obtained in step 1 to calibrate and align the camera and lidar;

[0011] Step 3: The transformation matrix obtained by calibration and registration fuses the image and point cloud;

[0012] Step 4: Use the fused color point cloud as a data set to train the point cloud segmentation neural network

[0013] Step 5: Reconstruct the surface of the segmented point cloud data, obtain the surface equation, and use the triple integral method to calculate the volume of the equation to obtain the precise volume of the object.

[0014] In step 1, at least 3 sets of laser radars and cameras are used for data collection. The laser radar uses non-repetitive scanning to obtain point clouds and images at different angles. The resolution of the point clouds and images is adjusted by setting the laser radar scanning time and the camera lens resolution to adapt to different scenes and objects.

[0015] In step 2, the camera intrinsic parameters are first calibrated. Multiple sets of pictures are taken from different distances and angles using a grid calibration plate, and the camera intrinsic parameter matrix is ​​calculated:

[0016] , (1)

[0017] And radial distortion parameters , and tangential distortion parameters , ;

[0018] After obtaining the intrinsic parameter matrix (1), fix the camera and radar, capture multiple sets of point cloud data and image data of objects, and use the key points of the object data to calculate the extrinsic parameter matrix of the camera and radar:

[0019] , (2)

[0020] In the external parameter matrix (2), is a 3x3 rotation matrix that describes the rotation from camera to radar. is a 3x1 translation vector that describes the translation from camera to radar. is a 1x3 vector of 0, is a scalar used to keep the matrix in homogeneous coordinate form;

[0021] After obtaining the intrinsic parameter matrix (1) and the extrinsic parameter matrix (2), the radars are aligned with each other. The coordinates of one laser radar are used as the reference coordinates, and the other radars are used as radars to be aligned. The calibration object is scanned at the same time, and at least 3 calibration points are collected to complete the rough alignment. Then, the ICP algorithm is used for fine alignment.

[0022] In step 3, the image is transformed from the image coordinate system to the camera coordinate system, and then from the camera coordinate system to the radar coordinate system through the transformation matrix obtained in step 2. Then the point cloud mask within the FOV is obtained to obtain a point cloud mask that can be projected on the image. The point cloud filtered by the mask is projected onto the image to obtain a depth map, thereby realizing the fusion of the image and the point cloud.

[0023] In step 4, a deep learning method is used to segment the point cloud and image. The image and point cloud training model are input separately, and the point cloud is segmented by comprehensive judgment. The color point cloud training model is input and the point cloud is segmented at the same time to obtain a complete point cloud of the object to be measured.

[0024] In step 5, after obtaining the complete point cloud, first ensure the closure of the point cloud. For incompletely closed models, use geometric methods to repair holes. Secondly, after ensuring the closure of the point cloud model, use the Poisson reconstruction method to reconstruct the surface and define an indicator function , which takes the value 1 inside the surface and 0 outside, using the point cloud and the corresponding normal information to estimate the gradient direction; for each point and its normal ,make exist The direction and Consistent, construct vector field As is an approximation of ; Using this vector field , seeking a scalar field , so that The gradient is as close as possible to , that is, solve:

[0025] , (3)

[0026] in, is the Laplace operator, is a vector field The divergence of is a scalar field defined everywhere, from which the isosurfaces are extracted, namely The surface is reconstructed to obtain a smooth and accurate three-dimensional model;

[0027] After the surface reconstruction is completed, the surface equations described mathematically are extracted from the closed surface model by the parametric surface method, and the parametric surface is expressed as a function ,in and is a parameter, , is a vector function that maps points on the parameter plane to the surface of three-dimensional space. ,in , , yes and Function of

[0028] The plane is parameterized as , the cylindrical surface is parameterized as ,in , the sphere is parameterized as ,in ; For more complex surfaces, use B-spline surfaces:

[0029] , (4)

[0030] in is the control point, and is the B-spline basis function, p and q are the degrees of the basis function;

[0031] Finally, the volume is calculated using a triple integral on the parameterized 3D surface equation:

[0032] , (5).

[0033] In step 5, the hole is repaired by a geometric method, which specifically includes the following steps:

[0034] The first step is to identify the boundaries of holes in the point cloud. Assume that the mesh of the model consists of a vertex set V and a face set F, where each face in F is defined by a vertex index; the boundary identification is implemented by the following algorithm:

[0035] ① Create an edge dictionary E, where the key is the vertex pair and the value is the number of times the edge appears;

[0036] ②Traverse each face and update the number of occurrences of each edge in E;

[0037] ③All edges that appear 1 times in E are boundary edges;

[0038] The second step is boundary smoothing to reduce noise and irregularities. The Laplace smoothing algorithm is used to make the irregularities more naturally integrated into the original model. The basic formula is:

[0039] , (6)

[0040] in, It is The position of the vertices, yes The set of adjacent vertices of is the number of neighbors, is the smoothing factor;

[0041] In the third step, for holes with regular boundaries, triangles are used to fill patches. For holes with complex boundaries, the minimum surface strategy and radial basis function (RBF) are used. The hole filling based on RBF is expressed as:

[0042] , (7)

[0043] is a point on the boundary. To find a point that satisfies Surface ,in is the coefficient to be determined, yes and The Euclidean distance between the two is determined by minimizing the distance between the surface and the surrounding mesh for each hole boundary point. .

[0044] It can be seen from the technical solution provided by the above invention that, compared with the existing measurement method, the above method reduces the contact between the staff and the goods. At the same time, the combination with deep learning enables the present invention to more accurately identify and classify different objects and features in the point cloud. The present invention can provide more accurate volume measurement by directly applying triple integrals in three-dimensional space. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0046] The following is a clear and complete description of the technical solutions in the implementation of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] like Figure 1 FIG. 1 is a schematic diagram of a volume measurement method based on surface reconstruction and triple integral provided by an embodiment of the present invention, wherein the method comprises:

[0048] 1. Multiple cameras and lidars acquire point clouds and images:

[0049] This step uses multiple sets of laser radars and cameras to shoot the object to be measured from different angles to obtain point clouds and images at different angles. In the present invention, at least three sets of equipment should surround the object to be measured, and the scanning range of each set of equipment is at most 120 degrees. The object is scanned from an oblique top view at a distance of at least 3 meters and at most 20 meters from the object. Each set of laser radars and cameras will not move after being fixed.

[0050] After the installation is complete, the multi-threaded startup program starts all devices simultaneously to collect data, obtaining a picture and a point cloud information for each angle.

[0051] In the specific implementation, you can choose to use three groups of equipment to place them at the vertices of an equilateral triangle, with the object to be measured placed in the middle of the triangle. The lidar and camera scan at a 45-degree angle looking down, and the distance between the equipment and the object is 5 meters. At this distance, better scanning accuracy can be obtained.

[0052] 2. Calibration and registration of images and point clouds:

[0053] The calibration and registration of images and point clouds includes the calibration of cameras and lidars and the registration between multiple lidars.

[0054] The calibration of cameras and lidars is used to measure and calibrate the spatial relationship and time synchronization between the two devices. The principle is to use specific points in the overlapping fields of view of the two devices, and the correspondence between the specific points to calculate the optimal rotation matrix R and translation vector t through the least squares method. This rotation matrix converts the points in the camera coordinate system to the radar coordinate system.

[0055] In the specific implementation, first install a fixed camera and lidar to ensure that the relative position of the camera and lidar does not change. Use a one-meter-long and one-meter-wide rectangular low-reflective foam as a calibration plate. Place the calibration plate at different positions in the field of view, take a photo and a point cloud at each position, and after taking 10 to 15 sets of data in total, mark the corner points of each set of data and record them. The image coordinates of the corner points are expressed as , the three-dimensional coordinates are expressed as , solving the optimal rotation matrix is ​​achieved by optimizing the following problem:

[0056] ,

[0057] The registration between LiDARs adopts a similar method. These two steps can be performed separately. After completing the calibration of each group of equipment, each group of equipment is installed according to step 1. At the same time, an object is scanned and at least 3 calibration points are recorded. The coordinate system of one group of equipment is used as the original coordinate system, and the other equipment is used as the equipment to be registered. The above optimization equation is used to obtain the transformation matrix of the rough registration, and then the ICP algorithm is used to complete the fine registration, so as to obtain the transformation matrix of each group of equipment to the original coordinate system.

[0058] 3. Fusion of image and point cloud:

[0059] The camera-to-lidar transformation matrix obtained in the second step can be used to color the point cloud, and the transformation matrix obtained by radar registration can be used to put all point clouds into the same coordinate system, thus obtaining a complete colored point cloud of the object to be measured.

[0060] In a specific implementation, the steps may be: first read the image data and point cloud data separately, transform the image from the image coordinate system to the camera coordinate system through the transformation matrix, and then from the camera coordinate system to the radar coordinate system, and then obtain the point cloud mask within the fov to obtain a point cloud mask that can be projected on the image, and project the point cloud after mask filtering onto the image to obtain a depth map, thereby realizing the fusion of the image and the point cloud.

[0061] 4. Neural network training and point cloud segmentation:

[0062] The deep learning method is used to segment the point cloud and image. The model used for segmentation can be trained separately for the image and point cloud, and finally the segmentation output point cloud is comprehensively judged. It can also be directly input for multi-dimensional colored point cloud training, and the colored point cloud is also segmented.

[0063] In this process, multimodal input allows the deep learning model to be trained more accurately. Compared with traditional algorithms and single-mode input, the deep learning model with multimodal input is better at processing difficult texture areas.

[0064] After completing the first three steps, you can customize the data set for a specific scenario based on the known application scenario. Using the model trained with a specific data set can achieve better point cloud segmentation results in a specific scenario.

[0065] 5. Reconstruction and volume calculation of point cloud:

[0066] The segmented point cloud cannot be used directly for mathematical calculations. There will inevitably be parts that cannot be scanned by the laser or are blocked by other objects. In this case, it is necessary to detect and fill holes in the point cloud. After filling, the closed point cloud is processed using a surface reconstruction algorithm for subsequent parameterization and calculation.

[0067] In the specific implementation, the process of point cloud reconstruction includes hole identification and filling as well as surface reconstruction. In the data collection process, when facing most irregular objects in reality, limited 3D scanning equipment often produces data missing due to various reasons (such as object occlusion, reflectivity problems, etc.). These missing areas appear as holes in the point cloud. Unrepaired holes may cause failure or misleading of surface reconstruction. Filling these areas with hole repair technology can improve the integrity and accuracy of the data.

[0068] When identifying hole boundaries, the point cloud needs to be preprocessed. First, use voxel downsampling and statistical outliers to remove useless and erroneous points in the reduced point cloud. Second, the point cloud can be converted into a triangular mesh using the Delaunay triangulation method to more intuitively and effectively identify hole boundaries. Once the triangular mesh is obtained, the hole boundary can be identified by analyzing the topological structure of the mesh:

[0069] Finding border edges: In a mesh, a border edge is an edge that belongs to only one triangle. You can identify border edges by traversing all triangles and counting the number of times each edge appears. The number of times a border edge appears is 1.

[0070] Connect boundary edges: By connecting boundary edges in sequence to form a closed ring structure, the boundary of the hole can be defined.

[0071] After the hole boundaries are determined, a Laplace smoothing algorithm is used to reduce noise and irregularities so that subsequent patches can blend more naturally into the original model:

[0072] ,

[0073] in, It is The position of the vertices, yes The set of adjacent vertices of is the number of neighbors, is the smoothing factor;

[0074] After hole identification and smoothing, different types of patches are used according to the size and shape of the holes. For small or relatively flat holes, flat patches are used. For complex or large holes, surface patches are used, such as the minimum surface strategy and radial basis function (RBF). The hole filling based on this method can be expressed as:

[0075] ,

[0076] is a point on the boundary. To find a point that satisfies Surface ,in is the coefficient to be determined, yes and The Euclidean distance between the two is determined by minimizing the distance between the surface and the surrounding mesh for each hole boundary point. .

[0077] After the holes are filled, Poisson reconstruction is used on the closed point cloud model. The core idea is to assume that an ideal surface can be reconstructed by solving an elliptic partial differential equation, which describes how the gradient field of a scalar field is aligned with the input normal vector.

[0078] Define an indicator function , which takes the value of 1 inside the surface and 0 outside. Then, the gradient is derived using the normal information of the point cloud. direction.

[0079] ,

[0080] in represents a point in the point cloud, Indicates at point The estimated normal at .

[0081] Poisson's equation is established using and , construct the vector field , the vector field is An approximation of :

[0082] ,

[0083] Next, find a scalar field , whose gradient Closest , which is close to the following equation:

[0084] ,

[0085] here is the Laplace operator, is a vector field The divergence of .

[0086] Solution of Poisson's equation. Use array method to solve the above Poisson's equation, such as finite element method or discrete difference method. The scalar field f obtained is continuous in the entire domain, so it can be extracted by isosurface to reconstruct the surface.

[0087] Once the surface reconstruction is completed, the surface equations describing the mathematical properties can be extracted from the closed surface model by using the parametric surface method.

[0088] For a plane the parameterization is: ;

[0089] The parameterization for the cylindrical surface is: , ;

[0090] The parameterization for a sphere is: , ;

[0091] For more complex surfaces, use B-spline surfaces:

[0092] ,

[0093] in is the control point, and is the B-spline basis function, p and q are the degrees of the basis function.

[0094] Finally, the volume is calculated using triple integrals on the parameterized 3D surface equation:

[0095] .

[0096] In summary, the method described in the embodiment of the present invention can recover a smooth surface from data containing noise, is applicable to incomplete or sparse point clouds, and has high accuracy and robustness. It is applicable to objects with complex topological structures, such as objects containing holes and tunnels, and the processing process is fully automated, with low technical requirements for operators, and can also avoid the possibility of damage to objects caused by contact, and is more widely used than existing technologies.

[0097] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field.

[0098] In addition, a person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing related hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0099] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

[0100] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in the field and are easily understood by the technical personnel in the field. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A volume measurement method based on surface reconstruction and triple integral, characterized in that: The method comprises the following steps: Step 1: Collect point cloud information and image information of the target object at different angles through multiple sets of laser radars and cameras; Step 2: Use the data obtained in step 1 to calibrate and align the camera and lidar; Step 3: The transformation matrix obtained by calibration and registration fuses the image and point cloud; Step 4: Use the fused color point cloud as a data set to train a point cloud segmentation neural network; Step 5: Reconstruct the surface of the segmented point cloud data, obtain the surface equation, calculate the volume of the equation using the triple integral method, and obtain the precise volume of the object; In step 5, after obtaining the complete point cloud, first ensure the closure of the point cloud. For the incompletely closed model, use geometric methods to repair the holes. Secondly, after ensuring the closure of the point cloud model, use the Poisson reconstruction method to reconstruct the surface and define an indicator function , which takes the value 1 inside the surface and 0 outside, using the point cloud and the corresponding normal information to estimate the gradient direction; for each point and its normal ,make exist The direction and Consistent, construct vector field As is an approximation of ; Using this vector field , seeking a scalar field , so that The gradient is as close as possible to , that is, solve: ，(3) in, is the Laplace operator, is a vector field The divergence of is a scalar field defined everywhere, from which the isosurfaces are extracted, namely The surface is reconstructed to obtain a smooth and accurate three-dimensional model; After the surface reconstruction is completed, the surface equations described mathematically are extracted from the closed surface model by the parametric surface method, and the parametric surface is expressed as a function ,in and is a parameter, , is a vector function that maps points on the parameter plane to the surface of three-dimensional space. ,in , , yes and Function of The plane is parameterized as , the cylindrical surface is parameterized as ,in , the sphere is parameterized as ,in ; For more complex surfaces, use B-spline surfaces: , (4) in is the control point, and is the B-spline basis function, p and q are the degrees of the basis function; Finally, the volume is calculated using a triple integral on the parameterized 3D surface equation: ，(5) In step 5, the hole is repaired by a geometric method, which specifically includes the following steps: The first step is to identify the boundaries of holes in the point cloud. Assume that the mesh of the model consists of a vertex set V and a face set F, where each face in F is defined by a vertex index; the boundary identification is implemented by the following algorithm: ① Create an edge dictionary E, where the key is the vertex pair and the value is the number of times the edge appears; ②Traverse each face and update the number of occurrences of each edge in E; ③All edges that appear 1 times in E are boundary edges; The second step is boundary smoothing to reduce noise and irregularities. The Laplace smoothing algorithm is used to make the irregularities more naturally integrated into the original model. The basic formula is: ，(6) in, It is The position of the vertices, yes The set of adjacent vertices of is the number of neighbors, is the smoothing factor; In the third step, for holes with regular boundaries, triangles are used to fill patches. For holes with complex boundaries, the minimum surface strategy and radial basis function (RBF) are used. The hole filling based on RBF is expressed as: ，(7) is a point on the boundary. To find a point that satisfies Surface ,in is the coefficient to be determined, yes and The Euclidean distance between the two is determined by minimizing the distance between the surface and the surrounding mesh for each hole boundary point. .

2. The volume measurement method based on surface reconstruction and triple integral according to claim 1, characterized in that: In step 1, at least 3 sets of laser radars and cameras are used for data collection. The laser radar uses non-repetitive scanning to obtain point clouds and images at different angles. The resolution of the point clouds and images is adjusted by setting the laser radar scanning time and the camera lens resolution to adapt to different scenes and objects.

3. The volume measurement method based on surface reconstruction and triple integral according to claim 1, characterized in that: In step 2, the camera intrinsic parameters are first calibrated. Multiple sets of pictures are taken from different distances and angles using a grid calibration plate, and the camera intrinsic parameter matrix is ​​calculated: ，(1) And radial distortion parameters , and tangential distortion parameters , ; After obtaining the intrinsic parameter matrix (1), fix the camera and radar, capture multiple sets of point cloud data and image data of objects, and use the key points of the object data to calculate the extrinsic parameter matrix of the camera and radar: ，(2) In the external parameter matrix (2), is a 3x3 rotation matrix that describes the rotation from camera to radar. is a 3x1 translation vector that describes the translation from camera to radar. is a 1x3 vector of 0, is a scalar used to keep the matrix in homogeneous coordinate form; After obtaining the intrinsic parameter matrix (1) and the extrinsic parameter matrix (2), the radars are aligned with each other. The coordinates of one laser radar are used as the reference coordinates, and the other radars are used as radars to be aligned. The calibration object is scanned at the same time, and at least 3 calibration points are collected to complete the rough alignment. Then, the ICP algorithm is used for fine alignment.

4. The volume measurement method based on surface reconstruction and triple integral according to claim 1, characterized in that: In step 3, the image is transformed from the image coordinate system to the camera coordinate system, and then from the camera coordinate system to the radar coordinate system through the transformation matrix obtained in step 2. Then the point cloud mask within the FOV is obtained to obtain a point cloud mask that can be projected on the image. The point cloud filtered by the mask is projected onto the image to obtain a depth map, thereby realizing the fusion of the image and the point cloud.

5. The volume measurement method based on surface reconstruction and triple integral according to claim 1, characterized in that: In step 4, a deep learning method is used to segment the point cloud and image. The image and point cloud training model are input separately, and the point cloud is segmented by comprehensive judgment. The color point cloud training model is input and the point cloud is segmented at the same time to obtain a complete point cloud of the object to be measured.

Images