A single building three-dimensional reconstruction method based on point cloud semantic segmentation and structure fitting

Abstract

The application provides a kind of monomer building three-dimensional reconstruction method based on point cloud semantic segmentation and structure fitting, mainly includes: building target point cloud data set pretreatment;Training building point cloud semantic segmentation network classifies building structure data;Geometric classification and three-dimensional parameterization fitting of building structure;Building structure prior information topological relationship construction and monomer building structure splicing.The application can cope with monomer building semantic reconstruction field problems, complete point cloud data to three-dimensional reconstruction of building structure graph model with semantic information, improve in segmentation effect, reconstruction accuracy, labor cost and the like.

Description

Technical Field

[0001] This invention relates to a method for 3D reconstruction of a single building based on point cloud semantic segmentation and structural fitting, belonging to the fields of computer vision and artificial intelligence technology. Background Technology

[0002] 3D building reconstruction refers to the process of constructing a 3D visualization model with the same characteristics as the physical structure of a building based on relevant data collected from the building target using computer processing technology. In the current digital age of booming real-world 3D construction, the creation of smart cities, digital blocks, and other urban 3D information spaces places higher demands on the realism, physicality, and refinement of 3D reconstruction of physical targets such as buildings and road networks. Furthermore, fields such as Building Information Modeling (BIM) and the protection and restoration of ancient buildings also require semantic and structured 3D reconstruction of individual building targets. Point clouds, as a typical data source for describing the 3D spatial structure of objects, are increasingly widely used in data acquisition, processing, and analysis. With the expansion of deep learning technology based on neural networks from 2D images to 3D point clouds, it has a significant impact on the intelligent processing and application research of 3D point clouds. Point cloud target detection and point cloud semantic segmentation have become research hotspots due to the needs of fields such as autonomous driving. Currently, the reconstruction of individual buildings mainly relies on software tools for manual annotation and entity modeling. This process is cumbersome, time-consuming, and labor-intensive, and lacks semantic information such as the building's structural attributes, failing to meet the accuracy and efficiency requirements of 3D reconstruction of building targets in many current applications. To address the above problems, this invention proposes a method for 3D reconstruction of individual buildings based on point cloud semantic segmentation and structural fitting. This method utilizes the semantic information of the building structure context for fitting and reconstruction, achieving efficient and high-precision automatic reconstruction of individual buildings. Summary of the Invention

[0003] The purpose of this invention is to address the problems of missing structural semantics in the modeling of individual buildings, low reconstruction efficiency, and poor accuracy. It proposes a three-dimensional reconstruction method for individual buildings based on point cloud semantic segmentation and structural fitting, so as to achieve efficient and high-precision reconstruction of individual buildings.

[0004] The objective of this invention is achieved as follows: a method for 3D reconstruction of a single building based on point cloud semantic segmentation and structural fitting, specifically implemented according to the following steps:

[0005] Step 1: Preprocessing of the building target point cloud dataset.

[0006] To address the issues of excessive noise and data redundancy in point cloud data, a correlation filtering operation is employed to process the point cloud dataset. First, a statistical filter is used to remove outliers and noise. Second, while preserving point cloud features, data redundancy is reduced by using a keypoint extraction algorithm based on normal vectors and an improved voxel conditional filtering point cloud downsampling algorithm to obtain keypoints in the point cloud and downsampled building sampling points. These are then combined and deduplicated to obtain a sampled and denoised single-building dataset. Finally, a random sampling algorithm is used to randomly sample a fixed number of points from each single building to ensure that the single-building point cloud dataset input to the semantic segmentation network has the same data dimension.

[0007] Step 2: Train a semantic segmentation network for building point clouds to classify building structure data.

[0008] A partial point cloud dataset of individual buildings is segmented and labeled to obtain training, validation, and test sets for the segmentation data of individual building structures. The training set of individual building point cloud data with annotation information is input into the PointNet++ network to train the semantic segmentation network model of building structures. Then, the validation set point cloud data is input into the network to adjust the network hyperparameters and improve the network segmentation effect. Finally, the obtained test set building structure point cloud is input into the trained segmentation network model for semantic segmentation, resulting in point cloud datasets of building structures for each part, including roof, facade, door, window, and column.

[0009] Step 3: Geometric classification and 3D parametric fitting of building structures.

[0010] Based on the building structure classification results completed by the segmentation network, and combined with the geometric characteristics of the building structure categories, the various building structure datasets are geometrically classified. Buildings include facades, roofs, doors, windows, columns, etc., and geometric categories include spatial geometric primitives such as planes, cylinders, and spheres. By establishing the correspondence between building structures and geometric primitives, appropriate algorithms are used to achieve 3D reconstruction of the geometric models of each building structure.

[0011] Step 4: Constructing the topological relationships of the building structure prior information and splicing individual building structures.

[0012] Based on the prior information of the building structure, a topological relationship map of the building structure is constructed. Then, combined with the structural models of each part of the building fitted in step three, as well as the obtained structural outlines and boundary points of the building parts, the structural models of each part of the building are spliced ​​together to obtain a complete three-dimensional model of the single building.

[0013] The details of the plan are as follows:

[0014] Step 1, laser scanning data acquisition point cloud information such as Figure 2As shown, statistical filtering is applied to the point cloud dataset to remove outliers and noise. Then, a feature point extraction algorithm based on normal vectors and an improved voxel conditional filtering point cloud downsampling algorithm are used to obtain point cloud feature points and downsampled sampling points. These are then concatenated and deduplicated to obtain the sampled and denoised single-building dataset, as shown below. Figure 3 As shown, the specific steps are as follows:

[0015] Step 1.1: Process the single building point cloud dataset using a statistical filter to remove outlier noise points. Specifically: Search all neighborhood points of each point using a kd-tree and calculate the X value for that point. i =(x i ,y i ,z i To the domain point X j =(x j ,y j ,z j Distance d ij Where i = [1,…,m] represents m search reference points, j = [1,…,k] represents the k nearest neighbor points to the reference points, and the nearest neighbor set X k ={x 1 ,x 2 …,x k Next, calculate the mean μ and standard deviation σ of the distance from the point to the neighboring points. The calculation method is as follows:

[0016]

[0017]

[0018] Therefore, the Gaussian distribution d ~ N(μ,σ) that the points in the point cloud conform to can be obtained. Point cloud noise filtering is achieved by eliminating point sets with excessively large mean distances from neighboring points within the Gaussian distribution confidence interval. The elimination condition formula is as follows:

[0019] or

[0020] Step 1.2: Before downsampling the data, based on the denoised point cloud data, extract key point information from the point cloud of a single building, such as boundary points, corner points, and intersection points between surfaces, to ensure that key information in the point cloud structure is preserved. Specifically, key point algorithms such as Harris3D and ISS can be used. The specific steps using the Harris3D feature point algorithm are as follows:

[0021] Step 1.2.1: Calculate the normal information for each point. The calculation process for normal information is as follows: Select the nearest points to the solution point, and use the covariance matrix calculation formula... Calculate each point p i The covariance matrix G of the points near it is given by the given information. Let k be the centroid of the set of points between the point and its nearest neighbors, and k be the number of nearest neighbors. Then, calculate the normal axis of the point using PCA principal component analysis. Finally, determine the normal direction using the viewpoint and calculate the coordinate system component vectors of the normal.

[0022] Step 1.2.2, based on the 2D Harris corner detection corner response value calculation formula, is applied to construct the normal correlation matrix and corner response values ​​of each point in the 3D point cloud. The calculation formula for the 3D point cloud corner response value is as follows:

[0023] R 3D =det(M 3D -0.04[trace(M 3D )] 2

[0024]

[0025] Where n is the number of neighboring points of the current point, n x n represents the X-axis direction component of the normal vector of a point calculated from its neighboring points. y n z These are the Y-axis and Z-axis vectors, respectively.

[0026] Step 1.2.3: Determine whether a point cloud corner or key point is determined based on the corner response threshold. If the response is greater than the threshold, it is a key point; otherwise, it is not a key point.

[0027] Step 1.2.4: Calculate the distances from data points in the original point cloud to each keypoint. If the distance of a data point to all keypoints is greater than a threshold, save that point as a keypoint to retain more point cloud structure information. Based on the supplemented set of keypoints, iteratively search for the remaining keypoint data. Obtain all keypoint data in the point cloud.

[0028] Step 1.3: Use the improved voxel conditional filtering algorithm to downsample the point cloud data, specifically as follows:

[0029] Step 1.3.1: Set the voxel grid resolution L. The voxel grid divides the point cloud data into several point cloud blocks. Calculate the centroid P of each point cloud block. After unifying the point mass, calculate the average coordinate of the point cloud in the voxel grid block, which is the centroid of the voxel block.

[0030] Step 1.3.2: Calculate the distance from the center of mass of each voxel block to the centroid, select the nearest point as the voxel center Q, and ensure that the voxel centroid is the actual point in the point cloud without changing the original point cloud data.

[0031] Step 1.3.3: Find the number of voxel grid points where the voxel center point Q is located. If it is lower than the voxel point cloud number threshold, delete the voxel center point Q to ensure that noise points are not retained as voxel sampling points.

[0032] Step 1.4: Merge the downsampled point cloud data with the Harris keypoint data, remove duplicate point sets, and perform point cloud data downsampling preprocessing.

[0033] Step 1.5: Based on the principle of the same dimension as the input point cloud for point cloud semantic segmentation, random sampling of a fixed number of points is performed on the point cloud data to sample the point cloud data of a single building to the same dimension, thus obtaining the building point cloud dataset. This completes the entire process of point cloud dataset preprocessing, resulting in point cloud data of a single building after retaining feature points and denoising and downsampling.

[0034] Step 2: Segment and label the single building point cloud dataset obtained after denoising and downsampling in Step 1 to obtain a single building structure segmentation data training set and a test set. Train the PointNet++ semantic segmentation network based on the single building semantic segmentation training set data with labeled information. Use a portion of the dataset as a validation set and test set to adjust parameters and verify the training effect. Apply the trained semantic segmentation network to the single building point cloud segmentation task to segment the unlabeled point cloud and obtain the structured segmented point cloud data of the building.

[0035] Step 2.1: Select the point cloud data of a single building acquired by the laser scanner, divide it into blocks according to the building structure, and label the structural categories, specifically including facade, roof, doors, windows, columns, etc. The point cloud data contains eight initial feature information: x, y, z, r, g, b, I, and C, representing the coordinate values ​​[x, y, z], color values ​​[r, g, b], intensity value I, and point cloud category C, respectively. The point cloud coordinate information, color information, and intensity information serve as the basic feature information of the points for training the point cloud semantic segmentation network model.

[0036] Step 2.2: Rearrange the labeled building point cloud dataset (generally no less than 200 sets) by building and randomly divide it into training set, validation set and test set according to the proportions of 80%, 10% and 10%.

[0037] Step 2.3: Input the training dataset into the PointNet++ network, using the point cloud 3D spatial coordinates, color information, and intensity information as inputs and the building structure category classification as output, to train the building structure semantic segmentation network model.

[0038] Step 2.4: Use the network parameters trained on the training set to validate the point cloud data. Adjust the relevant hyperparameters based on the classification effect on the validation set to optimize and improve the network segmentation effect.

[0039] Step 2.5: Apply the optimized hyperparameter segmentation network to the test set point cloud data to evaluate the generalization ability of the final model. If the segmentation effect is good, apply the segmentation network to the unlabeled single building point cloud set to obtain the segmentation results of each part of the structural point cloud data, segmenting the single building data into roof, facade, door, window, and column structural point cloud sets.

[0040] Step 3: Perform geometric classification on each building structure dataset, and use corresponding algorithms to achieve parametric fitting and 3D reconstruction of each building structure. The specific building category reconstruction methods are as follows:

[0041] Step 3.1: Classify the building structures segmented in Step 2. Doors, windows, and facade structures are classified as planar structure point clouds, column structures are classified as column structure point clouds, and roof point clouds are classified as planar structures or hemispherical structures. Calculate the mean angle of the normals of the roof point clouds. If the mean angle approaches zero, it is classified as a planar structure point cloud; otherwise, it is classified as a hemispherical structure point cloud.

[0042] Step 3.2, in the point cloud model fitting of the cylindrical structure, the parameters of the cylindrical model are represented as [x,y,z,dx,dy,dz,r], where (x,y,z) is a point on the central axis of the cylinder, and (dx,dy,dz) is the direction of the central axis and (dx,y,z) is the direction of the central axis. 2 +dy 2 +dz 2 =1), where r is the radius of the cylinder. The specific method is as follows:

[0043] Step 3.2.1, randomly select three points p on the cylindrical point set. 1 ,p 2 ,p 3 This is used to initialize and determine the central axis and radius. (Starting from point p) 1 ,p 2 ,p 3 The normal vector is perpendicular to the central axis. The direction (dx, dy, dz) of the central axis is calculated using the following formula:

[0044]

[0045] Among them, (n x1 ,n y1 ,n z1 Let p be a point. 1 The coordinate direction components of the normal vector, (n x2 ,n y2 ,n z2 Let p be a point. 2 The coordinate direction components of the normal vector.

[0046] Step 3.2.2: Transform the cylindrical coordinate system so that the central axis vector is parallel to the z-axis, and then transform point p on the cylindrical surface. 1 ,p 2 ,p 3 Projecting onto the xy-plane at this point, the three points are now the coordinates of points on the circle in the plane, as shown by the formula:

[0047] (x1-x0) 2 +(y1-y0) 2 =(x1-x0) 2 +(y1-y0) 2 =(x1-x0) 2 +(y1-y0) 2 =r 2

[0048] Calculate the radius r of the cylindrical model and the coordinates (x0, y0) of the center of the plane circle. Then, obtain the coordinates P(x, y, z) of a point at the axis of the cylinder through the inverse transformation of the original coordinates.

[0049] Step 3.2.3: Run the convex hull algorithm on the point cloud of the column structure to calculate the boundary of the point cloud of the column structure. Based on the boundary of the column and the parameters of the point cloud model of the cylindrical structure, generate the cylindrical structure model and complete the three-dimensional reconstruction of the column structure.

[0050] Step 3.3, the planar structure point cloud model fitting process is divided into the following steps: planar clustering, convex hull algorithm to obtain boundary lines, and planar fitting, specifically:

[0051] Step 3.3.1: Obtain a point P on the point cloud. Search for nearest neighbor data points in the remaining point cloud of the plane using a kd-tree. If the distance between the nearest neighbor points is less than a set threshold, and the angle between the point and the normal vector of P is less than an angle threshold, then classify it as a point in the same plane Q. Then, use the newly obtained point in the same plane as a reference to search for nearby points. If the condition is met, classify it as a point in the same plane. Repeat this process until no more points are added to the plane. Record the points contained in plane Q. Afterward, extract the remaining plane models using non-Q plane points as initial points. After completing the plane clustering, multiple single-plane point cloud data are obtained.

[0052] Step 3.3.2: Extract contour lines from single-plane point cloud data using the convex hull algorithm. The convex hull algorithm first finds the lowest point p0 in the z-axis direction of the point cloud. Then, it calculates the counterclockwise rotation angle between the line connecting p0 and the other points and the x-axis direction. The point p1 with the smallest angle is selected and recorded as the contour point. Then, taking p1 as the starting point, it calculates the counterclockwise rotation angle between the line connecting p1 and the other non-contour points and the direction from p1 to p0. This process is repeated to complete the extraction of planar contour line points.

[0053] Step 3.3.3, the process of obtaining the planar boundary and vertices from the planar contour line points, is as follows: Connect the planar contour points sequentially and calculate the normal vector of each contour line. Based on a contour line, calculate the angle between its normal vector and that of its adjacent contour lines. If the angle is less than a vertical threshold, calculate the sum of the lengths of the two contour lines and continue merging them with adjacent contour lines until the angle is greater than the vertical threshold. Then, merge line segments in the same direction starting from the perpendicular lines. After traversing the planar contour lines, compare the merged length values ​​of each boundary. Select the boundary with the longest length as the principal direction, calculate the average normal vector of the contour lines it contains, and use this as the principal direction normal vector of the plane. Its perpendicular direction is the secondary direction normal vector. Adjust the contour lines accordingly to obtain the planar boundary lines and the planar boundary vertices.

[0054] Step 3.3.4: For a single-plane point cloud, the RANSAC plane fitting algorithm is used to solve the plane model parameters. Three points are randomly selected from the point cloud to form an initial plane. The distance from other points to this plane is calculated. If it is less than the threshold T, it is considered that the point is in the same plane as the above points. If there are more than n points in the same plane, the plane parameters are recorded. Otherwise, the plane model parameters are updated according to the expanded plane points to obtain a more accurate plane model.

[0055] Step 3.3.5: Using the plane fitted in Step 3.3.4 and the boundary point contour lines extracted in Step 3.3.2, calculate the normal difference and line spacing between different planes. Set a threshold θ for the normal difference and a threshold γ for the line spacing. If the normal difference is less than the threshold θ and the line spacing is less than the spacing threshold γ, then the two related planes intersect; otherwise, they are non-intersecting planes. Calculate the intersection direction vector using the equations of the two related planes. And a point X = (x, y, z) on the intersection line, to obtain the parametric model of the plane intersection line. Calculate all intersection lines between planes and record them as inter-plane relationships.

[0056] Step 3.4, in the fitting of the spherical point cloud model, the parameters of the spherical model are represented as [x0, y0, z0, r], where (x0, y0, z0) represents the center of the sphere, and r is the radius of the spherical point. The Random Sample Consensus Algorithm (RANSAC) is used to fit the spherical model. The algorithm process is as follows:

[0057] Step 3.4.1: Select the minimum number of facet elements from the input point cloud data that can form a spherical model as the in-model points. Use these selected facets to calculate the formula using the spherical mathematical model:

[0058] (x-x0) 2 +(y-y0) 2 +(z-z0) 2 =r 2

[0059] Calculate the initial parameters of the spherical model: sphere radius r, sphere center (x0, y0, z0).

[0060] Step 3.4.2: Check the remaining points. If they conform to the calculated spherical model, classify them as in-model points. Then, use the obtained in-model points to re-estimate the model. Repeat the above process multiple times to estimate the fitted model. If the final number of in-model points is less than the model before the update, discard the model and re-execute the process. If the model has more in-model points, update it to the fitted spherical model.

[0061] Step 4: Based on the prior information of the building structure, construct the topological relationship map of the building structure. Then, combine the structural models of each part of the building fitted in Step 3 with the obtained structural outlines and boundary points of the building parts, and stitch together the structural models of each part of the building to obtain a complete three-dimensional model of the building.

[0062] Step 4.1: Use the ground coordinate system as the reference coordinate system for the building, and take the ground point projected from the centroid of the building point cloud as the origin. The z-axis of the coordinate system is perpendicular to the ground, the x-axis is the main direction of the building structure facade, and the y-axis is the secondary direction of the building perpendicular to the x-axis.

[0063] Step 4.2: Based on the prior information of the building structure, construct the building structure topology diagram as follows: Figure 6 As shown, the relevant relationships include: the facade is perpendicular to or parallel to the other facades at the same height; the top surface intersects with the other top surfaces at the same height; the doors, windows, columns and other building parts are parallel to each other of the same category at the same height; the top surface intersects with or is perpendicular to the facade; and doors and windows are embedded in the facade.

[0064] Step 4.3: Combining the architectural outline and boundary points obtained in Step 3, and using the architectural structural relationship map as a reference, calculate the distance and normal difference between the corner points or boundaries of the architectural structures. If both the distance and normal information meet the threshold conditions, the corner point or boundary is used as the common point or edge for splicing between structures, and the various structural models of a single building are spliced ​​in this way.

[0065] Step 4.4: Combining the structural model parameters of each part of the building fitted in Step 3 with the common corner information of the structural model splicing, construct a parametric 3D model of the single building as a whole, and finally obtain a complete 3D model of the single building as shown below. Figure 6 As shown.

[0066] Compared with the prior art, the beneficial effects of the present invention are: the present invention can realize the three-dimensional reconstruction of point cloud data into a building structure primitive model containing semantic information when dealing with the problem of semantic reconstruction of individual buildings, and has corresponding improvements in terms of segmentation effect, reconstruction accuracy and labor cost. Attached Figure Description

[0067] Figure 1This is a flowchart of the segmentation and reconstruction implementation steps of the present invention;

[0068] Figure 2 This is the input point cloud data for building reconstruction in this invention;

[0069] Figure 3 This refers to the point cloud data after point cloud preprocessing according to the present invention;

[0070] Figure 4 These are the reconstruction results of each part (roof, facade, window, door, column) after semantic segmentation in this invention;

[0071] Figure 5 This is a topological diagram showing the contextual splicing relationships of various parts of this invention;

[0072] Figure 6 This is the semantic segmentation and reconstruction result of a single building achieved by the present invention. Detailed Implementation

[0073] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0074] This invention is a method for reconstructing a single building based on semantic segmentation. The reconstruction process is as follows: Figure 1 As shown, it includes the following steps:

[0075] Step 1, laser scanning data acquisition point cloud information such as Figure 2 As shown, statistical filtering is applied to the point cloud dataset to remove outliers and noise. Then, a feature point extraction algorithm based on normal vectors and an improved voxel conditional filtering point cloud downsampling algorithm are used to obtain point cloud feature points and downsampled sampling points. These are then concatenated and deduplicated to obtain the sampled and denoised single-building dataset, as shown below. Figure 3 As shown, the specific steps are as follows:

[0076] Step 1.1: Process the single building point cloud dataset using a statistical filter to remove outlier noise points. Specifically: Search all neighborhood points of each point using a kd-tree and calculate the X value for that point. i =(x i ,y i ,z i To the domain point X j =(x j ,y j ,z j Distance d ij Where i = [1,…,m] represents m search reference points, j = [1,…,k] represents the k nearest neighbor points to the reference points, and the nearest neighbor set X k ={x 1 ,x 2 …,x kNext, calculate the mean μ and standard deviation σ of the distance from the point to the neighboring points. The calculation method is as follows:

[0077]

[0078]

[0079] Therefore, the Gaussian distribution d ~ N(μ,σ) that the points in the point cloud conform to can be obtained. Point cloud noise filtering is achieved by eliminating point sets with excessively large mean distances from neighboring points within the Gaussian distribution confidence interval. The elimination condition formula is as follows:

[0080] or

[0081] Step 1.2: Before downsampling the data, based on the denoised point cloud data, extract key point information from the point cloud of a single building, such as boundary points, corner points, and intersection points between surfaces, to ensure that key information in the point cloud structure is preserved. Specifically, key point algorithms such as Harris3D and ISS can be used. The specific steps using the Harris3D feature point algorithm are as follows:

[0082] Step 1.2.1: Calculate the normal information for each point. The calculation process for normal information is as follows: Select the nearest points to the solution point, and use the covariance matrix calculation formula... Calculate each point p i The covariance matrix C of its neighboring points, where Let k be the centroid of the set of points between the point and its nearest neighbors, and k be the number of nearest neighbors. Then, calculate the normal axis of the point using PCA principal component analysis. Finally, determine the normal direction using the viewpoint and calculate the coordinate system component vectors of the normal.

[0083] Step 1.2.2, based on the 2D Harris corner detection corner response value calculation formula, is applied to construct the normal correlation matrix and corner response values ​​of each point in the 3D point cloud. The calculation formula for the 3D point cloud corner response value is as follows:

[0084] R 3D =det(M 3D -0.04[trace(M 3D )] 2

[0085]

[0086] Where n is the number of neighboring points of the current point, n x n represents the X-axis direction component of the normal vector of a point calculated from its neighboring points. y n z These are the Y-axis and Z-axis vectors, respectively.

[0087] Step 1.2.3: Determine whether a point cloud corner or key point is determined based on the corner response threshold. If the response is greater than the threshold, it is a key point; otherwise, it is not a key point.

[0088] Step 1.2.4: Calculate the distances from data points in the original point cloud to each keypoint. If the distance of a data point to all keypoints is greater than a threshold, save that point as a keypoint to retain more point cloud structure information. Based on the supplemented set of keypoints, iteratively search for the remaining keypoint data. Obtain all keypoint data in the point cloud.

[0089] Step 1.3: Use the improved voxel conditional filtering algorithm to downsample the point cloud data, specifically as follows:

[0090] Step 1.3.1: Set the voxel grid resolution L. The voxel grid divides the point cloud data into several point cloud blocks. Calculate the centroid P of each point cloud block. After unifying the point mass, calculate the average coordinate of the point cloud in the voxel grid block, which is the centroid of the voxel block.

[0091] Step 1.3.2: Calculate the distance from the center of mass of each voxel block to the centroid, select the nearest point as the voxel center Q, and ensure that the voxel centroid is the actual point in the point cloud without changing the original point cloud data.

[0092] Step 1.3.3: Find the number of voxel grid points where the voxel center point Q is located. If it is lower than the voxel point cloud number threshold, delete the voxel center point Q to ensure that noise points are not retained as voxel sampling points.

[0093] Step 1.4: Merge the downsampled point cloud data with the Harris keypoint data, remove duplicate point sets, and perform point cloud data downsampling preprocessing.

[0094] Step 1.5: Based on the principle of the same dimension as the input point cloud for point cloud semantic segmentation, random sampling of a fixed number of points is performed on the point cloud data to sample the point cloud data of a single building to the same dimension, thus obtaining the building point cloud dataset. This completes the entire process of point cloud dataset preprocessing, resulting in point cloud data of a single building after retaining feature points and denoising and downsampling.

[0095] Step 2: Segment and label the single building point cloud dataset obtained after denoising and downsampling in Step 1 to obtain a single building structure segmentation data training set and a test set. Train the PointNet++ semantic segmentation network based on the single building semantic segmentation training set data with labeled information. Use a portion of the dataset as a validation set and test set to adjust parameters and verify the training effect. Apply the trained semantic segmentation network to the single building point cloud segmentation task to segment the unlabeled point cloud and obtain the structured segmented point cloud data of the building.

[0096] Step 2.1: Select the point cloud data of a single building acquired by the laser scanner, divide it into blocks according to the building structure, and label the structural categories, specifically including facade, roof, doors, windows, columns, etc. The point cloud data contains eight initial feature information: x, y, z, r, g, b, I, and C, representing the coordinate values ​​[x, y, z], color values ​​[r, g, b], intensity value I, and point cloud category C, respectively. The point cloud coordinate information, color information, and intensity information serve as the basic feature information of the points for training the point cloud semantic segmentation network model.

[0097] Step 2.2: Rearrange the labeled building point cloud dataset (generally no less than 200 sets) by building and randomly divide it into training set, validation set and test set according to the proportions of 80%, 10% and 10%.

[0098] Step 2.3: Input the training dataset into the PointNet++ network, using the point cloud 3D spatial coordinates, color information, and intensity information as inputs and the building structure category classification as output, to train the building structure semantic segmentation network model.

[0099] Step 2.4: Use the network parameters trained on the training set to validate the point cloud data. Adjust the relevant hyperparameters based on the classification effect on the validation set to optimize and improve the network segmentation effect.

[0100] Step 2.5: Apply the optimized hyperparameter segmentation network to the test set point cloud data to evaluate the generalization ability of the final model. If the segmentation effect is good, apply the segmentation network to the unlabeled single building point cloud set to obtain the segmentation results of each part of the structural point cloud data, segmenting the single building data into roof, facade, door, window, and column structural point cloud sets.

[0101] Step 3: Perform geometric classification on each building structure dataset, and use corresponding algorithms to achieve parametric fitting and 3D reconstruction of each building structure. The specific building category reconstruction methods are as follows:

[0102] Step 3.1: Classify the building structures segmented in Step 2. Doors, windows, and facade structures are classified as planar structure point clouds, column structures are classified as column structure point clouds, and roof point clouds are classified as planar structures or hemispherical structures. Calculate the mean angle of the normals of the roof point clouds. If the mean angle approaches zero, it is classified as a planar structure point cloud; otherwise, it is classified as a hemispherical structure point cloud.

[0103] Step 3.2, in the point cloud model fitting of the cylindrical structure, the parameters of the cylindrical model are represented as [x,y,z,dx,dy,dz,r], where (x,y,z) is a point on the central axis of the cylinder, and (dx,dy,dz) is the direction of the central axis and (dx,y,z) is the direction of the central axis. 2 +dy 2 +dz2 =1), where r is the radius of the cylinder. The specific method is as follows:

[0104] Step 3.2.1, randomly select three points p on the cylindrical point set. 1 ,p 2 ,p 3 This is used to initialize and determine the central axis and radius. (Starting from point p) 1 ,p 2 ,p 3 The normal vector is perpendicular to the central axis. The direction (dx, dy, dz) of the central axis is calculated using the following formula:

[0105]

[0106] Among them, (n x1 ,n y1 ,n z1 Let p be a point. 1 The coordinate direction components of the normal vector, (n x2 ,n y2 ,n z2 Let p be a point. 2 The coordinate direction components of the normal vector.

[0107] Step 3.2.2: Transform the cylindrical coordinate system so that the central axis vector is parallel to the z-axis, and then transform point p on the cylindrical surface. 1 ,p 2 ,p 3 Projecting onto the xy-plane at this point, the three points are now the coordinates of points on the circle in the plane, as shown by the formula:

[0108] (x1-x0) 2 +(y1-y0) 2 =(x1-x0) 2 +(y1-y0) 2 =(x1-x0) 2 +(y1-y0) 2 =r 2

[0109] Calculate the radius r of the cylindrical model and the coordinates (x0, y0) of the center of the plane circle. Then, obtain the coordinates P(x, y, z) of a point at the axis of the cylinder through the inverse transformation of the original coordinates.

[0110] Step 3.2.3: Run the convex hull algorithm on the point cloud of the column structure to calculate the boundary of the point cloud of the column structure. Based on the boundary of the column and the parameters of the point cloud model of the cylindrical structure, generate the cylindrical structure model and complete the three-dimensional reconstruction of the column structure.

[0111] Step 3.3, the planar structure point cloud model fitting process is divided into the following steps: planar clustering, convex hull algorithm to obtain boundary lines, and planar fitting, specifically:

[0112] Step 3.3.1: Obtain a point P on the point cloud. Search for nearest neighbor data points in the remaining point cloud of the plane using a kd-tree. If the distance between the nearest neighbor points is less than a set threshold, and the angle between the point and the normal vector of P is less than an angle threshold, then classify it as a point in the same plane Q. Then, use the newly obtained point in the same plane as a reference to search for nearby points. If the condition is met, classify it as a point in the same plane. Repeat this process until no more points are added to the plane. Record the points contained in plane Q. Afterward, extract the remaining plane models using non-Q plane points as initial points. After completing the plane clustering, multiple single-plane point cloud data are obtained.

[0113] Step 3.3.2: Extract contour lines from single-plane point cloud data using the convex hull algorithm. The convex hull algorithm first finds the lowest point p0 in the z-axis direction of the point cloud. Then, it calculates the counterclockwise rotation angle between the line connecting p0 and the other points and the x-axis direction. The point p1 with the smallest angle is selected and recorded as the contour point. Then, taking p1 as the starting point, it calculates the counterclockwise rotation angle between the line connecting p1 and the other non-contour points and the direction from p1 to p0. This process is repeated to complete the extraction of planar contour line points.

[0114] Step 3.3.3, the process of obtaining the planar boundary and vertices from the planar contour line points, is as follows: Connect the planar contour points sequentially and calculate the normal vector of each contour line. Based on a contour line, calculate the angle between its normal vector and that of its adjacent contour lines. If the angle is less than a vertical threshold, calculate the sum of the lengths of the two contour lines and continue merging them with adjacent contour lines until the angle is greater than the vertical threshold. Then, merge line segments in the same direction starting from the perpendicular lines. After traversing the planar contour lines, compare the merged length values ​​of each boundary. Select the boundary with the longest length as the principal direction, calculate the average normal vector of the contour lines it contains, and use this as the principal direction normal vector of the plane. Its perpendicular direction is the secondary direction normal vector. Adjust the contour lines accordingly to obtain the planar boundary lines and the planar boundary vertices.

[0115] Step 3.3.4: For a single-plane point cloud, the RANSAC plane fitting algorithm is used to solve the plane model parameters. Three points are randomly selected from the point cloud to form an initial plane. The distance from other points to this plane is calculated. If it is less than the threshold T, it is considered that the point is in the same plane as the above points. If there are more than n points in the same plane, the plane parameters are recorded. Otherwise, the plane model parameters are updated according to the expanded plane points to obtain a more accurate plane model.

[0116] Step 3.3.5: Using the plane fitted in Step 3.3.4 and the boundary point contour lines extracted in Step 3.3.2, calculate the normal difference and line spacing between different planes. Set a threshold θ for the normal difference and a threshold γ for the line spacing. If the normal difference is less than the threshold θ and the line spacing is less than the spacing threshold γ, then the two related planes intersect; otherwise, they are non-intersecting planes. Calculate the intersection direction vector using the equations of the two related planes. And a point X = (x, y, z) on the intersection line, to obtain the parametric model of the plane intersection line. Calculate all intersection lines between planes and record them as inter-plane relationships.

[0117] Step 3.4, in the fitting of the spherical point cloud model, the parameters of the spherical model are represented as [x0, y0, z0, r], where (x0, y0, z0) represents the center of the sphere, and r is the radius of the spherical point. The Random Sample Consensus Algorithm (RANSAC) is used to fit the spherical model. The algorithm process is as follows:

[0118] Step 3.4.1: Select the minimum number of facet elements from the input point cloud data that can form a spherical model as the in-model points. Use these selected facets to calculate the formula using the spherical mathematical model:

[0119] (x-x0) 2 +(y-y0) 2 +(z-z0) 2 =r 2

[0120] Calculate the initial parameters of the spherical model: sphere radius r, sphere center (x0, y0, z0).

[0121] Step 3.4.2: Check the remaining points. If they conform to the calculated spherical model, classify them as in-model points. Then, use the obtained in-model points to re-estimate the model. Repeat the above process multiple times to estimate the fitted model. If the final number of in-model points is less than the model before the update, discard the model and re-execute the process. If the model has more in-model points, update it to the fitted spherical model.

[0122] Step 4: Based on the prior information of the building structure, construct the topological relationship map of the building structure. Then, combine the structural models of each part of the building fitted in Step 3 with the obtained structural outlines and boundary points of the building parts, and stitch together the structural models of each part of the building to obtain a complete three-dimensional model of the building.

[0123] Step 4.1: Use the ground coordinate system as the reference coordinate system for the building, and take the ground point projected from the centroid of the building point cloud as the origin. The z-axis of the coordinate system is perpendicular to the ground, the x-axis is the main direction of the building structure facade, and the y-axis is the secondary direction of the building perpendicular to the x-axis.

[0124] Step 4.2: Based on the prior information of the building structure, construct the building structure topology diagram as follows: Figure 5 As shown, the relevant relationships include: the facade is perpendicular to or parallel to the other facades at the same height; the top surface intersects with the other top surfaces at the same height; the doors, windows, columns and other building parts are parallel to each other of the same category at the same height; the top surface intersects with or is perpendicular to the facade; and doors and windows are embedded in the facade.

[0125] Step 4.3: Combining the architectural outline and boundary points obtained in Step 3, and using the architectural structural relationship map as a reference, calculate the distance and normal difference between the corner points or boundaries of the architectural structures. If both the distance and normal information meet the threshold conditions, the corner point or boundary is used as the common point or edge for splicing between structures, and the various structural models of a single building are spliced ​​in this way.

[0126] Step 4.4: Combining the structural model parameters of each part of the building fitted in Step 3 with the common corner information of the structural model splicing, construct a parametric 3D model of the single building as a whole, and finally obtain a complete 3D model of the single building as shown below. Figure 6 As shown.

Claims

1. A method for 3D reconstruction of a single building based on point cloud semantic segmentation and structural fitting, characterized in that, The steps are as follows: Step 1: Preprocessing of the building target point cloud dataset. First, outlier noise is removed using a statistical filter. Second, while ensuring the preservation of point cloud features, data redundancy is reduced. Key points in the point cloud and building downsampling sampling points are obtained by using a key point extraction algorithm based on Harris3D corner detection algorithm and an improved voxel conditional filtering point cloud downsampling algorithm. These are combined and deduplicated to obtain a sampled and denoised single building dataset. Then, a random sampling algorithm is used to randomly sample a fixed number of points in a single building to ensure that the single building point cloud dataset input to the semantic segmentation network has the same data dimension. Step 2: Train the building point cloud semantic segmentation network to classify building structure data, and obtain training set, validation set and test set of single building structure segmentation data. Input the obtained single building point cloud data training set with annotation information into the PointNet++ network to train the building structure semantic segmentation network model. Then, input the validation set point cloud data into the network to adjust the network hyperparameters and improve the network segmentation effect. Finally, input the obtained test set building structure point cloud into the trained segmentation network model for semantic segmentation to obtain the building structure point cloud datasets of each part, including the roof, facade, door, window and column building structure point cloud sets. Step 3: Geometric classification and 3D parametric fitting of building structures. Based on the building structure classification results completed by the segmentation network, and combined with the planar, cylindrical, and spherical geometric primitive characteristics of the building structure categories of facade, roof, door, window, and column, the various building structure datasets are geometrically discriminated and classified. Step 4: Construction of the topological relationship of the building structure prior information and splicing of the individual building structure. Based on the building structure prior information, a topological relationship map of the building structure is constructed. Then, combined with the structural models of each part of the building fitted in Step 3 and the obtained structural outlines and boundary points of the building parts, the structural models of each part of the building are spliced ​​together to obtain a complete 3D model of the individual building.

2. The method for 3D reconstruction of a single building based on point cloud semantic segmentation and structural fitting according to claim 1, characterized in that, Step one specifically includes: Step 1.1: Process the single building point cloud dataset using a statistical filter to remove outlier noise points, specifically as follows: For each point, search all neighborhood points using a kd-tree and calculate the value of that point. To the field point distance ,in Let m represent the reference points for the search. The nearest neighbor set represents the k nearest neighboring points to the reference point. Secondly, calculate the average distance from the point to the neighboring points. and standard deviation The calculation method is as follows: Therefore, we can obtain the Gaussian distribution that corresponds to the midpoints of the point cloud. Point cloud noise filtering is achieved by removing point sets with excessively large mean distances from neighboring points using Gaussian distribution within the signal interval. The removal condition formula is as follows: Step 1.2: Before data downsampling, based on the denoised point cloud data, key point information is extracted from the point cloud of a single building. The specific steps of the Harris 3D feature point algorithm are as follows: Step 1.2.1: Calculate the normal information for each point. The calculation process for normal information is as follows: Select the nearest points to the solution point, and use the covariance matrix calculation formula... Calculate each point covariance matrix of its neighboring points ,in Let k be the centroid of the set of points between the point and its nearest neighbors, and k be the number of nearest neighbors. Then, calculate the axis of the normal to the point using PCA principal component analysis. Finally, determine the direction of the normal by the viewpoint and calculate the coordinate system component vectors of the normal. Step 1.2.2: Based on the 2D Harris corner detection corner response value calculation formula, this is applied to construct the normal correlation matrix and corner response values ​​of each point in the 3D point cloud; the calculation formula for the 3D point cloud corner response values ​​is as follows: Where n is the number of neighboring points of the current point. This represents the X-axis direction component of the normal vector of a point calculated from its neighboring points. These are the Y-axis and Z-axis vectors, respectively; Step 1.2.3: Determine whether a point cloud corner or key point is determined based on the corner response threshold. If the response is greater than the threshold, it is a key point; otherwise, it is not a key point. Step 1.2.4: Calculate the distance between data points in the original point cloud and each key point. If the distance between a data point and all key points is greater than the threshold, save the point as a key point to retain more point cloud structure information. Based on the supplemented set of key points, iteratively search for the remaining key point data to obtain all key point data in the point cloud. Step 1.3: Use the improved voxel conditional filtering algorithm to downsample the point cloud data, specifically as follows: Step 1.3.1: Set the voxel grid resolution L. The voxel grid divides the point cloud data into several point cloud blocks. Calculate the centroid P of each point cloud block. After unifying the point mass, calculate the average coordinate of the point cloud in the voxel grid block, which is the centroid of the voxel block. Step 1.3.2: Calculate the distance from the center of mass of each voxel block to the centroid, select the nearest point as the voxel center Q, and ensure that the voxel centroid is the actual point in the point cloud without changing the original point cloud data; Step 1.3.3: Find the number of voxel grid points where the voxel center point Q is located. If it is lower than the voxel point cloud number threshold, delete the voxel center point Q to ensure that noise points are not retained as voxel sampling points. Step 1.4: Merge the downsampled point cloud data with the Harris keypoint data, remove duplicate point sets, and perform downsampling preprocessing of the point cloud data. Step 1.5: Based on the principle of the same dimension as the input point cloud for point cloud semantic segmentation, a fixed number of points are randomly sampled from the point cloud data to sample the point cloud data of a single building to the same dimension, thus obtaining the building point cloud dataset. This completes the entire process of point cloud dataset preprocessing, resulting in point cloud data of a single building after retaining feature points and denoising and downsampling.

3. The method for 3D reconstruction of a single building based on point cloud semantic segmentation and structural fitting according to claim 1, characterized in that, Step two specifically includes: Step 2.1: Select the point cloud data of a single building acquired by the laser scanner, divide it into blocks according to the building structure, and label the structural parts, specifically including the facade, roof, doors, windows, and columns; the point cloud data contains 8 initial feature information. , representing the coordinates [x, y, z], color value [r, g, b], and intensity value of a point in the point cloud data, respectively. and point cloud categories Point cloud coordinate information, color information, and intensity information are used as the basic feature information of points to train the point cloud semantic segmentation network model; Step 2.2: Rearrange the labeled building point cloud dataset by building and randomly divide it into training set, validation set and test set according to the ratio of 80%, 10% and 10% respectively. Step 2.3: Input the training dataset into the PointNet++ network, using the point cloud 3D spatial coordinates, color information, and intensity information as inputs and the building structure category classification as output, to train the building structure semantic segmentation network model; Step 2.4: Use the network parameters trained on the training set to test the validation set point cloud data. Adjust the relevant hyperparameters based on the classification effect on the validation set to optimize and improve the network segmentation effect. Step 2.5: Apply the optimized hyperparameter segmentation network to the test set point cloud data to evaluate the generalization ability of the final model. If the segmentation effect is good, apply the segmentation network to the unlabeled single building point cloud set to obtain the segmentation results of each part of the structural point cloud data, and segment the single building data into the roof, facade, door, window and column building structural point cloud sets.

4. The method for 3D reconstruction of a single building based on point cloud semantic segmentation and structural fitting according to claim 1, characterized in that, Step three specifically includes: Step 3.1: Classify the building structures segmented in Step 2. Door, window, and facade structures are classified as planar structure point cloud, column structures are classified as column structure point cloud, and roof point clouds are classified as planar structure or hemispherical structure. Calculate the mean of the included angle of the roof point cloud normal. If the mean of the included angle approaches zero, it is classified as planar structure point cloud; otherwise, it is classified as hemispherical structure point cloud. Step 3.2, in the point cloud model fitting of the cylindrical structure, the parameters of the cylindrical model are expressed as follows: ,in( () is a point on the central axis of the cylinder, ), is the direction of the central axis and ( , The radius of the cylinder; the specific method is as follows: Step 3.2.1: Randomly select three points on the cylindrical point set. Used to initialize and determine the central axis and radius; by point The normal vector is perpendicular to the central axis. The direction of the central axis is calculated. The calculation formula is: in,( ) is a point The coordinate direction components of the normal vector, ( ) is a point The coordinate direction components of the normal vector; Step 3.2.2: Transform the cylindrical coordinate system so that the central axis vector is parallel to the z-axis, and then transform the points on the cylindrical surface. Projecting onto the xy-plane at this point, the three points are now the coordinates of points on the circle in the plane, as shown by the formula: Calculate the radius of the cylindrical model and the coordinates of the center of the circle in the plane ( Then, the coordinates P of a certain point at the axis of the cylinder are obtained through the inverse transformation of the original coordinates. ); Step 3.2.3: Run the convex hull algorithm on the point cloud of the column structure to calculate the boundary of the point cloud of the column structure. Based on the boundary of the column and the parameters of the point cloud model of the cylindrical structure, generate the cylindrical structure model and complete the three-dimensional reconstruction of the column structure. Step 3.3, the planar structure point cloud model fitting process is divided into the following steps: planar clustering, convex hull algorithm to obtain boundary lines, and planar fitting, specifically: Step 3.3.1: Obtain a point P on the point cloud. Search for the nearest neighbor data points of the remaining point clouds in the plane using the kd tree. If the distance between the nearest points is less than a set threshold and the angle between the point and the normal vector of P is less than the angle threshold, then classify it as a point in the same plane Q. Then, use the newly obtained point in the same plane as the reference to search for nearby points. If the condition is met, classify it as a point in the same plane. Repeat the process until the number of points in the plane no longer increases. Record the points contained in the plane Q. After that, use the non-Q plane points as the initial points to extract the remaining plane models. After completing the plane clustering, multiple single-plane point cloud data are obtained. Step 3.3.2: Extract the contour line from the single-plane point cloud data using the convex hull algorithm. The convex hull algorithm first finds the lowest point in the z-axis direction of the point cloud. Next, calculate the counterclockwise rotation angle between the line connecting this point and the other points and the x-axis, and select the point with the smallest angle. Record them as contour points, and then, with Starting from point, calculate The line connecting the other non-contour points and The counterclockwise rotation angle of the direction is used to extract the points of the planar contour line; Step 3.3.3, the process of obtaining the plane boundary and vertices from the plane contour line points is as follows: connect the plane contour points sequentially and calculate the normal vector of each contour line. Based on a contour line, calculate the angle between the normal vectors of its adjacent contour lines. If the angle is less than the vertical threshold, calculate the sum of the lengths of the two contour lines and continue to calculate and merge with adjacent contour lines until the angle is greater than the vertical threshold. Then, merge line segments in the same direction starting from the perpendicular lines. After traversing the plane contour lines, compare the merged length values ​​of each boundary, select the boundary with the longest length as the main direction, calculate the average normal value of the contour lines contained in it, and use it as the plane main direction normal vector; its vertical direction is the secondary direction normal vector. Adjust the contour lines in this way to obtain the plane boundary line and plane boundary vertices. Step 3.3.4: For a single-plane point cloud, the RANSAC plane fitting algorithm is used to solve the plane model parameters. Three points are randomly selected from the point cloud to form an initial plane. The distance from other points to the plane is calculated. If it is less than the threshold T, it is considered that the point is in the same plane as the above points. If there are more than n points in the same plane, the plane parameters are recorded. Otherwise, the plane model parameters are updated according to the expanded plane points to obtain a more accurate plane model. Step 3.3.5: Using the plane fitted in Step 3.3.4 and the boundary point contour lines extracted in Step 3.3.2, calculate the normal difference and line spacing between the contour lines of different planes, and set a threshold for the normal difference. and line spacing threshold If the normal difference is less than the threshold And the line spacing is less than the spacing threshold. If the two related planes intersect, they are considered intersecting; otherwise, they are considered non-intersecting. The direction vector of the intersection line is calculated using the equations of the two related planes. and a point on the intersection line The plane intersection parameter model is obtained; all inter-plane intersection lines are calculated and recorded as inter-plane relationships; Step 3.4, in the fitting of the spherical structure point cloud model, the parameters of the spherical model are expressed as follows: ,in( ) represents the center of the sphere. Let be the radius of the sphere; the Random Sampling Consensus Algorithm (RANSAC) is used to fit the spherical model; the algorithm process is as follows: Step 3.4.1: Select the minimum number of facet elements from the input point cloud data that can form a spherical model as the in-model points. Use these selected facets to calculate the formula using the spherical mathematical model: Calculate the initial parameters of the spherical model: sphere radius Sphere center ( ); Step 3.4.2: Check the remaining points. If they conform to the calculated spherical model, classify them as in-model points. Then, use the obtained in-model points to re-estimate the model. Repeat the above process multiple times to estimate the fitted model. If the final number of in-model points is less than the model before the update, discard the model and re-execute the process. If the model has more in-model points, update it to the fitted spherical model.

5. The method for 3D reconstruction of a single building based on point cloud semantic segmentation and structural fitting according to claim 1, characterized in that, Step four specifically includes: Step 4.1: Use the ground coordinate system as the reference coordinate system for the building, and take the ground point projected from the centroid of the building point cloud as the origin. The z-axis of the coordinate system is perpendicular to the ground, the x-axis is the main direction of the building structure facade, and the y-axis is the secondary direction of the building perpendicular to the x-axis. Step 4.2: Based on the prior information of the building structure, construct the topological relationship map of the building structure, where the relevant relationships include: the facade is perpendicular to and at the same height or parallel to the other facades, the top surface intersects with the other top surfaces and has the same bottom edge height, the door, window, column and building parts are parallel to each other of the same category and at the same height, the top surface intersects with or is perpendicular to the facade, and the doors and windows are embedded in the facade. Step 4.3: Combining the architectural outline and boundary points obtained in Step 3, and using the architectural structure relationship map as a reference, calculate the distance and normal difference between the corner points or boundaries of the architectural structures. If both the distance and normal information meet the threshold conditions, use the corner point or boundary as the common point or edge for splicing between structures, and splice the various structural models of a single building in this way. Step 4.4: Combining the structural model parameters of each part of the building fitted in Step 3 with the common corner information of the structural model splicing, construct the overall parametric 3D model of the single building, and finally obtain the complete 3D model of the single building.

Images