A method for building reconstruction from oblique photography point cloud based on deep neural network

Abstract

The present invention discloses a method for building reconstruction based on oblique photography point cloud based on deep neural network, which relates to the field of image data processing technology, including: constructing a single building point cloud data set, a building surface extraction network M1 and a loss function L; training M1 to obtain a building surface extraction model M2; for the single building point cloud to be reconstructed, using the RANSAC algorithm to obtain a point cloud set, and all point cloud sets constitute a set X1; using M2 to predict the predicted SDF value of each point in X1 to obtain a set X2; based on X2, performing surface reconstruction on the single building to be reconstructed. The present invention proposes an integrated, fully automatic building surface extraction and reconstruction method, which does not require manual participation and is based on reconstruction of fewer points, which can not only improve the reconstruction efficiency, but also simplify the number of planes as much as possible, so that the building representation is more concise, occupies less storage space, and has less data volume, thereby reducing the computational complexity.

Description

Technical Field

[0001] The present invention relates to the technical field of building reconstruction, and in particular to a method for reconstructing buildings from oblique photography point clouds based on a deep neural network. Background Art

[0002] At present, as a new type of geospatial data, matching point cloud data obtained based on oblique images from drones has been widely used in urban planning, urban emergency response, GIS services and other fields. However, in the current urban application scenarios, the three-dimensional modeling of a single building usually uses commercial modeling software to generate urban models. Such models are mostly grid models, commonly known as "a skin", that is, it is impossible to distinguish separate buildings or form a building structure composed of multiple faces. If you want to achieve refined modeling for a specific building, professionals are required to manually outline the building from the point cloud data and extract the CAD framework of the building and complete the reconstruction. In addition, some solutions are based on the classic Poisson surface reconstruction algorithm and its improved method, which combines the input unstructured point cloud with its normal information to complete the surface reconstruction. The reconstruction result consists of several triangle meshes.

[0003] However, the above methods have the following problems: 1. Existing point cloud building modeling methods are usually inefficient and have a high degree of manual intervention. If you need to model a single building in an urban scene, you need to first extract a single building from a large-scale point cloud, and then manually extract the outline of the building and form a structured model for the building. This method usually requires a lot of manpower. For the needs of batch building modeling, this modeling method is often very inefficient and slow to meet the needs of large-scale applications. 2. The reconstruction results based on classic reconstruction algorithms such as Poisson surface reconstruction or its improved algorithms are often composed of a large number of triangular facets. These triangular facets are difficult to effectively express the real surface of the building, and the reconstruction results are often rough, especially for buildings. It is difficult to form a compact and sealed model. Therefore, how to achieve automated and high-precision reconstruction of individual point cloud buildings has become an urgent need for urban planning, monitoring and management.

[0004] The RANSAC (Random sample consensus) algorithm, also known as the random sampling consensus algorithm, is a robust model fitting method that can handle data with a lot of noise and outliers. When fitting a plane, RANSAC randomly selects three points and then calculates the plane model determined by these three points. Then, RANSAC calculates the distance from all other points to the plane and determines whether these points conform to the plane model based on a preset threshold. This process is repeated many times, and finally the plane model with the most conforming points is selected as the final result.

[0005] Signed Distance Function (SDF): is a mathematical function that assigns a signed distance value to each point in space, representing the directed distance from the point to the nearest surface. The positive or negative sign indicates whether the point is inside or outside the surface, and the size of the distance indicates the distance from the point to the surface. In the context of point cloud 3D reconstruction, SDF is often used to represent implicit surface functions, which defines the surface as the zero set of SDF. Using deep learning to build a multi-layer perceptron to learn SDF, compared with explicit surface representations such as triangular meshes or point clouds, SDF can represent complex shapes more smoothly and the representation results are more compact, which meets the purpose of building reconstruction. Summary of the invention

[0006] The purpose of the present invention is to provide a method for reconstructing point cloud buildings using oblique photography based on deep neural networks, which can solve the above-mentioned problems and realize automated and high-precision reconstruction of individual point cloud buildings.

[0007] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is as follows: a method for reconstructing buildings from oblique photography point clouds based on a deep neural network, comprising the following steps;

[0008] S1, select a single building point cloud dataset, including multiple samples, each sample includes a point cloud set of a single building, and each point in the point cloud is marked with a three-dimensional coordinate and an SDF value, the SDF value includes a sign and a distance, the sign includes positive and negative, and the distance is the real distance from the point to the real plane to which it belongs;

[0009] S2, construct building surface extraction network M1;

[0010] The M1 includes a PointNet network, a first splicing layer, a second splicing layer and an MLP network;

[0011] The PointNet network is used to input a sample point cloud set P and generate a 256-dimensional implicit vector for each point;

[0012] The first splicing layer is used to splice the three-dimensional coordinates of the point and its implicit vector into a 259-dimensional first splicing vector;

[0013] The MLP includes N fully connected layers FC1~FC N and a nonlinear activation function layer, where the nth fully connected layer FC n The number of input nodes is 512, the number of output nodes is 253, and the number of input nodes and output nodes of the remaining fully connected layers are both 512, N ≥ 8 and N is an even number, n = N / 2; the first concatenation vector is sequentially passed through FC1~FC nThe intermediate vector of 253 dimensions is output, and the intermediate vector and the first concatenated vector are concatenated into a second concatenated vector of 512 dimensions by the second concatenated layer. The second concatenated vector is then concatenated into a second concatenated vector of 512 dimensions by the FC layer. n+1 ~FC N , and then the predicted SDF value is obtained after the nonlinear activation function layer;

[0014] S3, construct loss function L;

[0015] ,

[0016] ,

[0017] ,

[0018] Where, L r is the regression loss, L s is the cross entropy loss, is the L2 norm, p is a point in the point cloud set P, s is the real plane to which p belongs, d(p,s) is the real distance from p to s, and f o (p) and g(p) are the distance and sign of the predicted SDF value corresponding to p, respectively. is the logistic function used to convert g(p) into probability, is the cross entropy, S is the real plane set;

[0019] S4, using the single building point cloud dataset to minimize L to train the building surface extraction network M1, and obtain the building surface extraction model M2;

[0020] S5, determining a single building to be reconstructed in the scene, obtaining a three-dimensional point cloud of the scene based on oblique photography, and extracting a point cloud of the single building to be reconstructed;

[0021] S6, using the RANSAC algorithm to denoise the point cloud of the reconstructed single building and perform plane fitting to obtain several fitting planes, each of which corresponds to a point cloud set, and all point cloud sets constitute a set X1;

[0022] S7, send the set X1 to M2, output the predicted SDF value of each point, and take the points with a distance of 0 in the predicted SDF value to form the set X2;

[0023] S8, reconstructing the surface of the single building to be reconstructed based on X2.

[0024] Preferably, the method further includes step S9 of optimizing the reconstructed surface and performing post-processing to smooth the surface and delete artifacts.

[0025] Preferably, the optimization is to use a threshold value to merge line segments and facets, and the post-processing is surface smoothing.

[0026] As a preferred embodiment: in S5, extracting the point cloud of the single building to be reconstructed includes steps S51 to S53;

[0027] S51, preprocessing the three-dimensional point cloud of the scene to remove noise;

[0028] S52, extract the point cloud of individual buildings using semantic segmentation method;

[0029] S53, denoising the point cloud of a single building, is used to filter out the interference of trees, vegetation, outliers and noise.

[0030] Preferably, in S8, the reconstruction method includes an Alpha Shapes algorithm, a convex hull algorithm and a boundary tracking algorithm.

[0031] Compared with the prior art, the advantages of the present invention are:

[0032] 1. Extract the building surface by fitting the signed distance function through deep learning. Aiming at the problem of low efficiency of point cloud building surface reconstruction, the present invention proposes to construct a deep neural network using deep learning method to learn the method of directional distance function of buildings. The trained deep neural network is used as the building surface extraction model to fit the surface information of the building. Based on the powerful nonlinear fitting ability and feature extraction ability of deep neural network, the surface of point cloud buildings can be robustly and efficiently extracted.

[0033] 2. When the present invention reconstructs a single building, it first removes noise based on the RANSAC algorithm and obtains a point cloud set X1 by plane fitting, and then uses the building surface extraction model to process the points in X1, and only retains the points with a distance of 0 in the predicted SDF value to form a set X2. Finally, the present invention can generate a watertight model of a building with fewer points and planes.

[0034] 3. The building plane represented by rectangles or polygons in the present invention is more in line with the actual scene. Compared with the form of using triangular faces or multiple planes to represent buildings, the present invention can simplify the number of planes to a greater extent, which not only makes the representation of buildings more concise, but also takes up less storage space. In addition, the amount of data is small, which will further reduce the calculation complexity involved in subsequent calculations or uses.

[0035] 4. The present invention proposes an integrated, fully automatic building surface extraction and reconstruction method. (1) The existing methods are that technicians use CAD software to manually form the surface of the building based on the point cloud, which has a high degree of manual participation. This method is generated in an automated form. That is, it is fully automated from the input of the point cloud of the large scene to the final completion of the building surface reconstruction. (2) The existing research is also relatively scattered. For example, point cloud denoising, point cloud segmentation, surface reconstruction, etc. are all separate methods, and there is no such integrated, fully automatic solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the present invention;

[0037] Figure 2 Extract the structural diagram of network M1 for the building surface;

[0038] Figure 3 The present invention is a flow chart for processing a single building to be rebuilt. DETAILED DESCRIPTION

[0039] The present invention will be further described below in conjunction with the accompanying drawings.

[0040] Example 1: See Figures 1 to 3 ,A method for building reconstruction from oblique photography point cloud based on deep neural network, comprises the following steps;

[0041] S1, select a single building point cloud dataset, including multiple samples, each sample includes a point cloud set of a single building, and each point in the point cloud is marked with a three-dimensional coordinate and an SDF value, the SDF value includes a sign and a distance, the sign includes positive and negative, and the distance is the real distance from the point to the real plane to which it belongs;

[0042] S2, construct building surface extraction network M1;

[0043] The M1 includes a PointNet network, a first splicing layer, a second splicing layer and an MLP network;

[0044] The PointNet network is used to input a sample point cloud set P and generate a 256-dimensional implicit vector for each point;

[0045] The first splicing layer is used to splice the three-dimensional coordinates of the point and its implicit vector into a 259-dimensional first splicing vector;

[0046] The MLP includes N fully connected layers FC1~FC N and a nonlinear activation function layer, where the nth fully connected layer FC nThe number of input nodes is 512, the number of output nodes is 253, and the number of input nodes and output nodes of the remaining fully connected layers are both 512, N ≥ 8 and N is an even number, n = N / 2; the first concatenation vector is sequentially passed through FC1~FC n The intermediate vector of 253 dimensions is output, and the intermediate vector and the first concatenated vector are concatenated into a second concatenated vector of 512 dimensions by the second concatenated layer. The second concatenated vector is then concatenated into a second concatenated vector of 512 dimensions by the FC layer. n+1 ~FC N , and then the predicted SDF value is obtained after the nonlinear activation function layer;

[0047] S3, construct loss function L;

[0048] ,

[0049] ,

[0050] ,

[0051] Where, L r is the regression loss, L s is the cross entropy loss, is the L2 norm, p is a point in the point cloud set P, s is the real plane to which p belongs, d(p,s) is the real distance from p to s, and f o (p) and g(p) are the distance and sign of the predicted SDF value corresponding to p, respectively. is the logistic function used to convert g(p) into probability, is the cross entropy, S is the real plane set;

[0052] S4, using the single building point cloud dataset to minimize L to train the building surface extraction network M1, and obtain the building surface extraction model M2;

[0053] S5, determining a single building to be reconstructed in the scene, obtaining a three-dimensional point cloud of the scene based on oblique photography, and extracting a point cloud of the single building to be reconstructed;

[0054] S6, using the RANSAC algorithm to denoise the point cloud of the reconstructed single building and perform plane fitting to obtain several fitting planes, each of which corresponds to a point cloud set, and all point cloud sets constitute a set X1;

[0055] S7, send the set X1 to M2, output the predicted SDF value of each point, and take the points with a distance of 0 in the predicted SDF value to form the set X2;

[0056] S8, reconstructing the surface of the single building to be reconstructed based on X2.

[0057] Example 2: See Figures 1 to 3The present invention further comprises step S9, optimizing the reconstructed surface and performing post-processing to smooth the surface and remove artifacts. The optimization is to use a threshold to merge line segments and facets, and the post-processing is surface smoothing.

[0058] Example 3: See Figures 1 to 3 , based on Example 1 or Example 2, in step S5, extracting the point cloud of the single building to be reconstructed includes steps S51 to S53;

[0059] S51, preprocessing the three-dimensional point cloud of the scene to remove noise;

[0060] S52, extract the point cloud of individual buildings using semantic segmentation method;

[0061] S53, denoising the point cloud of a single building, is used to filter out the interference of trees, vegetation, outliers and noise.

[0062] In S8, the reconstruction methods include Alpha Shapes algorithm, convex hull algorithm and boundary tracking algorithm.

[0063] The SDF value includes the sign and distance. The distance is the real distance from the point to the real plane to which it belongs. A distance of 0 means that the point is exactly located in the real plane. The point is located inside or outside the real plane, which is represented by a sign. If the point is inside the real surface, it is negative, and if it is outside the real surface, it is positive.

[0064] In the present invention: Facet merging is a common technical means in three-dimensional model processing. For example, when the angle between the normal vectors of two or more facets is less than a preset threshold, these triangular facets can be considered to be on the same plane, and then they can be merged. This operation is very useful in scenes such as 3D modeling, game development, and rendering, because it can help simplify complex surface representations and reduce the amount of calculation. Finally, the value direction of the merged facets is revised. For example, most facets are based on vertical walls. Similarly, line segment merging is performed by presetting an angle threshold, and merging multiple line segments with angles less than the angle threshold into one.

[0065] Example 4: In order to illustrate the effect of the present invention, based on the simulation speed, we constructed some simple models to represent the point cloud of a single building, and reconstructed and compared them using the method of the present invention and the existing Poisson surface reconstruction method, and obtained Table 1:

[0066] Table 1. Comparison table between Poisson surface reconstruction and the present invention

[0067] Poisson surface reconstruction surface number The number of surfaces of the present invention Model 1 148 88 Model 2 298 178 Model 3 473 283

[0068] This is because the reconstruction of the building by Poisson surface reconstruction is composed of triangular facets, which requires a large number of face representations and is inaccurate. The surface reconstructed by the present invention is smaller than that of Poisson surface reconstruction and is more in line with the real surface of the building.

[0069] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for building reconstruction from oblique photography point cloud based on deep neural network, characterized in that: The steps include: S1, select a single building point cloud dataset, including multiple samples, each sample includes a point cloud set of a single building, and each point in the point cloud is marked with a three-dimensional coordinate and an SDF value, the SDF value includes a sign and a distance, the sign includes positive and negative, and the distance is the real distance from the point to the real plane to which it belongs; S2, construct building surface extraction network M1; The M1 includes a PointNet network, a first splicing layer, a second splicing layer and an MLP network; The PointNet network is used to input a sample point cloud set P and generate a 256-dimensional implicit vector for each point; The first splicing layer is used to splice the three-dimensional coordinates of the point and its implicit vector into a 259-dimensional first splicing vector; The MLP includes N fully connected layers FC1~FC N and a nonlinear activation function layer, where the nth fully connected layer FC n The number of input nodes is 512, the number of output nodes is 253, and the number of input nodes and output nodes of the remaining fully connected layers are both 512, N ≥ 8 and N is an even number, n = N / 2; the first concatenation vector is sequentially passed through FC1~FC n The intermediate vector of 253 dimensions is output, and the intermediate vector and the first concatenated vector are concatenated into a second concatenated vector of 512 dimensions by the second concatenated layer. The second concatenated vector is then concatenated into a second concatenated vector of 512 dimensions by the FC layer. n+1 ~FC N , and then the predicted SDF value is obtained after the nonlinear activation function layer; S3, construct loss function L; ， ， ， Where, L r is the regression loss, L s is the cross entropy loss, is the L2 norm, p is a point in the point cloud set P, s is the real plane to which p belongs, d(p,s) is the real distance from p to s, and f o (p) and g(p) are the distance and sign of the predicted SDF value corresponding to p, respectively. is the logistic function used to convert g(p) into probability, is the cross entropy, S is the real plane set; S4, using the single building point cloud dataset to minimize L to train the building surface extraction network M1, and obtain the building surface extraction model M2; S5, determining a single building to be reconstructed in the scene, obtaining a three-dimensional point cloud of the scene based on oblique photography, and extracting a point cloud of the single building to be reconstructed; S6, using the RANSAC algorithm to denoise the point cloud of the reconstructed single building and perform plane fitting to obtain several fitting planes, each of which corresponds to a point cloud set, and all point cloud sets constitute a set X1; S7, send the set X1 to M2, output the predicted SDF value of each point, and take the points with a distance of 0 in the predicted SDF value to form the set X2; S8, reconstructing the surface of the single building to be reconstructed based on X2.

2. The method for building reconstruction based on oblique photography point cloud based on deep neural network according to claim 1, characterized in that: The method further comprises step S9 of optimizing the reconstructed surface and performing post-processing to smooth the surface and remove artifacts.

3. The method for building reconstruction based on oblique photography point cloud based on deep neural network according to claim 2, characterized in that: The optimization is to use a threshold to merge line segments and facets, and the post-processing is surface smoothing.

4. The method for building reconstruction based on oblique photography point cloud based on deep neural network according to claim 1, characterized in that: In S5, extracting the point cloud of the single building to be reconstructed includes steps S51 to S53; S51, preprocessing the three-dimensional point cloud of the scene to remove noise; S52, extract the point cloud of individual buildings using semantic segmentation method; S53, denoising the point cloud of a single building, is used to filter out the interference of trees, vegetation, outliers and noise.

5. The method for building reconstruction based on oblique photography point cloud based on deep neural network according to claim 1, characterized in that: In S8, the reconstruction methods include Alpha Shapes algorithm, convex hull algorithm and boundary tracking algorithm.

Images