A 3D reconstruction method based on a real estate building cluster point cloud to MESH neural network
By converting point clouds of real estate building clusters into MESH neural networks, the problems of high computational load and strong hardware dependence in 3D reconstruction technology are solved, enabling efficient rendering and refined reconstruction of building cluster models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN REAL ESTATE & URBAN CONSTR DEV RES CENT
- Filing Date
- 2025-06-20
- Publication Date
- 2026-07-31
AI Technical Summary
Existing 3D reconstruction technologies require a large amount of computation in building complexes, and insufficient point cloud sampling leads to model voids and collapses. They are also highly dependent on hardware and make it difficult to render large-scale scenes.
A method based on converting real estate building cluster point clouds to MESH neural networks is adopted. Through uniform sampling, noise removal, mesh filtering, continuous multi-layer 3D convolutional neural networks, color consistency optimization, and texture distortion optimization, the point cloud data is reduced and its quality is improved. The local geometric patterns and features of buildings are automatically extracted, and color and texture mapping are optimized.
It achieves a reduction in the number of point clouds and an improvement in their quality, thereby enhancing the rendering and modeling accuracy and detail reconstruction of building complex models, and reducing hardware dependence.
Smart Images
Figure CN120707769B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of architectural single or multi-view 3D modeling and reconstruction technology, and in particular to a 3D reconstruction method based on converting real estate building cluster point clouds into MESH neural networks. Background Technology
[0002] Current 3D reconstruction techniques primarily rely on point cloud modeling, but the computational cost of point clouds is substantial, and insufficient point cloud sampling can lead to model collapse and voids. Due to the complexity of building complex models, these methods may be difficult to render within such complexes and are also heavily dependent on hardware limitations. Summary of the Invention
[0003] In order to overcome the shortcomings of the existing technology, the purpose of this invention is to provide a 3D reconstruction method based on the conversion of real estate building cluster point cloud to MESH neural network, so as to realize the rendering and modeling of large-scale scene of building cluster model.
[0004] To achieve the above objectives, the present invention provides the following solution:
[0005] A 3D reconstruction method based on real estate building cluster point cloud to MESH neural network includes:
[0006] The raw point cloud data is obtained by using uniform sampling to collect image data from map software;
[0007] Calculate the local normal vector of each point in the original point cloud data, and remove the data points corresponding to the local normal vectors whose angle difference in the neighborhood is greater than a preset threshold, to obtain noise-removed point cloud data and local filtered normal vectors;
[0008] The noise-removed point cloud data is subjected to grid filtering and curvature calculation to obtain curvature features;
[0009] When the product of the local filtering normal vector corresponding to the point to be processed and any point adjacent to the point to be processed in the noise removal point cloud data is less than 0, the direction of the local filtering normal vector corresponding to the point to be processed is reversed to obtain the normal vector feature.
[0010] The noise-removed point cloud data is subjected to density uniformization processing using a neighborhood density balancing strategy to obtain density uniform point cloud data.
[0011] The density-uniform point cloud data, along with the corresponding normal vector features and curvature features, are input into a continuous multi-layer 3D convolutional neural network to generate triangular meshes and optimize the topology, resulting in a triangular optimized mesh.
[0012] The color information in the original point cloud data is mapped to the point cloud of the triangular optimization mesh, and the color of each triangular face in the triangular optimization mesh is added by weighted averaging of vertex colors to obtain the triangular optimization mesh after color addition.
[0013] The color-added triangular optimization mesh is subjected to UV unwrapping and unwrapping optimization, and the texture image in the original point cloud data is mapped onto the unwrapped triangular optimization mesh to obtain the texture-added triangular optimization mesh;
[0014] The triangular optimization mesh after texture addition is then subjected to color consistency optimization and texture distortion optimization to obtain a 3D reconstruction model.
[0015] Preferably, the expression for uniform sampling is:
[0016]
[0017] Where S represents the original point cloud data; P i This refers to the i-th point cloud data in the original point cloud data; Let be the density gradient of the i-th point cloud data; ∈ is the preset sampling error threshold.
[0018] Preferably, the expression for calculating the local normal vector includes:
[0019]
[0020] Among them, N(P) i )={P j ∣d(P i ,P j )≤r};n i N(P) is the i-th local normal vector; i ) is P i The neighborhood of d(P); i ,P j ) is P i and P j The Euclidean distance between them; r is the radius threshold for neighborhood search; n is a variable within the maximum range of the number of vertices in the neighborhood.
[0021] Preferably, the expression for the grid filtering is:
[0022]
[0023] Among them, P new P represents the representative point of grid cell V; |V| is the number of points within the grid; m This represents the m-th point cloud data within grid cell V.
[0024] Preferably, the expression corresponding to the generation of the triangular mesh includes:
[0025]
[0026] F (l+1) =ReLU(W (l) *F (l) +g (l) ),
[0027]
[0028] Among them, F l+1 (x,y,z) represents the input feature map of the l-th layer; W(i,j,m1) represents the convolution kernel weights; g represents the bias term; k0 represents the size of the convolution kernel; F (l+1) W is the output feature map of the (l+1)th layer; (l) g represents the weights of the convolutional kernel in layer l. (l) f is the bias term for the l-th layer; local (V i ) is the vertex V i Local features; N(V) i () is point V i The neighborhood point set of W; j h is the neighborhood weight; h is the bias constant. This is the formula for maximizing constraints.
[0029] Preferably, the expression corresponding to the topology optimization includes:
[0030] E smooth =∑ o,q (|V o -V q | 2 ),
[0031]
[0032] L=λ1E smooth +λ2E geo ;
[0033] Where L is the total loss function; E smooth Energy loss due to surface smoothness; E geo The energy loss is geometrically consistent; λ1 and λ2 are the first loss weight and the second loss weight, respectively. It represents the first-order gradient of the Mesh surface.
[0034] Preferably, the expression for the vertex color-weighted average is:
[0035]
[0036] Among them, C(F)s ) is a triangular facet F s Average color; C(V) r () is point V r The corresponding colors; V1, V2, and V3 are triangular facets F s The first vertex, the second vertex, and the third vertex.
[0037] Preferably, the optimization objective expression corresponding to the expansion optimization is:
[0038]
[0039] Among them, E tex A is the texture distortion value. 3D (F x ) is a surface patch F in three-dimensional space x The area of A; 2D (F x ) represents the patch F in two-dimensional UV space x The area.
[0040] Preferably, the energy function expression for color consistency optimization is:
[0041]
[0042] Among them, E color For color consistency energy value; F y0 F z0 These are adjacent faces.
[0043] Preferably, the energy function expression for the texture distortion optimization is:
[0044]
[0045] Among them, E warp This represents the energy value for texture distortion. For the surface F x The original texture gradient; For the surface F x Mapped texture gradient.
[0046] The present invention discloses the following technical effects:
[0047] This invention provides a 3D reconstruction method based on a mesh neural network for converting point clouds of real estate building clusters. By using uniform sampling, noise removal, and mesh filtering, it solves the problem of excessive point cloud data volume in conventional techniques, achieving a reduction in the number of point clouds and an improvement in quality. Through continuous multi-layer 3D convolutional neural networks, it overcomes the shortcomings of conventional methods in feature processing, enabling automatic learning of local geometric patterns of buildings from data and layer-by-layer extraction of building features. By optimizing color consistency and texture distortion, it solves the problem of poor color and texture mapping effects in conventional techniques, achieving refined reconstruction of buildings. Attached Figure Description
[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0049] Figure 1 A schematic diagram of the 3D reconstruction process based on the conversion of real estate building cluster point cloud to MESH neural network provided in an embodiment of the present invention;
[0050] Figure 2 This is a schematic diagram of the technical framework provided for an embodiment of the present invention;
[0051] Figure 3 A schematic diagram of 3D reconstruction of a building provided in an embodiment of the present invention. Detailed Implementation
[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0053] The purpose of this invention is to provide a 3D reconstruction method based on the conversion of point cloud of real estate building clusters to MESH neural network, so as to realize the rendering and modeling of large-scale scene of building cluster model.
[0054] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0055] Figure 1 This is a schematic diagram of the 3D reconstruction process based on the conversion of real estate building cluster point cloud to MESH neural network provided in an embodiment of the present invention, as shown below. Figure 1As shown, this invention provides a 3D reconstruction method based on converting real estate building cluster point clouds to MESH neural networks, including:
[0056] Step 100: Use uniform sampling to collect image data from map software to obtain raw point cloud data;
[0057] Step 200: Calculate the local normal vector of each point in the original point cloud data, and remove the data points corresponding to the local normal vectors whose angle difference in the neighborhood is greater than a preset threshold, to obtain noise-removed point cloud data and local filtered normal vectors;
[0058] Step 300: Perform grid filtering and curvature calculation on the noise-removed point cloud data to obtain curvature features;
[0059] Step 400: When the product of the local filtering normal vectors corresponding to the point to be processed and any point adjacent to the point to be processed in the noise removal point cloud data is less than 0, the direction of the local filtering normal vector corresponding to the point to be processed is reversed to obtain the normal vector feature.
[0060] Step 500: Use the neighborhood density balancing strategy to perform density uniformization processing on the noise-removed point cloud data to obtain density uniform point cloud data;
[0061] Step 600: Input the density uniform point cloud data and the corresponding normal vector features and curvature features into a continuous multi-layer 3D convolutional neural network to generate triangular meshes and optimize the topology, so as to obtain a triangular optimized mesh.
[0062] Step 700: Map the color information in the original point cloud data to the point cloud of the triangular optimization mesh, and add color to each triangle facet in the triangular optimization mesh by weighted averaging of vertex colors to obtain the triangular optimization mesh after color addition.
[0063] Step 800: Perform UV unwrapping and unwrapping optimization on the color-added triangular optimization mesh, and map the texture image in the original point cloud data onto the unwrapped triangular optimization mesh to obtain the texture-added triangular optimization mesh;
[0064] Step 900: Perform color consistency optimization and texture distortion optimization on the triangular optimization mesh after texture addition to obtain the 3D reconstruction model.
[0065] Specifically, the expression for uniform sampling is:
[0066]
[0067] Where S represents the original point cloud data; P i The i-th point cloud data in the original point cloud data; Let be the density gradient of the i-th point cloud data; ∈ is the preset sampling error threshold.
[0068] Furthermore, the expression for calculating the local normal vector includes:
[0069]
[0070] Among them, N(P) i )={P j ∣d(P i ,P j )≤r};n i N(P) is the i-th local normal vector; i ) is P i The neighborhood of d(P); i ,P j ) is P i and P j The Euclidean distance between them; r is the radius threshold for neighborhood search; n is a variable within the maximum range of the number of vertices in the neighborhood.
[0071] Specifically, the expression for grid filtering is:
[0072]
[0073] Among them, P new P represents the representative point of grid cell V; |V| is the number of points within the grid; m This represents the m-th point cloud data within grid cell V.
[0074] Furthermore, the expression for generating the triangular mesh includes:
[0075]
[0076] F (l+1) =ReLU(W (l) *F (l) +g (l) ),
[0077]
[0078] Among them, F l+1 (x,y,z) represents the input feature map of the l-th layer; W(i,j,m1) represents the convolution kernel weights; g represents the bias term; k0 represents the size of the convolution kernel; F (l+1) W is the output feature map of the (l+1)th layer; (l) g represents the weights of the convolutional kernel in layer l. (l) f is the bias term for the l-th layer; local (V i ) is the vertex V i Local features; N(V) i () is point Vi The neighborhood point set of W; j h is the neighborhood weight; h is the bias constant. This is the formula for maximizing constraints.
[0079] Specifically, the expressions corresponding to topology optimization include:
[0080] E smooth =∑ o,q (|V o -V q | 2 ),
[0081]
[0082] L=λ1E smooth +λ2E geo ;
[0083] Where L is the total loss function; E smooth Energy loss due to surface smoothness; E geo The energy loss is geometrically consistent; λ1 and λ2 are the first loss weight and the second loss weight, respectively. It represents the first-order gradient of the Mesh surface.
[0084] Furthermore, the expression for the vertex color-weighted average is:
[0085]
[0086] Among them, C(F) s ) is a triangular facet F s Average color; C(V) r () is point V r The corresponding colors; V1, V2, and V3 are triangular facets F s The first vertex, the second vertex, and the third vertex.
[0087] Specifically, the optimization objective expression corresponding to the expansion optimization is as follows:
[0088]
[0089] Among them, E tex A is the texture distortion value. 3D (F x ) is a surface patch F in three-dimensional space x The area of A; 2D (F x ) represents the patch F in two-dimensional UV space x The area.
[0090] Furthermore, the energy function expression for color consistency optimization is:
[0091]
[0092] Among them, E color For color consistency energy value; F y0 F z0 These are adjacent faces.
[0093] Specifically, the energy function expression for texture distortion optimization is:
[0094]
[0095] Among them, E warp This represents the energy value for texture distortion. For the surface F x The original texture gradient; For the surface F x Mapped texture gradient.
[0096] refer to Figure 2 and Figure 3 The overall technical framework consists of four main processes:
[0097] 1) Point Cloud Sampling and Preprocessing: Point cloud data acquisition and preprocessing is the first step in any point cloud reconstruction task. Building clusters typically have complex geometries and highly structured features; the process mainly includes:
[0098] Point cloud sampling: First, point cloud sampling is performed from the map software. This data is usually sparse and contains noise. To ensure efficient processing, the original point cloud is typically sampled, and the most representative points are selected for subsequent calculations. The purpose of sampling is to reduce the amount of point cloud data while retaining necessary geometric information.
[0099] Noise Removal and Filtering: During the sampling process, point cloud data often contains noise, such as sensor errors or interference from the surrounding environment.
[0100] 2) Geometricization and Uniform Filtering of Point Clouds: A key step in converting point clouds to meshes is transforming the raw point cloud data into a structure with geometric features. Geometricization primarily involves uniform filtering and feature point selection to convert the point cloud into an effective data structure suitable for 3D reconstruction.
[0101] Geometric processing: Generate geometric features such as the normal vector and curvature of points by using the spatial relationship between adjacent points in the point cloud.
[0102] Uniform filtering: To make the point cloud more spatially uniform, uniform filtering can be applied to the sampling points. This process ensures that there are no areas with excessively high density in the point cloud during reconstruction, avoiding overly complex mesh models or excessive redundant data in the subsequent generation.
[0103] 3) POINT2MESH Construction Based on Continuous Multi-Layer 3D Convolutional Neural Networks: The conversion from point cloud to mesh can be achieved through deep learning. The goal is to extract the 3D structure of building clusters from point clouds using a 3D convolutional network, and then generate a triangular mesh. In terms of neural network architecture design, the network used should be able to handle high-dimensional spatial information and possess strong representational capabilities. This embodiment designs a method that can perform convolution operations within local regions of the point cloud to extract its geometric features. After the network's convolution operations, a set of feature maps about the geometric shape of the building cluster is obtained. These feature maps can reflect the details and overall form of the buildings.
[0104] Design of a Continuous Multi-Layer 3D Convolutional Network: First, the point cloud data is transformed into a format suitable for input into the 3D convolutional network. Local features (including spatial location, normal vector, curvature, etc.) are extracted from each sampling point in the point cloud. The continuous multi-layer 3D convolutional neural network can automatically learn geometric patterns from the local structure of the point cloud, automatically recognizing the shape, boundaries, and structural features of buildings during training.
[0105] 4) Facet Color and Texture Mapping: After generating the triangular faces, the final step is texture and color mapping to ensure a more realistic and detailed model. The appearance of a cluster of buildings depends not only on its geometry but also on visual information such as surface color, lighting, and texture. This process can be completed through the following steps:
[0106] Color Sampling and Distribution: Based on the color information or illumination distribution in the point cloud data, triangular facets are colored. By analyzing the color of each point in the point cloud, the color information is mapped onto the corresponding triangular facet. Texture Mapping and Optimization: For the texture parts of buildings, image texture colors can be applied to the 3D model through texture mapping. This process requires proper unfolding of the UV coordinates of the facets to match the acquired texture with the triangular facets.
[0107] Furthermore, the four main processes mentioned above are implemented as follows:
[0108] 1) Data preprocessing
[0109] In the point cloud to MESH neural network reconstruction scheme based on building clusters, the quality of the point cloud data directly affects the accuracy and effect of subsequent reconstruction. Therefore, in this stage, this embodiment needs to perform sampling, noise removal and uniform filtering on the original point cloud. This embodiment takes the point cloud of a building cluster from a certain map software as an example.
[0110] Furthermore, in the point cloud sampling of building complex images in map software, it is assumed that this embodiment obtains an information set containing a large number of points through point sampling on the image, each point being represented by its three-dimensional coordinates P. i=(x i ,y i ,z i The sampling method is denoted as , where i∈[1,N] is the index of the point, and N is the total number of points. The goal of sampling is to reduce the number of points in the point cloud while preserving important spatial structure information. Commonly used sampling methods include uniform sampling, random sampling, and region-based sampling. Specifically, for point cloud data of building clusters, this embodiment can use the uniform sampling method.
[0111] Specifically, the basic idea of uniform sampling is to select sample points evenly according to the density of the point cloud, ensuring that the number of sample points in each region is relatively uniform. The formula for uniform sampling is:
[0112]
[0113] The sampling process ensures a uniform distribution among the sampling points, thus avoiding the problem of excessively dense or sparse point clouds in certain areas. Next, noise is identified and removed by analyzing the distribution characteristics of points within the neighborhood of the point cloud.
[0114] Furthermore, assuming that the neighborhood of each point in the point cloud contains k nearest neighbors, then the local normal vector n i It can be obtained through fitting:
[0115]
[0116] This normal vector represents point P. i The planar orientation of the surrounding neighborhood. By calculating the normal vectors of all points, we can identify potential noise points in the point cloud and remove points whose normal vectors have significant angular differences from their neighboring points.
[0117] Specifically, even after noise removal, the density and distribution of point cloud data may still exhibit unevenness, especially in complex areas within building complexes. To avoid excessively dense point cloud data increasing the computational complexity of subsequent processing, uniform filtering is typically required. The goal of uniform filtering is to adjust the sampling interval of the point cloud using certain spatial rules, resulting in a more uniform distribution of the point cloud in space. This embodiment uses a mesh method (simple mesh, unlike the subsequent MESH) for spatial partitioning and filtering.
[0118] Specifically, in grid filtering, the point cloud space is divided into grid cells of a certain size, with each grid cell containing one point. If multiple points exist within a grid, a representative point (e.g., the center point or centroid) is selected to represent the geometric information of that grid region. The specific formula is:
[0119]
[0120] This method ensures a more uniform spatial distribution of point clouds while avoiding excessive refinement of local areas.
[0121] Preferably, normal vector calculation is a core step in point cloud processing, as it helps identify geometric features in the point cloud, such as planes, surfaces, and their morphological changes. Methods for calculating normal vectors typically rely on fitting local neighborhood points, especially crucial for subsequent 3D reconstruction in building cluster reconstruction. Assuming a point has k neighborhood points, this embodiment can fit a plane of these neighborhood points:
[0122]
[0123] Where a, b0, and c are the normal vectors of the plane, and d is the offset. The calculated normal vectors can be used to further analyze the curvature characteristics of the point cloud.
[0124] Furthermore, curvature is a measure of the degree of bending of the point cloud surface. For each point P i Curvature can be measured by calculating the minimum principal curvature κ1 and the maximum principal curvature κ2 of its local region, as shown in the following formula:
[0125]
[0126] Where, λ min and λ max Let P be the point i The eigenvalues of the minimum and maximum principal curvatures of the surface in question. These curvature features can help this embodiment identify details of building surfaces and structures.
[0127] By employing steps such as point cloud sampling, noise removal, uniform filtering, normal vector calculation, and curvature analysis, this embodiment can effectively improve the quality and usability of point cloud data, and these processes reduce data redundancy.
[0128] 2) Point cloud geometrization and uniform filtering
[0129] After point cloud data sampling and preliminary preprocessing, the obtained point cloud data is still discrete, and there may be uneven density or missing geometric features in some areas. Therefore, point cloud geometrization and uniform filtering are key steps in 3D reconstruction, aiming to further optimize the distribution of point cloud data, extract point cloud geometric features, and ensure better meshing and texture mapping in subsequent steps.
[0130] Specifically, the main purpose of geometric feature extraction in point clouds is to analyze and understand the distribution patterns of point clouds in three-dimensional space and identify key structural features (such as planes, curved surfaces, and boundaries). In point clouds of building clusters, geometric feature extraction is particularly important for identifying the facades, roofs, edges, and other structural features of buildings.
[0131] Preferably, in point cloud geometrization, neighborhood search is a key step used to identify the local geometric environment of each point. For any point P... i The set of neighboring points can be determined using the following formula.
[0132] N(P i )={P j ∣d(P i ,P j )≤r}
[0133] After determining the neighborhood point set, this embodiment fits the neighborhood points to obtain point P. i normal vector n i :
[0134]
[0135] Normal vector n i This is obtained by minimizing the sum of squared distances from all neighboring points to the best-fit plane. The two formulas for calculating the aforementioned normal vectors are applied to different data; the former is for a local point cloud, where the neighborhood is smaller.
[0136] Furthermore, the curvature of a point cloud describes the degree of bending of the surface near each point, serving as an important indicator for geometric feature extraction. In building clusters, curvature can help identify planar, curved, and complex shapes.
[0137] Mathematical expression of curvature:
[0138]
[0139] Where: Cur(P) i Let P be a point. i The curvature value, λ min λ is the smallest eigenvalue of the neighborhood covariance matrix. j The curvature is the sum of all eigenvalues. When the curvature value is close to 0, it indicates that the region tends to be flat; when the curvature value is large, it indicates that the region has significant bending characteristics.
[0140] Preferably, to further geometrize the point cloud, each local neighborhood needs to be fitted to a plane. For a local region in the point cloud, a plane equation can be used for fitting:
[0141] a1x1+b1y1+c1z1+d1=0
[0142] Where: a1, b1, c1 are the components of the plane normal vector, d1 is the offset of the plane from the origin, and x1, y1, z1 are the xyz-axis components of the fitted equation. The least squares method is used to fit the plane, with the objective of minimizing the distance from each point to the plane.
[0143]
[0144] By solving this optimization problem, the planar parameters (a2, b2, c2, d2) can be obtained, thereby locally geometrizing the point cloud into a plane or curved surface.
[0145] Preferably, in large-scale building point cloud processing, inconsistencies in normal vector directions may exist, which can interfere with subsequent mesh reconstruction. Therefore, normal vector consistency optimization is required. For the normal vector n of adjacent points... i1 and n j1 You can determine whether the normals are consistent by using the dot product:
[0146] n i1 ·n j1 ≥0
[0147] If the result is less than 0, then n j1 The direction is reversed.
[0148] Furthermore, a neighborhood density balancing strategy can be introduced to homogenize the point cloud. The neighborhood density of each point is defined as:
[0149]
[0150] Where: D(P) i2 ) represents the point cloud density. k represents the number of points in the neighborhood. V i2 Let be the volume of the neighborhood.
[0151] For areas with excessively high density, the density can be reduced by downsampling; for areas with low density, the number of points can be increased by interpolation.
[0152] Preferably, the point cloud data is subjected to in-depth geometrization and optimization processing through techniques such as neighborhood search, normal vector estimation, curvature calculation, and geometric feature extraction. The geometric features of the point cloud are effectively extracted, the density is homogenized, and the normal vector direction is consistently adjusted.
[0153] 3) Construction based on POINT2MESH of continuous multi-layer 3D convolutional neural networks
[0154] In the steps described above, this embodiment completed the sampling, preprocessing, and geometrization of point cloud data, resulting in a high-quality and uniformly distributed point cloud dataset. Next, a continuous multi-layer 3D convolutional neural network (C-3D-CNN) will be used to construct a POINT2MESH, converting the discrete point cloud vertex data (V1, V2, V3) into a continuous triangular mesh representation. This process is the core of 3D reconstruction, involving the extraction of spatial features, learning the topological structure between point clouds, and finally generating the triangular mesh.
[0155] Specifically, this embodiment designs a special continuous multi-layer 3D convolutional neural network (C-3D-CNN) deep learning model capable of feature extraction and learning from 3D data. Unlike 2D convolution, the 3D convolution in this embodiment introduces a depth dimension in space, enabling it to capture the spatial structure information of point clouds. The formula is as follows:
[0156]
[0157] Continuous 3D convolution operations can extract features simultaneously in three dimensions, making them suitable for processing 3D point cloud data.
[0158] Furthermore, in the POINT2MESH task, point cloud vertices V1, V2, and V3 serve as input to C-3D-CNN. Each vertex contains 3D coordinate information and optional color information. To adapt to C-3D-CNN, the point cloud data needs to be voxelized, that is, the continuous 3D point cloud is mapped onto a discrete 3D voxel mesh.
[0159] If a point exists in the point cloud within the voxel, then V(x, y, z) = 1; otherwise, it is 0.
[0160] In C-3D-CNN, each vertex is mapped to a corresponding voxel unit, followed by feature extraction and learning.
[0161] Preferably, the continuous C-3D-CNN network mainly includes the following modules:
[0162] Input layer: Inputs the voxelized point cloud data into the network.
[0163] Multi-layer 3D convolutional layers: extract spatial features and learn the topological structure of point clouds.
[0164] Pooling layer: Reduces the dimensionality of the feature map and reduces computation by using general mean pooling for dimensionality reduction.
[0165] Output layer: Generates the connection relationships between vertices and faces, and constructs a triangular mesh.
[0166] Specifically, the forward propagation process: for the input feature map F of the l-th layer... (l) and the output feature map F of layer l+1 (l +1) The formula for calculating convolution is:
[0167] F (l+1) =ReLU(W (l) *F (l) +g (l) )
[0168] 3D convolution operation. During 3D convolution, the network extracts the following features:
[0169] Local geometric features: the distribution pattern of points within a neighborhood.
[0170] Global structural features: the morphology and structural information of the entire point cloud.
[0171] Boundary features: Identify the connection relationships between boundary points and patches.
[0172] Local feature extraction formula:
[0173]
[0174] Furthermore, the construction of POINT2MESH:
[0175] After feature extraction is complete, the network needs to convert the point cloud vertices V v1 V v2 V v3 Connect them to form triangular facets. This process is accomplished using a connection strategy based on the arg-max constraint of the interior angles of a solid triangle.
[0176]
[0177] The goal is to maximize the minimum interior angle of the triangle to ensure the stability of the mesh structure.
[0178] Based on the initially generated triangular mesh, topology optimization is required to eliminate redundant faces and unreasonable connections. Surface smoothness:
[0179]
[0180] Geometric consistency:
[0181]
[0182] To optimize network performance, a loss function is defined to guide network training. The total loss function is:
[0183] L=λ1E smooth +λ2E geo
[0184] By employing a continuous multi-layer 3D convolutional neural network (C-3D-CNN) for feature extraction, learning, and topology optimization of point cloud vertices, a continuous triangular mesh was successfully constructed from a discrete point cloud. This process combines the powerful feature extraction capabilities of deep learning with the geometric constraints of traditional triangulation methods, ensuring the geometric accuracy and structural stability of the mesh.
[0185] 4) Patch color and texture mapping
[0186] After geometrizing, uniformly filtering, and constructing a POINT2MESH network based on a 3D convolutional network for the point cloud data, a structurally complete and topologically stable triangular mesh is obtained. However, this mesh only contains geometric structure information and does not include surface color and texture details. Therefore, to achieve high-fidelity 3D reconstruction, it is necessary to map the color information in the point cloud onto the faces of the triangular mesh to complete the reconstruction of color and texture. This step will elaborate on the theory and implementation process of color and texture mapping, and provide a rigorous description using formulas.
[0187] Specifically, in the point cloud sampling stage, color information can be represented as:
[0188] Color(V i3 )=(r i3 ,g i3 ,b i3 )
[0189] Among them, Color(V) i3 Vertex V i3 Color information. (r i3 ,g i3 ,b i3 ): The values of the three color channels: red, green, and blue.
[0190] To map color information from a point cloud onto triangular faces, we first need to determine the color source for each triangle. This can be done by weighted averaging of vertex colors:
[0191]
[0192] In actual rendering, the color on a facet requires not only vertex color information, but also a smooth transition of color across the entire facet. This embodiment uses bilinear interpolation.
[0193] UV Unwrapping: To accurately map a texture image onto triangular facets, the triangular mesh needs to be unwrapped using UV mapping. UV coordinates are coordinates in a two-dimensional texture space, commonly expressed by the following formula:
[0194] (U1,V1)=f(x3,y3,z3)
[0195] Where: (U1,V1) are the 2D texture coordinates. (x3,y3,z3) are the 3D mesh vertex coordinates. f(·) is the UV mapping function.
[0196] Preferably, in UV unwrapping, the goal is to minimize texture distortion and stretching between triangles. The energy function represents texture distortion:
[0197]
[0198] The goal is to minimize the above energy function, thereby achieving a uniform distribution of UV coordinates.
[0199] Furthermore, after UV unwrapping, a known texture image T(U, V) is mapped onto each triangular facet of the mesh. For any point on a triangular facet, its texture color can be sampled using the UV coordinates:
[0200] C(p)=T(U(p,V(p))
[0201] Where C(p): the color of point pp on the patch. T(U(p), V(p)): the color sampled from the texture image using UV coordinates.
[0202] Preferably, color consistency optimization: In practical applications, color discontinuities or texture seams may exist. To optimize color consistency, the following energy function can be used:
[0203]
[0204] By minimizing this energy function, color inconsistencies between patches can be significantly reduced.
[0205] Texture distortion optimization: Texture mapping may suffer from distortion, resulting in unnatural visual effects. Texture distortion can be addressed through the following optimization goals:
[0206]
[0207] Minimizing the above energy function can reduce texture distortion.
[0208] Preferably, by extracting and interpolating the color information of the point cloud, the color information is accurately mapped onto the triangular facets; through UV unwrapping and texture sampling, a high-resolution texture image is mapped onto the triangular mesh. Simultaneously, by optimizing color consistency and texture distortion, the quality of the final rendering effect is further improved.
[0209] The beneficial effects of this invention are as follows:
[0210] This invention reduces the amount of point cloud data and improves the quality of point cloud data by uniform sampling, noise removal, and mesh filtering; it improves the accuracy of the three-dimensional mesh model by automatically learning the local geometric patterns of buildings from the data and extracting building features layer by layer through continuous multi-layer 3D convolutional neural networks; and it optimizes the color and texture mapping effect by optimizing color consistency and texture distortion, thus achieving refined reconstruction of buildings.
[0211] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.
[0212] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A 3D reconstruction method based on a real estate building cluster point cloud to MESH neural network, characterized in that, include: The raw point cloud data is obtained by using uniform sampling to collect image data from map software; Calculate the local normal vector of each point in the original point cloud data, and remove the data points corresponding to the local normal vectors whose angle difference in the neighborhood is greater than a preset threshold, to obtain noise-removed point cloud data and local filtered normal vectors; The noise-removed point cloud data is subjected to grid filtering and curvature calculation to obtain curvature features; When the product of the local filtering normal vector corresponding to the point to be processed and any point adjacent to the point to be processed in the noise removal point cloud data is less than 0, the direction of the local filtering normal vector corresponding to the point to be processed is reversed to obtain the normal vector feature. The noise-removed point cloud data is subjected to density uniformization processing using a neighborhood density balancing strategy to obtain density uniform point cloud data. The density-uniform point cloud data, along with the corresponding normal vector features and curvature features, are input into a continuous multi-layer 3D convolutional neural network to generate triangular meshes and optimize the topology, resulting in a triangular optimized mesh. The color information in the original point cloud data is mapped to the point cloud of the triangular optimization mesh, and the color of each triangular face in the triangular optimization mesh is added by weighted averaging of vertex colors to obtain the triangular optimization mesh after color addition. The color-added triangular optimization mesh is subjected to UV unwrapping and unwrapping optimization, and the texture image in the original point cloud data is mapped onto the unwrapped triangular optimization mesh to obtain the texture-added triangular optimization mesh; The triangular optimized mesh after texture addition is then subjected to color consistency optimization and texture distortion optimization to obtain a 3D reconstructed model. The expression corresponding to the generation of the triangular mesh includes: 、 、 、 ； in, For the first The input feature map of the layer; These are the kernel weights; For bias terms; The size of the convolution kernel; For the first The output feature map of the layer; For the first Layer convolution kernel weights; For the first Layer bias terms; As vertices Local features; For point The neighborhood point set; Neighborhood weights; It is a bias constant; This is the formula for maximizing constraints.
2. The 3D reconstruction method based on real estate building cluster point cloud to MESH neural network according to claim 1, characterized in that, The expression for uniform sampling is: ； in, The original point cloud data; This refers to the i-th point cloud data in the original point cloud data; Let be the density gradient of the i-th point cloud data; This is the preset sampling error threshold.
3. The 3D reconstruction method based on a real estate building cluster point cloud to MESH neural network according to claim 2, characterized in that, The expression for calculating the local normal vector includes: ； in, ; Let i be the local normal vector; for ; for and Euclidean distance between them; This is the radius threshold for neighborhood search; This is a variable within the maximum range of the number of vertices in the domain.
4. The 3D reconstruction method based on a real estate building cluster point cloud to MESH neural network according to claim 3, characterized in that, The expression for the grid filtering is: ； in, For grid cells Representative point; This represents the number of points within the grid. For grid cells The m-th point cloud data within.
5. The 3D reconstruction method based on a real estate building cluster point cloud to MESH neural network according to claim 1, characterized in that, The expressions corresponding to the topology optimization include: 、 、 ； in, This is the total loss function; Energy loss due to surface smoothness; Energy loss due to geometric consistency; , These are the first loss weight and the second loss weight, respectively. It represents the first-order gradient of the Mesh surface.
6. The 3D reconstruction method based on real estate building cluster point cloud to MESH neural network according to claim 5, characterized in that, The expression for the vertex color-weighted average is: ； in, Triangular facet The average color; For point The corresponding color; They are triangular facets The first vertex, the second vertex, and the third vertex.
7. A 3D reconstruction method based on real estate building cluster point cloud to MESH neural network according to claim 6, characterized in that, The optimization objective expression corresponding to the expansion optimization is: ； in, This represents the texture distortion value. For a surface in three-dimensional space The area; A patch in two-dimensional UV space The area.
8. The 3D reconstruction method based on a real estate building cluster point cloud to MESH neural network according to claim 7, characterized in that, The energy function expression for color consistency optimization is: ； wherein, is a color consistent energy value; , is an adjacent facet.
9. The 3D reconstruction method based on a real estate building cluster point cloud to MESH neural network according to claim 8, characterized in that, The energy function expression for texture distortion optimization is: ； wherein, is a texture warp energy value; is a patch original texture gradient; is a patch mapped texture gradient.