3D reconstruction method based on real estate building cluster point cloud-to-MESH neural network

Abstract

The invention belongs to the technical field of building single or multi-view-angle to 3D modeling reconstruction, and provides a 3D reconstruction method based on a real estate building cluster point cloud to MESH neural network. Comprising the steps of original point cloud data acquisition, noise removal, grid filtering, normal vector optimization, density uniformity processing, continuous multilayer 3D convolutional neural network processing, color mapping, texture mapping, color consistency optimization and texture distortion optimization. Through uniform sampling, noise removal and grid filtering, the point cloud data volume is reduced, and the quality of the point cloud data is improved; through the continuous multi-layer 3D convolutional neural network, the local geometric pattern of the building is automatically learned from the data and the features of the building are extracted layer by layer, so that the precision of the three-dimensional grid model is improved; through color consistency optimization and texture distortion optimization, the color and texture mapping effect is optimized, and refined reconstruction of the building is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of converting a single or multi-viewpoint building into 3D modeling and reconstruction, and in particular to a 3D reconstruction method based on converting a point cloud of a real estate building cluster into a MESH neural network. Background Art

[0002] Current 3D reconstruction techniques primarily rely on point cloud modeling, but this requires significant computational effort. Furthermore, insufficient point cloud sampling can lead to model collapse and voids. Due to the complexity of building cluster models, these methods can be difficult to render in complex environments and are hardware-dependent. Summary of the Invention

[0003] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a 3D reconstruction method based on the conversion of real estate building cluster point cloud to MESH neural network, so as to realize the rendering modeling of large-scale scenes of building complex models.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A 3D reconstruction method based on converting real estate building cluster point cloud to MESH neural network, including:

[0006] Use uniform sampling to collect image data in mapping software to obtain original point cloud data;

[0007] Calculating the local normal vector of each point in the original point cloud data, and removing data points corresponding to the local normal vectors whose angle difference in the neighborhood is greater than a preset threshold, to obtain noise-removed point cloud data and local filtered normal vectors;

[0008] Performing grid filtering and curvature calculation on the noise-removed point cloud data to obtain curvature features;

[0009] When the product of the local filtered normal vectors corresponding to the to-be-processed point and any point adjacent to the to-be-processed point in the noise-removed point cloud data is less than 0, the direction of the local filtered normal vector corresponding to the to-be-processed point is reversed to obtain a normal vector feature;

[0010] Performing density uniformity processing on the noise-removed point cloud data using a neighborhood density balancing strategy to obtain density uniform point cloud data;

[0011] Inputting the density uniform point cloud data and the normal vector features and the curvature features corresponding to the density uniform point cloud data into a continuous multi-layer 3D convolutional neural network to perform triangular mesh generation and topology structure optimization to obtain a triangular optimized mesh;

[0012] Mapping the color information in the original point cloud data to the point cloud of the triangular optimized mesh, and adding color to each triangular face in the triangular optimized mesh by weighted averaging of vertex colors to obtain the triangular optimized mesh after color addition;

[0013] Performing UV unfolding and unfolding optimization on the triangular optimized mesh after color addition, and mapping the texture image in the original point cloud data to the unfolded triangular optimized mesh to obtain the triangular optimized mesh after texture addition;

[0014] Color consistency optimization and texture distortion optimization are performed on the triangular optimized mesh after texture addition to obtain a 3D reconstructed model.

[0015] Preferably, the expression of the uniform sampling is:

[0016]

[0017] Wherein, S is the original point cloud data; P i is the i-th point cloud data in the original point cloud data; is the density gradient of the i-th point cloud data; ∈ is the preset sampling error threshold.

[0018] Preferably, the calculation expression of the local normal vector includes:

[0019]

[0020] Among them, N(P i )={P j |d(P i ,P j )≤r};n i is the local normal vector of the i-th; N(P i ) is P i Neighborhood; d(P i ,P j ) is P i and P j The Euclidean distance between them; r is the radius threshold of the neighborhood search; n is a variable within the maximum value range of the number of vertices in the domain.

[0021] Preferably, the expression of the grid filtering is:

[0022]

[0023] Among them, P new is the representative point of the grid unit V; |V| is the number of points in the grid; P m is the mth point cloud data in the grid unit V.

[0024] Preferably, the expression corresponding to the triangular mesh generation includes:

[0025]

[0026] F (l+1) =ReLU(W (l) *F (l) +g (l) ),

[0027]

[0028] Among them, F l+1 (x, y, z) is the input feature map of layer l; W(i, j, m1) is the convolution kernel weight; g is the bias term; k0 is the size of the convolution kernel; F (l+1) is the output feature map of the l+1th layer; W (l) is the convolution kernel weight of the lth layer; g (l) is the bias term of the lth layer; f local (V i ) is the vertex V i Local characteristics of N(V i ) is point V i The neighborhood point set of W j is the neighborhood weight; h is the bias constant; is the maximization constraint formula.

[0029] Preferably, the expression corresponding to the topology optimization includes:

[0030] E smooth =∑ o,q (|V o -V q | 2 ),

[0031]

[0032] L=λ1E smooth +λ2E geo ;

[0033] Among them, L is the total loss function; E smooth is the surface smoothing energy loss; E geo is the geometric consistency energy loss; λ1 and λ2 are the first loss weight and the second loss weight respectively; is the first-order gradient of the Mesh patch.

[0034] Preferably, the expression of the weighted average of vertex colors is:

[0035]

[0036] Among them, C(Fs ) is a triangular patch F s The average color of C(V r ) is point V r Corresponding colors; V1, V2, V3 are triangle patches F s The first vertex, the second vertex, and the third vertex of .

[0037] Preferably, the optimization objective expression corresponding to the expansion optimization is:

[0038]

[0039] Among them, E tex is the texture distortion value; A 3D (F x ) is the patch F in three-dimensional space x The area of ​​​​A 2D (F x ) is the patch F in the two-dimensional UV space x area.

[0040] Preferably, the energy function expression for color consistency optimization is:

[0041]

[0042] Among them, E color is the color consistency energy value; F y0 、F z0 are adjacent patches.

[0043] Preferably, the energy function expression of the texture distortion optimization is:

[0044]

[0045] Among them, E warp is the texture distortion energy value; For the patch F x The original texture gradient; For the patch F x The gradient of the texture after mapping.

[0046] The present invention discloses the following technical effects:

[0047] The present invention provides a 3D reconstruction method based on the conversion of real estate building cluster point clouds into MESH neural networks. Through uniform sampling, noise removal and mesh filtering, the method solves the problem of excessive point cloud computing in conventional technologies, and achieves a reduction in the number of point clouds and an improvement in quality. Through continuous multi-layer 3D convolutional neural networks, the method solves the defect of poor feature processing capabilities of conventional methods, and realizes automatic learning of local geometric patterns of buildings from data and layer-by-layer extraction of building features. Through color consistency optimization and texture distortion optimization, the method solves the problem of poor color and texture mapping effects of conventional technologies, and realizes refined reconstruction of buildings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A schematic diagram of the 3D reconstruction process based on the conversion of real estate building cluster point cloud to MESH neural network provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of the technical framework provided by an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of 3D reconstruction of a building provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0053] The purpose of the present invention is to provide a 3D reconstruction method based on the conversion of real estate building cluster point cloud to MESH neural network, so as to realize the rendering modeling of large-scale scenes of building cluster models.

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Figure 1 The schematic diagram of the 3D reconstruction process based on the conversion of real estate building cluster point cloud to MESH neural network provided by the embodiment of the present invention is as follows: Figure 1As shown, the present invention provides a 3D reconstruction method based on converting real estate building cluster point cloud to MESH neural network, comprising:

[0056] Step 100: Collect image data in the mapping software using uniform sampling to obtain original point cloud data;

[0057] Step 200: Calculate the local normal vector of each point in the original point cloud data, and remove the data points corresponding to the local normal vectors whose angle difference in the neighborhood is greater than a preset threshold, to obtain noise-removed point cloud data and local filtered normal vectors;

[0058] Step 300: Perform grid filtering and curvature calculation on the noise-removed point cloud data to obtain curvature features;

[0059] Step 400: When the product of the local filtered normal vectors corresponding to the processing point and any point adjacent to the processing point in the noise-removed point cloud data is less than 0, the direction of the local filtered normal vector corresponding to the processing point is reversed to obtain a normal vector feature;

[0060] Step 500: performing density uniformity processing on the noise-removed point cloud data using a neighborhood density balance strategy to obtain density uniformity point cloud data;

[0061] Step 600: Input the density uniform point cloud data and the normal vector features and curvature features corresponding to the density uniform point cloud data into a continuous multi-layer 3D convolutional neural network to perform triangular mesh generation and topology structure optimization to obtain a triangular optimized mesh;

[0062] Step 700: Mapping the color information in the original point cloud data to the point cloud of the triangular optimized mesh, and adding color to each triangular face in the triangular optimized mesh by weighted averaging of vertex colors to obtain a triangular optimized mesh after color addition;

[0063] Step 800: UV unfolding and unfolding optimization are performed on the triangular optimized mesh after color addition, and the texture image in the original point cloud data is mapped to the unfolded triangular optimized mesh to obtain the triangular optimized mesh after texture addition;

[0064] Step 900: Perform color consistency optimization and texture distortion optimization on the triangular optimized mesh after texture addition to obtain a 3D reconstructed model.

[0065] Specifically, the expression for uniform sampling is:

[0066]

[0067] Among them, S is the original point cloud data; P i is the i-th point cloud data in the original point cloud data; is the density gradient of the i-th point cloud data; ∈ is the preset sampling error threshold.

[0068] Furthermore, the calculation expression of the local normal vector includes:

[0069]

[0070] Among them, N(P i )={P j |d(P i ,P j )≤r};n i is the i-th local normal vector; N(P i ) is P i Neighborhood; d(P i ,P j ) is P i and P j The Euclidean distance between them; r is the radius threshold of the neighborhood search; n is a variable within the maximum value range of the number of vertices in the domain.

[0071] Specifically, the expression of grid filtering is:

[0072]

[0073] Among them, P new is the representative point of the grid unit V; |V| is the number of points in the grid; P m is the mth point cloud data in the grid unit V.

[0074] Furthermore, the corresponding expressions for triangular mesh generation include:

[0075]

[0076] F (l+1) =ReLU(W (l) *F (l) +g (l) ),

[0077]

[0078] Among them, F l+1 (x, y, z) is the input feature map of layer l; W(i, j, m1) is the convolution kernel weight; g is the bias term; k0 is the size of the convolution kernel; F (l+1) is the output feature map of the l+1th layer; W (l) is the convolution kernel weight of the lth layer; g (l) is the bias term of the lth layer; f local (V i ) is the vertex V i Local characteristics of N(V i ) is point Vi The neighborhood point set of W j is the neighborhood weight; h is the bias constant; is the maximization constraint formula.

[0079] Specifically, the expressions corresponding to topology optimization include:

[0080] E smooth =∑ o,q (|V o -V q | 2 ),

[0081]

[0082] L=λ1E smooth +λ2E geo ;

[0083] Among them, L is the total loss function; E smooth is the surface smoothing energy loss; E geo is the geometric consistency energy loss; λ1 and λ2 are the first loss weight and the second loss weight respectively; is the first-order gradient of the Mesh patch.

[0084] Furthermore, the expression of the weighted average of vertex colors is:

[0085]

[0086] Among them, C(F s ) is a triangular patch F s The average color of C(V r ) is point V r Corresponding colors; V1, V2, V3 are triangle patches F s The first vertex, the second vertex, and the third vertex of .

[0087] Specifically, the optimization objective expression corresponding to the expansion optimization is:

[0088]

[0089] Among them, E tex is the texture distortion value; A 3D (F x ) is the patch F in three-dimensional space x The area of ​​​​A 2D (F x ) is the patch F in the two-dimensional UV space x area.

[0090] Furthermore, the energy function expression for color consistency optimization is:

[0091]

[0092] Among them, E color is the color consistency energy value; F y0 、F z0 are adjacent patches.

[0093] Specifically, the energy function expression for texture distortion optimization is:

[0094]

[0095] Among them, E warp is the texture distortion energy value; For the patch F x The original texture gradient; For the patch F x The gradient of the texture after mapping.

[0096] refer to Figure 2 and Figure 3 ,The overall technical framework is divided into four main processes:

[0097] 1) Point cloud sampling and preprocessing: Point cloud data collection and preprocessing are the first steps in any point cloud reconstruction task. Building clusters typically have complex geometric shapes and highly structured features. The process mainly includes:

[0098] Point cloud sampling: First, point clouds are sampled from mapping software. This data is often sparse and noisy. To ensure efficient processing, the original point cloud is typically sampled, selecting the most representative points for subsequent calculations. The goal of sampling is to reduce the number of point clouds while preserving necessary geometric information.

[0099] Noise removal and filtering: During the sampling process, point cloud data often contains noise, such as sensor errors or interference from the surrounding environment.

[0100] 2) Geometrization and Uniform Filtering of Point Clouds: The key step in point cloud to mesh conversion is converting the raw point cloud data into a structure with geometric features. Geometrization primarily involves converting the point cloud into a valid data structure for 3D reconstruction through uniform filtering and feature point selection.

[0101] Geometric processing: Generate geometric features such as point normals and curvatures through the spatial relationship between adjacent points in the point cloud.

[0102] Uniform filtering: To make the point cloud more evenly distributed, the sampling points can be uniformly filtered. This process ensures that the point cloud does not have areas of excessive density during reconstruction, preventing the subsequent mesh model from being overly complex or generating excessive redundant data.

[0103] 3) POINT2MESH construction based on continuous multi-layer 3D convolutional neural network: The conversion of point cloud to MESH can be achieved through deep learning methods. The goal is to use 3D convolutional networks to extract the three-dimensional structure of building clusters from point clouds, and then generate triangular facet meshes (MESH). In terms of neural network architecture design, the network used should be able to process high-dimensional spatial information and have strong representation capabilities. This embodiment designs a method that can perform convolution operations in local areas of point clouds to extract the geometric features of point clouds. After the convolution operation of the network, a set of feature maps about the geometric shapes of building clusters are obtained. These feature maps can reflect the details and overall form of the buildings.

[0104] Continuous multi-layer 3D convolutional network design: First, the point cloud data is converted into a format suitable for input into the 3D convolutional network. Local features (including spatial position, normal vector, curvature, etc.) are extracted from each sample point in the point cloud. The continuous multi-layer 3D convolutional neural network can automatically learn geometric patterns from the local structure of the point cloud, automatically identifying the shape, boundaries, and structural features of buildings during training.

[0105] 4) Facet Color and Texture Mapping: After generating the triangular facets, the final step is texture and color mapping to ensure a more realistic and detailed model. The appearance of a building cluster depends not only on its geometry but also on visual information such as surface color, lighting, and texture. This process can be completed in the following steps:

[0106] Color sampling and distribution: Triangles are shaded based on the color information or lighting distribution in the point cloud data. By analyzing the color of each point cloud point, the color information is mapped to the corresponding triangle. Texture mapping and optimization: For the textured parts of the building, image texture colors can be applied to the 3D model through texture mapping. This process requires properly unfolding the UV coordinates of the triangles to match the collected texture to the triangles.

[0107] Furthermore, the above four main processes are specifically implemented as follows:

[0108] 1) Data preprocessing

[0109] In the point cloud to MESH neural network reconstruction scheme based on building clusters, the quality of point cloud data directly affects the accuracy and effect of subsequent reconstruction. Therefore, at this stage, this embodiment needs to sample, remove noise, and perform uniform filtering on the original point cloud. This embodiment takes the point cloud of a building cluster in a certain map software as an example.

[0110] Furthermore, in the point cloud sampling of the building complex in the map software, it is assumed that this embodiment obtains an information set containing a large number of points by sampling points on the picture, and each point is represented by its three-dimensional coordinates P i=(x i ,y i ,z i ), where i∈[1, N] is the point index and N is the total number of points. The goal of sampling is to reduce the number of point clouds while preserving important spatial structure information. Common sampling methods include uniform sampling, random sampling, and region-based sampling. Specifically for point cloud data of building clusters, this embodiment uses the uniform sampling method.

[0111] Specifically, the basic idea of ​​uniform sampling is to uniformly select sample points based on the density of the point cloud to ensure that the number of sampling points in each area is relatively uniform. The formula for uniform sampling is:

[0112]

[0113] The sampling process ensures a uniform distribution between the sampling points, thus avoiding the problem of the point cloud being too dense or too sparse in some areas. Next, the noise is identified and removed by analyzing the distribution characteristics of the points in the point cloud neighborhood.

[0114] Furthermore, assuming that the neighborhood of each point in the point cloud contains k nearest neighbors, the local normal vector n i It can be obtained by fitting:

[0115]

[0116] The normal vector represents the point P i By calculating the normal vectors of all points, we can determine the possible noise points in the point cloud and remove points whose normal vectors have large angle differences with neighboring points.

[0117] Specifically, even after noise removal, the density and distribution of point cloud data may still be uneven, especially in complex areas of building complexes. To prevent overly dense point cloud data from increasing the computational complexity of subsequent processing, uniform filtering is often required. The goal of uniform filtering is to adjust the sampling interval of the point cloud based on certain spatial rules to achieve a more uniform distribution of the point cloud in space. This embodiment uses a grid method (a simple grid, different from the subsequent MESH) for spatial partitioning and filtering.

[0118] Specifically, in grid filtering, the point cloud space is divided into grid cells of a certain size, and one point is retained in each grid cell. If there are multiple points in the grid, a representative point in the grid (such as the center point or the center of gravity) is selected to represent the geometric information of the grid area. The specific formula is:

[0119]

[0120] This method ensures a more uniform spatial distribution of the point cloud while avoiding over-refinement of local areas.

[0121] Normal vector calculation is preferably a core step in point cloud processing, which can help identify geometric features in point clouds, such as planes, curved surfaces, and their morphological changes. The method of calculating normal vectors usually relies on fitting local neighborhood points. In particular, in the reconstruction of building clusters, normal vectors are crucial for subsequent 3D reconstruction. Assuming that there are k points in the neighborhood of a point, this embodiment can fit the plane of the neighborhood points:

[0122]

[0123] Where a, b0, and c are the normal vectors of the plane, and d is the offset. The calculated normal vectors can be used to further analyze the curvature characteristics of the point cloud.

[0124] Furthermore, curvature is a measure of the degree of curvature of the point cloud surface. i , the curvature can be measured by calculating the minimum principal curvature κ1 and the maximum principal curvature κ2 of its local area, as follows:

[0125]

[0126] Among them, λ min and λ max Point P i The characteristic values ​​of the minimum and maximum principal curvatures of the surface. These curvature features can help this embodiment identify details of building surfaces and structures.

[0127] Through steps such as point cloud sampling, noise removal, uniform filtering, normal vector calculation, and curvature analysis, this embodiment can effectively improve the quality and usability of point cloud data, and these processes reduce data redundancy.

[0128] 2) Point cloud geometry and uniform filtering

[0129] After sampling and initial preprocessing, the resulting point cloud data remains discrete and may have uneven density or missing geometric features in certain areas. Therefore, geometrization and uniform filtering of the point cloud are key steps in 3D reconstruction. These steps aim to further optimize the distribution of the point cloud data and extract its geometric features, ensuring better meshing and texture mapping in subsequent steps.

[0130] Specifically, the main purpose of geometric feature extraction from point clouds is to analyze and understand the distribution patterns of point clouds in three-dimensional space and identify key structural features (such as planes, curved surfaces, and boundaries). In building cluster point clouds, geometric feature extraction is particularly important for identifying building facades, roofs, edges, and other structures.

[0131] Preferably, in the point cloud geometry processing, neighborhood search is a key step for identifying the local geometric environment of each point. i , you can use the following formula to search and determine its neighboring point set.

[0132] N(P i )={P j |d(P i ,P j )≤r}

[0133] After determining the neighborhood point set, this embodiment performs fitting on the neighborhood points to obtain point P i Normal vector n i :

[0134]

[0135] Normal vector n i It is obtained by minimizing the sum of squares of distances from all neighborhood points to the best fitting plane. The two normal vector calculation formulas mentioned above are applied to different data respectively. The former is for local point clouds, where the scope is smaller.

[0136] Furthermore, the curvature of a point cloud describes the degree of curvature of the surface near each point and is an important indicator for geometric feature extraction. In building clusters, curvature can help identify planes, curved surfaces, and complex shapes.

[0137] Mathematical expression of curvature:

[0138]

[0139] Where: Cur(P i ) is point P i The curvature value, λ min is the minimum eigenvalue of the neighborhood covariance matrix, λ j is the sum of all eigenvalues. When the curvature value is close to 0, it indicates that the area is flat; when the curvature value is large, it indicates that the area has significant curvature features.

[0140] Preferably, in order to further geometrize the point cloud, each local neighborhood needs to be fitted into a plane. For a local area in the point cloud, the plane equation can be used for fitting:

[0141] a1x1+b1y1+c1z1+d1=0

[0142] Where: a1, b1, c1 are the components of the plane normal vector, d1 is the offset between the plane and the origin, x1, y1, z1 are the xyz axis components of the fitting equation. Use the least squares method to fit the plane, with the goal of minimizing the distance from each point to the plane:

[0143]

[0144] By solving this optimization problem, the plane parameters (a2, b2, c2, d2) can be obtained, thereby locally geometrizing the point cloud into a plane or a curved surface.

[0145] Preferably, in large-scale building point cloud processing, the normal vector direction may be inconsistent, which will interfere with the subsequent mesh reconstruction. Therefore, it is necessary to optimize the normal vector consistency. i1 and n j1 , we can use the dot product to determine whether the normals are consistent:

[0146] n i1 ·n j1 ≥0

[0147] If the result is less than 0, then n j1 direction is reversed.

[0148] Furthermore, in order to homogenize the point cloud, a neighborhood density balancing strategy can be introduced. The neighborhood density of each point is defined as:

[0149]

[0150] Where: D(P i2 ) is the point cloud density. k is the number of points in the neighborhood. V i2 is the neighborhood volume.

[0151] For areas with too high density, the density can be reduced by downsampling; for areas with low density, the number of points can be increased by interpolation.

[0152] Preferably, the point cloud data is deeply geometrized and optimized through techniques such as neighborhood search, normal vector estimation, curvature calculation, and geometric feature extraction. The geometric features of the point cloud are effectively extracted, the density is homogenized, and the normal vector direction is adjusted for consistency.

[0153] 3) POINT2MESH construction based on continuous multi-layer 3D convolutional neural network

[0154] In the above steps, this example completes the sampling, preprocessing, and geometrization of point cloud data, resulting in a high-quality and evenly distributed point cloud dataset. Next, a continuous multi-layer 3D convolutional neural network (C-3D-CNN) is used to construct POINT2MESH, converting the discrete point cloud vertex data (V1, V2, V3) into a continuous triangular mesh representation. This process, the core of 3D reconstruction, involves the extraction of spatial features, the learning of the topological structure between point clouds, and the final generation of a triangular mesh.

[0155] Specifically, this embodiment designs a special continuous multi-layer 3D convolutional neural network (C-3D-CNN) to create a deep learning model capable of extracting and learning features from three-dimensional data. Unlike 2D convolution, the 3D convolution in this embodiment introduces a depth dimension into the space, which can capture the spatial structure information of the point cloud. The formula is as follows:

[0156]

[0157] Continuous 3D convolution operations can extract features in three dimensions simultaneously and are suitable for processing three-dimensional point cloud data.

[0158] Furthermore, in the POINT2MESH task, point cloud vertices V1, V2, and V3 serve as input to the C-3D-CNN. Each vertex contains 3D coordinate information and optional color information. To adapt to the C-3D-CNN, the point cloud data needs to be voxelized, mapping the continuous 3D point cloud into a discrete 3D voxel grid.

[0159] If there is a point in the point cloud within the voxel, then V(x, y, z) = 1, otherwise it is 0.

[0160] Each vertex will be mapped to the corresponding voxel unit in C-3D-CNN, followed by feature extraction and learning.

[0161] Preferably, the continuous C-3D-CNN network mainly includes the following modules:

[0162] Input layer: Input the voxelized point cloud data into the network.

[0163] Multi-layer 3D convolutional layers: extract spatial features and learn the topological structure of point clouds.

[0164] Pooling layer: Reduces the dimension of the feature map, reduces the amount of calculation, and uses general mean pooling for dimensionality reduction.

[0165] Output layer: Generates the connection relationship between vertices and faces and constructs a triangular mesh.

[0166] Specifically, the forward propagation process: for the input feature map F of the lth layer (l) And the output feature map F of the l+1th layer (l +1) , the convolution calculation formula is:

[0167] F (l+1) =ReLU(W (l) *F (l) +g (l) )

[0168] 3D convolution operation. During the 3D convolution process, the network extracts the following features:

[0169] Local geometric characteristics: the distribution pattern of points within a neighborhood.

[0170] Global structural features: the morphological and structural information of the entire point cloud.

[0171] Boundary features: Identify the connection relationship between boundary points and patches.

[0172] Local feature extraction formula:

[0173]

[0174] Furthermore, the construction of POINT2MESH:

[0175] After feature extraction is completed, the network needs to transform the point cloud vertices V v1 ,V v2 ,V v3 Connecting into triangular patches. This process is accomplished by a connection strategy based on the arg-max constraint on the interior angles of solid triangles.

[0176]

[0177] The goal is to maximize the minimum interior angle of the triangle to ensure a stable mesh structure.

[0178] Based on the initially generated triangular mesh, topology optimization is required to eliminate redundant faces and unreasonable connections. Surface smoothness:

[0179]

[0180] Geometric consistency:

[0181]

[0182] In order to optimize network performance, a loss function is defined to guide network training. The total loss function is:

[0183] L=λ1E smooth +λ2E geo

[0184] By extracting features, learning, and optimizing the topology of point cloud vertices using a continuous multi-layer 3D convolutional neural network (C-3D-CNN), the team successfully constructed a continuous triangular mesh from the discrete point cloud. This process combines the powerful feature extraction capabilities of deep learning with the geometric constraints of traditional triangulation methods, ensuring the geometric accuracy and structural stability of the mesh.

[0185] 4) Patch color and texture mapping

[0186] After completing the geometry of the point cloud data, uniform filtering, and constructing the POINT2MESH based on a 3D convolutional network, a structurally complete and topologically stable triangular mesh is obtained. However, this mesh only contains geometric structure information and does not include surface color and texture details. Therefore, to achieve high-fidelity 3D reconstruction, it is necessary to map the color information in the point cloud onto the facets of the triangular mesh to complete the color and texture reconstruction. This step will detail the theory and implementation of color and texture mapping, and rigorously describe it through formulas.

[0187] Specifically, in the point cloud sampling stage, color information can be expressed as:

[0188] Color(V i3 )=(r i3 ,g i3 ,b i3 )

[0189] Among them, Color(V i3 ): Vertex V i3 Color information. (r i3 ,g i3 ,b i3 ): The values ​​of the three color channels: red, green, and blue.

[0190] In order to map color information from a point cloud to a triangle, we first need to determine the color source of each triangle. This can be done by weighted averaging the vertex colors:

[0191]

[0192] In actual rendering, the color on the patch not only requires vertex color information, but also needs to ensure that the color transitions smoothly across the entire patch. This embodiment uses bilinear interpolation.

[0193] UV Unwrapping: In order to accurately map the texture image to the triangle mesh, the triangle mesh needs to be UV-unwrapped. UV coordinates are coordinates in the two-dimensional texture space, and the commonly used formula is:

[0194] (U1,V1)=f(x3,y3,z3)

[0195] Where: (U1, V1) are the 2D texture coordinates. (x3, y3, z3) are the 3D mesh vertex coordinates. f(·) is the UV mapping function.

[0196] Preferably, in UV unwrapping, the goal is to minimize texture distortion and stretching between triangles. The energy function represents texture distortion:

[0197]

[0198] The goal is to minimize the above energy function to achieve a uniform distribution of UV coordinates.

[0199] Furthermore, after UV unwrapping, a known texture image T(U, V) is mapped to each triangle of the mesh. For any point on a triangle, its texture color can be sampled using the UV coordinates:

[0200] C(p)=T(U(p),V(p))

[0201] Where C(p) is the color of the point pp on the patch. T(U(p), V(p)) is the color sampled from the texture image using the UV coordinates.

[0202] Preferably, color consistency optimization: In practical applications, there may be color discontinuity or texture seam problems. To optimize color consistency, the following energy function can be used:

[0203]

[0204] By minimizing this energy function, the color inconsistency between patches can be significantly reduced.

[0205] Texture distortion optimization: Texture mapping may have distortion issues, resulting in unnatural visual effects. Texture distortion can be solved by the following optimization goals:

[0206]

[0207] Minimizing the above energy function can reduce texture distortion.

[0208] Optimally, by extracting and interpolating color information from the point cloud, the color information is accurately mapped to the triangles; through UV unwrapping and texture sampling, high-resolution texture images are mapped to the triangle mesh. At the same time, by optimizing color consistency and texture distortion, the quality of the final rendering effect is further improved.

[0209] The beneficial effects of the present invention are as follows:

[0210] The present invention reduces the amount of point cloud data and improves its quality through uniform sampling, noise removal, and grid filtering. Through continuous multi-layer 3D convolutional neural networks, it automatically learns the local geometric patterns of buildings from data and extracts the building features layer by layer, thereby improving the accuracy of the three-dimensional grid model. Through color consistency optimization and texture distortion optimization, it optimizes the color and texture mapping effects and realizes the refined reconstruction of the building.

[0211] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referenced to each other.

[0212] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A 3D reconstruction method based on real estate building cluster point cloud to MESH neural network, characterized by: include: Use uniform sampling to collect image data in mapping software to obtain original point cloud data; Calculating the local normal vector of each point in the original point cloud data, and removing data points corresponding to the local normal vectors whose angle difference in the neighborhood is greater than a preset threshold, to obtain noise-removed point cloud data and local filtered normal vectors; Performing grid filtering and curvature calculation on the noise-removed point cloud data to obtain curvature features; When the product of the local filtered normal vectors corresponding to the to-be-processed point and any point adjacent to the to-be-processed point in the noise-removed point cloud data is less than 0, the direction of the local filtered normal vector corresponding to the to-be-processed point is reversed to obtain a normal vector feature; Performing density uniformity processing on the noise-removed point cloud data using a neighborhood density balancing strategy to obtain density uniform point cloud data; Inputting the density uniform point cloud data and the normal vector features and the curvature features corresponding to the density uniform point cloud data into a continuous multi-layer 3D convolutional neural network to perform triangular mesh generation and topology structure optimization to obtain a triangular optimized mesh; Mapping the color information in the original point cloud data to the point cloud of the triangular optimized mesh, and adding color to each triangular face in the triangular optimized mesh by weighted averaging of vertex colors to obtain the triangular optimized mesh after color addition; Performing UV unfolding and unfolding optimization on the triangular optimized mesh after color addition, and mapping the texture image in the original point cloud data to the unfolded triangular optimized mesh to obtain the triangular optimized mesh after texture addition; Color consistency optimization and texture distortion optimization are performed on the triangular optimized mesh after texture addition to obtain a 3D reconstructed model.

2. A 3D reconstruction method based on real estate building cluster point cloud to MESH neural network according to claim 1, characterized in that: The expression of the uniform sampling is: Wherein, S is the original point cloud data; P i is the i-th point cloud data in the original point cloud data; is the density gradient of the i-th point cloud data; ∈ is the preset sampling error threshold.

3. The 3D reconstruction method based on the conversion of real estate building cluster point cloud to MESH neural network according to claim 2 is characterized in that: The calculation expression of the local normal vector includes: Among them, N(P i )={P j |d(P i ,P j )≤r};n i is the local normal vector of the i-th; N(P i ) is P i Neighborhood; d(P i ,P j ) is P i and P j The Euclidean distance between them; r is the radius threshold of the neighborhood search; n is a variable within the maximum value range of the number of vertices in the domain.

4. The 3D reconstruction method based on the conversion of real estate building cluster point cloud to MESH neural network according to claim 3 is characterized in that: The expression of the grid filter is: Among them, P new is the representative point of the grid unit V; |V| is the number of points in the grid; P m is the mth point cloud data in the grid unit V.

5. The 3D reconstruction method based on converting real estate building cluster point cloud to MESH neural network according to claim 1 is characterized in that: The expression corresponding to the triangular mesh generation includes: F (l+1) =ReLU(W (l) *F (l) +g (l) )、 Among them, F l+1 (x, y, z) is the input feature map of layer l; W(i, j, m1) is the convolution kernel weight; g is the bias term; k0 is the size of the convolution kernel; F (l+1) is the output feature map of the l+1th layer; W (l) is the convolution kernel weight of the lth layer; g (l) is the bias term of the lth layer; f local (V i ) is the vertex V i Local characteristics of N(V i ) is point V i The neighborhood point set of W j is the neighborhood weight; h is the bias constant; is the maximization constraint formula.

6. The 3D reconstruction method based on converting real estate building cluster point cloud to MESH neural network according to claim 5 is characterized in that: The expressions corresponding to the topology optimization include: E smooth =∑ o,q (|V o -V q | 2 )、 L=λ1E smooth +λ2E geo ; Among them, L is the total loss function; E smooth is the surface smoothing energy loss; E geo is the geometric consistency energy loss; λ1 and λ2 are the first loss weight and the second loss weight respectively; is the first-order gradient of the Mesh patch.

7. The 3D reconstruction method based on converting real estate building cluster point cloud to MESH neural network according to claim 6 is characterized in that: The expression of the weighted average of the vertex colors is: Among them, C(F s ) is a triangular patch F s The average color of C(V r ) is point V r Corresponding colors; V1, V2, V3 are triangle patches F s The first vertex, the second vertex, and the third vertex of .

8. The 3D reconstruction method based on converting real estate building cluster point cloud to MESH neural network according to claim 7 is characterized in that: The optimization objective expression corresponding to the expansion optimization is: Among them, E tex is the texture distortion value; A 3D (F x ) is the patch F in three-dimensional space x The area of ​​​​A 2D (F x ) is the patch F in the two-dimensional UV space x area.

9. The 3D reconstruction method based on converting real estate building cluster point cloud to MESH neural network according to claim 8 is characterized in that: The energy function expression of the color consistency optimization is: Among them, E color is the color consistency energy value; F y0 、F z0 are adjacent patches.

10. The 3D reconstruction method based on converting real estate building cluster point cloud to MESH neural network according to claim 9 is characterized in that: The energy function expression of the texture distortion optimization is: Among them, E warp is the texture distortion energy value; For the patch F x The original texture gradient; For the patch F x The texture gradient after mapping.

Images