A building facade three-dimensional reconstruction method and device based on multi-modal data

Abstract

The application discloses a kind of based on multi-modal data's building facade three-dimensional reconstruction method and device, the method includes: first, the multi-modal data to be reconstructed is obtained, and the multi-modal data at least includes normal map and grid point data;Then, the clustering center of three main directions is clustered based on the normal map, and all grid point data is rotated into standard coordinate system based on the clustering center of three main directions;Then, the clustering center is secondarily clustered, and a plurality of substructure clusters corresponding to the clustering center are obtained;Finally, all clustering centers and substructure clusters are plane fitting.It can improve the precision of building facade in three-dimensional reconstruction, avoid the situation that structure is not complete and boundary is fuzzy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of building three-dimensional reconstruction, and particularly relates to a building facade three-dimensional reconstruction method and device based on multi-modal data. BACKGROUND

[0002] The building facade is the most intuitive part of observing the building, and is also an important part of digital city modeling. In the prior art, the three-dimensional reconstruction of the building facade is usually performed by relying on point cloud geometric features or normal maps for segmentation and reconstruction. However, the three-dimensional reconstruction of the building facade in the prior art may contain a large amount of noise and occlusion data. The facade reconstructed by the traditional method is prone to incomplete structure or fuzzy boundary. In addition, the prior art cannot identify the repeated substructures in the same direction, such as multi-layer window sills and continuous window sills, and the three-dimensional reconstruction has low precision.

[0003] Therefore, how to improve the precision of the building facade in three-dimensional reconstruction and avoid incomplete structure and fuzzy boundary is a technical problem to be solved by those skilled in the art. SUMMARY

[0004] The application aims to solve the technical problem of low precision and incomplete structure or fuzzy boundary in the prior art.

[0005] To achieve the above technical purpose, in one aspect, the application provides a building facade three-dimensional reconstruction method based on multi-modal data, which comprises:

[0006] Obtaining multi-modal data to be reconstructed, wherein the multi-modal data at least comprises a normal map and grid point data;

[0007] Clustering the normal map to obtain the cluster centers of three main directions, and rotating all grid point data to a standard coordinate system based on the cluster centers of the three main directions;

[0008] Secondary clustering each cluster center to obtain a plurality of substructure clusters corresponding to each cluster center;

[0009] Fitting a plane to all cluster centers and substructure clusters.

[0010] Further, the clustering of the normal map to obtain the cluster centers of the three main directions specifically comprises:

[0011] Determining the normal vector of each pixel point in the building point cloud corresponding to the building to be reconstructed according to the normal map;

[0012] Mapping the three-dimensional unit vector of each pixel point to an integer interval by linear transformation and encoding as the encoded RGB value of the corresponding pixel point.

[0013] The spatial normal vector of each pixel is obtained by inverse mapping and normalizing the encoded RGB values ​​of each pixel.

[0014] Clustering algorithms are used to cluster all spatial normal vectors to obtain cluster centers in three principal directions.

[0015] Furthermore, the multimodal data also includes RGB images, and after determining the spatial normal vector of each pixel, the method further includes:

[0016] The original RGB values ​​of each pixel are determined based on the RGB image;

[0017] Invalid pixels are deleted based on the original RGB values.

[0018] Furthermore, the step of deleting invalid pixels based on the original RGB values ​​specifically includes:

[0019] Get the raw RGB values ​​of all pixels;

[0020] Pixels whose original RGB values ​​are represented as completely black are treated as invalid pixels and deleted.

[0021] Furthermore, the clustering centers based on the three principal directions transform the original point cloud values ​​into a standard coordinate system, specifically including:

[0022] After orthogonalizing the cluster centers in the three main directions using the singular value decomposition method, three mutually perpendicular direction vectors are generated.

[0023] Determine the projected area of ​​the grid point data corresponding to each direction vector on three planes in the standard coordinate system;

[0024] The coordinate axes corresponding to each direction vector are determined based on the three projected areas corresponding to each direction vector.

[0025] Transform the cluster centers corresponding to the direction vectors onto the coordinate axes corresponding to the direction vectors.

[0026] Furthermore, the step of determining the coordinate axes corresponding to each direction vector based on the three projected areas corresponding to each direction vector specifically includes:

[0027] The three projected areas corresponding to each direction vector on each projection plane are arranged in descending order to obtain the first projected area, the second projected area, and the third projected area. Each direction vector has a first projected area, a second projected area, and a third projected area.

[0028] If the value of the first projected area minus the second projected area corresponding to the direction vector is greater than a preset threshold, then the normal coordinate axis corresponding to the first projected area is used as the coordinate axis corresponding to the cluster center.

[0029] If the value of the first projected area minus the second projected area corresponding to the direction vector is not greater than a preset threshold, then the corresponding coordinate axis is assigned to the direction vector based on the linear allocation algorithm.

[0030] Furthermore, the step of transforming the cluster centers corresponding to the direction vectors onto the coordinate axes corresponding to the direction vectors specifically includes:

[0031] Each of the aforementioned direction vectors is taken as the source axis, while the coordinate axes in the standard coordinate system are taken as the target axes;

[0032] Construct a rotation matrix based on the source axis and the target axis;

[0033] Based on the rotation matrix, the grid point data corresponding to each cluster center is rotated to the corresponding coordinates.

[0034] Furthermore, the secondary clustering of each cluster center specifically includes:

[0035] The dot product of all grid point data in the standard coordinate system with the unit vector of each coordinate axis is performed to obtain the set of dot product values ​​corresponding to each coordinate axis.

[0036] Plot the set of dot product values ​​corresponding to each of the coordinate axes as a density distribution curve;

[0037] Based on the density distribution curve, multiple secondary cluster centers corresponding to each coordinate axis are determined;

[0038] Based on the secondary cluster centers, all grid point data on the corresponding coordinate axes are clustered again.

[0039] The present invention also provides a three-dimensional reconstruction device for building facades based on multimodal data, the device comprising:

[0040] The acquisition module is used to acquire the multimodal data to be reconstructed, wherein the multimodal data includes at least normal maps and grid point data;

[0041] The first clustering module is used to cluster three main directions of cluster centers based on the normal map, and to rotate all grid point data to the standard coordinate system based on the three main directions of cluster centers.

[0042] The second clustering module is used to perform secondary clustering on each of the clustering centers to obtain multiple substructure clusters corresponding to each of the clustering centers;

[0043] The fitting module is used to perform planar fitting on all cluster centers and substructure clusters.

[0044] This invention provides a method and apparatus for 3D reconstruction of building facades based on multimodal data. Compared with existing technologies, this method first acquires the multimodal data to be reconstructed, which includes at least a normal map and grid point data. Then, based on the normal map, cluster centers are formed in three main directions, and all grid point data are rotated to a standard coordinate system based on these three main directions. Next, secondary clustering is performed on each cluster center to obtain multiple substructure clusters corresponding to each cluster center. Finally, all cluster centers and substructure clusters are subjected to planar fitting. This method can improve the accuracy of 3D reconstruction of building facades and avoid incomplete structures and blurred boundaries. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 The diagram shown is a flowchart illustrating the three-dimensional reconstruction method for building facades based on multimodal data provided in the embodiments of this specification.

[0047] Figure 2 The diagram shown is a structural schematic of the three-dimensional reconstruction device for building facades based on multimodal data provided in the embodiments of this specification.

[0048] Figure 3 The diagram shown is a schematic diagram of cluster centers in three main directions in an embodiment of this specification;

[0049] Figure 4 The diagram shown is a schematic diagram of the secondary clustering in an embodiment of this specification. Detailed Implementation

[0050] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] like Figure 1The diagram illustrates a flowchart of a 3D reconstruction method for building facades based on multimodal data, as provided in the embodiments of this specification. While this specification provides the method operation steps or device structures shown in the embodiments or accompanying drawings, based on conventional methods or without creative effort, the method or device may include more or fewer operation steps or module units after partial merging. In steps or structures where there is no logically necessary causal relationship, the execution order of these steps or the module structure of the device are not limited to the execution order or module structure shown in the embodiments or accompanying drawings of this specification. When the method or module structure is applied in actual devices, servers, or terminal products, it can be executed sequentially or in parallel according to the method or module structure shown in the embodiments or accompanying drawings (e.g., in a parallel processor or multi-threaded processing environment, or even in a distributed processing or server cluster implementation environment).

[0052] The 3D reconstruction method for building facades based on multimodal data provided in the embodiments of this specification can be applied to terminal devices such as client and server devices, for example... Figure 1 As shown, the method specifically includes the following steps:

[0053] Step S101: Obtain the multimodal data to be reconstructed, wherein the multimodal data includes at least normal maps and grid point data.

[0054] Specifically, the multimodal data includes RGB images, depth images, normal maps, and grid point data. The RGB images represent color images of the building's exterior, each pixel in the depth image represents the distance between the camera and the building's surface, each pixel in the normal image records the direction of the three-dimensional spatial normal vector at that point, and the grid point data records the coordinates of the three-dimensional point corresponding to each pixel.

[0055] RGB images provide color texture information of the building's exterior, aiding in visualization and structural edge identification. During cluster analysis and result verification, RGB images can be overlaid with normal map results to clearly identify details on the building surface (such as shadows, windows, balcony railings, etc.), avoiding misidentification of these details as wall structures and thus ensuring the accuracy of the analysis.

[0056] Depth images record the depth of each pixel (distance from the camera to the building surface), providing higher density or more complete geometric information than point clouds (grid point data). When point cloud data is missing or noisy, depth images can be used to supplement, improve, or correct the point cloud; additionally, in some implementations, point cloud data can be calculated from depth maps as a source or verification of grid point data.

[0057] Normal map: Stores the spatial normal vector direction of each pixel on the building surface. It is the core data foundation for extracting the main orientation and performing the first clustering. The normal map directly reflects the surface orientation and can be used to identify the three main orthogonal directions of the building facade. It is a key basis for subsequent structural classification and coordinate system correction.

[0058] Mesh point cloud data contains the 3D coordinates of discrete points on the building surface and serves as the foundation for geometric calculations and model reconstruction. All spatial location-related operations (such as coordinate transformation, projected area calculation, density distribution analysis, and final planar fitting modeling) are based on this point cloud. By applying orientation information obtained from image analysis to the mesh point data, it is possible to accurately correlate the building's normal orientation with its 3D coordinates, thereby completing the 3D reconstruction of the facade.

[0059] The various modal data work together: the normal map provides structural orientation information, the RGB image and depth image ensure data integrity and result reliability, and the grid point data undertakes the actual 3D calculation. The results of multimodal analysis are applied to the reconstruction process, making the 3D reconstruction of the building facade more precise and reliable.

[0060] Step S102: Based on the normal map, cluster centers in three main directions are obtained, and all grid point data are rotated to the standard coordinate system based on the cluster centers in the three main directions.

[0061] Specifically, before performing 3D reconstruction of the building facade, it is necessary to determine the orientation of the building facade. The normal map is the basis for determining the surface orientation and structural classification. Therefore, the original point cloud is determined based on the normal map, with three main directions.

[0062] In this embodiment of the application, the step of clustering the clusters in three main directions based on the normal map specifically includes:

[0063] The normal vector of each pixel in the building point cloud corresponding to the building to be reconstructed is determined based on the normal map.

[0064] The three-dimensional unit vector of the normal vector of each pixel is mapped to an integer range through linear transformation and encoded as the RGB value of the corresponding pixel.

[0065] The spatial normal vector of each pixel is obtained by inverse mapping and normalizing the encoded RGB values ​​of each pixel.

[0066] Clustering algorithms are used to cluster all spatial normal vectors to obtain cluster centers in three principal directions.

[0067] Specifically, the three-dimensional normal vector information of each pixel is first extracted from the normal map. The normal map is a widely used input format in 3D reconstruction, which is usually obtained by performing local plane fitting on depth map or point cloud data: extract the neighborhood of each pixel as the center, use algorithms such as PCA (principal component analysis) to fit the local plane of the region, and then encode the normal vector of the plane into RGB values ​​and store it in the normal map.

[0068] Furthermore, for the normal vector (i.e., a three-dimensional unit vector containing x, y, and z components, each with values ​​ranging from -1 to 1) of each pixel on the building surface, a linear transformation is used to map it to the integer range of [0, 255], and then these values ​​are used as the R, G, and B channel values ​​for that pixel. For example, the transformation can be performed using the formula R = \frac{x+1}{2} \times 255 (the G and B channels are transformed similarly). After this encoding process, the spatial normal vector direction of each pixel is stored in the normal image as RGB color values.

[0069] The inverse mapping of RGB values ​​involves performing the inverse transformation of the linear mapping described above on the RGB values ​​of each pixel when reading the normal map, converting the color value back to a real number in the range [-1, 1]. For example, x = \frac{R}{255} \times 2 - 1 can be calculated (and y and z are calculated similarly). This recovers the initial normal vector components (x, y, z) corresponding to the pixel. Due to quantization errors, this vector may not be strictly unit length and requires further normalization (adjusting the vector length to 1). The normalized 3D vector is the spatial normal vector of that pixel. The spatial normal vector is specifically the actual normal direction vector of the object's surface in 3D space, i.e., the orientation of that point in the real-world coordinate system. The term "spatial normal vector" is mentioned in the encoding / decoding process to distinguish it from the RGB encoded values ​​stored in the image. Simply put, the spatial normal vector is the actual normal direction of a point on the building surface (e.g., the direction perpendicular to the wall or the upward direction of the roof), while the RGB value is just its encapsulated representation.

[0070] In this embodiment of the application, the multimodal data further includes an RGB image. After determining the spatial normal vector of each pixel, the method further includes:

[0071] The original RGB values ​​of each pixel are determined based on the RGB image;

[0072] Invalid pixels are deleted based on the original RGB values.

[0073] The step of deleting invalid pixels based on the original RGB values ​​specifically includes:

[0074] Get the raw RGB values ​​of all pixels;

[0075] Pixels whose original RGB values ​​are represented as completely black are treated as invalid pixels and deleted.

[0076] After reading the normal map, the RGB value of each pixel is first mapped back to a spatial normal vector (e.g., linearly mapping pixel values ​​from [0, 255] to [-1, 1]), and the resulting vectors are normalized to ensure that all normal vectors are unit vectors. Then, invalid regions (such as completely black pixels or regions occluded by a mask) are filtered out, and only the normal data within the valid building area is retained for analysis.

[0077] Specifically, after calculating the spatial normal vector for each pixel, invalid regions are filtered out. The specific implementation method is as follows: traverse all pixel normal vector data obtained from the normal map and check their corresponding original pixel values. If a pixel's RGB value is completely black (e.g., (0,0,0), which usually indicates that the point has no valid normal information and may belong to the background or a region where the sensor has no data), or if the pixel is pre-marked as an occluded / non-building area (i.e., not within the building facade), then the normal vector corresponding to that pixel is considered invalid and removed from subsequent analysis. In other words, only the normal vectors of building surface pixels with valid normal data (areas that are not completely black and not masked) are retained for subsequent clustering analysis. This filtering step eliminates the interference of noise points or irrelevant regions on the clustering results, ensuring that the reconstruction process focuses on the building facade itself.

[0078] Finally, a certain number of vector samples (e.g., 50,000) are randomly sampled from all valid normal vectors, and the KMeans clustering algorithm is used to divide these vectors into 3 classes. Each class of normal vectors represents a major directional distribution, initially corresponding to the three orthogonal directions in the building structure: usually the main structure such as walls and roofs along the X, Y, and Z axes.

[0079] In this embodiment of the application, the transformation of the original point cloud to a standard coordinate system based on the clustering centers of the three main directions specifically includes:

[0080] After orthogonalizing the cluster centers in the three main directions using the singular value decomposition method, three mutually perpendicular direction vectors are generated.

[0081] Determine the projected area of ​​the grid point data corresponding to each direction vector on three planes in the standard coordinate system;

[0082] The coordinate axes corresponding to each direction vector are determined based on the three projected areas corresponding to each direction vector.

[0083] Transform the cluster centers corresponding to the direction vectors onto the coordinate axes corresponding to the direction vectors.

[0084] Specifically, since the three principal directions obtained by KMeans may not be strictly orthogonal, this invention first uses the singular value decomposition (SVD) method to orthogonalize them, generating a set of mutually perpendicular direction vectors to satisfy the right-angle frame constraints commonly found in building structures.

[0085] In this embodiment of the application, determining the coordinate axis corresponding to each direction vector based on the three projected areas corresponding to each direction vector specifically includes:

[0086] The three projected areas corresponding to each direction vector on each projection plane are arranged in descending order to obtain the first projected area, the second projected area, and the third projected area. Each direction vector has a first projected area, a second projected area, and a third projected area.

[0087] If the value of the first projected area minus the second projected area corresponding to the direction vector is greater than a preset threshold, then the normal coordinate axis corresponding to the first projected area is used as the coordinate axis corresponding to the cluster center.

[0088] If the value of the first projected area minus the second projected area corresponding to the direction vector is not greater than a preset threshold, then the corresponding coordinate axis is assigned to the direction vector based on the linear allocation algorithm.

[0089] Specifically, if the area of ​​each cluster differs significantly in the three projection directions, the axis pointing to the largest area is directly assigned to that cluster; if there are multiple possible overlaps or matching conflicts among the three clusters, a linear assignment algorithm (such as the Hungarian algorithm) is used to maximize the overall assigned area score to obtain the optimal cluster-coordinate axis mapping relationship.

[0090] Furthermore, it is necessary to establish a correspondence between these three directions and the X, Y, and Z axes of the standard coordinate system. To this end, this invention calculates two key indicators for the 3D point cloud clustered in each principal direction:

[0091] Variance calculation: Calculate the spatial coordinate variance of each cluster in the X, Y, and Z directions, reflecting the extent of the point cloud's expansion in each direction. A smaller variance in a direction usually means that the points in that cluster are distributed on a plane orthogonal to that direction.

[0092] Projected area calculation: Project the point cloud of each cluster onto the YZ, XZ, and XY planes respectively. For each projection plane, construct a two-dimensional grid histogram, count the number of non-empty cells, multiply by the grid area, and estimate the projected area. The direction with the larger projected area usually indicates that the cluster represents the structural surface normal vector in that direction.

[0093] The coordinate axes corresponding to each direction vector are determined by the three projected areas corresponding to the direction vectors, and finally the cluster centers corresponding to the direction vectors are transformed onto the coordinate axes corresponding to the direction vectors.

[0094] It should be noted that the purpose of calculating the spatial coordinate variance in this application is to characterize the distribution characteristics of the point clouds of each cluster along different axes. This variance serves as auxiliary information and provides a reference during the coordinate axis assignment process. Although the variance value does not directly appear in the subsequent threshold comparison and axis assignment decision, the variance analysis results help to verify and optimize the correspondence between the principal direction and the coordinate axis. For example, if the variance of the point cloud of a certain cluster is significantly smaller along a certain coordinate axis, it means that the points of that cluster are mainly distributed on a plane orthogonal to that axis (with very little expansion along that axis). This usually suggests that the normal vector of that cluster is close to that axis. In this invention, this can be used to verify the judgment results of the projected area method. For example, if the variance of a cluster in the Z-axis direction is extremely small, it indicates that the points in this cluster are almost on the same height plane, possibly corresponding to a horizontal building structure (such as a roof or floor slab), and its normal should be close to vertically upward (Z-axis). Correspondingly, if the variance of a cluster in the X-axis direction is very small, then the points in this cluster are roughly distributed on a plane perpendicular to the X-axis, implying that its normal is close to the X-axis direction. When the comparison of projected areas cannot create a significant difference, that is, the value of the first projected area minus the second projected area is not greater than a preset threshold, and a linear allocation algorithm is needed for decision-making, the variance distribution of each cluster can also be considered as one of the factors. Directions with smaller variances often correspond to clearer normal orientations, while directions with larger variances indicate that the point cloud is scattered, and the normal direction may not belong to the main axis. Variance calculation provides guidance for matching the main direction with the coordinate axis. When the projected area method can make a clear judgment, variance plays a corroborating role; when the projected area method encounters ambiguous situations, variance information helps to improve the rationality and stability of the matching decision.

[0095] In the above, "non-empty" refers to a grid cell that contains at least one projection point in the grid division of the projection plane. When calculating the projected area, we project the point cloud, i.e., the grid point data of each cluster center, onto a coordinate plane (such as the YZ, XZ, or XY plane) and divide the grid into regular grids on that plane. If a grid cell falls into at least one point cloud projection, the cell is called "non-empty"; otherwise, if no point falls into the projection, it is called an "empty" cell.

[0096] To determine the grid area, a suitable grid resolution is usually pre-set during algorithm implementation. For example, each grid cell can be chosen to represent a fixed size in the actual space (e.g., ΔX × ΔY, assuming each cell covers an area of ​​ΔX meters multiplied by ΔY meters) based on the building size and point cloud density. Thus, the actual area corresponding to each grid cell is ΔA = ΔX × ΔY. Subsequently, by counting the total number of non-empty grid cells N and multiplying it by the cell area ΔA, the coverage area of ​​the point cloud on the projection plane can be approximated: Area approx N × ΔA.

[0097] In short: the criterion for non-empty cells is that the presence of point cloud projections within a grid cell is considered non-empty; the grid area is determined by a preset cell size, with each non-empty cell contributing a fixed cumulative area value. By selecting an appropriate grid size, this method can accurately estimate the distribution area of ​​the point cloud on the projection plane, which can be used to compare the relative sizes of projections in each principal direction.

[0098] In the above, the normal coordinate axis corresponding to the first projected area is used as the coordinate axis corresponding to the cluster center. Specifically, this means that the projection plane itself corresponds to a specific normal coordinate axis. Each projection plane is perpendicular to a certain standard coordinate axis, for example:

[0099] The YZ plane is perpendicular to the X-axis, so the projected area of ​​the YZ plane reflects the degree of expansion of the point cloud relative to the X-axis direction (normal direction is X).

[0100] The XZ plane is perpendicular to the Y-axis, so the projected area of ​​the XZ plane corresponds to the unfolding of the point cloud relative to the Y-axis direction (normal direction is Y).

[0101] The XY plane is perpendicular to the Z-axis, so the projected area of ​​the XY plane reflects the distribution scale of the point cloud relative to the Z-axis direction (normal direction is Z).

[0102] Therefore, when comparing the projected areas of the three planes, the coordinate axis corresponding to the plane with the largest projected area should be determined as the coordinate axis direction that the cluster center should match. For example, if a cluster has the largest projected area in the YZ plane, then the principal direction vector of the cluster corresponds to the X-axis; if the projected area in the XZ plane is the largest, it corresponds to the Y-axis; and if the projected area in the XY plane is the largest, it corresponds to the Z-axis.

[0103] The linear allocation algorithm mentioned above specifically includes:

[0104] Constructing the weight matrix: Calculate the projected area of ​​each clustering direction vector relative to each coordinate axis, forming a 3×3 matrix M. The element M_{ij} in the matrix represents the size of the projected area of ​​the i-th clustering direction vector when it corresponds to the j-th coordinate axis. This area value can be used as the "score" or "weight" for applying the assignment algorithm. Generally, a larger projected area indicates a better match, and a score matrix can be constructed accordingly.

[0105] Finding the optimal match: A linear assignment algorithm (such as the classic Hungarian algorithm) is used to find the optimal one-to-one matching for the above matrix. The algorithm comprehensively considers all possible assignment combinations to find the matching scheme that maximizes the total score (or minimizes the total "cost"), that is, choosing a different coordinate axis for each clustering direction to maximize the sum of the corresponding projected areas.

[0106] Determining the axis assignment result: Through the above optimization, an optimal mapping relationship is obtained for the three cluster directions corresponding to the X, Y, and Z axes, respectively. For example, the algorithm may result in "cluster 1 → X-axis, cluster 2 → Y-axis, cluster 3 → Z-axis" (the specific result depends on the distribution of the projected area). This assignment scheme guarantees global optimality under the projected area data, ensuring that each cluster is aligned with the most suitable coordinate axis.

[0107] This ensures that the correspondence between the three principal directions and the X / Y / Z axes is most reasonable in terms of the overall projected area, avoiding pairing errors that may be caused by subjective or local judgments, such as... Figure 3 The diagram shows the cluster centers that are clustered in three main directions.

[0108] It should be noted that the above allocation algorithm is an example of this specification. Those skilled in the art can use other linear allocation algorithms to allocate corresponding coordinate axes to the direction vector according to the actual working scenario or actual needs. This application does not impose any restrictions.

[0109] In this embodiment of the application, the step of transforming the cluster centers corresponding to the direction vectors to the coordinate axes corresponding to the direction vectors specifically includes:

[0110] Each of the aforementioned direction vectors is taken as the source axis, while the coordinate axes in the standard coordinate system are taken as the target axes;

[0111] Construct a rotation matrix based on the source axis and the target axis;

[0112] Based on the rotation matrix, the grid point data corresponding to each cluster center is rotated to the corresponding coordinates.

[0113] Specifically, after the allocation is completed, this invention constructs a three-dimensional rotation matrix to transform the original point cloud and normal direction from the input coordinate system to the standard XYZ coordinate system. This rotation matrix is ​​calculated based on the orientation transformation relationship between the matched cluster centers (source axis) and the standard orthogonal axis (target axis), and is usually constructed through QR decomposition and determinant adjustment to form an orthogonal matrix.

[0114] More specifically, the source and target axes are determined: the three mutually perpendicular cluster center direction unit vectors (denoted as \mathbf{u}_1, \mathbf{u}_2, \mathbf{u}_3) obtained through SVD orthogonalization correspond to the X, Y, and Z axes of the standard coordinate system, respectively. In other words, \mathbf{u}_1 is close to a horizontal direction, \mathbf{u}_2 is close to a vertical direction, and \mathbf{u}_3 is close to another horizontal direction, and a one-to-one correspondence has been established with the standard axes.

[0115] Construct the direction matrix: Take the components of these three unit vectors in the original coordinate system as column vectors to form a 3×3 matrix \mathbf{U} (the first column is the coordinate component of \mathbf{u}_1, and so on). Since \mathbf{u}_1, \mathbf{u}_2, and \mathbf{u}_3 are orthogonal, the matrix \mathbf{U} should be close to an orthogonal matrix.

[0116] Calculating the rotation matrix: Ideally, the column vectors of matrix `mathbf{U}` represent the basis transformation of the original coordinate system relative to the standard coordinate system. To rotate the original coordinate system to the standard coordinate system, we can take the rotation matrix `mathbf{R} = mathbf{U}^{-1}`. Since `mathbf{U}` is nearly orthogonal, it can be orthogonalized (e.g., through QR decomposition) to become an orthogonal matrix, resulting in `mathbf{U}^{-1} = \mathbf{U}^T`. Therefore, `mathbf{R}` can be approximated as `mathbf{U}^T` (adjusted to ensure it is a right-handed system). To understand it more intuitively, the row vectors of mathbf{R} are the transposes of the vectors mathbf{u}_1,\mathbf{u}_2,\mathbf{u}_3. This means that mathbf{R} maps the source coordinate axis basis vectors mathbf{u}_1,\mathbf{u}_2,\mathbf{u}_3 to the standard coordinate system (1,0,0),(0,1,0),(0,0,1).

[0117] Implementation: In practical calculations, the rotation matrix R is usually obtained by orthogonally decomposing the matrix U. For example, performing QR decomposition on U yields an orthogonal matrix (adjusting the determinant sign if necessary to ensure the use of a right-handed coordinate system), which can then be used as the desired rotation matrix R.

[0118] After completing the above calculations, applying the rotation matrix `mathbf{R}` to the coordinate vector of each point in the original point cloud transforms all points to the standard XYZ coordinate system. This rotation transformation aligns the main orientation axis of the building facade with the standard coordinate axes, laying the foundation for subsequent secondary clustering and precise plane fitting.

[0119] Step S103: Perform secondary clustering on each cluster center to obtain multiple substructure clusters corresponding to each cluster center.

[0120] Specifically, the first clustering only determines the main direction of the building facade. However, the building facade is not a smooth plane. There are multiple substructures on the building facade, such as window sills, parapet walls, etc. Therefore, it is necessary to perform clustering again to make the substructures on the building facade clearer and more accurate.

[0121] In this embodiment of the application, the secondary clustering of each cluster center specifically includes:

[0122] The dot product of all grid point data in the standard coordinate system with the unit vector of each coordinate axis is performed to obtain the set of dot product values ​​corresponding to each coordinate axis.

[0123] Plot the set of dot product values ​​corresponding to each of the coordinate axes as a density distribution curve;

[0124] Based on the density distribution curve, multiple secondary cluster centers corresponding to each coordinate axis are determined;

[0125] Based on the secondary cluster centers, all grid point data on the corresponding coordinate axes are clustered again.

[0126] Specifically, first, the point cloud in the current direction is multiplied by the unit vector of each coordinate axis to obtain the "projection value" of each point in that direction. For example, if the Z direction is being processed, the dot product result represents the height distribution of that point on the Z-axis. If multiple structural surfaces are stacked vertically (such as window sills or parapet walls), their dot product values ​​will be concentrated in different numerical ranges.

[0127] More specifically, this step actually involves calculating the coordinate component values ​​of the point cloud along each coordinate axis. Since the X, Y, and Z axes of the standard coordinate system are orthogonal to each other, each axis can be considered as a normal direction perpendicular to the corresponding projection plane. For example, the unit vector along the X-axis is (1,0,0), along the Y-axis (0,1,0), and along the Z-axis (0,0,1). When calculating the dot product projection:

[0128] Along the X-axis: Take the dot product of the coordinates (x, y, z) of each point in the standard coordinate system with (1, 0, 0). The result is 1 ≤ x + 0 ≤ y + 0 ≤ z = x, which is the X-coordinate value of that point. Perform this operation on the entire point cloud to obtain the set of X-coordinates of all points {x_i}. This is the sequence of projection values ​​of the point cloud along the X-axis.

[0129] Along the Y-axis: dot product of (0,1,0) and (x,y,z) yields y, which is the Y-coordinate of the point. Similarly, the sequence of projection values ​​{y_i} of the point cloud along the Y-axis can be obtained.

[0130] Along the Z-axis: dot product (0,0,1) and (x,y,z) yields z, which is the Z-coordinate of the point, forming the projection value sequence {z_i} of the point cloud along the Z-axis.

[0131] Therefore, the dot product value is essentially the extraction of all coordinate component values ​​of the point cloud along a certain coordinate axis. These values ​​reflect the distribution of the point cloud along that axis. By plotting the projected values ​​along the same axis as a density distribution curve, peaks (high-density areas) can be observed. These peaks often correspond to substructures that repeatedly appear along that direction on the building facade (e.g., floor structures distributed vertically, which will appear as peaks at several heights in the density curve). Based on this, the locations of multiple secondary cluster centers can be determined, and point clouds adjacent to each other along the corresponding axis can be grouped into one class, completing the secondary clustering.

[0132] Therefore, the dot product of the point cloud with the unit vector of the coordinate axis extracts the coordinates of each point in that direction, thus providing a basis for subsequent identification of substructures through density curve analysis.

[0133] Then, this invention uses the **kernel density estimation (KDE)** method on these dot product values ​​to plot a smooth density distribution curve. This curve reflects the density levels of the point cloud at different locations along the current direction. The presence of multiple parallel structures is typically manifested as multiple "distinct peaks" in the curve, each peak corresponding to a spatial layer with a concentrated distribution of points.

[0134] Following the identification of peaks, this invention uses these peaks as the initial centers for KMeans clustering, performing a second clustering of the point cloud in that direction. The role of KMeans at this stage is to divide the point cloud into several groups based on their projection values, creating a compactly distributed "substructure layer" for each group in that direction. For example, in a facade with five window sills, KDE will detect five density peaks. This invention will use these five peaks as the initial cluster centers to perform clustering, ultimately dividing the point cloud into five categories, each corresponding to one of the five window sills. Figure 4 The diagram shown is a schematic of the results after secondary clustering.

[0135] The above operation is based on the fact that parallel structures have a consistent normal vector geometrically, but are discrete in spatial location, and can therefore be distinguished by their "positional differences" in the principal direction. The entire process does not require a preset number of structures, nor does it rely on shape templates, and has a high degree of adaptability.

[0136] Step S104: Perform plane fitting on all cluster centers and substructure clusters.

[0137] Specifically, the point cloud of each sub-cluster is fitted to a plane to generate a mesh structure file that can be exported as .obj. Each cluster (main direction + sub-direction) can be visualized using different colors and superimposed on the original image and normal map to compare the structure boundary.

[0138] In addition, the results can be imported into 3D modeling tools such as PyVista or Rhino to perform structural consistency checks and verify the alignment of each face with the actual building.

[0139] Based on the above-described method for 3D reconstruction of building facades using multimodal data, one or more embodiments of this specification also provide a platform or terminal for 3D reconstruction of building facades using multimodal data. This platform or terminal may include devices, software, modules, plug-ins, servers, clients, etc., using the methods described in the embodiments of this specification, combined with necessary hardware implementation. Based on the same innovative concept, the systems in one or more embodiments provided in this specification are as described in the following embodiments. Since the implementation schemes and methods for solving the system problem are similar, the specific system implementation in the embodiments of this specification can refer to the implementation of the aforementioned methods. Repeated details will not be repeated. The terms "unit" or "module" used below can refer to a combination of software and / or hardware that achieves a predetermined function. Although the systems described in the following embodiments are preferably implemented in software, hardware implementation, and a combination of software and hardware, are also possible and contemplated.

[0140] Specifically, Figure 2 This is a schematic diagram of the module structure of one embodiment of the 3D reconstruction device for building facades based on multimodal data provided in this specification, as shown below.Figure 2 As shown, the 3D reconstruction device for building facades based on multimodal data provided in this specification includes:

[0141] The acquisition module 201 is used to acquire the multimodal data to be reconstructed, wherein the multimodal data includes at least normal maps and grid point data;

[0142] The first clustering module 202 is used to cluster three main directions of clustering centers based on the normal map, and to rotate all grid point data to the standard coordinate system based on the three main directions of clustering centers.

[0143] The second clustering module 203 is used to perform secondary clustering on each of the clustering centers to obtain multiple substructure clusters corresponding to each of the clustering centers;

[0144] The fitting module 204 is used to perform planar fitting on all cluster centers and substructure clusters.

[0145] It should be noted that the system described above may include other implementation methods based on the description of the corresponding method embodiments. The specific implementation methods can be referred to the description of the corresponding method embodiments above, and will not be elaborated here.

[0146] This application also provides an electronic device, including:

[0147] processor;

[0148] Memory used to store the processor's executable instructions;

[0149] The processor is configured to perform the methods provided in the embodiments described above.

[0150] The electronic device provided in this application stores executable instructions for a processor in a memory. When the processor executes these instructions, it first acquires the multimodal data to be reconstructed, which includes at least a normal map and grid point data. Then, based on the normal map, it clusters three main directions into cluster centers, and rotates all grid point data to a standard coordinate system based on these three main directions. Next, it performs secondary clustering on each cluster center to obtain multiple substructure clusters corresponding to each cluster center. Finally, it performs planar fitting on all cluster centers and substructure clusters. This improves the accuracy of building facades during 3D reconstruction and avoids incomplete structures and blurred boundaries.

[0151] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0152] The methods or apparatus described in the embodiments provided in this specification can implement business logic through a computer program and record it on a storage medium. The storage medium can be read and executed by a computer to achieve the effects of the solutions described in the embodiments of this specification, such as:

[0153] Acquire the multimodal data to be reconstructed, wherein the multimodal data includes at least normal plots and grid point data;

[0154] Based on the normal map, cluster centers are generated in three main directions, and all grid point data are rotated to the standard coordinate system based on the cluster centers in the three main directions.

[0155] Secondary clustering is performed on each of the cluster centers to obtain multiple sub-structure clusters corresponding to each cluster center;

[0156] All cluster centers and substructure clusters are subjected to plane fitting.

[0157] The storage medium can include physical devices for storing information, typically digitizing the information and then storing it using electrical, magnetic, or optical methods. The storage medium can include: devices that store information using electrical energy, such as various types of memory, like RAM and ROM; devices that store information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memory, bubble memory, and USB flash drives; and devices that store information using optical methods, such as CDs or DVDs. Of course, there are other readable storage media, such as quantum memories and graphene memories.

[0158] The embodiments in this specification are not limited to conforming to industry communication standards, standard computer resource data update and data storage rules, or the situations described in one or more embodiments of this specification. Slightly modified implementations based on certain industry standards or custom methods or embodiments can also achieve the same, equivalent, or similar, or predictable, implementation effects as described above. Embodiments that utilize these modified or modified methods for data acquisition, storage, judgment, and processing still fall within the scope of optional implementations of the embodiments in this specification.

[0159] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0160] The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or plug-ins may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0161] These computer program instructions can also be loaded onto a computer or other programmable resource data updating device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0163] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for three-dimensional reconstruction of building facades based on multimodal data, characterized in that, The method includes: Acquire the multimodal data to be reconstructed, wherein the multimodal data includes at least normal plots and grid point data; Based on the normal map, cluster centers are generated in three main directions, and all grid point data are rotated to the standard coordinate system based on the cluster centers in the three main directions. Secondary clustering is performed on each of the cluster centers to obtain multiple sub-structure clusters corresponding to each cluster center; Perform plane fitting on all cluster centers and substructure clusters; Specifically, the clustering centers based on the three main directions will transform the original point cloud values ​​into a standard coordinate system, including: After orthogonalizing the cluster centers in the three main directions using the singular value decomposition method, three mutually perpendicular direction vectors are generated. Determine the projected area of ​​the grid point data corresponding to each direction vector on three planes in the standard coordinate system; The coordinate axes corresponding to each direction vector are determined based on the three projected areas corresponding to each direction vector. Transform the cluster centers corresponding to the direction vectors onto the coordinate axes corresponding to the direction vectors; Specifically, determining the coordinate axes corresponding to each direction vector based on the three projected areas corresponding to each direction vector includes: The three projected areas corresponding to each direction vector on each projection plane are arranged in descending order to obtain the first projected area, the second projected area, and the third projected area. Each direction vector has a first projected area, a second projected area, and a third projected area. If the value of the first projected area minus the second projected area corresponding to the direction vector is greater than a preset threshold, then the normal coordinate axis corresponding to the first projected area is used as the coordinate axis corresponding to the cluster center. If the value of the first projected area minus the second projected area corresponding to the direction vector is not greater than a preset threshold, then the corresponding coordinate axis is assigned to the direction vector based on the linear allocation algorithm; Specifically, transforming the cluster centers corresponding to the direction vectors to the coordinate axes corresponding to the direction vectors includes: Each of the aforementioned direction vectors is taken as the source axis, while the coordinate axes in the standard coordinate system are taken as the target axes; Construct a rotation matrix based on the source axis and the target axis; Based on the rotation matrix, the grid point data corresponding to each cluster center is rotated to the corresponding coordinates.

2. The method for three-dimensional reconstruction of building facades based on multimodal data as described in claim 1, characterized in that, The process of clustering three main directions based on the normal map specifically includes: The normal vector of each pixel in the building point cloud corresponding to the building to be reconstructed is determined based on the normal map. The three-dimensional unit vector of the normal vector of each pixel is mapped to an integer range through linear transformation and encoded as the RGB value of the corresponding pixel. The spatial normal vector of each pixel is obtained by inverse mapping and normalizing the encoded RGB values ​​of each pixel. Clustering algorithms are used to cluster all spatial normal vectors to obtain cluster centers in three principal directions.

3. The method for three-dimensional reconstruction of building facades based on multimodal data as described in claim 2, characterized in that, The multimodal data also includes RGB images. After determining the spatial normal vector of each pixel, the method further includes: The original RGB values ​​of each pixel are determined based on the RGB image; Invalid pixels are deleted based on the original RGB values.

4. The method for three-dimensional reconstruction of building facades based on multimodal data as described in claim 3, characterized in that, The step of deleting invalid pixels based on the original RGB values ​​specifically includes: Get the raw RGB values ​​of all pixels; Pixels whose original RGB values ​​are represented as completely black are treated as invalid pixels and deleted.

5. The method for three-dimensional reconstruction of building facades based on multimodal data as described in claim 1, characterized in that, The secondary clustering of each cluster center specifically includes: The dot product of all grid point data in the standard coordinate system with the unit vector of each coordinate axis is performed to obtain the set of dot product values ​​corresponding to each coordinate axis. Plot the set of dot product values ​​corresponding to each of the coordinate axes as a density distribution curve; Based on the density distribution curve, multiple secondary cluster centers corresponding to each coordinate axis are determined; Based on the secondary cluster centers, all grid point data on the corresponding coordinate axes are clustered again.

6. A three-dimensional reconstruction device for building facades based on multimodal data, characterized in that, The device includes: The acquisition module is used to acquire the multimodal data to be reconstructed, wherein the multimodal data includes at least normal maps and grid point data; The first clustering module is used to cluster three main directions of cluster centers based on the normal map, and to rotate all grid point data to the standard coordinate system based on the three main directions of cluster centers. The second clustering module is used to perform secondary clustering on each of the clustering centers to obtain multiple substructure clusters corresponding to each of the clustering centers; The fitting module is used to perform planar fitting on all cluster centers and substructure clusters. Specifically, the clustering centers based on the three main directions will transform the original point cloud values ​​into a standard coordinate system, including: After orthogonalizing the cluster centers in the three main directions using the singular value decomposition method, three mutually perpendicular direction vectors are generated. Determine the projected area of ​​the grid point data corresponding to each direction vector on three planes in the standard coordinate system; The coordinate axes corresponding to each direction vector are determined based on the three projected areas corresponding to each direction vector. Transform the cluster centers corresponding to the direction vectors onto the coordinate axes corresponding to the direction vectors; Specifically, determining the coordinate axes corresponding to each direction vector based on the three projected areas corresponding to each direction vector includes: The three projected areas corresponding to each direction vector on each projection plane are arranged in descending order to obtain the first projected area, the second projected area, and the third projected area. Each direction vector has a first projected area, a second projected area, and a third projected area. If the value of the first projected area minus the second projected area corresponding to the direction vector is greater than a preset threshold, then the normal coordinate axis corresponding to the first projected area is used as the coordinate axis corresponding to the cluster center. If the value of the first projected area minus the second projected area corresponding to the direction vector is not greater than a preset threshold, then the corresponding coordinate axis is assigned to the direction vector based on the linear allocation algorithm; Specifically, transforming the cluster centers corresponding to the direction vectors to the coordinate axes corresponding to the direction vectors includes: Each of the aforementioned direction vectors is taken as the source axis, while the coordinate axes in the standard coordinate system are taken as the target axes; Construct a rotation matrix based on the source axis and the target axis; Based on the rotation matrix, the grid point data corresponding to each cluster center is rotated to the corresponding coordinates.

Images