A building point cloud roof segmentation and structured fitting method based on a dual-mode

Abstract

The application discloses a building point cloud roof segmentation and structured fitting method based on a dual-mode, which comprises the following steps: acquiring large-scene point cloud data of a target area and corresponding image data registered therewith, performing image segmentation on the corresponding image data based on a visual basic model to obtain a mask of a building instance; performing spatial correlation on the mask of the building instance and the large-scene point cloud data to separate out point cloud clusters belonging to different building instances and obtain at least one independent single building point cloud; combining semantic information provided by the image data and geometric information provided by the point cloud data to perform dual-mode roof structure fitting on the single building point cloud to generate a geometric element set describing a roof structure. Through the dual-mode strategy of fusing the semantic information provided by the image data and the geometric information provided by the point cloud data, the accuracy and robustness of building roof segmentation and structure reconstruction are improved, and the problem that it is difficult to generate a roof model which is geometrically consistent and topologically closed without large-scale labeled data is solved.

Description

Technical Field

[0001] This invention relates to the field of building point cloud roof processing technology, and in particular to a method for building point cloud roof segmentation and structured fitting based on dual-modality. Background Technology

[0002] With the widespread acquisition of urban-scale point cloud data (such as airborne LiDAR, terrestrial laser scanning, and multi-view image reconstructed point clouds), how to convert large-scale, unorganized building point clouds into structured roof models usable in CAD / GIS has become a core issue in 3D urban modeling. Existing methods can be mainly divided into three categories: model-driven methods, data-driven methods, and end-to-end methods based on deep learning.

[0003] Model-driven methods typically use predefined geometric primitives (such as planes and cylinders) to describe roof structures, exhibiting good topological regularity and interpretability. However, real-world urban building roof morphologies are complex and diverse, making it difficult for fixed models to cover all situations, resulting in insufficient fitting ability in complex scenarios. Data-driven methods directly utilize point cloud geometric constraints (such as RANSAC plane detection, region growing, and normal clustering) to extract roof planes and topological structures, and further optimize the structure. These methods are highly efficient but sensitive to noise, uneven point density, and parameter settings, easily leading to oversegmentation and missed detection of small-scale slopes. In recent years, with the development of deep models such as PointNet++, DGCNN, and sparse convolutional networks, end-to-end roof reconstruction methods based on deep learning have emerged. These frameworks can predict roof vertices and boundaries end-to-end, possessing high expressive power, but they heavily rely on large-scale labeled data, and there are often domain differences between the training set and the real scene, thus limiting their generalization ability in real-world urban point clouds.

[0004] Furthermore, with the development of visual foundational models (such as the SAM series), robust instance segmentation capabilities can now support zero-shot building segmentation. However, existing methods still struggle to simultaneously meet the following requirements: accurately capturing complex, multi-story, and small-scale building roof structures; maintaining high reconstruction accuracy without a large training set; and providing structured outputs that meet the geometric consistency requirements of CAD / GIS applications. Therefore, there is an urgent need in this field for a method that combines robustness, high accuracy, requires no large amount of training data, and can generate structured roof models. Summary of the Invention

[0005] The purpose of this invention is to provide a bimodal method for roof segmentation and structured fitting of building point clouds. By fusing semantic information provided by image data and geometric information provided by point cloud data in a bimodal strategy, the accuracy and robustness of roof segmentation and structural reconstruction are improved, and the technical problem of existing methods being unable to generate geometrically consistent and topologically closed roof models under the condition of no large-scale labeled data is solved.

[0006] To address the aforementioned technical problems, a first aspect of this invention provides a method for roof segmentation and structured fitting of building point clouds based on a dual-modal approach, comprising the following steps: Acquire large-scale point cloud data of the target area and its corresponding image data, and perform image segmentation on the corresponding image data based on the visual basic model to obtain the mask of the building instance. Spatially correlate the mask of the building instance with the large scene point cloud data to separate point cloud clusters belonging to different building instances, and obtain at least one independent single building point cloud. By combining the semantic information provided by the image data and the geometric information provided by the point cloud data, a bimodal roof structure fitting is performed on the point cloud of the single building to generate a set of geometric elements describing the roof structure.

[0007] Furthermore, the acquisition of large-scale scene point cloud data of the target area and its corresponding image data, and the image segmentation of the image data based on a visual basic model to obtain the mask of the building instance, includes: The corresponding image data is input into a pre-trained visual base model, which is a model capable of zero-shot general segmentation. The visual base model processes the input image data and outputs semantic masks for all building instances in the image data.

[0008] Further, the step of spatially associating the mask of the building instance with the large scene point cloud data to separate point cloud clusters belonging to different building instances and obtain at least one independent single building point cloud includes: Based on the registration relationship between the image data and the large scene point cloud data, the semantic mask of each building instance is back-projected into three-dimensional space, and all three-dimensional points located within the projection range of the semantic mask are extracted from the large scene point cloud data to form a candidate three-dimensional point set for the building instance. For each of the building instances, a density-based spatial clustering analysis is performed on the candidate 3D point set. Based on a preset neighborhood radius threshold and a minimum number of points threshold, a core point set with dense spatial distribution is identified from the candidate 3D point set, and each core point set and its density-reachable points are divided into a core point cloud cluster. For a building instance that is divided into multiple core point cloud clusters, the minimum spatial distance between the bounding boxes of different core point cloud clusters is calculated. If the minimum spatial distance is less than a preset merging threshold, the spatially adjacent core point cloud clusters are merged to obtain a single building point cloud corresponding to a single building instance.

[0009] Furthermore, by combining the semantic information provided by the image data and the geometric information provided by the point cloud data, a bimodal roof structure fitting is performed on the point cloud of the individual building to generate a set of geometric elements describing the roof structure, including: Robust global plane fitting is performed on the point cloud of the individual building. The dominant roof plane is extracted and optimized by distance threshold and normal vector direction constraint to obtain the first plane set. The interior points belonging to the first plane set are removed from the point cloud of the individual building to obtain the residual point cloud. Perform clustering analysis based on local normal vectors on the residual point cloud. By calculating the local normal vector of each point in the residual point cloud and clustering in the normal vector space, fit plane parameters to each cluster to obtain a second set of planes composed of small-scale or fragmented surfaces. The planes in the first set of planes and the second set of planes are merged to generate a unified set of plane hypotheses; Project the point cloud corresponding to each plane in the set of plane assumptions onto its own two-dimensional plane, identify boundary points through angle distribution analysis and perform polygonization and simplification to generate the two-dimensional boundary contour of each plane. Based on the set of plane assumptions and the two-dimensional boundary contours of each plane, a set of geometric elements describing the roof structure is formed.

[0010] Furthermore, the robust global plane fitting is performed on the point cloud of the individual building, and the dominant roof plane is extracted and optimized through distance threshold and normal vector direction constraints to obtain a first set of planes, including: For the point cloud of the individual building, a random sampling consensus algorithm is used to iteratively generate a plane hypothesis, and interior points belonging to the plane hypothesis are filtered based on a preset first distance threshold. Calculate the cosine of the angle between the normal vector of the assumed plane and the vertical direction, and determine whether the cosine of the angle is greater than or equal to a preset direction threshold. If the cosine value of the included angle is less than the preset direction threshold, the current plane is assumed to be a vertical plane and is eliminated. If the cosine value of the included angle is greater than or equal to the preset direction threshold, the current plane assumption is retained, and the plane parameters are refitted based on all its interior points to obtain the optimized dominant roof plane. Traverse the point cloud of the individual building until all dominant planes that satisfy the first distance threshold and the first direction threshold constraints are extracted to form the first plane set.

[0011] Furthermore, the step of iteratively generating the plane hypothesis using a random sampling consensus algorithm includes: Based on the registration relationship between the image data and the point cloud of the individual building, determine the set of roof candidate points in the point cloud of the individual building that correspond to the semantics of the roof area in the image data; In each iteration of the random sampling consensus algorithm, initial sampling points required to generate the planar hypothesis are preferentially selected from the set of roof candidate points; Plane parameters are calculated using the preferred initial sampling points to generate a plane hypothesis.

[0012] Further, the step of performing clustering analysis based on local normal vectors on the residual point cloud involves calculating the local normal vector of each point in the residual point cloud and clustering them in the normal vector space. Plane parameters are fitted to each cluster to obtain a second set of planes composed of small-scale or fragmented surfaces, including: For each point in the residual point cloud, the local normal vector of the point is calculated by principal component analysis based on its preset number of nearest neighbors. In the normal vector space composed of all the local normal vectors, normal vector points with similar directions are clustered to obtain multiple normal vector clusters, wherein the judgment of similar directions is based on the angle between the normal vectors being less than a preset angle similarity threshold. For each normal vector cluster, the original residual points corresponding to all normal vectors belonging to the normal vector cluster are grouped into the same 3D point cloud subset; For each subset of the three-dimensional point cloud, perform plane fitting, calculate the corresponding plane parameters, and add all the fitted planes to the second plane set.

[0013] Further, merging the planes in the first set of planes and the second set of planes includes: Calculate the angle between the normal vectors of any two planes to be merged. If the angle is less than a first preset threshold, the normal vectors are determined to be similar. Calculate the centroid distance between the two planes to be merged. If the centroid distance is less than a second preset threshold, they are determined to be spatially adjacent. If two planes simultaneously satisfy the conditions of similar normal vectors and spatial proximity, they are merged into one plane, and the parameters of the merged plane are refitted.

[0014] Further, the step of projecting the point cloud corresponding to each plane in the set of plane assumptions onto its own two-dimensional plane, identifying boundary points through angle distribution analysis and performing polygonization and simplification to generate the two-dimensional boundary contour of each plane includes: For each plane in the set of plane assumptions, its corresponding point cloud is projected onto the corresponding two-dimensional coordinate system of the plane to obtain a two-dimensional projection point set. For each projection point in the two-dimensional projection point set, calculate its azimuth angle with all neighboring projection points, and generate an ordered angle sequence according to the angle size. Calculate the angle gap between adjacent angles in the ordered angle sequence, and determine the maximum angle gap; Determine whether the maximum angular gap is greater than a preset angular threshold. If so, mark the projection point as a boundary candidate point. Sort and connect all the marked boundary candidate points according to their spatial connectivity to form the initial boundary polygon; The initial boundary polygon is simplified by applying the Douglas-Puk algorithm, which reduces the number of polygon vertices while meeting the preset simplification distance tolerance, thereby generating a simplified two-dimensional boundary contour.

[0015] Furthermore, after describing the set of geometric elements constituting the roof structure, it also includes: For the intersecting planes in the set of plane assumptions, calculate their theoretical intersection lines, and project the adjacent corner points on the two-dimensional boundary contour onto the corresponding theoretical intersection lines for position correction; A spatial index structure is constructed for all corrected corner points, and duplicate corner points whose distance from each other is less than a preset deduplication threshold are retrieved and merged. Based on the merged global corner point set and the aforementioned planar assumption set, a topologically closed three-dimensional surface model of the building roof is constructed.

[0016] Further, the step of projecting adjacent corner points on the two-dimensional boundary contour onto the corresponding theoretical intersection line for position correction includes: For a plane in the set of plane assumptions that intersects with other planes by a theoretical intersection line, the corner points on its two-dimensional boundary contour that are adjacent to the theoretical intersection line are projected onto the corresponding theoretical intersection line for position correction. For an independent plane in the set of plane assumptions that does not intersect with any other plane, for a corner point on its two-dimensional boundary contour, search for its nearest neighbor within a preset range in the point cloud of the individual building, and optimize the position of the corner point based on the spatial distribution of the nearest neighbor.

[0017] Furthermore, the spatial index structure is a KD tree; The merging of duplicate corner points whose distance to each other is less than a preset deduplication threshold includes: For all duplicate corner points retrieved through the KD tree that are less than the preset deduplication threshold, calculate the arithmetic mean of their coordinates, and use the arithmetic mean as the new corner point position after merging.

[0018] Accordingly, a second aspect of the present invention provides an electronic device, including: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the above-described bimodal building point cloud roof segmentation and structured fitting method.

[0019] Accordingly, a third aspect of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-described method for roof segmentation and structured fitting of building point cloud based on bimodality.

[0020] The above-described technical solutions of the embodiments of the present invention have the following beneficial technical effects: 1. By employing a pre-trained zero-shot visual basic model for image instance segmentation and utilizing the registration relationship between image and point cloud data to achieve cross-modal instance association and point cloud purification, this invention achieves highly robust and high-precision automated extraction of building instances in complex large scenes without relying on large-scale scene-specific labeled data. It effectively overcomes the problems of traditional data-driven methods being sensitive to parameters, susceptible to noise interference, and having weak generalization ability and dependence on a large amount of labeled data for deep learning models. 2. By designing and implementing a two-stage, two-strategy roof plane detection mechanism that combines "robust global plane fitting" and "local normal vector clustering analysis", and merging and optimizing the extracted planes based on quantization criteria, it can completely and accurately capture multi-level roof geometry from large-scale dominant slopes to small-scale fragmented surfaces, significantly improving the integrity and geometric accuracy of complex roof shape reconstruction, and avoiding the defects of plane over-segmentation, missed detection and model oversimplification common in traditional methods. 3. By extracting and simplifying the boundary contours of all roof planes in a unified manner, and finally performing systematic geometric consistency processing including intersection correction, independent plane corner point optimization, and global deduplication based on spatial index, this invention can directly output a three-dimensional surface model of a building roof with topological closure, rigorous geometric structure, and water tightness. This meets the requirements of CAD / GIS systems for direct use of structured model data and realizes end-to-end high-quality reconstruction from raw point cloud to usable engineering model. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method for roof segmentation and structured fitting of building point cloud based on dual-modality provided in this embodiment of the invention; Figure 2 This is an overall flowchart of the method for roof segmentation and structured fitting of building point cloud based on dual-modality provided in this embodiment of the invention; Figure 3 This is an algorithm effect diagram of the building point cloud roof segmentation and structured fitting method based on dual-modality provided in the embodiments of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0023] The dual-modal building point cloud roof segmentation and structured fitting method in this invention is suitable for automatically reconstructing high-precision building roof models from mixed data obtained from city-level airborne lidar scanning or large-scale oblique photogrammetry. This supports applications with strict requirements for 3D geographic information, such as smart cities, real estate registration, and communication network planning. The following detailed explanation of each step is provided in conjunction with specific implementation methods.

[0024] Please refer to Figure 1 The first aspect of this invention provides a method for roof segmentation and structured fitting of building point clouds based on a dual-modal approach, comprising the following steps: Step S100: Obtain large-scale scene point cloud data of the target area and its corresponding image data, and perform image segmentation on the corresponding image data based on the visual basic model to obtain the mask of the building instance.

[0025] In typical urban mapping projects, large-scale point cloud data of the target area is acquired using LiDAR sensors mounted on an aerial platform. This point cloud contains hundreds of millions to billions of 3D points, while corresponding high-resolution remote sensing images are acquired by digital cameras on the same platform or through strict spatiotemporal synchronization. To ensure geometric consistency in subsequent cross-modal processing, precise spatial registration of the point cloud and images is performed using known sensor calibration parameters, flight trajectories, and aerial triangulation results, ensuring that each pixel has a clear projection relationship in 3D space. Subsequently, the registered digital orthophotos are input into a visual foundation model (e.g., Segment Anything Model2, or SAM2) pre-trained on a large-scale general image dataset. This model possesses zero-sample transfer capability, requiring no additional training or fine-tuning for building targets. Through semantic understanding of the image content, it automatically outputs pixel-level precision building instance segmentation masks. These masks accurately distinguish the outlines of individual buildings in the image and effectively separate buildings from other features such as roads, vegetation, and water bodies, providing reliable 2D semantic guidance for the subsequent precise location of individual buildings from dense and complex large-scale point clouds.

[0026] Step S200: Spatially associate the mask of the building instance with the large scene point cloud data, separate the point cloud clusters belonging to different building instances, and obtain at least one independent single building point cloud.

[0027] Based on the precise registration relationship established in step S100, the two-dimensional polygon mask of each building instance is inversely calculated to three-dimensional space according to the collinearity equation or projection matrix, forming a projection volume covering the approximate area of ​​the building. Within this projection volume, all three-dimensional points are extracted from the large scene point cloud to form the initial candidate three-dimensional point set for the building. However, this set usually contains background noise points from the building facade, attachments (such as air conditioner outdoor units, solar panels), and background noise points (such as trees, ground points) caused by registration residuals or occlusion. To obtain a clean three-dimensional point cloud that only represents the building roof and main structure, density-based spatial clustering analysis is performed on each candidate set, specifically using an algorithm such as DBSCAN. This algorithm presets two key parameters based on the average density of the point cloud: the neighborhood search radius and the minimum number of neighborhood points required to become a core point. After the algorithm runs, it can identify the core point clusters that are spatially dense and conform to the continuous shape of the building roof, while sparse and discrete points are marked as noise and filtered out. For cases where the point cloud of a building instance is segmented into multiple spatially separated but otherwise belonging to the same entity due to roof obstruction or missing point cloud data, the minimum Euclidean distance between the 3D bounding boxes of these clusters is further calculated. If this distance is less than a preset merging threshold (which is typically set based on the building point cloud sampling interval and typical gap size), these clusters are determined to belong to the same building instance and are merged. The final output is a single building point cloud object with clear geometric boundaries, a clean point cloud, and complete point cloud structure, laying the foundation for subsequent refined roof structure analysis.

[0028] Step S300: Combining the semantic information provided by the image data and the geometric information provided by the point cloud data, perform bimodal roof structure fitting on the point cloud of a single building to generate a set of geometric elements describing the roof structure.

[0029] Step S300 is crucial for transforming point clouds into structured geometric models; its essence is a refined fitting process that deeply integrates semantic priors and geometric constraints. The individual building point clouds generated in step S200 are used as the direct processing object, while continuously referencing the semantic information contained in the image data (such as the overall shape of the roof and boundary orientation). First, robust global plane fitting is performed, using a random sampling consensus algorithm to iteratively search for dominant planes in the point cloud. To improve efficiency and accuracy, a dual-modal guidance is introduced: the initial sampling points when generating plane hypotheses are preferentially selected from a subset of point clouds that semantically correspond to the roof region in the image. For each generated plane hypothesis, not only are interior points filtered using a preset distance threshold, but the cosine of the angle between its normal vector and the vertical direction is also calculated and compared with a preset direction threshold close to 1, thus resolutely eliminating wall hypotheses with near-vertical normal vectors, ensuring that only roof planes (including horizontal and sloping surfaces) are retained. After extracting all dominant planes (the first set of planes), their interior points are removed from the original point cloud to obtain the residual point cloud. Subsequently, a refined clustering analysis based on local normal vectors is performed on the residual point cloud: for each residual point, a precise local normal vector is calculated using principal component analysis based on its k-nearest neighbor set; then, the normal vectors of all points are clustered in the normal vector space (a unit sphere), grouping normal vectors with similar directions (angle less than a preset angle threshold) into one category; the original 3D points corresponding to each normal vector category are considered to belong to the same potential secondary roof patch (such as small-scale slopes, dormer window sides, etc.), thus forming a second set of planes. Next, all planes in the first and second sets are merged and optimized based on their normal vector angles and the distance between their 3D centroids to eliminate redundancy, forming a complete and conflict-free set of roof plane hypotheses. Finally, the point cloud in each plane is orthogonally projected onto the 2D local coordinate system defined by that plane. Boundary points are identified by calculating the polar angle between each projected point and its neighboring points and finding the maximum angular gap. The Douglas-Puk algorithm is then used to simplify the initial polygons obtained by connecting them, ultimately generating a concise and accurate 2D boundary profile for the plane. These planar parameters and their two-dimensional boundary profiles together constitute a set of elements that describe the geometry of the roof and can be further edited and applied.

[0030] Specifically, step S100 involves acquiring large-scale scene point cloud data of the target area and its corresponding image data, performing image segmentation on the image data based on a visual baseline model, and obtaining the mask of the building instance, including: Step S110: Input the corresponding image data into the pre-trained visual base model. The visual base model is a model that can perform zero-shot general segmentation.

[0031] In practice, the acquired "corresponding image data" is generally a digital orthophoto map that has undergone geometric and radiometric correction, and its coverage is completely consistent with the "large-scene point cloud data". The aforementioned "pre-trained visual foundation model" is a key technological carrier. It refers to a deep neural network model that has been trained and has its parameters frozen on ultra-large-scale, diverse, and general-purpose image datasets (such as LAION, COCO, etc.). A typical example is the SAM (SegmentAnything Model) series or its iterations released by Meta. The core capability of this type of model is "zero-shot general segmentation," meaning that without any specific training for "building" category or target area images (i.e., zero-shot), the model can perform open-set instance segmentation on unfamiliar input images solely based on its generalized visual feature representation and reasoning capabilities learned from massive amounts of general-purpose data. In practice, the entire or segmented orthophoto image is input into the model. The model extracts multi-level features through its encoder and generates initial segmentation results based on the decoder and prompting mechanism.

[0032] Step S120: The visual base model processes the input image data and outputs the semantic mask of all building instances in the image data.

[0033] After processing the input image, the model outputs pixel-level "semantic masks." Each mask is a binary image matrix, where a pixel value of 1 precisely corresponds to the projection range of an independent, complete building instance in the image, while a pixel value of 0 belongs to the background or other features. "All building instances" here means that the model can distinguish adjacent buildings, even those that are similar in appearance and closely connected, and segment them into different instance objects. This is crucial for generating independent 3D models for each building. These semantic masks not only contain the building's outline information but also high-level semantic information directly parsed from the image—that this is a building. This semantic information is a prerequisite for subsequent fusion with point cloud geometry. For example, in images of urban blocks with complex vegetation occlusion or industrial areas with diverse building styles, it can effectively distinguish the main building elements such as roofs and walls from distracting elements such as trees and vehicles, providing clean and semantically labeled 2D guiding information for subsequent steps.

[0034] Specifically, in step S200, the mask of the building instance is spatially correlated with the large scene point cloud data to separate point cloud clusters belonging to different building instances, resulting in at least one independent single building point cloud, including: Step S210: Based on the registration relationship between the image data and the large scene point cloud data, the semantic mask of each building instance is back-projected into the three-dimensional space, and all three-dimensional points within the semantic mask projection range are extracted from the large scene point cloud data to form a candidate three-dimensional point set for the building instance.

[0035] This registration relationship is typically established through sensor joint adjustment or feature point matching algorithms to ensure that each pixel in the image can be accurately back-calculated to a ray in 3D space using collinearity equations or projection matrices. In implementation, for each building instance's semantic mask (i.e., one or more polygonal regions) obtained in steps S110-S120, its projection range in 3D space is calculated based on the row and column numbers of the pixels it covers, combined with the image's interior and exterior orientation elements, forming one or more view frustums or projection patches. Then, in the large-scene point cloud data, a spatial range query is performed to extract all points whose 3D coordinates fall within this projection range. These points constitute the "candidate 3D point set" for the building instance. However, due to occlusion during point cloud acquisition (such as trees, other buildings), the presence of facade points, and unavoidable registration residual errors, this candidate set not only includes real roof points but also some wall points, attachment points, and noise points from the ground or other adjacent features. By leveraging precise geometric constraints, a subset of 3D points related to specific building semantics can be quickly and initially selected from a massive point cloud of billions of points in a scene. This provides a foundation for subsequent fine-grained filtering and is far more efficient than performing semantic segmentation or clustering of the entire scene directly in 3D space.

[0036] Step S220: Perform density-based spatial clustering analysis on the candidate 3D point set of each building instance. Based on the preset neighborhood radius threshold and minimum point number threshold, identify the core point set with dense spatial distribution from the candidate 3D point set, and divide each core point set and its density reachable points into a core point cloud cluster.

[0037] The coarse-grained candidate 3D point set obtained in step S210 is purified and initially segmented to identify and retain point sets with typical spatial distribution characteristics of building rooftops, while filtering out noise. Density-based spatial clustering analysis is used here, with DBSCAN being the most commonly used algorithm. The neighborhood radius threshold defines the distance scale for determining whether points are "density connected," typically set based on the average density of the point cloud (points / square meter), for example, 2-3 times the average point spacing. The minimum number of points threshold defines the minimum number of neighboring points required to become a "core point," used to distinguish dense regions from sparse noise, and is usually set to 5-10. The algorithm iterates through each point in the candidate point set. If the number of points within a point's neighborhood radius is greater than the minimum number of points threshold, the point is marked as a core point. All points connected by core points through density reachability relationships are grouped into the same core point cloud cluster. Sparse points that cannot be covered by the neighborhood of any core point are judged as noise points and discarded. After this step, the candidate point cloud of a building instance may be divided into one or more spatially continuous core point cloud clusters with high internal point density. Each cluster represents a continuous area on the roof, while a large amount of discrete noise (such as isolated vegetation points and flying points) has been effectively removed.

[0038] Step S230: For a building instance that is divided into multiple core point cloud clusters, calculate the minimum spatial distance between the bounding boxes of different core point cloud clusters. If the minimum spatial distance is less than a preset merging threshold, merge the spatially adjacent core point cloud clusters to obtain the single building point cloud corresponding to the single building instance.

[0039] Existing methods are prone to incorrect segmentation in the following situations: Severe occlusion (such as tall trees penetrating the roof), missing point clouds, or shadows under eaves preventing laser penetration can cause a physically continuous building roof to be segmented into several spatially separated blocks in the point cloud. Step S220 may divide these blocks into different core point cloud clusters. The purpose of step S230 is to merge these discrete clusters belonging to the same building entity based on spatial proximity criteria. In practice, for a building instance divided into multiple core point cloud clusters in step S220, the minimum spatial distance between the bounding boxes (or convex hulls) of any two clusters is calculated. This minimum distance reflects the degree of separation between the two clusters in three-dimensional space. A "preset merging threshold" is set, which is typically slightly larger than the average sampling interval of the point cloud and considers the continuity of typical building structures (e.g., less than 1 meter). If the calculated minimum spatial distance is less than this merging threshold, the two clusters are determined to be spatially adjacent, and the gap between them is likely caused by missing data rather than actual physical separation (such as courtyards or patios), therefore they are merged into one point cloud cluster. By iterating through this process, one or more integrated, clean point clouds of individual buildings will eventually be generated for each building instance (for buildings that do contain structures such as atriums).

[0040] Specifically, in step S300, the semantic information provided by the image data and the geometric information provided by the point cloud data are combined to perform bimodal roof structure fitting on the point cloud of a single building, generating a set of geometric elements describing the roof structure, including: Step S310: Perform robust global plane fitting on the point cloud of individual buildings, extract and optimize the dominant roof plane through distance threshold and normal vector direction constraints to obtain the first plane set, and remove the interior points belonging to the first plane set from the point cloud of individual buildings to obtain the residual point cloud.

[0041] To efficiently and robustly extract large-scale planes constituting the main roof structure, such as main pitched or flat roofs, from the point cloud of a single building, a random sample consensus algorithm is used as the core framework. A plane hypothesis is initialized by randomly sampling three points from the point cloud, and the Euclidean distance from all points in the point cloud to this hypothetical plane is calculated. By setting a preset distance threshold (usually determined based on point cloud accuracy and noise level, e.g., 0.1 meters), points with distances less than this threshold are classified as inliers of the plane. However, distance constraints alone can incorrectly extract vertical walls as roof planes; therefore, a "normal vector direction constraint" is introduced as the key to semantic-geometric fusion: the unit normal vector of the current plane hypothesis is calculated, and its dot product (i.e., the cosine of the angle) with the vertical direction (usually the Z-axis) is calculated. A direction threshold close to 1 (e.g., 0.8) is set; if the dot product is less than this threshold, it indicates that the plane is too steep, close to the facade, and is resolutely rejected; otherwise, it is retained as a candidate roof plane. For the retained planes, the plane parameters are refitted and optimized using the least squares method with all their inliers. This process is iterated until all dominant planes satisfying the above dual constraints are extracted, forming the first set of planes. Subsequently, all interior points of these identified planes are removed from the original single-unit point cloud to obtain the residual point cloud, which mainly contains secondary slopes, small-scale structures (such as skylights and chimneys), and noise.

[0042] Step S320: Perform clustering analysis based on local normal vectors on the residual point cloud. By calculating the local normal vector of each point in the residual point cloud and performing clustering in the normal vector space, fit plane parameters to each cluster to obtain a second set of planes composed of small-scale or broken surface patches.

[0043] Leveraging the characteristic that even broken roof panels maintain a consistent local orientation, the residual point cloud left over from step S310 is processed to recover roof structures missed by the global fitting strategy due to their small area, sparse point cloud, or irregular geometric shape. First, for each point in the residual point cloud, the covariance matrix is ​​calculated using principal component analysis based on its nearest neighbors (e.g., 10-20). The eigenvector corresponding to the smallest eigenvalue is the local normal vector of that point, providing the most subtle geometric orientation information. Then, the unit normal vectors of all points are considered as points located on a unit sphere, and cluster analysis is performed in this normal vector space (e.g., using K-means with Euclidean distance or angle-based clustering). An angular similarity threshold (e.g., 10 degrees) is set, and normal vectors with directional angles smaller than this threshold are grouped into the same category, forming a normal vector cluster. Each such cluster signifies the existence of a group of points with similar normal vectors in three-dimensional space, likely belonging to the same physical plane, regardless of whether these points are spatially continuous. Next, each cluster of normal vectors is mapped back to the original 3D space, and all its corresponding residual points are grouped into a subset of a 3D point cloud. Finally, plane fitting is performed on each subset to obtain its plane equation. All these planes recovered from the residual point cloud constitute a second set of planes, effectively capturing the details and complex components of the roof.

[0044] Step S330: Merge the planes in the first plane set and the second plane set to generate a unified plane hypothesis set.

[0045] Since steps S310 and S320 may produce slightly different fitting results for the same physical plane due to point cloud distribution (e.g., the edge of the dominant plane and adjacent broken patches actually belong to the same slope), or there may be spurious small planes due to noise, merging is necessary. The merging process is based on explicit geometric criteria. For any two planes to be judged, firstly, the angle between their unit normal vectors is calculated. If the angle is less than a preset first threshold (e.g., 5 degrees), the normal vectors of the two planes are considered similar, i.e., their orientations are basically the same. Secondly, the Euclidean distance between the centroids of the two planes (i.e., the mean of the three-dimensional coordinates of their respective interior points) is calculated. If this distance is less than a preset second threshold (e.g., 1 meter), the two planes are considered spatially adjacent. Only when two planes simultaneously satisfy both the conditions of normal vector similarity and spatial proximity are they considered to be the same roof physical plane, and thus merged. During merging, all interior points of the two planes are collected, and a new, better plane parameter is refitted. After traversing all plane combinations, a unified set of plane assumptions is output, where each plane represents a unique and geometrically consistent patch on the roof.

[0046] Step S340: Project the point cloud corresponding to each plane in the set of plane assumptions onto its own two-dimensional plane, identify boundary points through angle distribution analysis and perform polygonization and simplification to generate the two-dimensional boundary contour of each plane.

[0047] For each plane in the hypothetical set of planes, all its interior points (i.e., the 3D points supporting the plane) are first projected onto the plane's own 2D local coordinate system (UV coordinate system) through orthogonal projection transformation, resulting in a 2D point set. To identify the boundary of the planar region from this scattered 2D point set, an angle distribution-based analysis method is used: for each point in the point set, the angle between its vector to all neighboring points (within a certain radius) and a reference direction (such as the U-axis) is calculated and sorted by size to form a closed angle sequence. Theoretically, if the point is located inside the region, its neighboring points should surround it, and the maximum gap in the angle sequence should be small; conversely, if the point is a boundary point, there will be a lack of neighboring points in a certain direction, resulting in a large gap in the angle sequence. By setting an angle threshold (such as 90 degrees), if the maximum gap exceeds this threshold, the point is marked as a boundary candidate point. All boundary candidate points are sorted and connected according to their natural connection order to form an initial polygon that may contain redundant vertices. Finally, the Douglas-Puk algorithm is applied to simplify the polygon. This algorithm recursively removes vertices that contribute the least to the overall shape, while ensuring that the maximum vertical distance between the simplified polygon and the original boundary point set does not exceed the preset simplification distance tolerance. This results in a concise, accurate, and less vertices-rich two-dimensional boundary profile.

[0048] Step S350: Based on the set of planar assumptions and the two-dimensional boundary contours of each plane, a set of geometric elements describing the roof structure is constructed.

[0049] The intermediate geometric results obtained from the aforementioned processing (i.e., the set of planar assumptions and the two-dimensional boundary contours of each plane) are integrated into a unified structured data set, called the "geometric element set," which fully describes the composition of the roof: the set of planar assumptions defines the spatial position and orientation of each roof surface, while the two-dimensional boundary contours accurately depict the planar shape and extent of each surface.

[0050] Further, in step S310, robust global plane fitting is performed on the point cloud of individual buildings. The dominant roof plane is extracted and optimized using distance thresholds and normal vector direction constraints to obtain a first set of planes, including: Step S311: For the point cloud of a single building, a random sampling consensus algorithm is used to iteratively generate a plane hypothesis, and the interior points belonging to the plane hypothesis are filtered based on a preset first distance threshold.

[0051] Step S311 is the initialization and hypothesis generation stage for robust global plane fitting. In typical building point cloud processing scenarios, a single building point cloud usually contains tens of thousands to hundreds of thousands of 3D points, including points that constitute the various planes of the roof, as well as a small number of residual noise points. To identify the most dominant planar structure from these points, a random sample consensus algorithm is used. This algorithm generates a candidate planar model hypothesis by randomly selecting a minimum sample set (typically three non-collinear points for a plane) from the point cloud. Then, based on a preset distance threshold (e.g., 0.15 meters), the distance from all points in the point cloud to the hypothetical plane is calculated, and points with a distance less than the threshold are classified as "interior points" of the hypothesis. This process can evaluate the size of the point set supporting the hypothesis in a single iteration. Through numerous random iterations, the algorithm can probabilistically explore all possible plane combinations, thus laying the foundation for discovering the true dominant plane in the point cloud.

[0052] Step S312: Calculate the cosine of the angle between the normal vector of the plane assumption and the vertical direction, and determine whether the cosine of the angle is greater than or equal to the preset direction threshold.

[0053] After the RANSAC algorithm generates a hypothetical plane and initially filters out interior points, it's necessary to determine whether this hypothetical plane represents a roof. Since the normal vector of a building roof (whether flat or pitched) typically differs significantly from the absolute vertical direction (flat roofs are nearly horizontal, pitched roofs are sloping), while the normal vector of a building wall (facade) is generally perpendicular to the horizontal plane, this step calculates the cosine of the angle between the unit normal vector of the current hypothetical plane and the vertical direction in 3D space (usually the unit vector along the Z-axis). Simultaneously, a preset direction threshold is established (e.g., 0.8, corresponding to an angle of approximately 37 degrees). By determining whether the calculated cosine of the angle is greater than or equal to this preset threshold, a quantitative assessment is made as to whether the plane is a nearly horizontal roof surface or a nearly vertical wall surface. This step transforms the semantic information from images or common sense—that "a roof should not be vertical"—into a computable geometric constraint.

[0054] Step S313: If the cosine value of the included angle is less than the preset direction threshold, the current plane is assumed to be a vertical elevation and is removed.

[0055] When the calculated cosine of the angle is less than a preset direction threshold, it indicates that the angle between the normal vector of the current plane assumption and the vertical direction is too large, and the plane is spatially closer to a vertical state. In the context of urban building point clouds, such a plane most likely corresponds to the exterior facade of a building, rather than the extracted roof structure. Mistaking it for a roof plane will seriously affect the accuracy of subsequent roof model reconstruction. Therefore, once this condition is triggered, the current plane assumption will be immediately identified as a "vertical facade" and explicitly removed in this and subsequent iterations.

[0056] Step S314: If the cosine value of the included angle is greater than or equal to the preset direction threshold, then retain the current plane assumption and refit the plane parameters based on all its interior points to obtain the optimized dominant roof plane.

[0057] When the cosine of the included angle of the hypothetical plane satisfies the condition of being greater than or equal to a preset direction threshold, the hypothesis is initially identified as a potentially valid roof plane. However, the plane parameters initially generated based on three random points may not be accurate enough. To obtain a better plane model, the plane parameters are refitted using numerical optimization methods such as least squares, utilizing all the "interior points" (i.e., points supporting the hypothesis) selected in step S311. This fitting utilizes more reliable data points belonging to the plane, and the resulting new plane equations (normal vector and intercept) more accurately reflect the true geometry of the roof segment, thus obtaining the optimized dominant roof plane.

[0058] Step S315: Traverse the point cloud of individual buildings until all dominant planes that satisfy the first distance threshold and direction threshold constraints are extracted to form the first plane set.

[0059] The RANSAC algorithm requires multiple iterations (e.g., thousands) to cover all possible dominant planes with a high probability. Steps S311 to S314 describe the complete process of generating, evaluating, filtering, and optimizing a plane hypothesis in one iteration, controlling this process to run cyclically until a preset stopping condition is met (e.g., reaching the maximum number of iterations, or no new dominant planes being found consecutively). After the traversal is complete, all retained and optimized plane hypotheses are checked to ensure they are geometrically significant to each other (usually determined by the number of interior points) and satisfy both distance and orientation thresholds. Finally, all these confirmed dominant planes are grouped together to form the "first plane set".

[0060] Furthermore, step S311, which iteratively generates the plane hypothesis using a random sampling consensus algorithm, includes: Step S3111: Based on the registration relationship between the image data and the point cloud of individual buildings, determine the set of candidate roof points in the point cloud of individual buildings that correspond to the semantics of the roof area in the image data.

[0061] In steps S100 and S200, images precisely registered with the point cloud of individual buildings and semantic masks of building roofs segmented from these images have been obtained. Using this registration relationship, the two-dimensional roof semantic mask is back-projected into three-dimensional space. Specifically, based on the camera imaging model and known spatial transformation parameters, the three-dimensional ray or ground projection area corresponding to each pixel covered by the semantic mask is calculated. Subsequently, spatial querying and filtering are performed on the individual building point cloud to extract all three-dimensional points located within this projection area, forming a subset called the "roof candidate point set." This point set should theoretically mainly contain points from building roofs, largely excluding building facades, balconies, and nearby ground features that may be misprojected, providing a semantically clean and target-clear three-dimensional data foundation for subsequent planar fitting.

[0062] Step S3112: In each iteration of the random sampling consensus algorithm, the initial sampling points required to generate the plane hypothesis are preferentially selected from the set of roof candidate points.

[0063] The semantic prior information provided in step S3111 is embedded into the core mechanism of the Random Sample Consensus (RANSAC) algorithm to significantly improve its efficiency and accuracy in finding the correct plane hypothesis. In the standard RANSAC algorithm, the initial sampling points required to generate the plane hypothesis are randomly selected from the entire input point cloud. In this invention, the algorithm is improved so that, in each iteration, when three points need to be randomly selected to initialize a plane hypothesis, selection is preferentially made from the "roof candidate point set" determined in step S3111. That is, the starting point of the algorithm's exploration is limited to the point cloud portion that is semantically more likely to be the roof region. This "preferential selection" strategy greatly reduces the probability of the algorithm generating invalid plane hypotheses in the early stages due to random sampling of wall points, ground points, or noise points, thereby concentrating computational resources more on exploring truly valid roof planes, accelerating the convergence speed, and improving the reliability of finding all dominant planes within a finite number of iterations.

[0064] Step S3113: Calculate the plane parameters using the preferred initial sampling points to generate a plane hypothesis.

[0065] In step S3112, after selecting three initial sampling points (ensuring they are non-collinear), an initial plane equation is directly calculated and determined based on the 3D coordinates of these three points using analytical geometry. Specifically, a vector is constructed using two of the points, and another vector is constructed using the third point. The cross product of these two vectors is calculated to obtain the initial normal vector of the plane. Then, the coordinates of one point are substituted to solve for the constant term of the plane equation. This generates a parameterized plane model, i.e., a "plane hypothesis." This plane hypothesis will be used as the object of evaluation and optimization in this round of RANSAC iterations. Subsequently, its interior points will be selected in the entire point cloud using a preset distance threshold, and their quality will be judged based on criteria such as the size of the interior points.

[0066] Further, in step S320, clustering analysis based on local normal vectors is performed on the residual point cloud. This involves calculating the local normal vector of each point in the residual point cloud and clustering it in the normal vector space. Plane parameters are fitted to each cluster to obtain a second set of planes composed of small-scale or fragmented surfaces, including: Step S321: For each point in the residual point cloud, calculate the local normal vector of the point through principal component analysis based on its preset number of nearest neighbors.

[0067] The residual point cloud mainly consists of points that failed to be captured by the global plane fitting. These points may belong to small, sparse, or irregularly shaped roof components, such as small slopes, the sides of dormer windows, or the surface of chimneys. To accurately calculate the local normal vector of a point, its local neighborhood needs to be determined first. By setting a "preset number" (e.g., 15), the K nearest neighbors of the point are determined. Then, based on the 3D coordinates of these K neighboring points, their covariance matrix is ​​constructed. By performing principal component analysis on this covariance matrix, three eigenvalues ​​and their corresponding eigenvectors are obtained. The eigenvector corresponding to the smallest eigenvalue indicates the direction in which the local point cloud distribution is least significant, i.e., the normal direction of the local surface. Normalizing this eigenvector yields the "local normal vector" of the point.

[0068] Step S322: In the normal vector space composed of all local normal vectors, cluster the normal vector points with similar directions to obtain multiple normal vector clusters. The judgment of similar directions is based on the fact that the angle between the normal vectors is less than a preset angle similarity threshold.

[0069] Points with similar local orientations are grouped together to identify potential small-scale planar regions with consistent orientations, regardless of whether these points are continuous in 3D space. The unit normal vectors of all points can be considered as points distributed on a unit sphere, this spherical space being the "normal vector space." Clustering algorithms (such as K-means clustering based on Euclidean distance or spectral clustering) are used to group these normal vector points. The core criterion for clustering is similar orientation, specifically defined as the angle between two unit normal vectors being less than a preset angular similarity threshold (e.g., 10 or 15 degrees). By setting this threshold, points with very small angles between their normal vectors can be grouped into the same class, forming a normal vector cluster. Even though the original 3D residual points corresponding to these points may be spatially separated (e.g., small skylights facing the same direction but located at different locations on a roof), their local surface orientations are highly consistent, thus they are judged to likely belong to the same type of geometric plane.

[0070] Step S323: For each normal vector cluster, the original residual points corresponding to all normal vectors belonging to the normal vector cluster are grouped into the same 3D point cloud subset.

[0071] For each normal vector cluster obtained in step S322, there are several eigenvectors with similar directions. Each eigenvector has a definite source in step S321, namely a specific 3D point in the residual point cloud. Therefore, by tracing this correspondence, all the original 3D residual points corresponding to all eigenvectors belonging to the same normal vector cluster can be found and grouped together to form a subset of the 3D point cloud. Although the points in this subset may not be spatially continuous, they have highly consistent local normal vector directions, thus strongly indicating that they originate from the same physical plane (or a set of parallel planes).

[0072] Step S324: Perform plane fitting on each 3D point cloud subset, calculate the corresponding plane parameters, and add all the fitted planes to the second plane set.

[0073] For each subset of 3D point cloud generated in step S323, assuming all points are approximately located on the same plane, a least-squares plane fitting method is used to fit all 3D points in the subset as a whole, calculating an optimal plane equation parameter, including the unit normal vector and the distance from the plane to the origin. This fitted plane represents a small-scale or fragmented surface recovered from the residual point cloud. Collecting all the planes obtained through this process constitutes the "second plane set." This set complements the first plane set obtained in step S310, together covering the complete roof geometry from the main slope to small structural components.

[0074] Further, merging the planes in the first plane set and the second plane set in step S330 includes: Step S331: Calculate the angle between the normal vectors of any two planes to be merged. If the angle is less than the first preset threshold, the normal vectors are determined to be similar.

[0075] In building roof point clouds, due to uneven point cloud density, occlusion, or noise, the same continuous roof slope may be segmented or fitted into multiple hypothetical planes with slight orientation differences. To quantify these orientation differences, this step calculates the unit normal vectors of the two planes to be evaluated and the spatial angle (or cosine value) between them. A small first preset threshold is set (e.g., 5 degrees or 10 degrees). If the calculated angle is less than this threshold, the normal vectors of the two planes are considered similar. This threshold is based on the geometric regularity of typical building roof structures; for example, the local normal vectors of a continuous pitched roof should be basically consistent, and small differences usually originate from fitting errors or local perturbations in the point cloud.

[0076] Step S332: Calculate the centroid distance between the two planes to be merged. If the centroid distance is less than the second preset threshold, they are determined to be spatially adjacent.

[0077] Even if two planes have similar normal vectors, if they are far apart in three-dimensional space, they may belong to different structures in different locations on a building (such as similar sloping roofs of the main building and the annex). Therefore, the "centroids" of the two planes are calculated separately, which are the arithmetic mean of the three-dimensional coordinates of all their interior points. Then, the Euclidean distance between these two centroids is calculated as an indicator of their spatial proximity. A second preset threshold is set (e.g., 1.5 meters or 2 meters), which is typically based on the typical dimensions of the building structure, the point cloud sampling interval, and the maximum break distance that the same roof surface may experience due to occlusion. If the calculated centroid distance is less than this threshold, the two planes are considered "spatially close."

[0078] Step S333: If two planes simultaneously satisfy the conditions of similar normal vectors and spatial proximity, then merge them into one plane and refit the parameters of the merged plane.

[0079] Only when two planes simultaneously satisfy the normal vector similarity condition in step S331 and the spatial proximity condition in step S332 can they be considered to be highly probable redundant representations of the same physical roof plane due to data or algorithmic reasons, and thus can be merged. The merging operation includes data aggregation and model refitting: First, all interior points of the two original planes are aggregated to form a unified, larger 3D point set, which more completely represents the roof surface. Then, using this merged point set, numerical optimization methods such as least squares are employed to recalculate an optimal plane equation (i.e., normal vector and intercept). The parameters of this newly fitted plane more accurately reflect the true geometry of the roof surface than either of the original planes because it is based on more comprehensive and spatially distributed data. Finally, this new plane replaces the original two planes and is added to the optimized set of plane assumptions.

[0080] Further, step S340 involves projecting the point cloud corresponding to each plane in the set of plane hypotheses onto its own two-dimensional plane, identifying boundary points through angular distribution analysis, and performing polygonization and simplification to generate the two-dimensional boundary contour of each plane, including: Step S341: For each plane in the set of plane assumptions, project its corresponding point cloud onto the corresponding two-dimensional coordinate system of the plane to obtain a two-dimensional projection point set.

[0081] For each identified roof plane in the hypothetical set of planes (defined by its normal vector and distance to the origin), it is first necessary to obtain all 3D interior points supporting that plane. To analyze the distribution shape of these 3D points on the plane, they need to be transformed from 3D space to the plane's own 2D parametric coordinate system. Specifically, a local 2D Cartesian coordinate system (UV coordinate system) is established for the plane, where the origin can be chosen as the projection center of the point set in the plane, and the U-axis and V-axis are any set of orthogonal unit vectors on the plane. Subsequently, the 2D coordinates of each 3D interior point in the UV coordinate system are calculated through orthogonal projection, thus obtaining a 2D projected point set. This transformation reduces the 3D boundary identification problem to a point set contour extraction problem on a 2D plane, significantly simplifying the computational complexity while preserving the relative positional relationships of the points in the plane, providing a suitable data foundation for subsequent boundary analysis.

[0082] Step S342: For each projection point in the two-dimensional projection point set, calculate its azimuth angle with all neighboring projection points, and generate an ordered angle sequence by sorting the angles according to their magnitudes.

[0083] For each projected point in the point set, firstly, a neighborhood is defined around it (e.g., a circular area with a fixed radius centered on the point, or its K nearest neighbors). Then, the vector from the point to each of its neighboring points is calculated, and the azimuth angle (i.e., polar angle) between each vector and a reference direction (usually the positive U-axis) is calculated, typically ranging from 0 to 360 degrees. Next, all these calculated azimuth angles are sorted in ascending order to form a closed, cyclical "ordered angle sequence." This sequence visually reflects the distribution of neighboring points around the point in various directions: if the point is located inside the point set, its neighboring points should surround it, and the angle sequence will relatively evenly cover the entire angle range; if the point is located near the boundary, one side may lack neighboring points, thus forming a large blank area in the angle sequence.

[0084] Step S343: Calculate the angle gap between adjacent angles in the ordered angle sequence and determine the maximum angle gap.

[0085] After obtaining the ordered angle sequence, the difference between any two adjacent angle values ​​in the sequence is calculated sequentially; this is known as the "angle gap." Since the angle sequence is cyclical (e.g., from 359 degrees to 0 degrees is considered a gap), the gaps between the beginning and end of the sequence require special handling. After calculating the gaps between all adjacent angles, the largest gap value is identified; this is called the "maximum angle gap." The size of this maximum angle gap directly reflects whether a point has a significantly missing neighboring point in one direction within its neighborhood. If the maximum angle gap is large, it indicates that the point has no neighboring points within a large angular range on one side, strongly suggesting that the point may be located on the boundary of the point set.

[0086] Step S344: Determine whether the maximum angular gap is greater than the preset angular threshold. If so, mark the projection point as a candidate boundary point.

[0087] First, a preset angle threshold (e.g., 90 degrees or 120 degrees) is set. This threshold is used to distinguish between internal points and boundary candidate points. The maximum angular gap of each projected point calculated in step S343 is compared with this threshold. If the maximum angular gap of a point is greater than the preset angle threshold, it is determined that the point has a sufficiently large angular blank area in its neighborhood, and therefore it is marked as a boundary candidate point. Conversely, if the maximum angular gap is less than or equal to the threshold, the neighborhood around the point is considered to be relatively uniformly distributed, belonging to an internal point, and it is not marked. By adjusting the preset angle threshold, the sensitivity of boundary detection can be controlled, balancing boundary integrity and noise resistance.

[0088] Step S345: Sort and connect all marked boundary candidate points according to their spatial connection relationship to form the initial boundary polygon.

[0089] All points marked as boundary candidates are discretely distributed on a two-dimensional plane. They need to be sorted and concatenated according to their inherent spatial connectivity to form closed or open boundary lines. Common methods are based on the nearest neighbor principle or constructing a Delaunay triangulation of the point set to extract its outer edges. By traversing the boundary candidate points, the points closest to each other are connected sequentially to form an ordered sequence of points. This sequence, when connected, constitutes an initial boundary polygon. This initial polygon may contain all detected boundary points, so its number of vertices is often large, and it may contain some minor undulations caused by point cloud noise or local unevenness, making the boundary less smooth and concise.

[0090] Step S346: The Douglas-Puk algorithm is applied to simplify the initial boundary polygon, reducing the number of polygon vertices while meeting the preset simplification distance tolerance, and generating a simplified two-dimensional boundary contour.

[0091] The Douglas-Puk algorithm is employed recursively to remove vertices that contribute the least to the overall shape, ensuring that the maximum perpendicular distance between the simplified polygon and the original polygon does not exceed a preset simplification distance tolerance. Specifically, a baseline is formed by connecting the first and last points of the polygon. The perpendicular distance from all intermediate points to this baseline is calculated, and the point with the maximum distance is identified. If this maximum distance is less than the tolerance, all intermediate points are discarded, and the baseline is used to replace the curve segment. If it is greater than the tolerance, the curve is divided into two segments using this point as the boundary, and the same operation is recursively performed on both segments. Ultimately, the algorithm outputs a simplified polygon with a significantly reduced number of vertices, while retaining key shape features—the final 2D boundary contour.

[0092] Furthermore, after the set of geometric elements describing the roof structure in step S350, the following is also included: Step S361: For intersecting planes in the set of plane assumptions, calculate their theoretical intersection lines, and project the adjacent corner points on the two-dimensional boundary contour onto the corresponding theoretical intersection lines for position correction.

[0093] In building roof structures, different slopes or component planes often intersect in the form of ridge lines, gutters, etc. Step S340 generates a two-dimensional boundary profile for each plane independently. The corner points located at the intersection, after being projected into three-dimensional space, may not strictly fall on the theoretical intersection line of the two intersecting planes. This is due to the discretization of point clouds, fitting errors, and the independent boundary extraction process. First, based on the equations of any two intersecting planes in the plane hypothesis set, the precise "theoretical intersection line" (a straight line in three-dimensional space) is calculated. Then, for each plane boundary profile, a corner point close to this theoretical intersection line (determined by distance) is orthogonally projected onto this theoretical intersection line, and the projected point replaces the original corner point coordinates. This mandatory constraint ensures that the boundary lines from different planes are strictly collinear at the intersection, eliminating model gaps or overlapping patches caused by small point position deviations, and laying a geometric foundation for constructing rigorous topological relationships.

[0094] Step S362: Construct a spatial index structure for all corrected corner points, and retrieve and merge duplicate corner points whose distance from each other is less than a preset deduplication threshold.

[0095] Because multiple planes intersect at a single point (such as the intersection of roof ridges) or are adjacent at boundaries, after correction in step S361, multiple corner points representing the same vertex may exist in very close spatial locations. These "duplicate corner points" can cause degenerate triangles or redundant vertices in the generated 3D mesh, compromising the simplicity and topological correctness of the model. To address this issue, an efficient spatial index structure is first constructed for all corrected 3D corner points, most commonly a KD-Tree. This structure accelerates spatial proximity queries. Subsequently, a small "preset deduplication threshold" (e.g., 0.01 meters) is set, and all groups of corner points with a distance less than this threshold are quickly retrieved using the spatial index. For each such group of duplicate corner points, the arithmetic mean of its 3D coordinates is calculated, and this average coordinate is used as a new, unique corner point, replacing all original corner points in that group. This operation merges redundant vertices with sub-centimeter precision, ensuring that there is only one unique point identifier at each geometric vertex of the model.

[0096] Step S363: Based on the merged global corner point set and the planar assumption set, construct a topologically closed three-dimensional surface model of the building roof.

[0097] The input to the 3D surface model of a building roof consists of a "global corner set" (a unique set of 3D vertices) and a "planar hypothesis set," obtained after geometric correction in step S361 and global deduplication in step S362. The output is a "topologically closed 3D surface model of a building roof." The construction process is based on boundary representation: each roof plane (from the planar hypothesis set) is defined by its 2D boundary contour (after projection and simplification, its corners are mapped to 3D vertices in the global corner set). Based on the correspondence between these contours and vertices, one or more planar polygon patches are generated for each plane (e.g., complex contours are triangulated into triangular meshes). Because the vertices undergo global consistency processing, different planar polygons will necessarily reference the same vertices at shared edges, thus naturally ensuring seamless connections between patches, forming a closed, seamless, manifold 3D surface. The model can be output in a standard 3D file format (such as OBJ, CityGML), which is geometrically accurate and topologically correct, and can be directly imported into CAD software for editing or connected to a GIS platform for spatial analysis.

[0098] Furthermore, step S361, which involves projecting nearby corner points on the two-dimensional boundary contour onto the corresponding theoretical intersection line for position correction, includes: Step S3611: For planes in the set of plane assumptions that intersect with other planes by theoretical intersection lines, project the corner points of their two-dimensional boundary contours that are adjacent to the theoretical intersection lines onto the corresponding theoretical intersection lines for position correction.

[0099] In typical pitched roof buildings, two or more roof slopes intersect along the ridge or valley lines. During the preliminary boundary extraction step, the independently generated 2D contour corner points of each plane often do not precisely lie on the theoretical intersection line of the two planes in 3D space, resulting in tiny gaps or overlapping surfaces at the intersection points in the constructed 3D model. To solve this problem, the precise theoretical intersection line is first calculated based on the mathematical equations (normal vector and intercept) of the two intersecting planes; this is an infinitely extending 3D straight line. Then, the corner points on the 2D boundary contour of each plane closest to this theoretical intersection line (i.e., the vertices that geometrically constitute the roof boundary) are identified. Through orthogonal projection calculations, the 3D coordinates of these corner points are forcibly constrained and moved to this theoretical intersection line, ensuring that they precisely satisfy the equations of the two planes, thus geometrically guaranteeing that the boundary lines of the intersecting planes completely coincide.

[0100] Step S3612: For an independent plane in the set of plane assumptions that does not intersect with any other plane, for the corner point on its two-dimensional boundary contour, search for its nearest neighbor within a preset range in the point cloud of a single building, and optimize the position of the corner point based on the spatial distribution of the nearest neighbor.

[0101] For independent planes in a roof structure that do not intersect with other planes, such as isolated flat roofs or a single pitched roof of a building annex, there are no theoretical intersection lines with other planes to serve as geometric constraints. However, there may still be room for optimization in the corner positions of their boundary profiles to make them more closely resemble the true distribution of the original point cloud, especially in areas where the point cloud density is uneven or the boundary extraction is affected by noise. Therefore, for each corner point on the two-dimensional boundary profile of an independent plane, its spatial nearest neighbor is searched in the original single-building point cloud within a preset range (e.g., a spherical area with a radius of 0.5 meters) centered on its current three-dimensional coordinates. The spatial distribution of these nearest neighbors is analyzed (e.g., calculating their centroids or fitting a local plane), and the position of the original corner point is fine-tuned and optimized based on this distribution information to make it closer to the true physical boundary position represented by the actual point cloud data.

[0102] Furthermore, the spatial index structure is a KD tree. Correspondingly, in step S362, merging duplicate corner points whose distance to each other is less than a preset deduplication threshold specifically involves: for all duplicate corner points retrieved through the KD tree whose distance to each other is less than the preset deduplication threshold, calculating the arithmetic mean of their coordinates, and using the arithmetic mean as the position of the new corner point after merging.

[0103] First, a KD-tree spatial index structure is constructed for all geometrically corrected 3D corner points. This structure efficiently organizes the point set through recursive spatial partitioning, laying the foundation for fast proximity queries. Then, using this index, and with a preset deduplication threshold as the search radius, all spatially too close duplicate corner point groups are retrieved in batches. For each group of duplicate corner points, the arithmetic mean of its X, Y, and Z coordinate components is calculated, and this is used as the new merged corner point position, located at the geometric center of the duplicate point group. Finally, in the global corner point set and all reference relationships, this new corner point uniformly replaces all original corner points within the group.

[0104] like Figure 2 and Figure 3 As shown, the above method will be described in detail below with a specific embodiment: (1) Extraction of dual-modal instances from single building point clouds Extracting point cloud data for a single building using bimodal information from images and point clouds: 1) Input imagery and generate building instance masks using SAM2 or a similar base model; 2) Project the mask onto the point cloud and filter out the 3D points belonging to the same instance; 3) Use spatial clustering (such as DBSCAN) to further remove noise and non-roof points.

[0105] The output is a single building point cloud. ,in .

[0106] (2) Two-stage roof plan inspection method To improve robustness and integrity, this invention proposes a two-stage planar detection strategy.

[0107] 1) Phase 1: Robust Global Plane Fitting The potential plane is calculated using the global plane assumption and statistical constraints: The plane equation is expressed as: ; in It is the unit normal vector.

[0108] Using residuals on candidate planes: ; Construct an interior set, threshold Based on adaptive setting of residual distribution.

[0109] To avoid interference from the wall, the angle between the constraint normal and the vertical direction must satisfy: ; in, .

[0110] The first stage can detect the main roof slopes and horizontal planes.

[0111] 2) Phase 2: Clustering Fitting Based on Normals Local normals are estimated for the remaining point cloud, and small-scale or fragmented roof structures are detected by normal clustering.

[0112] A new plane is fitted to each type of point cloud, and then further merged into the complete set of planes.

[0113] (3) Plane merging and unified optimization For any two planes If the following conditions are met: 1) The included angle of the normal vectors is less than the threshold; 2) The distance between the centroids of the planes is close; Then merge the interior point sets of the two planes and re-estimate the normals: ; (4) Boundary point detection and polygon extraction For each plane, project the point onto the corresponding two-dimensional plane and select a reference point to calculate the polar angle of the projected point: ; Find the difference between the sorted angle sequences: ; like If the corresponding point is a candidate boundary point, all candidate points form a boundary point set, and then the RDP polygon simplification algorithm is used to generate the final boundary polygon.

[0114] (5) Global corner point fusion and topology consistency processing There are geometric redundancies and inconsistencies among the generated planar polygons. This invention employs a global optimization strategy to address these issues uniformly. 1) Plane intersection correction For planes with intersecting lines, according to ; Calculate the intersection line and optimize the position of the corresponding corner point using the neighborhood point set so that it falls precisely on the intersection line.

[0115] 2) Nearest Neighbor Consistency Constraint For simple roofs or corner points with weak geometric relationships, their positions on the planar boundary are adjusted through nearest neighbor search to ensure optimal consistency with the point cloud.

[0116] 3) Global corner deduplication based on KD-Tree Constructing the KD-Tree search radius Repeated corner points within the same area are merged using the average value: ; The final set of corner points possesses global geometric consistency and structural uniqueness.

[0117] (6) Model output The final output includes: 1) Normal directions, center points, and plane equations of each plane of the roof; 2) The polygonal boundary of each plane; 2) A unified global list of corner points and topological relationships.

[0118] Based on the corrected vertex set and plane The final structured 3D model is constructed according to topological relationships (such as triangular meshes or polygonal patches) and output in standard formats such as OBJ, DXF or CityGML.

[0119] Accordingly, a second aspect of the present invention provides an electronic device, including: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the above-described bimodal building point cloud roof segmentation and structured fitting method.

[0120] Accordingly, a third aspect of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-described method for roof segmentation and structured fitting of building point cloud based on bimodality.

[0121] The embodiments of this invention aim to protect a method for roof segmentation and structured fitting of building point clouds based on dual-modality, which has the following effects: 1. By employing a pre-trained zero-shot visual basic model for image instance segmentation and utilizing the registration relationship between image and point cloud data to achieve cross-modal instance association and point cloud purification, this invention achieves highly robust and high-precision automated extraction of building instances in complex large scenes without relying on large-scale scene-specific labeled data. It effectively overcomes the problems of traditional data-driven methods being sensitive to parameters, susceptible to noise interference, and having weak generalization ability and dependence on a large amount of labeled data for deep learning models. 2. By designing and implementing a two-stage, two-strategy roof plane detection mechanism that combines "robust global plane fitting" and "local normal vector clustering analysis", and merging and optimizing the extracted planes based on quantization criteria, it can completely and accurately capture multi-level roof geometry from large-scale dominant slopes to small-scale fragmented surfaces, significantly improving the integrity and geometric accuracy of complex roof shape reconstruction, and avoiding the defects of plane over-segmentation, missed detection and model oversimplification common in traditional methods. 3. By extracting and simplifying the boundary contours of all roof planes in a unified manner, and finally performing systematic geometric consistency processing including intersection correction, independent plane corner point optimization, and global deduplication based on spatial index, this invention can directly output a three-dimensional surface model of a building roof with topological closure, rigorous geometric structure, and water tightness. This meets the requirements of CAD / GIS systems for direct use of structured model data and realizes end-to-end high-quality reconstruction from raw point cloud to usable engineering model.

[0122] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0124] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0125] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for roof segmentation and structured fitting of building point clouds based on dual-modality, characterized in that, Includes the following steps: Acquire large-scale point cloud data of the target area and its corresponding image data, and perform image segmentation on the corresponding image data based on the visual basic model to obtain the mask of the building instance. Spatially correlate the mask of the building instance with the large scene point cloud data to separate point cloud clusters belonging to different building instances, and obtain at least one independent single building point cloud. By combining the semantic information provided by the image data and the geometric information provided by the point cloud data, a bimodal roof structure fitting is performed on the point cloud of the single building to generate a set of geometric elements describing the roof structure.

2. The method for roof segmentation and structured fitting of building point cloud based on dual-modality as described in claim 1, characterized in that, The process of acquiring large-scale scene point cloud data of the target area and its corresponding image data for registration, and performing image segmentation on the image data based on a visual basic model to obtain a mask for building instances includes: The corresponding image data is input into a pre-trained visual base model, which is a model capable of zero-shot general segmentation. The visual base model processes the input image data and outputs semantic masks for all building instances in the image data.

3. The method for roof segmentation and structured fitting of building point clouds based on dual-modality as described in claim 1, characterized in that, The step of spatially associating the mask of the building instance with the large scene point cloud data to separate point cloud clusters belonging to different building instances and obtain at least one independent single building point cloud includes: Based on the registration relationship between the image data and the large scene point cloud data, the semantic mask of each building instance is back-projected into three-dimensional space, and all three-dimensional points located within the projection range of the semantic mask are extracted from the large scene point cloud data to form a candidate three-dimensional point set for the building instance. For each of the building instances, a density-based spatial clustering analysis is performed on the candidate 3D point set. Based on a preset neighborhood radius threshold and a minimum number of points threshold, a core point set with dense spatial distribution is identified from the candidate 3D point set, and each core point set and its density-reachable points are divided into a core point cloud cluster. For a building instance that is divided into multiple core point cloud clusters, the minimum spatial distance between the bounding boxes of different core point cloud clusters is calculated. If the minimum spatial distance is less than a preset merging threshold, the spatially adjacent core point cloud clusters are merged to obtain a single building point cloud corresponding to a single building instance.

4. The method for roof segmentation and structured fitting of building point cloud based on dual-modality according to any one of claims 1-3, characterized in that, The semantic information provided by the image data and the geometric information provided by the point cloud data are combined to perform bimodal roof structure fitting on the point cloud of the individual building, generating a set of geometric elements describing the roof structure, including: Robust global plane fitting is performed on the point cloud of the individual building. The dominant roof plane is extracted and optimized by distance threshold and normal vector direction constraint to obtain the first plane set. The interior points belonging to the first plane set are removed from the point cloud of the individual building to obtain the residual point cloud. Perform clustering analysis based on local normal vectors on the residual point cloud. By calculating the local normal vector of each point in the residual point cloud and clustering in the normal vector space, fit plane parameters to each cluster to obtain a second set of planes composed of small-scale or fragmented surfaces. The planes in the first set of planes and the second set of planes are merged to generate a unified set of plane hypotheses; Project the point cloud corresponding to each plane in the set of plane assumptions onto its own two-dimensional plane, identify boundary points through angle distribution analysis and perform polygonization and simplification to generate the two-dimensional boundary contour of each plane. Based on the set of plane assumptions and the two-dimensional boundary contours of each plane, a set of geometric elements describing the roof structure is formed.

5. The method for roof segmentation and structured fitting of building point cloud based on dual-modality as described in claim 4, characterized in that, The robust global plane fitting of the point cloud of the individual building is performed, and the dominant roof plane is extracted and optimized through distance threshold and normal vector direction constraints to obtain a first set of planes, including: For the point cloud of the individual building, a random sampling consensus algorithm is used to iteratively generate a plane hypothesis, and interior points belonging to the plane hypothesis are filtered based on a preset first distance threshold; Calculate the cosine of the angle between the normal vector of the assumed plane and the vertical direction, and determine whether the cosine of the angle is greater than or equal to a preset direction threshold. If the cosine value of the included angle is less than the preset direction threshold, the current plane is assumed to be a vertical plane and is eliminated. If the cosine value of the included angle is greater than or equal to the preset direction threshold, the current plane assumption is retained, and the plane parameters are refitted based on all its interior points to obtain the optimized dominant roof plane. Traverse the point cloud of the individual building until all dominant planes that satisfy the first distance threshold and the first direction threshold constraints are extracted to form the first plane set.

6. The method for roof segmentation and structured fitting of building point cloud based on dual-modality as described in claim 5, characterized in that, The method of iteratively generating the plane hypothesis using a random sampling consensus algorithm includes: Based on the registration relationship between the image data and the point cloud of the individual building, determine the set of candidate roof points in the point cloud of the individual building that correspond to the semantics of the roof area in the image data; In each iteration of the random sampling consensus algorithm, initial sampling points required for generating the planar hypothesis are preferentially selected from the set of roof candidate points; Plane parameters are calculated using the preferred initial sampling points to generate a plane hypothesis.

7. The method for roof segmentation and structured fitting of building point cloud based on dual-modality as described in claim 4, characterized in that, The step involves performing clustering analysis based on local normal vectors on the residual point cloud. This is achieved by calculating the local normal vector of each point in the residual point cloud and clustering it in the normal vector space. Plane parameters are then fitted to each cluster to obtain a second set of planes composed of small-scale or fragmented surfaces, including: For each point in the residual point cloud, the local normal vector of the point is calculated by principal component analysis based on its preset number of nearest neighbors. In the normal vector space composed of all the local normal vectors, normal vector points with similar directions are clustered to obtain multiple normal vector clusters, wherein the judgment of similar directions is based on the angle between the normal vectors being less than a preset angle similarity threshold. For each normal vector cluster, the original residual points corresponding to all normal vectors belonging to the normal vector cluster are grouped into the same 3D point cloud subset; For each subset of the three-dimensional point cloud, perform plane fitting, calculate the corresponding plane parameters, and add all the fitted planes to the second plane set.

8. The method for roof segmentation and structured fitting of building point cloud based on dual-modality as described in claim 4, characterized in that, The step of merging the planes in the first set of planes and the second set of planes includes: Calculate the angle between the normal vectors of any two planes to be merged. If the angle is less than a first preset threshold, the normal vectors are determined to be similar. Calculate the centroid distance between the two planes to be merged. If the centroid distance is less than a second preset threshold, they are determined to be spatially adjacent. If two planes simultaneously satisfy the conditions of similar normal vectors and spatial proximity, they are merged into one plane, and the parameters of the merged plane are refitted.

9. The method for roof segmentation and structured fitting of building point cloud based on dual-modality as described in claim 4, characterized in that, The step of projecting the point cloud corresponding to each plane in the set of plane hypotheses onto its own two-dimensional plane, identifying boundary points through angle distribution analysis and performing polygonization and simplification to generate the two-dimensional boundary contour of each plane includes: For each plane in the set of plane assumptions, its corresponding point cloud is projected onto the corresponding two-dimensional coordinate system of the plane to obtain a two-dimensional projection point set. For each projection point in the two-dimensional projection point set, calculate its azimuth angle with all neighboring projection points, and generate an ordered angle sequence according to the angle size. Calculate the angle gap between adjacent angles in the ordered angle sequence, and determine the maximum angle gap; Determine whether the maximum angular gap is greater than a preset angular threshold. If so, mark the projection point as a boundary candidate point. Sort and connect all the marked boundary candidate points according to their spatial connectivity to form the initial boundary polygon; The initial boundary polygon is simplified by applying the Douglas-Puk algorithm, which reduces the number of polygon vertices while meeting the preset simplification distance tolerance, thereby generating a simplified two-dimensional boundary contour.

10. The method for roof segmentation and structured fitting of building point cloud based on dual-modality according to claim 4, characterized in that, Following the set of geometric elements constituting the description of the roof structure, it also includes: For the intersecting planes in the set of plane assumptions, calculate their theoretical intersection lines, and project the adjacent corner points on the two-dimensional boundary contour onto the corresponding theoretical intersection lines for position correction; A spatial index structure is constructed for all corrected corner points, and duplicate corner points whose distance from each other is less than a preset deduplication threshold are retrieved and merged. Based on the merged global corner point set and the aforementioned planar assumption set, a topologically closed three-dimensional surface model of the building roof is constructed.

Images