Method and system for reconstructing building information modeling (BIM) data for piping systems - Patents.com
The method uses machine learning to automatically reconstruct BIM models for piping systems from point cloud data, addressing inefficiencies and inaccuracies in existing methods by embedding neighborhood features and performing semantic labeling and connectivity analysis, achieving efficient and accurate model updates.
Patent Information
- Application Number
- JP2024531422
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-26
- Filing Date
- 2022-11-21
- Publication Date
- 2024-11-15
AI Technical Summary
Existing methods for reconstructing BIM models of piping systems are inefficient and prone to errors due to the need for manual preprocessing, parameter adjustments, and assumptions about pipe orientation and size, leading to lengthy and inaccurate model updates.
A method and system using machine learning models to automatically reconstruct BIM for piping systems from unstructured point cloud data without preprocessing, by embedding neighborhood features and predicting pipe points, centerlines, and parameters, utilizing a deep learning network (PipeNet) for semantic labeling and graph-based connectivity analysis.
Enables fast and accurate reconstruction of BIM models for piping systems, reducing manual effort and eliminating the need for prior knowledge, resulting in improved efficiency and accuracy.
Smart Images

Figure 2024542591000001_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to Singapore Patent Application No. 10202113171Y, filed on November 26, 2021, the contents of which are incorporated herein by reference in their entirety for all purposes.
[0002] The present invention generally relates to a method and system for reconstructing building information modeling (BIM) for a piping system. [Background technology]
[0003] Documentation of buildings and facilities was a long, tedious and laborious task in the past when all components were represented by 2D schematic diagrams. With the development of computer-aided design (CAD) technology, documentation has become faster and documentation revisions and updates are easier to track and share among designers, builders, and other stakeholders. CAD further lays the foundation for BIM, which stores not only geometry but also other metadata or attributes, such as material and energy specifications, manufacturers, and costs. Metadata can provide the necessary information through different stages of a facility lifecycle, from planning and construction, operation and maintenance (O&M), to demolition. Leveraging parametric design characteristics, changes to instances of components can be easily updated within BIM. Thus, as an integrated information platform, BIM bridges the gap in communication, collaboration, and interoperability among multiple stakeholders across multiple disciplines.
[0004] Among all the stages of a building life cycle, O&M lasts the longest and has been found to be the biggest source of cost waste. Mechanical, electrical, and plumbing (MEP), as a critical and complex system that provides necessary services in every building, usually requires the most regular maintenance. It has been reported that the cost of MEP system maintenance can account for up to 50% of the total investment in large public projects. Inadequate maintenance of aging MEP systems can cause significant operational, economic, and even environmental impacts. The application of BIM to MEP management has attracted increasing attention in construction inspection and O&M. BIM-assisted MEP O&M can effectively control the waste of maintenance costs. The integrated information platform improves the efficiency of data query and data exchange. 3D geometry and structured metadata provide accurate visualization in time, helping managers make comprehensive decisions. Digital twins enable virtual simulation and monitoring for operation optimization. These benefits can be further amplified by integrating the Internet of Things (IoT) with machine learning methods to achieve strategized maintenance frameworks such as predictive maintenance that provide the necessary maintenance to minimize equipment downtime and extend the operational life of the equipment.
[0005] To utilize BIM for O&M of MEP systems, a properly updated BIM model is required. Although BIM has been gradually advocated and implemented for new buildings and facilities, the BIM model designed by architects and engineers, or the as-designed BIM model, usually contains deviations from the actual built environment due to construction errors or adaptation to site conditions. Traditionally, during project handover, the owner typically sends an inspector to verify the constructed facility against the as-designed BIM model, and requests the builder to correct the errors if necessary and update the as-designed BIM model according to the built conditions. Thus, the updated BIM model is called as-built BIM, which reflects the actual built environment. The as-built verification process is mostly performed manually in the current industry practice, which is time-consuming and inefficient. Surveyors go to the site and scan the site, typically using terrestrial laser scanners, also called Light Detection and Ranging (LiDAR). Point cloud data is generated, which contains a large number (e.g., millions) of points, each coded with the relative position of the contact point of the laser with the object surface from the scanner and sometimes with the surface color or laser reflection intensity. The modeler then imports the point cloud into a BIM authoring software such as REVIT, overlays the point cloud on the as-designed BIM model, and revises the model accordingly. This process applies equally to existing buildings with outdated as-built BIM models due to maintenance and renovation works to be carried out. For older buildings without BIM documentation, the modeler has to create the entire BIM model from scratch, referencing the point cloud. Thus, traditionally, the so-called scan-to-BIM process typically takes several months depending on the estate scale and complexity, of which modeling time can take up to 90%.MEP systems can be more complex and prone to errors in the manual modeling process due to their small size, varying dimensions, relatively free-formed shapes, and hidden or self-occluded locations compared to structural and architectural components such as slabs, walls and doors.
[0006] Many studies have been conducted to enable MEP scan-to-BIM processing. Most of these studies focused on cylindrical piping, the most frequent element in MEP systems, and formulated the problem as cylinder extraction. They relied heavily on manually created geometric features, such as point normals or curvatures, to classify the cylindrical points and calculate the cylindrical parameters. These features are not available in the raw data but are obtained by additional pre-processing steps, which not only consume computational time but may also introduce errors, especially in noisy data. Furthermore, the calculation of these features usually requires a significant step of parameter setting. In environments with complex MEP systems, the setting of multiple parameters may not be straightforward and subsequent parameter tuning needs to be performed. Also, some studies made assumptions regarding the pipe orientation and dimensions that limit the generalization of the method. Furthermore, few studies attempt to maximize the integrity of the resulting piping system connections and therefore do not address various related issues such as incomplete scans and severe occlusion that may be common in multiple point clouds. Several software packages are available on the market to assist the modeling process by extracting some primitive shapes, such as planes and cylinders, from the point cloud. However, a large amount of manual work is still required to make it into a complete BIM model, such as setting parameters, building object connection relationships, and assigning metadata, which is considered to be the bottleneck of the whole modeling process. Moreover, software or tools usually have specific requirements for point cloud data, such as structured point clouds or proprietary formats.
[0007] Reconstructing the piping Point Cloud Feature Extraction Normal vectors are the most used features in 3D geometry processing. For example, we fit planes or higher-order surfaces to calculate surface normals, or perform Principal Component Analysis (PCA) to assign the eigenvector corresponding to the smallest eigenvalue as the normal. These methods often face the problem of neighborhood size selection. Especially when the input point cloud is noisy, crowded, or incomplete, it is important to set the neighborhood size parameter, i.e., too large a neighborhood leads to unreliable local geometric properties, and too small a neighborhood leads to being heavily influenced by noise. The selection of neighborhood size is typically performed experimentally. Several methods have been proposed to identify optimal results by exploring the characteristics of multi-scale neighborhood sizes. For example, a method has been disclosed to improve the estimation of normals by inspecting multiple neighborhoods with a list of pre-set neighborhood sizes. In each neighborhood, a surface roughness value was calculated as the average deviation distance to the best-fitted circle. The neighborhood scale with the smallest surface roughness was used to derive the final normal. Another method was to partition the scene into the largest possible consistent planar neighborhoods by searching top-down through an octree of the point cloud and applying RANSAC plane fitting. Then, all points in the same neighborhood have the plane normal as their normal. Previous work has also presented a deterministic MM-estimator approach that selects multiple neighboring points and then performs PCA to estimate the normals.
[0008] Methods of applying deep learning to point cloud analysis have been disclosed. For example, methods of pure deep learning and methods combining deep learning with multiple geometric feature descriptors or prior knowledge have been proposed for point normal estimation. A combination of PCA and convolutional neural network (CNN) for estimating point normals has also been disclosed. A query point, along with its neighbors, was projected onto a plane defined by the normals calculated by PCA to generate a height map. All height maps were trained with a CNN to classify multiple query points as edge points or multiple non-edge points, and another two CNNs were trained to predict normals for edge and non-edge points, respectively. It has also been disclosed to use a graph neural network to iteratively refine the weights of multiple neighbors, followed by weighted least-square plane fitting. In general, methods based on deep learning can outperform traditional methods and be more robust to noise. Nevertheless, the root mean square (RMS) angular error can vary between 6 and 23 degrees with noise levels smaller than 10% of the object bounding box size.
[0009] Curvature is another commonly used geometric feature. Ideally, for a point on a cylindrical surface, its maximum principal curvature value is equal to the inverse of the cylinder radius with its direction along the surface normal, and its minimum curvature value is equal to zero with its direction along the cylinder centerline axis. Different surface fitting methods have been used to estimate point curvatures, such as NURBS (non-uniform rational basis spline) and quadric surfaces. A method is also disclosed for reducing the 3D curvature estimation problem to iterative circle fitting on a projected surface that contains both the query points and the point normal. The circles with the maximum and minimum radii were selected to determine the final principal curvatures.
[0010] Pipe detection Object detection from point clouds involves extraction and object classification of points corresponding to target objects. For piping, the problem can be formulated as cylinder primitive detection. A common instance segmentation method is region growing, which clusters points close to seed points with similar point features, e.g., point normal to best fitting circle and uniformity. For example, a method is disclosed for extracting pipe segments by finding connected point clusters in both a projection plane and a k-nearest neighbors (k-NN) graph in 3D space, and then classifying the clusters by circle detection in the refined projection plane. Also disclosed is a method for extracting pipe instances by detecting common orientations of pipes in a scene using Gaussian spheres and detecting circles in the projection planes. The multiple clusters are examined again to filter out non-cylindrical objects, for example, circle detection in projections along the main directions, RANSAC (RANdom SAmple Consensus) cylinder fitting, projection analysis based on eigenvectors, curvature analysis, or checking the curvature, i.e., cylinder radius, against existing part catalog P&IDs (Piping and Instrumentation Diagrams). Some other studies used existing CAD or BIM models for target object extraction.
[0011] Several studies have proposed applying deep learning neural networks to point cloud segmentation. For example, a method combining deep learning and convex optimization has been proposed to segment random objects into planes. The goal of such studies is to segment point clouds into different structural and architectural objects, such as walls, slabs, and fixtures, through training on similar datasets. However, due to the lack of MEP components in available open source datasets, there is little literature on applying deep learning methods to MEP-related object detection. For example, a method has been disclosed for creating an industrial MEP equipment dataset and training a customized PointNet++ for MEP object segmentation. However, the dataset has not been made publicly accessible. A method has also been disclosed for training PointNet on point clouds synthesized from as-designed BIM models to perform point cloud semantic classification, and then using Euclidean clustering to further segment the points of each class into object instances. It is further disclosed how to build a DeepPipe neural network with a PointCNN backbone and train the neural network on a synthetic dataset to semantically classify multiple MEP objects in a point cloud and predict, for each pipe point, its radius and offset vector from the point to the centerline. The radius prediction is formatted as a classification task using a predefined discrete list. It is also disclosed how to develop a semantic classification network consisting of a CNN followed by a recurrent neural network (RNN) and train the network on privately scanned and manually labeled data. However, all of these works directly applied deep learning networks to multiple point clouds of the entire scene.Due to the limited memory capacity of conventional computing power, in most cases raw point clouds cannot be directly loaded onto devices and must be preprocessed first. In this regard, methods have been disclosed that divide point clouds into smaller parts, e.g., 1m×1m blocks with equal number of points sampled in each block, and directly downsample the point clouds with a grid (voxel) size of 3cm or 4cm. Therefore, these methods are essentially not applicable to relatively large scenes with small pipes (e.g., with a diameter of less than 5cm) because the small pipes tend to be filtered out in the downsampling process. Besides the above-mentioned method of applying deep learning networks to point clouds, a method has also been disclosed that uses a CNN network, DeepLabv3+, trained on manually labeled 2D MEP images to detect MEP components from 2D images. The classification results were then transferred to a point cloud and DBSCAN was applied to further segment the points of each class into instances.
[0012] Piping Modeling Common methods used in piping modeling are RANSAC and Hough transform (HT). For example, a RANSAC cylinder fitting method has been proposed, which uses two random data points along with their normal vectors. The direction of the cylinder centerline was found by the cross product of those normal vectors, and the radius was the distance from one of those data points to the centerline. However, in practice, many false positives may occur when the RANSAC method is directly applied to the point cloud of a large-scale scene with multiple pipes. To remove the false positives, some thresholds can be set, such as the number, length, radius, and angle point range of inlier points. Apart from RANSAC cylinder fitting, a method combining RANSAC plane or circle fitting with a Gaussian sphere has also been proposed. This is because this method has the property that a Gaussian sphere with multiple normal vectors of multiple points from a cylinder exhibits a 2D circular pattern. RANSAC plane fitting is usually applied to a Gaussian sphere to detect the cylinder centerline direction, and RANSAC circle fitting is used to estimate the cylinder radius. For example, a method is disclosed that formulates the 3D cylinder detection problem into a 2D circle detection, assuming the cylinder direction is either perpendicular or parallel to the ground plane.
[0013] The HT method detects primitive shapes by a parameter voting mechanism. The large memory space required by the 5-dimensional Hough accumulator prohibits the direct application of HT for 3D cylinder detection, especially with input data that may contain millions of points. For example, a 2-step sequential HT has been proposed to reduce the Hough space dimensionality. The cylinder centerline direction is first estimated using a 3D Hough Gaussian sphere, and then the radius and position estimation are formulated as a HT circle detection problem on the projection plane. The Manhattan-world assumption has been applied in some studies, i.e., multiple pipes are assumed to be installed along three orthogonal directions in the scene. For example, a method has been disclosed that utilizes the Manhattan-world assumption and applies 2D HT circle detection on the projection of multiple orthogonal slices at a given distance interval. In practice, the Hough accumulator needs to be discretized at a certain granularity, resulting in a dilemma: a higher granularity provides a more accurate estimate at the expense of computation time, while a lower granularity increases computation speed and requires less memory space, but compromises the accuracy of the results. In an attempt to solve this problem, several modifications to sequential HT have been proposed, such as iteratively increasing the granularity in multiple regions of interest in Hough space, or sampling cells more densely with a preference for multiple orientations along which the cylinders may be located according to the user's prior knowledge of the input scene.
[0014] There is another category of skeleton-based methods, which follows the principle that pipes, including straight cylindrical segments and connectors such as elbows and tees, can be represented in an abstract skeleton form defined by a centerline and a radius. Skeleton nodes were proposed to be estimated by curvature calculations, Voronoi diagram-based methods, and RANSAC sphere fitting. The Voronoi diagram method implicitly required relatively large angular scan coverage of the pipes, and the RANSAC sphere fitting method required prior knowledge of the cylinder diameter range to adjust the neighborhood size for candidate generation. Pipes with large radii cannot be detected, since a large neighborhood increases the computation time and a small neighborhood is potentially misclassified as planar. For example, a method has been disclosed in which centerline nodes are estimated by applying circle fitting to cross-sectional slices at predetermined interval steps along orthogonal axes and other known directions, and then DBSCAN is used to cluster the centerline nodes.
[0015] Other statistical analysis methods for cylinder modeling have also been proposed, such as PCA for centerline orientation estimation and circle fitting on 2D projections. PCA orientation estimation may be inaccurate for short pipe segments. It is also observed that in real scan data, circle fitting on 2D projections may be inaccurate for long and small pipes, because they do not always lie perfectly straight due to gravity or installation errors. A method has been proposed to fit ellipses to the intersections of random slice planes and point clouds, and generate cylinder candidates based on the geometric relationship between the ellipses and cylinders. The cylinder candidates were then clustered by maximizing the pairwise similarities of all candidates determined by the distances between the cylinders. However, generating random slice planes itself is non-trivial for unoriented point clouds of large scenes with complex pipe systems.
[0016] Some studies have reconstructed pipe connectivity relationships after the process of pipe modeling. Most of them simply connected pipes with line distance or end point distance less than some given threshold. For example, a method has been proposed to approach the problem by analyzing the connectivity patterns of centerline nodes and angles formed by a query node and its two neighbors. These methods may not perform well when the input data is incomplete or when a complex pipe system crowds the scene. A method based on a graph has also been proposed, where the nodes are the detected pipe segments and connectors and the edge weights are defined by the pairwise Euclidean distance of the nodes. Although this systematic method can handle complex pipe systems, the graph was constructed purely based on the results from the preceding DeepPipe neural network processing and did not involve any interpolation analysis to maximize the completeness of the pipe system when the input data was incomplete.
[0017] Therefore, with regard to BIM reconstruction of piping systems, it has been found that the existing or traditional methods have several problems, including: Geometric point features, e.g. point normals or curvatures, are required, therefore pre-processing of the input point cloud is required. The first step of feature calculation is prone to errors that may affect later steps. The additional processing also increases the computation time.
[0018] Parameter tuning is required, e.g., the granularity of the Hough space and the neighborhood size for feature computation, object segmentation and primitive fitting. Incorrect parameters can affect both the accuracy and the execution time of the algorithm. Moreover, setting these scene-dependent parameters is not straightforward for inexperienced users.
[0019] Prior assumptions or knowledge, e.g., Manhattan World assumptions regarding pipe direction, pipe radius range, and existing 3D drawings or item lists, are applied. However, in many application cases, prior knowledge is not available and the assumptions are not always true, e.g., some sewer pipes in buildings are intentionally constructed with a gradual slope to ensure fluid mobility.
[0020] Therefore, there is a need to provide a method and system for reconstructing BIM for piping systems that seeks to overcome or at least ameliorate one or more deficiencies in conventional methods of BIM reconstruction for piping systems, and more particularly, to improve efficiency and effectiveness in BIM reconstruction for piping systems, such as automated BIM reconstruction of piping systems. Against this background, the present invention has been developed. Summary of the Invention
[0021] According to a first aspect of the present invention, there is provided a method for reconstructing a BIM for a piping system using at least one processor, the method comprising: obtaining a point cloud patch from point cloud data obtained for the piping system, the point cloud patch relating to a seed point for the point cloud patch, the point cloud patch including the seed point and a first plurality of neighboring points for the seed point; For each of the first neighboring points, embedding a plurality of neighboring features related to the neighboring point into the neighboring point to form a neighboring feature embedded point cloud patch; generating a point cloud sub-patch including a plurality of second neighboring points from the plurality of first neighboring points of the neighbor feature embedded point cloud patch, where each of the plurality of second neighboring points is determined to belong to a same pipe segment as the seed point using a first machine learning model; determining whether the seed point is a pipe point based on the point cloud subpatch using a second machine learning model; and determining a pipe centerline point associated with the seed point based on the point cloud subpatch and based on determining that the seed point is a pipe point.
[0022] According to a second aspect of the present invention, there is provided a system for reconstructing a BIM for a piping system, the system comprising: At least one memory; and at least one processor communicatively connected to said at least one memory and configured to execute the method for reconstructing a BIM according to the above-mentioned first aspect of the present invention.
[0023] According to a third aspect of the present invention, there is provided a computer program product embodied in one or more non-transitory computer readable storage media comprising a plurality of instructions executable by at least one processor to perform a method for reconstructing a BIM according to the above-mentioned first aspect of the present invention. [Brief description of the drawings]
[0024] BRIEF DESCRIPTION OF THE DRAWINGS Several embodiments of the present invention will be better understood and readily apparent to those skilled in the art from the following description, given by way of example only, in conjunction with the drawings in which: FIG. [Figure 1] FIG. 1 illustrates a schematic flow diagram of a method for reconstructing a BIM for a piping system using at least one processor, according to various embodiments of the present invention. [Diagram 2] FIG. 2 illustrates a schematic block diagram of a system for reconstructing BIM for a piping system according to various embodiments of the present invention. [Diagram 3]FIG. 3 illustrates a schematic block diagram of an exemplary computer system that may be used to realize or implement a system for reconstructing a BIM for a piping system according to various embodiments of the present invention. [Figure 4] FIG. 4 illustrates a schematic flow diagram of a method for reconstructing a BIM for a piping system, according to various exemplary embodiments of the present invention. [Diagram 5] FIG. 5 illustrates a schematic diagram of an example architecture of PipeNet configured for cylindrical piping detection, in accordance with various example embodiments of the present invention. [Figure 6] FIG. 6 illustrates the set abstraction (SA) and feature propagation (FP) layers in PipeNet's local point embedding (LPE), in accordance with various exemplary embodiments of the present invention. [Figure 7A-7C] 7A-7C illustrate an exemplary processing of an input point cloud patch into selected point cloud sub-patches output by PipeNet's Dynamic Scale Manager (DSM), according to various exemplary embodiments of the present invention. [Figure 8] FIG. 8 illustrates several examples of simulated training data (exemplary simulated pipes and ducts) according to various exemplary embodiments of the present invention. [Figure 9] FIG. 9 illustrates a simulated radial distribution of multiple pipes, according to various exemplary embodiments of the present invention. [Figure 10] FIG. 10 illustrates a table (Table 1) containing the final composition of the training dataset for PipeNet, according to various exemplary embodiments of the present invention. [Figures 11A-11C] 11A-11C show an example of a small pipe (a pipe of interest) in actual test scan data of a scene, according to various exemplary embodiments of the present invention. [Figure 12] FIG. 12 illustrates line distances and intersections at twist locations in a centerline connectivity reconstruction, according to various exemplary embodiments of the present invention. [Figure 13]FIG. 13 illustrates three types of splices in a centerline connectivity reconstruction, according to various exemplary embodiments of the present invention. [Figure 14] FIG. 14 illustrates an example of forming a minimum spanning forest, according to various exemplary embodiments of the present invention. [Figure 15] FIG. 15 illustrates a visualization of neighborhood selection results from a DSM according to various exemplary embodiments of the present invention compared to neighborhood point selection results using traditional k-NN methods. [Figure 16] FIG. 16 shows the qualitative results of the two models for ablation and a comparison study on a synthetic parallel dataset. [Figure 17] FIG. 17 shows a table (Table 2) summarizing the quantitative results of the synthetic parallel dataset study. [Figure 18A-18B] 18A-18B show box plots of radius and centerline node errors predicted by PipeNet for synthetic data at different scan coverage angles θ and low and high noise levels, in accordance with various exemplary embodiments of the present invention. [Fig. 18C-18D] 18C-18D show box plots of radius and centerline node errors predicted by PipeNet for synthetic data at different scan coverage angles θ and low and high noise levels, in accordance with various exemplary embodiments of the present invention. [Figure 19A-19B] 19A-19B show plots of mean absolute radial error and mean relative radial error, maximum centerline end point displacement, and centerline angle for the final reconstructed piping models at all σ and θ conditions in accordance with various exemplary embodiments of the present invention. [Fig. 19C-19D] 19C-19D show plots of mean absolute radial error and mean relative radial error, maximum centerline end point displacement, and centerline angle for the final reconstructed piping models at all σ and θ conditions in accordance with various exemplary embodiments of the present invention. [Figure 20] FIG. 20 shows a table (Table 3) containing basic statistics of the test scenes and scan data. [Figures 21A-21D] 21A-21D show a point cloud, predicted potential piping points extracted by PipeNet's coarse sampling process, a reconstructed IFC model, and an IFC model superimposed on the potential piping points, respectively, for a laboratory scene in accordance with various exemplary embodiments of the present invention. [Figures 22A-22C] 22A-22C show the point cloud, predicted possible piping points extracted by PipeNet's coarse sampling process, and the reconstructed IFC model, respectively, for a warehouse scene in accordance with various exemplary embodiments of the present invention. [Diagram 23] FIG. 23 shows a table (Table 4) containing the processing times consumed by PipeNet and model reconstruction, respectively, according to various exemplary embodiments of the present invention. [Figure 24] FIG. 24 shows a table (Table 5) containing results for both the laboratory scene and the warehouse scene, according to various exemplary embodiments of the present invention. [Diagram 25] FIG. 25 shows a table (Table 6) comparing the method of the present invention with existing studies. [Figures 26A-26D] 26A-26D show an IFC model overlaid on point cloud data for a laboratory scene according to various exemplary embodiments of the present invention, a false positive pipe generated in a lighting cable, a false positive pipe generated in a curved section of a phone booth, and a close-up of a missing pipe. [Figures 27A-27D] 27A-27D show close-ups of an IFC model superimposed on potential piping points, examples of failed piping segments, and examples of false positives generated on a person's limbs for a warehouse scene, according to various exemplary embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0025] Various embodiments of the present invention provide a method and system for reconstructing building information modeling (BIM) for a piping system.
[0026] For example, as described in the Background, with respect to BIM reconstruction of piping systems, it has been found that existing or conventional methods have several problems that result in inefficiency and ineffectiveness in the BIM reconstruction of piping systems. Accordingly, various embodiments of the present invention provide a method and system for reconstructing BIM for piping systems that seeks to overcome or at least improve one or more deficiencies in conventional methods for reconstructing BIM for piping systems, and more specifically, to improve the efficiency and effectiveness in the reconstruction of BIM for piping systems, such as an automated BIM reconstruction of piping systems.
[0027] 1 illustrates a schematic flow diagram of a method 100 for reconstructing a BIM for a piping system using at least one processor, according to various embodiments of the present invention. The method 100 includes obtaining (at 102) a point cloud patch for a seed point for the point cloud patch from acquired point cloud data for the piping system, the point cloud patch including the seed point and a plurality of first neighboring points for the seed point, and for each of the plurality of first neighboring points, embedding (at 104) a plurality of neighboring features for the neighboring point into the neighboring point to form a neighboring feature embedded point cloud patch, and generating (at 106) a point cloud sub-patch including a plurality of second neighboring points from the plurality of first neighboring points of the neighboring feature embedded point cloud patch. The method includes generating a plurality of second neighboring points, each of which has been determined (e.g., predicted) to belong to the same piping segment as the seed point using a first machine learning model; determining (at 108) whether the seed point is a piping point based on the point cloud sub-patch using a second machine learning model; and determining (at 110) a piping centerline point associated with the seed point based on the point cloud sub-patch and based on the determination (at 108) that the seed point is a piping point (e.g., if the seed point is determined (at 108) to be a piping point).
[0028] Thus, in various embodiments, the above-described neighborhood feature embedded point cloud patch includes a seed point and a number of first neighborhood points, each of which has a number of neighborhood features associated with it embedded therein.
[0029] In various embodiments, multiple points (or multiple adjacent points) belonging to the same pipe segment may refer to multiple points belonging to a common part, section, or portion of the pipe. For example, if two points belong to two adjacent pipes, these two points do not belong to the same pipe segment because the two adjacent pipes are different parts of a pipe system. As another example, in the case of two points that belong to two sections (or portions) on different sides of a pipe having a 90-degree elbow configuration, such two points may belong to the same pipe, but they belong to different sections (or portions) of the same pipe and therefore do not belong to the same pipe segment.
[0030] Thus, the method 100 for reconstructing a BIM for a piping system according to various embodiments of the present invention advantageously improves efficiency and effectiveness in BIM reconstruction of a piping system, such as automatic BIM reconstruction of a piping system. In particular, by embedding a plurality of neighborhood features related to the neighborhood point into the neighborhood point for each of a plurality of first neighboring points to form a neighborhood feature embedded point cloud patch, and generating a point cloud subpatch including a plurality of second neighboring points from the plurality of first neighboring points of the neighborhood feature embedded point cloud patch, each of the plurality of second neighboring points has been determined to belong to the same pipe segment as the seed point using a first machine learning model, which advantageously enables, facilitates, or promotes the generation of an optimized (e.g., optimal) point cloud subpatch including a plurality of neighboring points having a plurality of neighborhood features that are more or truly related (e.g., belong to the same pipe segment as the seed point), or in other words, reduces or minimizes a plurality of neighboring points that are not related or less related (e.g., do not belong to the same pipe segment as the seed point). As a result, subsequent prediction processes, such as determining whether a seed point is a piping point and determining various piping parameters based on such point cloud sub-patch, are advantageously performed based on more accurate or relevant data (e.g., the optimized point cloud sub-patch), thereby enabling more accurate prediction results. Moreover, advantageously, no pre-processing is required to extract geometric features from the point cloud data, no assumptions are made about the piping direction and size, and no prior knowledge, such as an existing BIM or CAD model or catalog of components, is used. Thus, the method 100 advantageously provides a fully automatic solution for reconstructing a BIM for a piping system from unstructured point cloud data. These advantages or technical effects, and / or other advantages or technical effects, will become more apparent to those skilled in the art when the method 100 for reconstructing a BIM for a piping system, as well as a corresponding system for reconstructing a BIM for a piping system, are described in more detail according to various embodiments and exemplary embodiments of the present invention.
[0031] In various embodiments, the neighborhood features for the neighboring points are learned latent space features for the local neighborhood of the neighboring points. In various embodiments, the neighborhood points for the query point (such neighborhood points constituting or defining the local neighborhood (e.g., query neighborhood points) of the query point) may be obtained (or extracted) by using a ball query method or a k-nearest neighbors (k-NN) method for the query point. According to the ball query method, for example, all points whose distance to the query point is less than a predetermined distance may be grouped to form or define the local neighborhood for the query point.
[0032] In various embodiments, determining (at 110) a piping centerline point associated with the seed point as described above includes determining (e.g., predicting) a piping radius associated with the seed point based on the point cloud subpatch using a third machine learning model; determining (e.g., predicting) a point normal associated with the seed point based on the point cloud subpatch using a fourth machine learning model; and determining a piping centerline point associated with the seed point based on the seed point, the determined piping radius, and the determined point normal.
[0033] In various embodiments, the method 100 further includes using a fifth machine learning model to determine (e.g., predict) a flow direction of the pipe associated with the seed point based on the point cloud subpatch.
[0034] In various embodiments, the first machine learning model is a first classification model trained to determine (e.g., predict) for each of a plurality of first neighboring points whether the neighboring points belong to the same pipe segment as the seed point. In various embodiments, the second machine learning model is a second classification model trained to determine (e.g., predict) whether the seed point is a pipe point based on the point cloud subpatch. In various embodiments, the third machine learning model is a first regression model trained to determine (e.g., predict) a pipe radius associated with the seed point based on the point cloud subpatch. In various embodiments, the fourth machine learning model is a second regression model trained to determine (e.g., predict) a point normal associated with the seed point based on the point cloud subpatch. In various embodiments, the fifth machine learning model is a third regression model trained to determine (e.g., predict) a pipe flow direction associated with the seed point based on the point cloud subpatch.
[0035] In various embodiments, obtaining (at 102) a point cloud patch for the seed point as described above includes extracting a number of neighboring points for the seed point from the point cloud data using a ball query method or a k-nearest neighbors (k-NN) method.
[0036] In various embodiments, for each of a plurality of point cloud patches from the point cloud data, the method 100 performs the steps of obtaining (at 102) a point cloud patch for a seed point for the point cloud patch as described above, for each of a plurality of first neighboring points, embedding (at 104) a plurality of neighboring features for the neighboring point into the neighboring point as described above, generating (at 106) a point cloud sub-patch as described above, determining (at 108) whether the seed point is a pipe point as described above, and determining (at 110) a pipe centerline point associated with the seed point based on the point cloud sub-patch and based on determining (at 108) that the seed point is a pipe point as described above, and determining a pipe flow direction associated with the seed point as described above. In this regard, the method 100 further comprises generating, for each of the plurality of point cloud patches including the seed points determined to be pipe points, a centerline parameter set including a determined piping centerline point associated with the seed point for the point cloud patch, a determined piping radius associated with the seed point for the point cloud patch, and a determined piping flow direction associated with the seed point for the point cloud patch. In this regard, the determined piping radius and the determined piping flow direction based on the point cloud patch are associated with the determined piping centerline point based on the point cloud patch. Thus, in various embodiments, only piping parameters associated with the plurality of seed points determined (or predicted) to be the plurality of pipe points at 108 are included in the centerline parameter set.
[0037] In various embodiments, the method 100 further includes generating a set of centerline candidates based on the centerline parameter set, refining the set of centerline candidates to obtain a refined set of centerline candidates, and generating a piping system model for the piping system based on the refined set of centerline candidates.
[0038] In various embodiments, generating the set of centerline candidates as described above includes, for each centerline point in the centerline parameter set, dividing the centerline parameter set into a plurality of centerline parameter subsets based on the centerline point and a piping flow direction associated with the centerline point, and, for each of the plurality of centerline parameter subsets, applying line fitting to the centerline parameter subset to generate centerline candidates associated with the centerline parameter subset to obtain the set of centerline candidates.
[0039] In various embodiments, each centerline candidate in the set of centerline candidates has two end points, a direction vector, a radius, and a set of inlier centerline points associated with it.
[0040] In various embodiments, refining the set of centerline candidates includes: for each of the plurality of common centerline groups, setting a longest centerline candidate in the set of centerline candidates that does not yet belong to any of the plurality of common centerline groups as a seed centerline candidate associated with the common centerline group and adding the longest centerline candidate to the common centerline group; and for each centerline candidate in the set of centerline candidates that does not yet belong to any of the plurality of common centerline groups, adding the centerline candidate to the common centerline group based on determining that the centerline candidate satisfies a common centerline condition between the centerline candidate and the seed centerline candidate, and adding the added centerline candidate to the common centerline group. as a new seed centerline candidate associated with the common centerline group; and for each new seed centerline candidate set in the common centerline group, based on determining that, for each centerline candidate in the set of centerline candidates that does not yet belong to any of the multiple common centerline groups, the centerline candidate satisfies a common centerline condition between the centerline candidate and the seed centerline candidate (i.e., the new seed centerline candidate), adding the centerline candidate to the common centerline group and setting the added centerline candidate as a new seed centerline candidate (i.e., a further new seed centerline candidate) associated with the common centerline group.
[0041] In various embodiments, the common centerline condition includes multiple subconditions including a first subcondition based on an angle formed by multiple directional vectors associated with the centerline candidate and the seed centerline candidate, respectively, a second subcondition based on a distance between the centerline candidate and the seed centerline candidate, and a third subcondition based on a difference between multiple radii associated with the centerline candidate and the seed centerline candidate, respectively.
[0042] In various embodiments, refining the set of centerline candidates as described above further includes, for each of the plurality of common centerline groups, applying line fitting to the common centerline group to form refined centerline candidates associated with the common centerline group to obtain the refined set of centerline candidates.
[0043] In various embodiments, refining the set of centerline candidates as described above further includes, for each of the plurality of common centerline groups, determining a radius associated with the refined centerline candidate associated with the common centerline group by applying circle fitting based on the extracted segment of the point cloud data corresponding to the length of the refined centerline candidate.
[0044] In various embodiments, generating the piping system model as described above includes performing a graph-based centerline connectivity reconstruction based on the refined set of centerline candidates, where performing the centerline connectivity reconstruction includes constructing a graph including a plurality of nodes representing the refined set of centerline candidates, where each node represents a corresponding refined centerline candidate of the refined set of centerline candidates, with each pair of nodes of the plurality of nodes having an edge weight assigned to the pair of nodes; generating a minimum spanning forest including a plurality of minimum spanning trees based on the graph; and performing a plurality of centerline connections for the refined set of centerline candidates based on the minimum spanning forest.
[0045] In various embodiments, performing the graph-based centerline connectivity reconstruction as described above further includes, for each pair of nodes of the plurality of nodes, determining an edge weight assigned to the pair of nodes based on the Euclidean distance between the refined pair of centerline candidates of the refined set of centerline candidates corresponding to the pair of nodes, the scan data density in the possible connection region between the refined pair of centerline candidates, the orientation angle difference between the refined pair of centerline candidates, and the radius difference between the refined pair of centerline candidates.
[0046] Figure 2 shows a schematic block diagram of a system 200 for reconstructing a BIM for a piping system according to various embodiments of the present invention, which corresponds to the above-described method 100 for reconstructing a BIM for a piping system described above with reference to Figure 1. The system 200 comprises at least one memory 202 and at least one processor 204 communicatively coupled to the at least one memory 202 and configured to execute the above-described method 100 for reconstructing a BIM according to various embodiments of the present invention. Thus, the at least one processor 204 is configured to perform the following: obtain a point cloud patch for a seed point for the point cloud patch from the obtained point cloud data for the piping system, the point cloud patch including the seed point and a plurality of first neighboring points for the seed point; for each of the plurality of first neighboring points, embedding a plurality of neighboring features for the neighboring point into the neighboring point to form a neighboring feature embedded point cloud patch; generate a point cloud sub-patch including a plurality of second neighboring points from the plurality of first neighboring points of the neighboring feature embedded point cloud patch, each of the plurality of second neighboring points being determined (e.g., predicted) to belong to the same piping segment as the seed point using a first machine learning model; determine whether the seed point is a piping point based on the point cloud sub-patch using the second machine learning model; and determine a piping centerline point associated with the seed point based on the point cloud sub-patch and based on the determination that the seed point is a piping point.
[0047] Those skilled in the art will appreciate that the at least one processor 204 may be configured to perform various functions or operations through a set or sets of instructions (e.g., software modules) executable by the at least one processor 204 to perform various functions or operations. Thus, as shown in FIG. 2 , the system 200 includes a point cloud patch acquisition module (or point cloud patch acquisition circuit) 212 configured to acquire a point cloud patch for a seed point for the point cloud patch from point cloud data acquired for the piping system, the point cloud patch including the seed point and a plurality of first neighboring points for the seed point, a neighboring feature embedding module (or neighboring feature embedding circuit) 214 configured to, for each of the plurality of first neighboring points, embed a plurality of neighboring features for the neighboring point into the neighboring point to form a neighboring feature embedded point cloud patch, and a neighboring feature embedding circuit 215 configured to embed a plurality of neighboring features for the neighboring point into the neighboring point to generate a point cloud sub-patch including a plurality of second neighboring points from the plurality of first neighboring points of the neighboring feature embedded point cloud patch. the second plurality of neighboring points having been determined to belong to the same pipe segment as the seed point using a first machine learning model; a piping point determination module (or piping point determination circuit) 218 configured to determine whether the seed point is a piping point based on the point cloud subpatch using the second machine learning model; and a piping centerline point determination module (or piping centerline point determination circuit) 220 configured to determine a piping centerline point associated with the seed point based on the point cloud subpatch and based on the determination that the seed point is a piping point.
[0048] Those skilled in the art will appreciate that the above-mentioned modules are not necessarily separate modules, and that two or more modules may be realized by or implemented as one functional module (e.g., a circuit or software program) as desired or appropriate without departing from the scope of the present invention. For example, two or more modules, such as the point cloud patch acquisition module 212, the neighborhood feature embedding module 214, the point cloud sub-patch generation module 216, the piping point determination module 218, and the piping centerline point determination module 220, may be realized (e.g., compiled together) as one executable software program (e.g., referred to as a software application or simply "app"), which may be stored, for example, in at least one memory 202 and executable by at least one processor 204 to perform various functions / operations described herein according to various embodiments of the present invention.
[0049] In various embodiments, the system 200 for reconstructing a BIM corresponds to the method 100 for reconstructing a BIM described above with reference to Fig. 1 according to various embodiments, i.e. various functions or operations configured to be executed by the at least one processor 204 may correspond to various steps or operations of the method 100 for reconstructing a BIM described above according to various embodiments and therefore need not be repeated with respect to the system 200 for reconstructing a BIM for the sake of clarity and brevity. In other words, various embodiments described herein in the context of these methods are equally valid for the corresponding systems and vice versa.
[0050] For example, in various embodiments, the at least one memory 202 may store a point cloud patch acquisition module 212, a neighborhood feature embedding module 214, a point cloud sub-patch generation module 216, a piping point determination module 218, and / or a piping centerline point determination module 220, each of which corresponds to various steps (or operations or functions) of the method 100 for reconstructing a BIM as described herein according to various embodiments and are executable by the at least one processor 204 to perform the corresponding functions or operations as described herein.
[0051] A computing system, controller, microcontroller, or any other system providing processing capabilities may be provided according to various embodiments of the present disclosure. Such a system may be construed as including one or more processors and one or more computer-readable storage media. For example, the system 200 for reconstructing a BIM described above may include at least one processor (or controller) 204 and at least one computer-readable storage medium (or memory) 202, which are used in various processes performed, for example, as described herein. The memory or computer-readable storage medium used in various embodiments may be a volatile memory, such as a Dynamic Random Access Memory (DRAM), a non-volatile memory, such as a Programmable Read Only Memory (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), a flash memory, such as a floating gate memory, a charge trapping memory, a Magnetoresistive Random Access Memory (MRAM), or a Phase Change Random Access Memory (PCRAM).
[0052] In various embodiments, a "circuit" may be understood as any kind of logic implementation entity, which may be a dedicated circuit or a processor that executes software, firmware, or any combination thereof stored in a memory. Thus, in one embodiment, a "circuit" may be a hardwired logic circuit or a programmable processor, such as a microprocessor (e.g., a Complex Instruction Set Computer (CISC) processor or a Reduced Instruction Set Computer (RISC) processor). A "circuit" may also be a processor that executes software, such as any kind of computer program, such as a computer program using virtual machine code, such as Java. Any other kind of implementation of the respective functions may also be understood as a "circuit" according to various embodiments. Similarly, a "module" may be part of a system according to various embodiments, may include the "circuit" described above, or may be understood as any kind of logic implementation entity.
[0053] Some portions of this disclosure are presented explicitly or implicitly in terms of algorithms and functional or symbolic representations of operations on data within a computer memory. These algorithmic descriptions and functional or symbolic representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is herein generally conceived to be a self-consistent sequence of steps leading to a desired result. These steps require physical manipulations of physical quantities, such as electrical, magnetic, or optical signals, capable of being stored, transferred, combined, compared, and otherwise manipulated.
[0054] Unless otherwise indicated, and as will become apparent below, throughout this specification, statements or explanations utilizing terms such as "obtain," "embed," "generate," "seek," "extract," "execute," "refine," "divide," "apply," "set," "add," "construct," and the like will be understood to refer to the operations and processing of a computer system or similar electronic device that manipulates and converts data represented as physical quantities within a computer system into other data similarly represented as physical quantities within a computer system or other information storage, transmission, or display device.
[0055] This specification also discloses systems (which may also be embodied as devices or apparatuses, for example) such as the system 200 for reconstructing BIM for performing various operations / functions of various methods described herein. Such systems may be specially constructed for the required purposes, or may comprise a general-purpose computer or other device selectively activated or reconfigured by a computer program stored in the computer. The algorithms presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose machines may be used with computer programs according to the teachings of this specification. Alternatively, construction of more specialized apparatuses to perform various method steps may be appropriate.
[0056] In addition, this specification also at least implicitly discloses computer programs or software / functional modules in that it will be apparent to one skilled in the art that the individual steps of the various methods described herein may be implemented by computer code. The computer program is not intended to be limited to any particular programming language and its implementation. It will be understood that various programming languages and their coding may be used to implement the teachings of the present disclosure contained herein. Furthermore, the computer program is not intended to be limited to any particular control flow. There are many other variations of computer programs that may use different control flows without departing from the scope of the present invention. It will be understood by one skilled in the art that the various modules described herein (e.g., point cloud patch acquisition module 212, neighborhood feature embedding module 214, point cloud sub-patch generation module 216, piping point determination module 218, and / or piping centerline point determination module 220) may be one or more software modules realized by one or more computer programs or a set or sets of instructions executable by a computer processor to perform the required functions, or may be one or more hardware modules that are one or more functional hardware units designed to perform the required functions. It will also be appreciated that a combination of hardware and software modules may be implemented.
[0057] Furthermore, one or more of the steps of the computer programs / modules or methods described herein may be executed in parallel rather than sequentially. Such computer programs may be stored in any computer-readable medium. The computer-readable medium may include storage devices such as magnetic or optical disks, memory chips, or other storage devices suitable for interfacing with a computer. When the computer programs are loaded and executed in such a computer, a system or apparatus for performing various steps of the methods described herein is effectively realized.
[0058] In various embodiments, a computer program product embodied in one or more computer readable storage media (non-transitory computer readable storage media) is provided, which includes a number of instructions (e.g., point cloud patch acquisition module 212, neighborhood feature embedding module 214, point cloud sub-patch generation module 216, piping point determination module 218, and / or piping centerline point determination module 220) executable by one or more computer processors to execute the method 100 for reconstructing a BIM described herein with reference to FIG. 1 according to various embodiments. Thus, various computer programs or modules described herein may be stored in a computer program product receivable by a system, such as the system 200 for reconstructing a BIM as shown in FIG. 2, for execution by at least one processor 204 of the system 200 to perform various functions.
[0059] The software or functional modules described herein may be implemented as hardware modules. More specifically, in the hardware sense, a module is a functional hardware unit designed for use with other components or modules. For example, a module may be implemented using individual electronic components or may form part of an entire electronic circuit, such as an ASIC (Application Specific Integrated Circuit). Numerous other possibilities exist. Those skilled in the art will appreciate that the software or one or more functional modules described herein may also be implemented as a combination of hardware and software modules.
[0060] In various embodiments, the system 200 for reconstructing a BIM may be realized by any computer system (e.g., a desktop or portable computer system) including at least one processor and at least one memory, for example, an exemplary computer system 300 as shown generally in FIG. 3 by way of example only and not limitation. The various methods / steps or functional modules may be implemented as software, such as computer programs, that execute within the computer system 300 and instruct the computer system 300 (particularly the processor or processors therein) to perform various functions or operations as described herein according to various embodiments. The computer system 300 may comprise a system unit 302, input devices such as a keyboard and / or touch screen 304 and a mouse 306, and multiple output devices such as a display 308. The system unit 302 may be connected to a computer network 312 via a suitable transmission / reception device 314 to enable access to, for example, the Internet or other network systems such as a LAN (Local Area Network) or a WAN (Wide Area Network). The system unit 302 may include a processor 318 for executing various instructions, a Random Access Memory (RAM) 320, and a Read Only Memory (ROM) 322. The system unit 302 may further include several input / output (I / O) interfaces, such as an I / O interface 324 to a display device 308 and an I / O interface 326 to a keyboard 304. The various components of the system unit 302 typically communicate via an interconnected bus 328 in a manner known to those skilled in the art.
[0061] It will be understood by those skilled in the art that the terms used herein are merely for the purpose of describing various embodiments and are not intended to limit the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising", as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.
[0062] Any reference herein to an element or feature using a designation such as "first," "second," etc. does not limit the quantity or order of such elements or features, unless otherwise stated or unless the context otherwise requires. For example, such designations may be used herein as a convenient way of distinguishing between two or more elements or instances of an element. Thus, reference to a first element and a second element does not necessarily imply that only two elements may be employed or that the first element must precede the second element. In addition, a phrase referring to "at least one of" a list of items refers to any single item therein, or any combination of two or more items therein.
[0063] In order to make the present invention easily understood and to obtain practical effects, various exemplary embodiments of the present invention are described below by way of example and not by way of limitation. However, the present invention is not limited to the exemplary embodiments disclosed below, and may be embodied in various different forms. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present invention to those skilled in the art.
[0064] BIM is gradually being recognized and promoted as an emerging standard practice in the construction industry as well as the built environment. MEP systems play a vital role in the operation and maintenance of buildings as systems that require regular maintenance, whereby their failure can cause significant operational, economic and even environmental impacts. As-built BIM is required for efficient BIM-enabled facility management of buildings that were built before BIM existed or with outdated BIM. As explained in the background, as-built BIM modeling is traditionally performed in a very manual and tedious manner, requiring significant time and effort even for experienced modelers. To address these issues related to as-built BIM reconstruction of piping systems, various exemplary embodiments of the present invention provide a technical solution to automatically reconstruct as-built BIM of multiple piping systems in multiple buildings from LiDAR scanned point cloud data. Compared to various existing works, for example, a method for reconstructing a BIM of one or more piping systems according to various exemplary embodiments (sometimes referred to herein simply as the method) does not require additional data other than an unstructured point cloud having an XYZ field (i.e., a three-dimensional space in a Cartesian coordinate system having three axes, sometimes referred to as the x-axis, y-axis, and z-axis), does not require data pre-processing, and does not require prior knowledge of pipe orientation or dimensions. According to various exemplary embodiments, the method includes two stages. In the first stage, a deep learning network (sometimes referred to herein as PipeNet) is provided that is capable of detecting pipes and predicting pipe centerline points along with other pipe parameters, regardless of the size of the input data and the scale of the target scene. In various exemplary embodiments, PipeNet is configured to consume a raw point cloud and output a plurality of semantic labels and associated pipe parameters.In the second stage, a piping system model (which may be simply referred to as a piping model) is reconstructed (i.e., generating a piping system model that is a reconstruction of the physical piping system) through line fitting, refinement, and graph-based connectivity analysis constrained by domain knowledge that maximizes the coherence of the piping system model. In various exemplary embodiments, multiple piping centerlines are extracted and refined from the network prediction, and their connection relationships are reconstructed using a systematic graph-based approach. The piping system model may then be converted to an Industry Foundation Classes (IFC) format (e.g., BIM IFC 4.0) (i.e., a platform-neutral format used in the BIM industry) that is neutrally acceptable in the BIM industry. That is, the reconstructed piping system model may be output in IFC format. The method is validated against both synthetic and real scan data, and the results (described in detail below) demonstrate its robustness, high speed, and high recognition rate and accuracy, thereby demonstrating the improved efficiency and effectiveness of the method for reconstructing BIM of a piping system according to various embodiments and exemplary embodiments of the present invention.
[0065] Thus, the various exemplary embodiments advantageously provide a fully automatic solution for reconstructing as-built BIM of multiple piping systems from unstructured point clouds, which does not require data pre-processing or any orientation or dimensional assumptions. For better understanding, and without limitation, a method for reconstructing BIM for piping systems will now be described in further detail, in accordance with various exemplary embodiments of the present invention.
[0066] Intelligent and automatic piping BIM reconstruction According to various exemplary embodiments, an intelligent and automatic solution is provided for reconstructing an as-built BIM model of a piping system from unstructured point cloud data. To maximize the generalization of the method, advantageously, no pre-processing is required to extract geometric features from the input data, no assumptions are made about the pipe direction and size, and no prior knowledge such as existing BIM or CAD models or catalogs of components is used. Various exemplary embodiments note that a complete pipe line may be defined by multiple consecutive centerline segments with radius parameters and connection relationships. In this regard, various exemplary embodiments provide a two-stage solution for cylindrical piping reconstruction. FIG. 4 shows a schematic flow diagram of a method 400 for reconstructing a BIM for a piping system according to various exemplary embodiments of the present invention. As mentioned above, in the first stage, a PipeNet 402 is provided that is configured to receive raw point cloud data with xyz fields and output multiple semantic labels and multiple centerline node predictions along with other piping parameters (e.g., pipe radius). In the second stage, a complete piping model or models are reconstructed from the PipeNet predictions through several processes including centerline candidate generation 406, candidate refinement 410, and graph-based connectivity reconstruction 414. Finally, the generated piping system model or models are converted to the BIM IFC format using an IFC converter 418.
[0067] It should be noted that although RGB color and intensity values other than the XYZ fields may be generated in the point cloud by most modern scanners, they are not used in the method 400 since various exemplary embodiments address the uncertainty in these two information fields. Although there are commonly used color codes for piping systems of certain functions, for example, white for drain and sewer pipes, yellow for gas pipes, and red for fire sprinkler pipes, these color codes are not mandatory and may vary from country to country. Furthermore, multiple pipes may be intentionally painted to be the same color as the ceiling or walls to visually minimize their presence, such as two scenes in the actual scanning test performed according to various exemplary embodiments of the present invention (described below) where all pipes are painted black in the laboratory and all fire sprinkler pipes are white in the warehouse. Furthermore, the color of the pipe surface may fade over time due to environmental effects such as weather, and the color of the pipe in the point cloud also depends on the scanner used and the lighting of the scene. With regard to the intensity information, it may be modified depending on the scanner used, the distance of the pipe to the scanner, and the surface material of the pipe. Due to these uncertainties, among the various fields of information (or metadata) originally embedded in the point cloud data generated by the scanner, various exemplary embodiments use only the XYZ information (XYZ field) from the point cloud data to detect multiple pipes.
[0068] PipeNet402 5 shows a schematic diagram of an example architecture of PipeNet 402 according to various example embodiments of the present invention that may be configured for cylindrical pipe detection. As shown in FIG. 5, PipeNet 402 includes two processing modules, namely, local point embedder (LPE) 502 (e.g., corresponding to neighborhood feature embedding module 214 as described above according to various examples of the present invention) and dynamic scale manager (DSM) 512 (e.g., corresponding to point cloud subpatch generation module 216 as described above according to various examples of the present invention), and two output modules, namely, semantic classifier 522 (e.g., corresponding to pipe point determination module 218 as described above according to various examples of the present invention) and centerline node regressor 532 (e.g., corresponding to pipe centerline point determination module 220 as described above according to various examples of the present invention), which will be described in more detail below. In FIG. 5 , the numbers in brackets associated with each multilayer perceptron (MLP) indicate the number of output features after each respective MLP layer, the numbers in brackets associated with each set abstraction (SA) layer and each feature propagation (FP) layer indicate the number of subsampled points and the number of output features, respectively, and the numbers in brackets associated with the multiple nodes / points indicate the configuration or shape of the patch array or matrix of those nodes / points (e.g., point patch (N,3) indicates a patch array having a configuration or shape of N×3 (N rows and 3 columns), where N indicates the number of nodes / points, “3” indicates the number of feature channels (XYZ), and pipe direction (3,) indicates a 1D array of length 3). PipeNet 402 is configured to receive multiple point cloud patches as input. In various exemplary embodiments, the multiple point cloud patches are each generated using a ball query to generate a patch array of each seed point p iFor example, to obtain each point cloud patch, a seed point p for the point cloud patch may be sampled from the raw point cloud data. i A spherical point cloud patch is formed around a seed point by grouping neighboring points whose distance to the seed point is less than a predetermined radius. Thus, each point cloud patch includes a corresponding seed point and neighboring points around the seed point. Thus, PipeNet402 can be viewed as a multi-tasking classification network that provides multiple seed points for multiple input point cloud patches.
[0069] In various exemplary embodiments, the segmentation of pipes may be performed in a bottom-up manner by applying PipeNet402 to batches of point cloud patches sampled from the entire point cloud. For example, according to the bottom-up manner, for each point cloud sub-patch generated by DSM512, a seed point of the point cloud sub-patch may be classified by a semantic classifier522 based on the point cloud sub-patch to determine (e.g., predict) whether the seed point is a pipe point, and the seed points determined to be pipe points may be grouped to form semantically segmented point clusters. In contrast, in a conventional top-down segmentation, classification is performed on the entire point cloud and points are extracted from the point cloud to generate point clouds. In various exemplary embodiments, a coarse-to-fine strategy or technique for sampling seed points is provided to increase the implementation speed, as described later in the "Actual Scan Validation" section below. For example, compared to existing scene segmentation networks that consume downsampled point clouds of either the entire scene or specific blocks cropped from the scene, PipeNet402 can advantageously consume more detail from the point cloud patch inputs and thus detect pipes of various sizes regardless of the scene scale and raw point cloud size.
[0070] Local Point Embedder (LPE) 502 In various exemplary embodiments, for each point cloud patch of the received plurality of point cloud patches, the point cloud patch first passes through the LPE 502, which individually extracts a plurality of neighborhood features (sometimes referred to as a plurality of local neighborhood features) for each neighboring point (relative to a seed point) in the point cloud patch. In particular, the LPE 502 is configured to embed, for each of the plurality of neighboring points of the point cloud patch, the plurality of neighborhood features for the neighboring point into the neighboring point to form a neighborhood feature embedded point cloud patch. It will be appreciated that the present invention, and more specifically, the PipeNet 402, is not restricted or limited to any point feature embedding network, so long as the point feature embedding network can be utilized to embed, for each of the plurality of neighboring points of the point cloud patch, the plurality of neighborhood features for the neighboring point into the neighboring point. In various exemplary embodiments, PointNet++ as described in, by way of example and not limitation, CR Qi, L. Yi, H. Su, and LJ Guibas, “PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space,” June 2017, doi: 10.1016 / S0039-9140(96)02179-0, may be used for its simplicity and demonstrated success for various tasks such as regular estimation, object detection, and large scene segmentation. In this regard, various exemplary embodiments may apply a hierarchical feature learning architecture with a multi-scale grouping strategy proposed in PointNet++, namely, a set abstraction (SA) layer and a feature propagation (FP) layer, as shown in FIG. 6. In each SA layer, a portion of the query points is randomly selected from the patch points or query points from the previous layer. Then, for each query point, a point-wise kernel function can take the location offset from each neighboring point to the query point, along with the learned neighboring points' features from the previous layer, if any, and feed them into a series of 1-D convolutional layers to learn latent space features in the local neighborhood of the query point.The FP layer propagates the features of the query points back to the input patch points using an inverse distance weighted average based on the k-NN relationship. For example, FIG. 5 illustrates an example data processing flow of the LPE 502. For example, in the example case of the LPE 502 including a first SA layer, a second SA layer, a first FP layer, and a second FP layer, data can flow through the first SA layer, the second SA layer, the first FP layer, and the second FP layer in sequence as shown in FIG. 5. The illustrated dashed arrow indicates a skip-link. For example, the first FP layer may refer to the output from the first FP layer (i.e., it does not use the output features, but only uses the positions of the subsampled query points) to propagate the features of the subsampled points in the second SA layer back to those points in the first SA layer. Thus, after these FP layers, each neighboring point in the point cloud patch is embedded with the neighboring features of their respective local neighborhoods. As a result, the LPE module 502 embeds each patch point (e.g., each neighbor in the point cloud patch relative to a seed point of (or for) the point cloud patch) with rich local neighborhood information in addition to its original XYZ fields, which advantageously enables the DSM module 512 to better recognize outliers, if any, and the output modules (semantic classifier 522 and centerline node regressor 532) to make more accurate predictions according to various exemplary embodiments of the present invention. Thus, for each neighbor in the point cloud patch, the above-mentioned multiple neighborhood features for the neighbor are multiple latent space features learned for the local neighborhood of the neighbor. In various example embodiments, a number of neighboring points to a query point (constituting a local neighborhood of the query point (e.g., query neighboring points)) may be obtained (or extracted) by using a ball query method for the query point, such as in a manner the same or similar to the manner in which a number of neighboring points to a seed point are obtained (or extracted) from the point cloud data, whereby all points having a distance to the query point less than a predetermined distance may be grouped together to form a local neighborhood about the query point.
[0071] Dynamic scale manager (DSM)512 In various exemplary embodiments, the DSM 512 may include a series of multilayer perceptrons (MLPs) that use a rectified linear unit as an activation function. In this regard, the DSM 512 may determine, for each neighboring point in the input point cloud patch, whether that neighboring point is a seed point p i In various exemplary embodiments, the positively predicted neighbors are considered to be true neighbors for the seed point. These true neighbors may then form an optimized (e.g., optimal) point cloud subpatch for the seed point. Thus, in various exemplary embodiments, the DSM 512 may be specifically trained on one or more piping systems, as described in more detail below in the "Training Details" section. In this regard, for each neighbor in the input point cloud patch, the neighbors are compared to the seed point p for the input point cloud patch. iIt will be understood by those skilled in the art that the DSM 512 configured or trained to predict whether a seed point p belongs to the same piping segment as the seed point to form a point cloud subpatch (i.e., the optimized point cloud subpatch described above) will still output or generate such a point cloud subpatch for any seed point (i.e., whether the seed point is a piping point or not), simply being specifically trained for one or more piping systems. In other words, such a generated point cloud subpatch will be understood to include a plurality of neighboring points, each of which is determined (or predicted) to belong to the same piping segment as the seed point, so long as the determination (or prediction) is made by the DSM 512 configured or trained to do so as described above. In this regard, as described below in accordance with various exemplary embodiments, the semantic classifier module 522 may determine whether a seed point p belongs to the same piping segment as the seed point. i Whether or not the seed point p is on the pipe i is a plumbing point or not. In various exemplary embodiments, a slicing operation may be performed to obtain a point cloud sub-patch, whereby a neighborhood mask is multiplied with the neighbor feature embedded point cloud patch to extract the positively predicted neighbor points, as shown in FIG. 5. In various exemplary embodiments, a random N 1points may be selected, in other words, the optimal point cloud sub-patch may be modified or adjusted (or trimmed) to have the selected number of points. For example, this may be due to GPU constraints, whereby to place the network on a GPU for parallel processing, the number of points in each layer must be the same. In various exemplary embodiments, the first three channels, i.e., XYZ fields, of the feature vectors of these sub-patch points are transformed into their position displacements from the patch seed point before being fed to the subsequent prediction modules (e.g., semantic classifier 522 and centerline node regressor 532). For example, as shown in FIG. 5, the sub-patch transformation of the point cloud sub-patch relative to the seed point may be performed based on a minus operator. Thus, the DSM 512 is configured to generate a point cloud sub-patch including a plurality of second neighboring points from a plurality of first neighboring points of the neighboring feature embedded point cloud patch, and each of the plurality of second neighboring points is determined (or predicted) to belong to the same piping segment (or piping portion) as the seed point using a first machine learning model, e.g., a classification model trained to determine (or predict) for each of the plurality of neighboring points of the point cloud patch (e.g., the neighboring feature embedded point cloud patch), whether the neighboring point belongs to the same piping segment (or piping portion) as the seed point of the point cloud point.
[0072] Thus, the DSM 512 is advantageous in the neighbor extraction process because it does not require a kernel with a predetermined shape and size. In addition, because the DSM 512 is configured to generate a point cloud subpatch that includes neighbors that are determined (e.g., predicted) to belong to the same pipe segment as the seed point, it advantageously enables, facilitates, or promotes the generation of an optimized (e.g., optimal) point cloud subpatch that includes neighbors that have more or truly related neighbor features, or in other words, reduces or minimizes neighbors that have unrelated or less related neighbor features. As a result, the point cloud subpatch output by the DSM 512 advantageously enables subsequent prediction modules (e.g., the semantic classifier 522 and the centerline node regressor 532) to be run based on more accurate or related data, thereby generating better prediction results.
[0073] For purposes of illustration only and not limitation, Figures 7A-7C illustrate an exemplary process from an input point cloud patch to a selected point cloud sub-patch (e.g., corresponding to the optimized point cloud sub-patch described above) output by DSM 512. In particular, Figure 7A illustrates a point cloud of two closely parallel pipes (belonging to different pipe segments (or portions)), and Figure 7B illustrates an input point cloud patch to DSM 512 including multiple neighboring points 702 that include multiple pipe points belonging to the two closely parallel pipes, whereby seed points for the point cloud patch are indicated by dark black dots (the seed points are shown enlarged in Figures 7A-7C for clarity and ease of reference only, and will be understood by those skilled in the art to have no relevance as to the size of the seed points). 7C illustrates a point cloud subpatch (optimized point cloud subpatch) generated by DSM 512 (a brighter patch with a seed point) from an input point cloud patch, whereby neighboring points 704 of the input point cloud patch determined to belong to the same pipe segment (or portion) are included in the point cloud subpatch and neighboring points 706 of the input point cloud patch determined not to belong to the same pipe segment (or portion) are excluded from the point cloud subpatch 704. Thus, it can be seen that DSM 512 advantageously enables, facilitates, or promotes the generation of optimized (e.g., optimal) point cloud subpatch that includes neighboring points 704 having more or truly related neighboring features, or in other words reduces or minimizes neighboring points 706 having unrelated or less related neighboring features.
[0074] For example, when a seed point is in a crowded area where multiple pipes are in close proximity, such as the example shown in FIG. 7A, extracting multiple neighboring points using k-NN or ball query may necessarily include multiple points from other objects (e.g., other pipe segments or parts). In this regard, various exemplary embodiments note that those outliers may affect the accuracy of parameter prediction or estimation. Although tuning the parameters of the k-NN or ball query, typically by reducing the k value or ball radius, may filter out those outsiders in some cases, some true pipe neighbors may also be filtered out, leading to less reliable cylinder parameter estimation. In contrast, the DSM 512 according to various exemplary embodiments of the present invention dynamically adapts to the pipe of interest by operating to automatically filter out noise and irrelevance while advantageously operating to maintain multiple true neighbors as much as possible. Furthermore, the DSM 512 advantageously does not rely on any fixed parameters, so task-dependent parameter tuning is not required.
[0075] Output Module In various exemplary embodiments, the final outputs of PipeNet 402 are predicted based on the selected point cloud sub-patches (corresponding to the above-mentioned optimized point cloud sub-patches having the above-mentioned selected number of points) output by DSM 512 using a semantic classifier module 522 and a centerline node regressor module 532. In various exemplary embodiments, as described above, the semantic classifier module 522 uses the seed points p i Whether or not the seed point p is on the pipe i In various exemplary embodiments, the centerline node regressor 532 may be configured to output a binary prediction of whether n is a pipe point or not. i , pipe radius r i , and flow direction for non-oriented pipes iIn this regard, various exemplary embodiments configure pipe radius prediction as a regression task rather than a classification task so that application of PipeNet402 is not limited to a fixed set of pipe sizes. In various exemplary embodiments, the seed point p i Center line point c associated with i (or the centerline point parameters) are then determined by equation (1) below:
[0076]
number
[0077] Thus, the semantic classifier module 522 (e.g., corresponding to a second machine learning model as described above according to various embodiments) may classify the seed points p based on the point cloud sub-patches generated by the DSM 512. i In addition, the centerline node regressor 532 can be trained to determine (or predict) whether a seed point p is a pipe point based on the point cloud subpatch. i and a pipe radius regression model 534 (e.g., corresponding to the third machine learning model described above in accordance with various embodiments) trained to determine (or predict) a pipe radius (or a pipe radius parameter) associated with a seed point p based on the point cloud subpatch. i and a point normal regression model 536 (e.g., corresponding to the fourth machine learning model described above in accordance with various embodiments) trained to determine (or predict) a point normal (or point normal parameter) associated with a seed point p based on the point cloud subpatch. i and a pipe direction regression model 538 (e.g., corresponding to the fifth machine learning model described above in accordance with various embodiments) trained to determine (or predict) a pipe flow direction (or a pipe flow direction parameter) associated with a seed point p i Related to pipe centerline point c i According to the above formula (1), etc., the seed point p i, the calculated pipe radius r i , and the calculated point normal n i It is determined (or predicted) based on
[0078] Thus, for each of a plurality of point cloud patches from the point cloud data, the point cloud patch is obtained and processed by PipeNet 402 as described above with reference to FIG. 5 to obtain, for each of a plurality of point cloud patches that include a seed point determined to be a piping point (i.e., for each point cloud patch whose seed point is determined by the semantic classifier 522 to be a piping point), a determined piping centerline point c associated with the seed point for the point cloud patch is obtained. i , the calculated pipe radius r associated with the seed point for the point cloud patch i , and the calculated pipe flow direction l associated with the seed point for the point cloud patch. i A centerline parameter set (e.g., C:{c i ,r i ,l i Therefore, in the center line parameter set C, the calculated pipe radius r i and the flow direction of the pipe obtained based on the point cloud patch l i is the pipe centerline point c obtained based on the point cloud patch. i , e.g., based on or linked by the same index i (associated with the point cloud patch). Thus, in various exemplary embodiments, only piping parameters associated with seed points determined (or predicted) to be piping points by the semantic classifier module 522 are included or retained in the centerline parameter set.
[0079] Loss function In various exemplary embodiments, PipeNet402 may be trained to minimize a multi-task loss defined in the following exemplary equations (2)-(5). The multi-task loss may include five loss terms, each of which supervises one type of output. In the following equations (2)-(5), {{Ai L ∈ i[0,5]}} are fixed weights for each of the five loss terms that are empirically set. ins and L sem are the neighborhood mask classification of patch points by the DSM 512 and the p by the semantic classifier 522, respectively. i We use the classical cross-entropy loss for both terms.
[0080]
number
[0081] are the ground truth label and the predicted label, respectively. w indicates the loss weight for each class. s(x) indicates "1" if x is true, and "0" otherwise. N, N sim , N sim pipe , and N pos denote the total number of seed points, the number of seed points from simulated data, the number of seed points from simulated pipes, and the number of patch points predicted as positive, respectively.
[0082] In various exemplary embodiments, only the positively predicted patch points of the simulated data are included in L insIn this case, the patch points are counted in the classification order, i.e., negatively labeled patch points that are positively predicted are penalized, but not vice versa. This soft classification allows the DSM 512 to select a subset of true neighborhoods (e.g., patch points from the same pipe segment (or part)) as point cloud sub-patches. Since the DSM 512 is co-supervised by all other loss terms, the point cloud sub-patches are essentially trained to be optimized (e.g., optimal) neighborhoods for seed point classification by the semantic classifier 522 and cylinder parameter estimation by the centerline node regressor 532.
[0083] Predicted normal n i and ground truth
[0084]
number
[0085] The Euclidean distance difference between nor Become a teacher of L r represents radius loss and includes two terms: (1) absolute difference and (2) relative difference. Because a small absolute difference will produce a large relative difference for small pipes, but a small relative difference will produce a large absolute difference for large pipes, various exemplary embodiments have found that using such two terms advantageously helps to better train the radius regression by pipe radius regression model 534 for both large and small pipes. In various exemplary embodiments, A 5 is applied to moderate the value of the relative difference loss; otherwise, it is observed that the radius relative difference far outperforms all other losses in the initial training phase, which can lead to a rapid increase in the loss. dir indicates the flow loss of the pipe. According to various exemplary embodiments, since the predicted flow direction is not directed, the predicted L is used to calculate the flow loss of the pipe. i and ground truth
[0086]
number
[0087] The minimum Euclidean distance difference between,and,is used. Furthermore, all simulated data is counted in this loss.
[0088]
number
[0089] Training Details In experiments performed according to various exemplary embodiments, PipeNet402 is trained on both a simulated dataset and the open source datasets ScanNet (A. Dai, AX Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nieβner, “ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes,” 2017) and S3DIS ((I. Armeni, S. Sax, AR Zamir, and S. Savarese, “Joint 2D-3D-Semantic Data for Indoor Scene Understanding,” ArXiv e-prints, February 2017). The simulated dataset is created due to the lack of MEP categories in multiple open source datasets. Furthermore, since the training dataset required by the network contains both semantic labels and piping parameters, it is impractical to manually label and model piping using real scanned data. According to various exemplary embodiments, simulation of point cloud data is identified as an ideal workaround.
[0090] In various exemplary embodiments, the dataset simulation includes two steps: pipe model simulation and point cloud synthesis. Both steps are implemented in BLENDER software. MEP models are simulated using a modified pipeMare add-on. In this regard, the features added include round elbows with different curvatures, pipes in close parallel arrangement, and ducts with rectangular cross-sectional shapes. Each model includes pipes and / or ducts arranged in 2-4 layers along the depth axis within an area of 5m x 5m. The radius range is 1-20cm, covering common pipe sizes in residential and commercial buildings. Each simulated pipe has a fixed radius throughout and does not have any pipe fittings such as flanges and valves.
[0091] Point cloud synthesis is performed using the BlenSor add-on. Five scanner locations are randomly added for each model. These scanner locations are restricted to one side of the piping model at distances between 1 and 4 m. This is to simulate a real-world scanning scenario where a laser scanner scans multiple pipes installed under the ceiling or close to the wall. Three point cloud data are generated for each model by randomly selecting 1 to 3 random scans to increase data diversity in density and scan coverage. The ground truth is generated by sampling multiple points along multiple centerlines at 0.1 mm intervals and assigning each scan point its closest centerline point along with the corresponding radius, point normal, and pipe direction parameters. The point normal ground truth is calculated as the normalized offset vector from the scan point to the centerline point. The centerline node spacing size of 0.1 mm is selected to keep the angular error of the point normal ground truth within 1 degree. For illustrative purposes only, some examples of simulated training data are shown in Figure 8. In Fig. 8, the left column is a perspective view of an exemplary simulated pipe and duct, and the right column is a top view of the exemplary simulated pipe and duct. Additionally, Fig. 8 also shows ground truth centerline points through the exemplary simulated pipe and duct.
[0092] Although noise augmentation is performed, it is noted through experiments that PipeNet402 trained only on simulated data may not perform well on real scans, especially when the data is noisy. However, performance is improved according to various exemplary embodiments after adding ScanNet and S3DIS of real scan datasets as complementary training data. As an example, the first 100 data are used in ScanNet, and 10 toilet point clouds in S3DIS, which are more likely to have multiple pipes, are selected. Since these two public datasets do not contain the required plumbing-related ground truth, they are used as semantically negative data, i.e., they are used with L as the negative class. sem In total, 270 simulation models were created, including 1003 pipes and 616 ducts. Figure 9 shows the radius distribution of the simulated pipes. 810 synthetic point clouds are created. 20% of the point clouds were used for network training and the rest for testing. As an example, the final composition of the training dataset for PipeNet402 is summarized in Table 1 in Figure 10.
[0093] PipeNet402 was trained for 100 epochs with a batch size of 256 using the Adam optimizer. In each epoch, 1024 patches were randomly sampled from each point cloud. The initial learning rate was 0.001 and decayed at a rate of 0.8 every 4 epochs. As an example, the loss term weights used are [A 0 ,A 1 ,A 2 ,A 3 ,A 4 ,A 5 ]=[2,2.5,2,0.5,1,20]. The training took a total of 22 hours on an NVIDIA RTX A6000 GPU. Due to the imbalance in the number of non-piping and piping instances in the training dataset, various exemplary embodiments use L semLoss weights w of “1” and “2” were applied for pipes and non-pipes in, respectively. The input patch radius was set to 0.1 m and contained 512 points. 85 points (1 / 6 of the number of patch points) were selected from the point cloud subpatch to be used in multiple output modules. Gaussian noise was added to the running data with zero mean and standard deviation σ between 1 and 4 mm.
[0094] Model Reconstruction The as-built BIM model is then processed through multiple processes including centerline candidate generation 406, candidate refinement 410, and connectivity reconstruction 414 to obtain the centerline node set C:{c i ,r i ,l i Specifically, according to various exemplary embodiments, the method 400 of BIM reconstruction includes generating a set of centerline candidates CC based on the centerline node set C, refining the set of centerline candidates CC to obtain a refined set of centerline candidates, and generating a piping system model for the piping system based on the refined set of centerline candidates.
[0095] Center line candidate generation 406 In various exemplary embodiments, the centerline node set C may be divided into multiple centerline node subsets based on the centerline node and the piping flow direction associated with the centerline node for each centerline node (sometimes referred to herein as centerline point) in the centerline node set. For example, the centerline node set C may be divided into multiple centerline node subsets based on the spatial similarity and the piping flow direction l i The centerline node set C can be segmented using a region growing method based on a similarity criterion of C = 1, C = 2, C = 3, C = 4, C = 5, C = 6, C = 7, C = 8, C = 9, C = 10, C = 11, C = 12, C = 13, C = 14, C = 15, C = 16, C = 17, C = 18, C = 19, C = 20, C = 21, C = 22, C = 23, C = 24, C = 25, C = 26, C = 27, C = 28, C = 29, C = 30, C = 31, C = 32, C = 19, C = 25, C = 33, C = 15, C = 26, C = 19, C = 27, C = 34, C = 19, C = 28, C = 19, C = 29, C = 35, C = 18, C = 25, C = 26, C = 35, C = 19, C = 27, C = 36, C = 28, C = 37, C = 19, C = 29, C = 38, C = 29, C = 39, C = 19, C = 25, C = 35, C = 19, C = 25, C = 35 i, to obtain a set of centerline candidates CC. For example, RANSAC line fitting may be applied to each cluster (centerline node subset) to generate the centerline candidate set CC. In various exemplary embodiments, each centerline candidate cl i is the two end points
[0096]
number
[0097] , unit direction (or direction vector) D iri , radius R i , and a set of inlier centerline nodes C in It can be defined as R i is C in It should be noted that, although various exemplary embodiments may directly apply RANSAC line fitting to the centerline node set C to generate multiple centerline candidates, this method is found to require more computation time and generate more noisy candidates, creating more workload in the subsequent process of generating a piping system model of the piping system.
[0098] Centerline Refinement 410 According to various exemplary embodiments, candidate refinement is performed because multiple false positive candidates are generated and the region growing method may over-segment the pipe into multiple candidates. In various exemplary embodiments, refining the set of centerline candidates CC includes: for each of the multiple common centerline groups, setting a longest centerline candidate in the set of centerline candidates CC that does not already belong to any of the multiple common centerline groups as a seed centerline candidate associated with the common centerline group and adding the longest centerline candidate to the common centerline group; and based on determining that the centerline candidate satisfies a common centerline condition between the centerline candidate and the seed centerline candidate, adding the centerline candidate to the common centerline group and setting the added centerline candidate as a new seed centerline candidate associated with the common centerline group. Further, for each set new seed centerline candidate in the common centerline group, then, for each centerline candidate in the set of centerline candidates CC that does not yet belong to any of the multiple common centerline groups, based on a determination that the centerline candidate satisfies a common centerline condition between the centerline candidate and the seed centerline candidate (i.e., new seed centerline candidate), add the centerline candidate to the common centerline group and set the added centerline candidate as a new seed centerline candidate (i.e., further new seed centerline candidate) associated with the common centerline group. In various example embodiments, the common centerline condition includes multiple sub-conditions including a first sub-condition based on an angle formed by direction vectors associated with the centerline candidate and the seed centerline candidate, respectively, a second sub-condition based on a distance between the centerline candidate and the seed centerline candidate, and a third sub-condition based on a difference between multiple radii associated with the centerline candidate and the seed centerline candidate, respectively.
[0099] In other words, in an exemplary implementation according to various exemplary embodiments of the present invention, the centerline candidates in the set of centerline candidates CC are selected such that the longest line candidate in the set of centerline candidates CC is the seed line.
[0100]
number
[0101] The subset of centerline candidates (i.e., satisfying this exemplary common centerline condition) can be iteratively merged so that
[0102]
number
[0103] are added to a group (which may be called a common centerline group), where <·> denotes the angle formed by the two vectors, and D(·) denotes the distance between the two candidate lines, calculated by the distance from the nearest end point on one line to the other line. α, β, and γ are fixed scalars empirically set such that all thresholds are on the candidate radius. The adaptive thresholds can improve the generalization of the algorithm in multiple piping scenes with large dimensional diversity. The centerline candidates added to the group are then selected as new seeds, and the above process is repeated until all centerline candidates in the set of centerline candidates CC that satisfy the common centerline condition are added to the group (common centerline group). This forms multiple groups (common centerline groups) with the centerline candidates added as described above.
[0104] In various exemplary embodiments, after obtaining the plurality of common centerline groups, for each of the plurality of common centerline groups, a line fitting may be applied to the common centerline group to form a refined centerline candidate associated with the common centerline group to obtain a refined set of centerline candidates. For example, for each common centerline group, a refined centerline may be fitted to all inlier nodes of all centerline candidates in the group. A radius associated with the refined centerline candidate associated with each of the plurality of common centerline groups may then be determined by applying a circle fitting based on an extracted segment of the point cloud data corresponding to the length of the refined centerline candidate. For example, a least squares circle fitting may be applied to refine the candidate radius using the scan data corresponding to these inlier centerline nodes. In various exemplary embodiments, a segment of up to 1 m in length from the scan data with the best completeness is extracted for circle fitting. In this regard, various exemplary embodiments have found that such segment extraction is advantageous because it may be unreliable to directly apply a circle fitting method to an entire scan of a pipe having a long length and small diameter. For example, some pipes, such as fire sprinkler pipes with a diameter of about 4-5 cm, are not perfectly straight but bend slightly along their length due to gravity or differences in the positions or lengths of the hanging cables. For illustration, Fig. 11A-C show an example of a small pipe (pipe of interest) in actual test scan data of a scene. In particular, Fig. 11A shows a perspective view of the pipe of interest in a remote scene, Fig. 11B shows a close-up view of the pipe of interest (dark black pipe shown in a dotted box), and Fig. 11C shows a projection of the pipe of interest along its first principal direction obtained by PCA. It can be seen that the projections do not show a clear circular pattern.In various exemplary embodiments, to remove noise, since it is observed that noisy centerlines usually have poor inlier support or small length-to-radius ratio, the refined centerlines are further inspected by length-to-radius ratio and inlier density, which is defined as the number of inlier nodes per unit length.
[0105] Connectivity Reconstruction 414 In various exemplary embodiments, after obtaining the refined set of centerline candidates, a piping system model of the piping system can be generated, which includes performing a graph-based centerline connectivity reconstruction based on the refined set of centerline candidates. In various exemplary embodiments, performing the graph-based centerline connectivity reconstruction includes: constructing a graph including a plurality of nodes to represent the refined set of centerline candidates, each node representing a corresponding refined centerline candidate of the refined set of centerline candidates, each pair of nodes of the plurality of nodes having an edge weight assigned thereto; generating a minimum spanning forest including a plurality of minimum spanning trees based on the graph; and performing a plurality of centerline connections for the refined set of centerline candidates based on the minimum spanning forest. Thus, the centerline connection relationship is reconstructed, i.e., grouping multiple centerlines in the same piping system. In various exemplary embodiments, since the centerlines of connected piping segments are not guaranteed to be coplanar in 3D space, the intersections are calculated as follows and as shown in Figure 12, whereby both lines are extended and the closest points on one line to the other are found. The middle of the two closest points is defined as the skew line intersection. In particular, Figure 12 shows the distance and intersection of the lines at the skew, with the left side of Figure 12 showing a side view and the right side of Figure 12 showing a top view.
[0106] In various exemplary embodiments, in constructing the graph described above, the graph G(V,E) may be constructed such that each vertex V (which may be referred to herein as a node) represents a refined set of centerline candidates (which may be referred to herein as a refined centerline candidate set) corresponding to a centerline candidate (or more specifically, a refined centerline candidate). In various exemplary embodiments, for each pair of nodes of the plurality of nodes, an edge weight assigned to the pair of nodes is determined based on a Euclidean distance between the refined pair of centerline candidates of the refined set of centerline candidates corresponding to the pair of nodes, a scan data density in a potential connection region between the refined pair of centerline candidates, an orientation angle difference between the refined pair of centerline candidates, and a radius difference between the refined pair of centerline candidates. For example, in contrast to calculating edge weights based solely on multiple Euclidean distances between multiple centerlines, various exemplary embodiments consider multiple additional factors including scan data support, orientation angle difference, and radius difference in a potential connection region, as shown in the following Equations (8)-(11), where {B i │i∈{0,1,2}} is a constant weight for each term, ε and θ are constant parameters set empirically, and DR(V i ,V j ) respectively represent the sum of the ratios of the distances between the endpoints and intersections of a line to its radius, and N conn represents the number of scan data at the connection position. Therefore, for a certain DR(V i ,V j ), N conn The larger the W scan becomes smaller, and DR(V i ,V j ) CA denotes a set of common connection angles used in piping systems, such as 45 degrees and 90 degrees. In various exemplary embodiments, these three factors, i.e., W scan , W r , and W ang is designed to distinguish true connections in congested areas where complex piping systems exist.
[0107]
number
[0108] In various exemplary embodiments, each edge may be labeled according to one of a number of connection types, such as direct-joint, extended-joint, and complement-joint, based on the total endpoint distance and line distance on the radius. In various exemplary embodiments, if the endpoints of two lines are close enough and their line distance is within a preset threshold, the two endpoints may be labeled as direct joints. If the endpoint distance is larger, the two endpoints may be labeled as extended joints, and if their line distance is larger, they may be labeled as complement joints. For example, the above-mentioned predetermined threshold may be set by a user depending on the data quality and the scale of the scene, e.g., if it is a large-scale scene and the point cloud data has a large amount of missing areas (e.g., due to occlusion), the predetermined threshold may be set relatively large. In various exemplary embodiments, the predetermined threshold may be, for example, a multiplier of the radius of the longer pipe between the two pipes. By way of example only and not limitation, an exemplary predetermined threshold for the direct junction type may be "2" (e.g., distances smaller than pipe radius x 2 may be assigned as direct junction type), an exemplary predetermined threshold for the extended junction threshold type may be "15" for the laboratory example (e.g., pipe radius x 2<distance<15), and "25" for the warehouse example (e.g., pipe radius x 2<distance<25), and an exemplary predetermined threshold for the complementary junction threshold may be "1" and "20" (e.g., pipe radius x 1<line distance<radius x 20). It will be understood by those skilled in the art that the multiple thresholds for the multiple connection types are not limited to the exemplary thresholds mentioned above, and may be appropriately set as desired or for the intended purpose. For example, the complementary junction type may be to recover short multiple piping segments that may sometimes be filtered out due to misclassification as noise in the previous process due to a small length-to-radius ratio or insufficient centerline node support. For illustration, FIG. 13 shows the three types of junctions mentioned above.In various exemplary embodiments, in the graph-based centerline connectivity reconstruction, among the three types of junctions mentioned above, direct junction edges are prioritized over the other two types. In various exemplary embodiments, a minimum spanning forest including multiple minimum spanning trees (MSTs) is computed from the graph according to the junction priority order (i.e., in the order of direct junction edges→extended junction edges→complementary junction edges (starting with the highest priority)). In various exemplary embodiments, to maximize completeness, certain constraints based on domain knowledge of piping system design are applied to the MST computation, including, for example, the following i-iv:
[0109] i. Each endpoint of a centerline may be joined to a maximum of two other centerlines. ii. Two closely parallel centerlines with similar radii are mutually exclusive if they are not directly joined, i.e., they should be in different trees. (By way of example only and not limitation, a threshold of 0.1 may be set for radius similarity (meaning that the difference in radii should be less than 0.1 of the radius of the longer pipe between the two pipes), and a threshold of "6" may be set for the closeness of two parallel centerlines. It will be understood by those skilled in the art that the thresholds for radius similarity and centerline closeness are not limited to the exemplary thresholds mentioned above and can be set as desired or appropriate for the intended purpose.).
[0110] iii. If a pair of connected centerlines each have a mutually exclusive line, then these two exclusive lines should be connected as well, if there is an edge between them, regardless of the edge weight and type. Two pairs of connected centerlines should not collide with each other.
[0111] iv. Each tree created inherits all the exclusions of its member nodes, i.e., it is not possible to add an exclusionary line of all member nodes to a tree. For better understanding and illustration, FIG. 14 illustrates the formation of a minimum spanning forest according to various exemplary embodiments of the present invention. As shown in FIG. 14, each centerline cl i are nodes in the graph (e.g., associated with or indicated by the same index i). The edge weight W(V i ,V j ) are determined according to equations (8)-(11) above. Note that the edge weights shown in FIG. 14 are for illustration only and may not be the actual edge weights determined according to equations (8)-(11) above. That is, for ease of illustration, the edge weights are shown to correspond approximately to the end-point distances between each pair of centerlines. In accordance with constraint (ii) above, cl 1 and cl 2 are parallel, close together, and have similar radii, so are mutually exclusive, and cl 3 and cl 4 The same is true for E. Then, the prioritized direct junction edges are inspected, and E 2,3 Then, according to the constraint (iii) above, cl 2 and cl 3 The exclusion of, i.e., cl 1 and cl 4 Meanwhile, Edge E 1,4 Then, the extended joint edges are tested starting from those with the smallest edge weights. Due to the inheritance of exclusion by the constraint (iv) above, E 1,3 and E 2,4 is not selected. As a result, two minimum spanning trees that constitute a minimum spanning forest are generated, as shown in Figure 14. In various exemplary embodiments, each minimum spanning tree may be further inspected and filtered if it has, on average, too many anomalous connection angles or too few inlier centerline nodes. After performing this graph-based centerline connectivity reconstruction, the generated piping system model may be converted to an IFC format.
[0112] Experiments and Verification Next, validation of the method (sometimes simply referred to herein as the method) for reconstructing the BIM of one or more piping systems according to various exemplary embodiments on both synthetic and real scan data sets is presented and described. The synthetic data set allows for the generation of experiments and quantitative results under different controlled conditions. In the experiments, the synthetic data is subjected to two enhancements: scan coverage angle variation and noise addition. The point cloud of each pipe is cropped to make the scan coverage angle θ limited from 60° to 180° in steps of 60°. Gaussian noises with zero mean and standard deviation σ corresponding to 0.5% to 2% of the pipe radius are added.
[0113] DSM Verification FIG. 15 shows a visualization of the neighborhood selection results generated by DSM512 according to various exemplary embodiments of the present invention compared to the neighborhood selection results generated using the conventional k-NN method. In the neighborhood selection results shown in FIG. 15, each seed point is illustrated by a dark black dot (it will be understood by those skilled in the art that each seed point is shown enlarged in FIG. 15 only for clarity and ease of reference and does not have any relevance regarding the size of the seed point). In FIG. 15, the first column shows the input point cloud patches (i.e., the selected neighborhood points collected by the ball query method) obtained by the ball query method using a radius of 0.1 m, the second column and the third column show the neighborhood selection results (i.e., the selected sub-patch points) using the conventional k-NN method (where k=85, which is the same as the PipeNet setting described above in the "Training Details" section) and DSM512, respectively, where the neighborhood selection results (selected or predicted true neighborhood points) are shown in dark black, and the filtered (excluded) points are shown in gray. Therefore, the output from DSM512 is visualized to verify whether DSM512 actually learns to distinguish multiple neighboring points from different objects (i.e., different pipe segments or parts), as shown in FIG. 15. The first two rows show the results on synthetic data, and the bottom two rows show the results on real scan data. From FIG. 15, it can be seen that DSM512 is effective in filtering out (filtering) points from different ones (i.e., neighboring points that do not belong to the same pipe segment as the seed point) while keeping the most true neighboring points (i.e., neighboring points that belong to the same pipe segment as the seed point). In FIG. 15, an example input patch containing a 90-degree elbow is shown, and DSM512 determined or predicted points on the other side of the elbow (the other side relative to the seed point) as outliers, even though they are from the same pipe system. That is, DSM512 successfully predicted those points as not belonging to the same pipe segment (or pipe part). This is consistent with expectations, since pipe parameter estimation only depends on multiple points from the same pipe segment or cylinder as the seed point.
[0114] DSM512 is further evaluated through comparison with ablation networks, namely baseline networks PointNet++ and NestiNet. The two ablation networks are created by replacing DSM512 with k-NN and ball query methods while keeping the rest of the network structure and parameters the same. In various exemplary embodiments, the input to PipeNet402 is the patch point cloud obtained by ball query, so the ball query neighbor point selection is essentially randomly sampling a subset within the input patch. PointNet++ multi-scale classification network model is used for comparison. Both PointNet++ and NestiNet have final fully connected multiple layers that are replicated to produce the same output as the current PipeNet402. The synthetic dataset used for validation includes 20 models of pipes and ducts with dimensions ranging from 2 cm to 40 cm in 2 cm steps. In each model, three pipes and one duct of the same dimensions (diameter for pipes, width and height for ducts) are arranged in parallel with 3 cm intervals. Planes were also created behind the pipes and ducts to simulate real-world environments where pipes and ducts are installed near walls or ceilings. Following the procedure described above in the "Training Details" section, three scans are synthesized for each model. FIG. 16 shows the qualitative results of two models with dimensions of 4 cm and 38 cm, respectively, for ablation and comparison studies on the synthetic parallel dataset. In FIG. 16, the pipe points are dark black, the duct points are dark gray, the plane points are light gray, and the predicted pipe centerline points are white. It can be seen that the centerline point predictions of PipeNet402 according to various exemplary embodiments of the present invention are clearer and less out of line compared to the other methods shown. FIG. 17 shows a table (Table 2) summarizing the quantitative results of the synthetic parallel dataset testing.Across all noise levels, PipeNet402 according to various exemplary embodiments of the present invention is able to consistently predict more accurate centerline nodes and pipe radii, showing, on average, 15.5%-37.6% improvement for centerline point prediction and 15.7%-32.4% improvement for radius prediction.
[0115] Sensitivity analysis The sensitivity of the method to scan coverage angle and noise is evaluated. The synthetic dataset used for the sensitivity analysis consists of multiple independent models of straight pipes with radii ranging from 1 cm to 20 cm in steps of 1 cm. The point cloud synthesis procedure remains the same as described above in the "Training Details" section. Robustness to variations in scan coverage angle can be important, since real pipes are typically installed adjacent to walls or ceilings or within confined spaces, resulting in inevitable loss of visibility of parts of the scan data, which may affect the reconstruction results.
[0116] The accuracy of both PipeNet prediction and reconstruction is tested. Figures 18A-18D show box plots of radius and centerline node errors predicted by PipeNet402 for synthetic data under different scan coverage angles θ and low and high noise levels. The horizontal axis represents the scan coverage angle θ. Figures 18A and 18C correspond to a low σ=0.5% noise level, while Figures 18B and 18D correspond to a high σ=2% noise level. It can be observed that both errors decrease as θ increases from 60° to 120° and then remain fairly stable for the following θ cases. As expected, both the error and the error distribution increase for high noise cases. Several outliers are observed for θ between 60° and 120°. The outliers are from several pipes with two smallest radii (1-2 cm) for θ=60° and several pipes with a radius of 1 cm for θ={90°,120°}. 19A-19D show plots of the mean absolute and mean relative radial errors, maximum centerline end point displacements, and centerline angles for the final reconstructed pipe models under all σ and θ conditions. Each curve corresponds to a different noise level σ, and the horizontal axis represents the scan coverage angle θ. The angular error is in degrees. The error generally decreases with increasing θ and decreasing σ, and remains at a fairly low level for the condition θ≧120°, proving the robustness of the method. However, it is noted that the method according to various exemplary embodiments may not be able to reconstruct a pipe with a radius of 1 cm at θ≦120°, and a pipe with a radius of 2 cm at θ=60°. To address this, for small pipes with a radius of 2 cm or less, various exemplary embodiments capture them at a scan coverage angle greater than 120° to ensure successful detection and reconstruction.
[0117] Actual Scan Verification Data and study details The method is tested on two real scan data. The first scene is a laboratory at Nanyang Technological University. The laboratory is a fully operational laboratory with equipment and staff working around. The second scene is an empty warehouse. Basic statistics of the test scenes and scan data are summarized in Table 3 of FIG. 20. The tests are performed using one NVIDIA RTX A6000 GPU. In the implementation of PipeNet402, two seed point sampling rounds with coarse and fine grid sizes are adopted, respectively. The grid size in the coarse sampling round (or the first sampling round) is 0.1 m. Only the patch points of the seed points that are semantically predicted as pipes are kept and used in the fine sampling round (or the second sampling round). That is, in the coarse sampling round, the seed points are sampled from the input raw point cloud data with a large seed point sampling grid size. Then, all the seed points predicted as pipe points are taken along with their point cloud patch points and provided to a fine sampling round (i.e., corresponding to the "coarse" to "fine" approach for seed point sampling described above). The coarse sampling acts as a fast screen of the input point clouds to isolate potential pipe points, thus significantly reducing the total implementation time. The fine sampling round uses a smaller grid size of 0.02 m to scrutinize the potential pipe data and predict the final pipe points and their centerline nodes along with other pipe parameters. In various exemplary embodiments, the predictions described above are also performed using PipeNet402. Thus, in various exemplary embodiments, PipeNet402 may be implemented in a coarse sampling round (sometimes referred to as PipeNet's coarse sampling round) and then subsequently in a fine sampling round (PipeNet's fine sampling round). The results from the fine sampling round are used in the subsequent model reconstruction stage.
[0118] result 21A-21D show the point cloud (with the ceiling manually removed to better visualize the interior), the predicted potential piping points extracted by PipeNet's coarse sampling process, the reconstructed IFC model, and the IFC model superimposed on the potential piping points for a laboratory, respectively. 22A-22C show the point cloud (with the ceiling manually removed to better visualize the interior), the predicted potential piping points extracted by PipeNet's coarse sampling process, and the reconstructed IFC model, respectively, for a warehouse. Since the warehouse area is large, the scan was taken without color to save time. For example, comparing FIG. 21A with FIG. 21B, we can see that PipeNet's coarse sampling effectively removes most of the data points belonging to structural objects and fixtures.
[0119] The processing times for PipeNet402 and model reconstruction, respectively, are shown in Table 4 of FIG. 23. Various exemplary embodiments evaluate the reconstruction results using recall, precision, and F1 score, as defined in the following equations (12)-(14). The results for both scenes are summarized in Table 5 of FIG. 24. In the laboratory scene, 187 out of 197 pipes are successfully reconstructed, generating 11 false positives. The recall, precision, and F1 scores are calculated as 0.95, 0.90, and 0.92. In the warehouse scene, 160 out of 171 pipes are successfully reconstructed, creating 9 false positives. The recall, precision, and F1 scores are calculated as 0.94, 0.89, and 0.91. Thus, the method is demonstrated or verified to be capable of efficiently and robustly reconstructing multiple pipes, even in crowded and large scenes. For example, compared to the existing studies listed in Table 6 of FIG. 25, although the test data used is different, the present method achieves the third best recall and the best precision and F1 score. In particular, Table 1 includes the following existing studies:
[0120] [Table 1]
[0121] Compared to Lee et al., supra, which had the best reported recall, the processing time of our method is much shorter per point and is therefore significantly more efficient.
[0122]
number
[0123] In the laboratory scene, selected close-up views of the reconstructed piping model superimposed on the point cloud data are shown in Fig. 26A-D, with boxes highlighting errors. In particular, Fig. 26A shows a close-up view of the IFC model superimposed on the point cloud data, Fig. 26B shows false positive piping is generated at the light cable, Fig. 26C shows false positive piping is generated at the curved part of the phone booth, and Fig. 26D shows missing piping. There are mainly two types of false positive sources. The first is when the electric cables of the light are misclassified as piping by PipeNet402, as shown in Fig. 26B. The second source is furniture (fixtures) with partial surfaces of circularly curved shapes, such as the headrest and armrest of the sofa and the legs of the office chair shown in Fig. 17C. This misclassification is because PipeNet402 has a limited field of view of objects from input patches with limited radii. FIG. 26D shows a failed pipe segment with a length of 12 cm and a scan with a small angle range, where the failed pipe segment is erroneously changed to a ceiling. This type of short pipe segment is more likely to fail in the centerline candidate generation step due to its small number of centerline nodes. Nevertheless, if a structural model is available, it can be recovered by adding structural components as additional nodes in the graph. The failed pipe segment and the ceiling can be considered as complementary junctions, and their edge weights can be calculated in the same way as described above in the "connectivity reconstruction" section.
[0124] In the warehouse scene, Fig. 27A-D show selected close-up views of an IFC model superimposed on multiple pipe points, with multiple boxes highlighting errors. Specifically, Fig. 27A shows a close-up view of an IFC model superimposed on a potential pipe point, Fig. 27B and Fig. 27C show examples of failed pipe segments, and Fig. 27D shows an example of a false positive generated on a human limb. The missing pipe in Fig. 27B is due to a short length of about 8 cm. Although the warehouse is an empty space without fixtures, most pipes are small in size and installed near the ceiling at a height of about 10 m, making it difficult for both scanning and reconstruction. As shown in Fig. 27C, sparse scans, scans with small angle ranges, and failed (missing) scans are frequently observed, especially for multiple pipes near the wall, which is due to the limited range of laser incidence angles from the scanner placed on the ground to these pipes. False positives typically occur in objects with surfaces of circular or nearly circular cross-sectional shapes, such as stair railings and people's limbs captured in the scan, as shown in Figure 27D. These types of false positives can be filtered by analyzing the relative positions between the generated piping and structural components, if a structural model is available.
[0125] Thus, various exemplary embodiments of the present invention advantageously provide a method for automatically reconstructing as-built BIM models of multiple piping systems from a point cloud. For example, the method does not require any additional data other than the point cloud with XYZ fields, and does not require any pre-processing on the input data, such as normal estimation, or prior knowledge of the piping, such as their orientation. In various exemplary embodiments, the method includes two stages, and in the first stage (corresponding to PipeNet 402), a deep learning network is developed to extract multiple piping points from the input data and predict the piping centerline points and other piping parameters, including the radius and undirected flow direction of each sampled point. In particular, the DSM module 512 can be configured to select optimal true neighboring points that belong to the same piping segment as the seed point, thus helping to better predict multiple piping parameters. In the second stage, the predicted centerline nodes are further processed to output an IFC model through multiple processes, including centerline candidate generation using region growing and RANSAC line fitting, centerline refinement using iterative line merging and circle fitting, and finally, graph-theoretic based connectivity reconstruction constrained by domain knowledge.
[0126] As described in detail above, the method is validated against both synthetic data and real scans. The DSM module 512 is validated against synthetic data, where pipes are placed close to each other along with non-pipe objects, by comparing with an ablation model, a baseline model, and a SOTA model. The results prove that the DSM 512 is effective in filtering outliers, and that PipeNet 402 can predict multiple centerline nodes and pipe radii that are consistently accurate in all noise conditions. The sensitivity of the method of the present invention to scan angle range and noise is also evaluated for synthetic data. The results show that scans with an angle range of 120° or more are preferred to reconstruct small pipes with radii of 2 cm or less, but the method generally shows robustness to both conditions. In the real scan tests, one scene is a fully operational laboratory with a complex piping system as well as fixtures and people around. The other scene is an empty warehouse with an 11 m high ceiling and a fire protection piping system installed directly below the ceiling. Our method successfully works on both scenes, achieving a total recall of 0.94, precision of 0.89, and F1 score of 0.92, which outperforms most existing works. Reconstruction takes 9.53 and 24.14 minutes, respectively, which is significantly more efficient than manual modeling.
[0127] In various exemplary embodiments, due to the design of the patches input to PipeNet402, the neighborhood information available to the seed points is limited. This limitation may cause the generation of false positives on non-pipe objects with partial circular or near-circular cross-sectional surfaces. To address this, various exemplary embodiments may apply a point embedder to the entire or larger size of the point cloud data. This would then allow each point to be embedded with features of a larger neighborhood, thus providing richer information to the seed points. Furthermore, in various exemplary embodiments, pipes may be assumed to be dominated by straight line segments. While this assumption holds true for most of the piping systems, it may fail for trap components such as U-traps or S-traps, which may be common in sewer systems. To address this, various exemplary embodiments perform reconstruction of these kinds of irregular components by matching the BIM model with point clouds in potential areas. For example, short centerline candidates are usually generated in those components that imply the presence of such components.
[0128] Although the embodiments of the present invention have been particularly shown and described with reference to certain embodiments, it should be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention as defined by the appended claims. The scope of the present invention is therefore indicated by the appended claims, and all changes that come within the meaning and range of equivalence of the claims are therefore intended to be embraced.
Claims
1. 1. A method for reconstructing a building information modeling (BIM) for a piping system using at least one processor, comprising: obtaining a point cloud patch from point cloud data obtained for the piping system, the point cloud patch relating to a seed point for the point cloud patch, the point cloud patch including the seed point and a first plurality of neighboring points relating to the seed point; For each of the first neighboring points, embedding a plurality of neighboring features related to the neighboring point into the neighboring point to form a neighboring feature embedded point cloud patch; generating a point cloud sub-patch including a plurality of second neighboring points from the plurality of first neighboring points of the neighbor feature embedded point cloud patch, where each of the plurality of second neighboring points is determined to belong to the same pipe segment as the seed point using a first machine learning model; determining whether the seed point is a pipe point based on the point cloud subpatch using a second machine learning model; and determining a piping centerline point associated with the seed point based on the point cloud subpatch and based on determining that the seed point is a piping point.
2. The method of claim 1 , wherein the plurality of neighborhood features for the neighboring points are latent space features learned for a local neighborhood of the neighboring points.
3. Determining the pipe centerline points associated with the seed points comprises: determining a pipe radius associated with the seed point based on the point cloud subpatch using a third machine learning model; and determining a point normal associated with the seed point based on the point cloud subpatch using a fourth machine learning model; and and determining the pipe centerline point associated with the seed point based on the seed point, the determined pipe radius, and the determined point normal.
4. The method of claim 3 , further comprising: determining a pipe flow direction associated with the seed point based on the point cloud subpatch using a fifth machine learning model.
5. the first machine learning model is a first classification model trained to determine, for each of the first neighboring points, whether the neighboring point belongs to the same pipe segment as the seed point; the second machine learning model is a second classification model trained to determine whether the seed point is a pipe point based on the point cloud subpatch; the third machine learning model is a first regression model trained to determine the pipe radius associated with the seed point based on the point cloud subpatch; the fourth machine learning model is a second regression model trained to determine the point normals associated with the seed points based on the point cloud sub-patches; The method of claim 4 , wherein the fifth machine learning model is a third regression model trained to determine a flow direction of the pipe associated with the seed point based on the point cloud subpatch.
6. Obtaining the point cloud patch for the seed point comprises: The method of any one of claims 1 to 5, comprising extracting the plurality of neighbouring points about the seed point from the point cloud data using a ball query method or a k-nearest neighbour (k-NN) method.
7. For each of a plurality of point cloud patches from the point cloud data, the method comprises: obtaining the point cloud patch for the seed point for the point cloud patch; for each of the first neighboring points, embedding the neighboring points with the neighboring features associated with the first neighboring points; generating the point cloud sub-patch; determining whether the seed point is a piping point; determining the piping centerline point associated with the seed point based on the point cloud subpatch and based on determining that the seed point is a piping point; determining a flow direction of the pipe associated with the seed point; The method comprises: generating, for each of the plurality of point cloud patches that include the seed point determined to be a pipe point, a centerline parameter set including the determined pipe centerline point associated with the seed point for that point cloud patch, the determined pipe radius associated with the seed point for that point cloud patch, and the determined pipe flow direction associated with the seed point for that point cloud patch; The method according to any one of claims 4 to 6, wherein the pipe radius determined based on the point cloud patches and the flow direction of the pipe determined are associated with a pipe centerline point determined based on the point cloud patches.
8. generating a set of centerline candidates based on the centerline parameter set; refining the set of centerline candidates to obtain a refined set of centerline candidates; The method of claim 7 , further comprising: generating a piping system model for the piping system based on the refined set of centerline candidates.
9. Generating the set of centerline candidates comprises: for each centerline point in the centerline parameter set, dividing the centerline parameter set into a plurality of centerline parameter subsets based on the centerline point and a flow direction of the pipe associated with the centerline point; 9. The method of claim 8, comprising: for each of the plurality of centerline parameter subsets, applying line fitting to the centerline parameter subset to generate centerline candidates associated with the centerline parameter subset to obtain the set of centerline candidates.
10. The method of claim 9 , wherein each centerline candidate in the set of centerline candidates is associated with two end points, a direction vector, a radius, and a set of inlier centerline points.
11. Refining the set of centerline candidates includes, for each of a plurality of common centerline groups: setting a longest centerline candidate in the set of centerline candidates that does not already belong to any of the plurality of common centerline groups as a seed centerline candidate associated with the common centerline group, and adding the longest centerline candidate to the common centerline group; for each centerline candidate in the set of centerline candidates that does not yet belong to any of the plurality of common centerline groups, based on determining that the centerline candidate satisfies a common centerline condition between the centerline candidate and the seed centerline candidate, adding the centerline candidate to the common centerline group and setting the added centerline candidate as a new seed centerline candidate associated with the common centerline group; 11. The method of claim 10, comprising: for each new seed centerline candidate set in the common centerline group, then, for each centerline candidate in the set of centerline candidates that does not yet belong to any of the plurality of common centerline groups, adding the centerline candidate to the common centerline group based on determining that the centerline candidate satisfies the common centerline condition between the centerline candidate and the seed centerline candidate, and setting the added centerline candidate as a new seed centerline candidate associated with the common centerline group.
12. The common centerline condition is:
12. The method of claim 11, further comprising: a first subcondition based on an angle formed by the plurality of directional vectors associated with the centerline candidate and the seed centerline candidate, respectively; a second subcondition based on a distance between the centerline candidate and the seed centerline candidate; and a third subcondition based on a difference between the plurality of radii associated with the centerline candidate and the seed centerline candidate, respectively.
13. Refining the set of centerline candidates comprises:
13. The method of claim 11 or 12, further comprising, for each of the plurality of common centerline groups, applying line fitting to the common centerline group to form refined centerline candidates associated with the common centerline group to obtain the set of refined centerline candidates.
14. Refining the set of centerline candidates comprises:
14. The method of claim 13, further comprising: for each of the plurality of common centerline groups, determining a radius associated with the refined centerline candidate associated with that common centerline group by applying circle fitting based on an extracted segment of the point cloud data corresponding to a length of the refined centerline candidate.
15. Generating the piping system model includes: performing a graph-based centerline connectivity reconstruction based on the refined set of centerline candidates; Performing the graph based centerline connectivity reconstruction includes: constructing a graph comprising a plurality of nodes representing the refined set of centerline candidates, where each node represents a corresponding refined centerline candidate of the refined set of centerline candidates, and where each pair of nodes of the plurality of nodes has an edge weight assigned to the pair of nodes; generating a minimum spanning forest based on the graph, the minimum spanning forest including a plurality of minimum spanning trees; and performing multiple centerline connections on the refined set of centerline candidates based on the minimum spanning forest.
16. Performing the graph based centerline connectivity reconstruction includes:
16. The method of claim 15, further comprising: for each pair of nodes of the plurality of nodes, determining the edge weights assigned to the pair of nodes based on a Euclidean distance between a refined pair of centerline candidates of the refined set of centerline candidates corresponding to the pair of nodes, a scan data density in a potential connection region between the refined pair of centerline candidates, an orientation angle difference between the refined pair of centerline candidates, and a radius difference between the refined pair of centerline candidates.
17. A system for reconstructing building information modeling (BIM) for a piping system, comprising: At least one memory; A system comprising: at least one processor communicatively connected to the at least one memory and configured to execute the method for reconstructing a BIM according to any one of claims 1 to 16.
18. A computer program product implemented on one or more non-transitory computer-readable storage media comprising a plurality of instructions executable by at least one processor to perform the method for reconstructing a BIM according to any one of claims 1 to 16.
Citation Information
Patent Citations
Pipeline three-dimensional reconstruction method and system based on deep learning, medium and equipment
CN111429563A
Three-dimensional point cloud pipeline extraction and modeling method for self-adaptive search radius
CN112884886A
Pipe reconstruction from unorganized point cloud data
US20110304628A1
Site management method and site management device
WO2013051080A1
