A scene 3D intelligent reconstruction system and method based on BIM and deep learning
Through a three-dimensional intelligent reconstruction system based on BIM and deep learning, combined with RGB-D sensors and improved PointNet++ network, the problems of limited range and low accuracy in the existing technology are solved, and efficient three-dimensional reconstruction and semantic segmentation are achieved. It also provides technical support for comparing the differences between design and construction in construction.
Patent Information
- Application Number
- CN202210734738.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-27
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-06-27
AI Technical Summary
The existing three-dimensional reconstruction system has limited reconstruction scope in building indoor projects, and the efficiency and accuracy of three-dimensional reconstruction and semantic segmentation are not high, making it difficult to effectively compare the differences between the design stage and the construction stage.
Adopt a scene 3D intelligent reconstruction system based on BIM and deep learning, including deep data acquisition and processing module, 3D reconstruction module, semantic segmentation module and BIM fusion diagnosis module. Data is collected through RGB-D sensors and semantic segmentation is used for improved PointNet++ network. and three-dimensional reconstruction, and compare the results with the BIM model to achieve the difference between the construction and design stages.
Improve the efficiency and accuracy of three-dimensional reconstruction and semantic segmentation, enhance the informatization and visualization level of construction management, better compare the differences between design and construction stages, and is suitable for building completion comparison, expand the reconstruction scope and improve equipment scanning. Position stability within the field of view.
Smart Images

Figure CN115147545B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a scene three-dimensional intelligent reconstruction system and method based on BIM and deep learning, belonging to the technical field of scene three-dimensional reconstruction. Background Art
[0002] With the rapid development of science and technology and information technology, the construction industry is facing new changes with the emergence and application of technologies such as BIM (Building Information Modeling), 3D laser scanning, deep learning, AR (Augmented Reality), VR (Virtual Reality), and 3D printing.
[0003] 3D reconstruction is a basic work in the field of 3D computer vision. It reconstructs the 3D model of an object or scene based on a 2D image using the third-dimensional depth data information. The focus of 3D reconstruction technology is how to obtain the depth information of the target scene or object. Under the condition that the depth information of the scene is known, the 3D reconstruction of the scene can be achieved only through the registration and fusion of the point cloud data. The commonly used 3D reconstruction system is KinectFusion, but research has found that the reconstruction range of the KinectFusion modeling system is very limited. If it is applied to the 3D reconstruction of indoor projects, the complete model needs to be re-stitched between the block point clouds. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a scene three-dimensional intelligent reconstruction system and method based on BIM and deep learning, thereby improving the accuracy of building three-dimensional reconstruction.
[0005] The present invention adopts the following technical solutions to solve the above technical problems:
[0006] A scene 3D intelligent reconstruction system based on BIM and deep learning, the system comprising a deep data acquisition and processing module, a 3D reconstruction module, a semantic segmentation module and a BIM fusion diagnosis module;
[0007] The depth data acquisition and processing module is used to collect depth data during the construction phase, pre-process the depth data, convert the pre-processed depth data into point cloud data, and perform HHA encoding processing on the depth data;
[0008] The three-dimensional reconstruction module is used to complete the point cloud data and perform three-dimensional reconstruction based on the completed point cloud data to obtain a three-dimensional model;
[0009] The semantic segmentation module is used to perform semantic segmentation on the depth data after HHA encoding processing using the improved PointNet++ semantic segmentation network, and to annotate the semantic information;
[0010] The BIM fusion diagnosis module is used to fuse the three-dimensional model obtained by the three-dimensional reconstruction module with the semantic information obtained by the semantic segmentation module, and compare the fused model with the building BIM model to achieve difference comparison between the construction stage and the design stage.
[0011] As a preferred solution of the system of the present invention, the depth data acquisition and processing module includes an RGB-D sensor, a preprocessing unit, a point cloud conversion unit and an HHA encoding processing unit;
[0012] The working process of the depth data acquisition and processing module is as follows:
[0013] 1) Calibrate the internal and external parameters of the RGB-D sensor, and collect depth data during the construction phase through the calibrated RGB-D sensor;
[0014] 2) Using the preprocessing unit to filter the collected depth data;
[0015] 3) using a point cloud conversion unit to convert the filtered depth data into point cloud data, so as to prepare for the three-dimensional reconstruction module;
[0016] 4) Perform HHA encoding processing on the depth data using the HHA encoding processing unit to prepare for the semantic segmentation module.
[0017] As a preferred solution of the system of the present invention, the working process of the three-dimensional reconstruction module is as follows:
[0018] First, the point cloud data is completed through the point cloud completion neural network, and then the completed point cloud data is reconstructed in three dimensions through the BundleFunsion method to obtain a three-dimensional model.
[0019] As a preferred solution of the system of the present invention, the improved PointNet++ semantic segmentation network includes a first sampling and grouping layer, a first improved PointNet network layer, a second sampling and grouping layer, a second improved PointNet network layer, a first reverse interpolation layer, a third improved PointNet network layer, a second reverse interpolation layer, a fourth improved PointNet network layer and an output layer connected in sequence, the output of the first sampling and grouping layer and the output of the second reverse interpolation layer serve as inputs to the fourth improved PointNet network layer, and the output of the second sampling and grouping layer and the output of the first reverse interpolation layer serve as inputs to the third improved PointNet network layer;
[0020] The first improved PointNet network layer includes a spatial alignment layer, a first fully connected network, a feature alignment layer, an attention mechanism module, a second fully connected network, a maximum pooling layer, a third fully connected network, a fourth fully connected network and an output layer connected in sequence; for the N×3 point cloud input, the first improved PointNet network layer first aligns it spatially through the spatial alignment layer, then maps it to a 64-dimensional high-dimensional space through the first fully connected network, and then aligns it through the feature alignment layer to obtain a 64-dimensional local feature. The 64-dimensional local feature is obtained through the attention mechanism module, the second fully connected network and the maximum pooling layer. After the pooling layer, a 1024-dimensional global feature is obtained, and the 64-dimensional local feature and the 1024-dimensional global feature are combined to form a 1088-dimensional space, which is then passed through the third fully connected network and the fourth fully connected network in turn to obtain the output of the first improved PointNet network layer; where N is the number of deep data after HHA encoding, the dimension of the first fully connected network is 64×64, the dimension of the second fully connected network is 64×128×1024, the dimension of the third fully connected network is 512×256×128, and the dimension of the fourth fully connected network is 128×M, where M is the number of categories of point cloud data;
[0021] The structures of the first improved PointNet network layer, the second improved PointNet network layer, the third improved PointNet network layer and the fourth improved PointNet network layer are all the same.
[0022] As a preferred solution of the system of the present invention, the working process of the BIM fusion diagnosis module is as follows:
[0023] The 3D model obtained by the 3D reconstruction module is fused with the semantic information obtained by the semantic segmentation module and converted into a point cloud file. The point cloud file is imported into the Revit software for comparison with the BIM model in the design phase, thereby realizing the fusion of the 3D semantic information and the BIM model, and providing the difference comparison information between the construction phase and the design phase.
[0024] A scene 3D intelligent reconstruction method based on the above-mentioned scene 3D intelligent reconstruction system based on BIM and deep learning, the scene 3D intelligent reconstruction method comprises the following steps:
[0025] Step 1, collecting depth data during the construction phase, preprocessing the depth data, converting the preprocessed depth data into point cloud data, and performing HHA encoding processing on the depth data;
[0026] Step 2, completing the point cloud data, and performing 3D reconstruction based on the completed point cloud data to obtain a 3D model;
[0027] Step 3: Use the improved PointNet++ semantic segmentation network to perform semantic segmentation on the depth data after HHA encoding and annotate the semantic information;
[0028] Step 4: fuse the three-dimensional model obtained in step 2 with the semantic information obtained in step 3, and compare the fused model with the building BIM model to achieve a difference comparison between the construction stage and the design stage.
[0029] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned scene three-dimensional intelligent reconstruction method are implemented.
[0030] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned scene three-dimensional intelligent reconstruction method.
[0031] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:
[0032] 1. The present invention improves the efficiency of 3D reconstruction and semantic segmentation, integrates them with the BIM model, and applies them to construction management, thereby improving the informationization, real-time and visualization levels of the construction and acceptance stages, in order to establish a 3D intelligent reconstruction system for scenes based on BIM and deep learning, and accelerate the coordinated development of intelligent construction and building industrialization.
[0033] 2. The present invention overcomes the problem of low efficiency and accuracy of 3D reconstruction and semantic segmentation of buildings in the prior art, making 3D reconstruction and semantic segmentation more convenient and timely. At the same time, the innovative use of RGB-D sensors for data collection has the characteristics of high efficiency, moderate accuracy and low cost, and the deep neural network algorithm is used to complete the depth data, further improving the accuracy of subsequent 3D reconstruction. The present invention can be applied to the comparison of completed buildings, and can well compare the differences between buildings in the design stage and the construction stage.
[0034] 3. The present invention re-adopts the 3D reconstruction method based on BundleFusion, mainly studies and improves the algorithms of point cloud registration and point cloud fusion, and the remaining stages refer to the methods used in KinectFusion. This method can expand the original reconstruction range, realize the tracking and positioning of the camera posture within the device scanning field of view, and improve the camera tracking stability, so it can better meet the actual engineering modeling needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1It is an architecture diagram of a scene three-dimensional intelligent reconstruction system based on BIM and deep learning of the present invention;
[0036] Figure 2 It is a flow chart of a scene three-dimensional intelligent reconstruction system based on BIM and deep learning of the present invention;
[0037] Figure 3 This is the structural diagram of the improved PointNet++ semantic segmentation network;
[0038] Figure 4 This is the structure diagram of the PointNet network with the attention mechanism introduced;
[0039] Figure 5 It is the structural diagram of the attention mechanism module. DETAILED DESCRIPTION
[0040] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.
[0041] like Figure 1 , Figure 2 As shown, the present invention proposes a scene 3D intelligent reconstruction system based on BIM and deep learning, which includes a depth data acquisition and processing module, a 3D reconstruction module, a semantic segmentation module and a BIM fusion diagnosis module; wherein:
[0042] The depth data acquisition and processing module collects depth data through the RGBD sensor and processes the data through algorithms to achieve the acquisition and processing of depth data;
[0043] The 3D reconstruction module is based on the data collected by the depth data acquisition and processing module. It first enriches the collected depth information through the point cloud completion neural network, and then completes the 3D reconstruction process of the depth information through BundleFunsion, and renders the 3D model at the same time;
[0044] The semantic segmentation module improves the original PointNet++ neural network by introducing the attention mechanism to further improve the accuracy of network segmentation. It is then trained based on the data collected by the depth data acquisition and processing module to perform semantic segmentation on the depth information, annotate the semantic information, and divide the information of each component.
[0045] The BIM fusion diagnosis module achieves model fusion by fusing the 3D model of the 3D reconstruction module with the semantic information of the semantic segmentation module and comparing it with the BIM model in the design phase, thus providing a technical solution for the construction progress management and completion comparison system.
[0046] The depth data acquisition and processing module collects depth image data through the RGB-D sensor, and then processes the data to realize the acquisition of depth data. The acquisition and processing methods are as follows:
[0047] Step (1), calibrating the internal and external parameters of the RGB-D camera. The accuracy of the calibration result and the stability of the algorithm directly affect the accuracy of the results produced by the RGB-D camera.
[0048] Step (2), collecting RGB data and depth data using a calibrated RGB-D camera;
[0049] Step (3), filtering the collected data and converting it into point cloud data;
[0050] Step (4), perform HHA encoding on the depth data to prepare for the subsequent semantic segmentation module.
[0051] The 3D reconstruction module uses the data collected by the depth data acquisition and processing module to perform 3D reconstruction. This module mainly studies and improves the algorithms of the point cloud registration and point cloud fusion stages. This module can expand the original reconstruction range, realize the tracking and positioning of the camera posture within the device scanning field of view, and improve the camera tracking stability, so it can better adapt to the actual engineering modeling needs. The 3D reconstruction module mainly adopts the BundleFusion method. The core of this method is a robust posture estimation strategy. By considering the complete scan image of the RGB-D input, an efficient hierarchical method is adopted. On the basis of sparse features and dense correspondence, the constantly changing posture estimation flow is monitored, and the reconstruction is updated through dynamic integration and de-integration. Each frame of the global camera posture set is optimized to achieve real-time solution to the global registration problem and achieve the purpose of 3D reconstruction.
[0052] The semantic segmentation module is an improvement on the original PointNet++ network. The PointNet++ network extracts local features using the PointNet network. The present invention improves PointNet++ by using the PointNet with the attention mechanism to improve the accuracy of semantic segmentation. The data collected by the depth data acquisition and processing module is trained to obtain the result of semantic segmentation. Finally, the result of semantic segmentation is annotated in the three-dimensional reconstructed model by matching feature points. The segmentation results of each object in the scene are intuitively reflected.
[0053] Specifically, the semantic segmentation module improves the original PointNet++ network, such as Figure 3As shown in FIG, the PointNet++ network extracts local features using the PointNet network. The present invention improves PointNet++ by using the PointNet with the attention mechanism to improve the accuracy of semantic segmentation. Specifically, Figure 4 As shown in Figure 1, for each N×3 point cloud input, the PointNet network first aligns it spatially (rotates it to the front) through a T-Net, then maps it to a 64-dimensional high-dimensional space through an MLP, and then aligns it again in the 64-dimensional space, so that it can be passed as shown in Figure 1. Figure 5 The attention mechanism module shown in the figure is finally mapped to a 1024-dimensional space. The 64-dimensional local features and the 1024-dimensional global features are then combined to form a 1088-dimensional space, and another MLP change is performed. Finally, each point is classified into M categories, which is equivalent to outputting M scores to obtain the result of semantic segmentation. The attention mechanism module also uses average pooling and maximum pooling to aggregate the point cloud input features along the N dimension of the feature channel number, generating feature descriptors at different angles, and then uses MLP to refine the point cloud features. The aggregated features are trained using a double hidden layer MLP with shared parameters to generate attention weights.
[0054] The BIM fusion diagnosis module integrates the 3D reconstructed model with the results of semantic segmentation, converts it into a point cloud file, imports it into the Revit software for comparison with the BIM model, realizes the integration of 3D semantic information and BIM model, and provides technical solutions for construction progress management and completion comparison system.
[0055] Based on the same inventive concept, the present invention also proposes a scene 3D intelligent reconstruction method based on BIM and deep learning, comprising the following steps:
[0056] Step 1: collect depth data during the construction phase, preprocess the depth data, convert the preprocessed depth data into point cloud data, and perform HHA encoding on the depth data;
[0057] Step 2, completing the point cloud data, and performing 3D reconstruction based on the completed point cloud data to obtain a 3D model;
[0058] Step 3: Use the improved PointNet++ semantic segmentation network to perform semantic segmentation on the depth data after HHA encoding and annotate the semantic information;
[0059] Step 4: fuse the three-dimensional model obtained in step 2 with the semantic information obtained in step 3, and compare the fused model with the building BIM model to achieve a difference comparison between the construction stage and the design stage.
[0060] Based on the same inventive concept, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned scene three-dimensional intelligent reconstruction method are implemented.
[0061] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned scene three-dimensional intelligent reconstruction method are implemented.
[0062] The above embodiments are only for illustrating the technical idea of the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.
Claims
1. A scene 3D intelligent reconstruction system based on BIM and deep learning, characterized in that: The system includes a depth data acquisition and processing module, a three-dimensional reconstruction module, a semantic segmentation module and a BIM fusion diagnosis module; The depth data acquisition and processing module is used to collect depth data during the construction phase, pre-process the depth data, convert the pre-processed depth data into point cloud data, and perform HHA encoding processing on the depth data; The three-dimensional reconstruction module is used to complete the point cloud data and perform three-dimensional reconstruction based on the completed point cloud data to obtain a three-dimensional model; The semantic segmentation module is used to perform semantic segmentation on the depth data after HHA encoding processing using the improved PointNet++ semantic segmentation network, and to annotate the semantic information; The BIM fusion diagnosis module is used to fuse the three-dimensional model obtained by the three-dimensional reconstruction module with the semantic information obtained by the semantic segmentation module, and compare the fused model with the building BIM model to achieve a difference comparison between the building construction stage and the design stage; The improved PointNet++ semantic segmentation network includes a first sampling and grouping layer, a first improved PointNet network layer, a second sampling and grouping layer, a second improved PointNet network layer, a first reverse interpolation layer, a third improved PointNet network layer, a second reverse interpolation layer, a fourth improved PointNet network layer and an output layer connected in sequence, the output of the first sampling and grouping layer and the output of the second reverse interpolation layer serve as inputs to the fourth improved PointNet network layer, and the output of the second sampling and grouping layer and the output of the first reverse interpolation layer serve as inputs to the third improved PointNet network layer; The first improved PointNet network layer includes a spatial alignment layer, a first fully connected network, a feature alignment layer, an attention mechanism module, a second fully connected network, a maximum pooling layer, a third fully connected network, a fourth fully connected network and an output layer connected in sequence; for the N×3 point cloud input, the first improved PointNet network layer first aligns it spatially through the spatial alignment layer, then maps it to a 64-dimensional high-dimensional space through the first fully connected network, and then aligns it through the feature alignment layer to obtain a 64-dimensional local feature. The 64-dimensional local feature is obtained through the attention mechanism module, the second fully connected network and the maximum pooling layer. After the pooling layer, a 1024-dimensional global feature is obtained, and the 64-dimensional local feature and the 1024-dimensional global feature are combined to form a 1088-dimensional space, which is then passed through the third fully connected network and the fourth fully connected network in turn to obtain the output of the first improved PointNet network layer; where N is the number of deep data after HHA encoding, the dimension of the first fully connected network is 64×64, the dimension of the second fully connected network is 64×128×1024, the dimension of the third fully connected network is 512×256×128, and the dimension of the fourth fully connected network is 128×M, where M is the number of categories of point cloud data; The structures of the first improved PointNet network layer, the second improved PointNet network layer, the third improved PointNet network layer and the fourth improved PointNet network layer are all the same.
2. The scene 3D intelligent reconstruction system based on BIM and deep learning according to claim 1, characterized in that: The depth data acquisition and processing module includes an RGB-D sensor, a preprocessing unit, a point cloud conversion unit and an HHA encoding processing unit; The working process of the depth data acquisition and processing module is as follows: 1) Calibrate the internal and external parameters of the RGB-D sensor, and collect depth data during the construction phase through the calibrated RGB-D sensor; 2) Using the preprocessing unit to filter the collected depth data; 3) using a point cloud conversion unit to convert the filtered depth data into point cloud data, so as to prepare for the three-dimensional reconstruction module; 4) Perform HHA encoding processing on the depth data using the HHA encoding processing unit to prepare for the semantic segmentation module.
3. The scene 3D intelligent reconstruction system based on BIM and deep learning according to claim 1, characterized in that: The working process of the 3D reconstruction module is as follows: First, the point cloud data is completed through the point cloud completion neural network, and then the completed point cloud data is reconstructed in three dimensions through the BundleFunsion method to obtain a three-dimensional model.
4. The scene 3D intelligent reconstruction system based on BIM and deep learning according to claim 1, characterized in that: The working process of the BIM fusion diagnosis module is as follows: The 3D model obtained by the 3D reconstruction module is fused with the semantic information obtained by the semantic segmentation module and converted into a point cloud file. The point cloud file is imported into the Revit software for comparison with the BIM model in the design phase, thereby realizing the fusion of the 3D semantic information and the BIM model, and providing the difference comparison information between the construction phase and the design phase.
5. A method for scene 3D intelligent reconstruction based on the scene 3D intelligent reconstruction system based on BIM and deep learning according to any one of claims 1 to 4, characterized in that: The scene 3D intelligent reconstruction method comprises the following steps: Step 1, collecting depth data during the construction phase, preprocessing the depth data, converting the preprocessed depth data into point cloud data, and performing HHA encoding processing on the depth data; Step 2, completing the point cloud data, and performing 3D reconstruction based on the completed point cloud data to obtain a 3D model; Step 3: Use the improved PointNet++ semantic segmentation network to perform semantic segmentation on the depth data after HHA encoding and annotate the semantic information; Step 4: fuse the three-dimensional model obtained in step 2 with the semantic information obtained in step 3, and compare the fused model with the building BIM model to achieve a difference comparison between the construction stage and the design stage.
6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the scene three-dimensional intelligent reconstruction method as claimed in claim 5 are implemented.
7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the scene three-dimensional intelligent reconstruction method as claimed in claim 5 are implemented.
