Method and system for constructing live-action three-dimensional model of building

Abstract

The invention provides a live-action three-dimensional model construction method and system of a building, and relates to the technical field of building modeling. According to the method, feature extraction and semantic segmentation are carried out on the two types of point clouds through the deep learning model, so that point cloud registration is completed more quickly and accurately, and the error problem caused by manual marking is avoided; and meanwhile, intelligent registration and fusion are performed on two types of point clouds obtained by oblique photogrammetry and laser scanning by using a deep learning model, manual participation is not needed, and automatic, accurate and rapid fusion of the point clouds inside and outside the building is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of building modeling, and in particular to a method and system for constructing a real-scene three-dimensional model of a building. Background Art

[0002] With the development of computer vision technology, real-life 3D modeling has become a key technical means and plays a vital role in infrastructure planning, emergency rescue, cultural heritage protection, film and television production and other fields. As a comprehensive 3D information carrier, the real-life 3D model of a building presents the internal structure and external form of the building in an all-round digital way, providing intuitive and accurate 3D spatial data for various applications.

[0003] At present, the real-life 3D model of a building is usually constructed by combining the real-life 3D model of the building's interior with the real-life 3D model of the exterior. In the prior art, this fusion process generally relies on manually marked fusion points, which act as a bridge between the internal and external models, and through them, the internal and external models are connected and integrated.

[0004] However, manual labeling is prone to introduce subjective errors, which will accumulate during the model fusion process and thus affect the accuracy of the final model. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies in the prior art, the present invention provides a method and system for constructing a real-life three-dimensional model of a building, which solves the technical problem that the existing fusion method requires manual marking of fusion points and is prone to introducing errors.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] In a first aspect, the present invention provides a method for constructing a real-scene three-dimensional model of a building, comprising:

[0010] Acquire external real scene image data and internal real scene cloud data of the building at multiple angles; and process the external real scene image data at multiple angles to obtain external real scene cloud data;

[0011] Based on the deep learning model, semantic segmentation and feature extraction are performed on the external real-life point cloud data and the internal real-life point cloud data to obtain the internal structure, internal point cloud features, external structure and external point cloud features;

[0012] The internal structure, internal point cloud features, external structure and external point cloud features are learned using the graph pooling method. The distance between the external point cloud and the internal point cloud is measured using the triplet loss function. The geometric relationship and hierarchical relationship between the external point cloud and the internal point cloud are learned using the convolutional neural network.

[0013] According to the geometric relationship and hierarchical relationship between the external point cloud and the internal point cloud, the external real point cloud data and the internal real point cloud data are fused to obtain a three-dimensional model of the building, and the inner and outer surfaces of the three-dimensional model of the building are reconstructed to obtain a real-scene three-dimensional model of the building.

[0014] Preferably, the external real-scene image data is external real-scene image data of the building at multiple angles collected based on oblique photography technology.

[0015] Preferably, the processing of the external real scene image data at multiple angles to obtain the external real scene point cloud data includes: processing the external real scene image data at multiple angles by spatial triangulation to obtain the external real scene point cloud data.

[0016] Preferably, fusing the external real-time point cloud data and the internal real-time point cloud data to obtain a three-dimensional model of the building includes:

[0017] The 3D point cloud data of the exterior and interior of the building are fused using a three-stage point cloud fusion method, which includes:

[0018] The first stage: Use the curvature-bounding box method to thin out the two types of point clouds in turn, including:

[0019] Step 1: Select internal real-time point cloud data and divide the point cloud into several clusters through clustering algorithm;

[0020] Step 2: Calculate the average curvature C of each point in the cluster point cloud using formula (1): avg , and then calculate the average curvature K of the cluster point cloud through formula (2) avg , and determine K avg and C avg The size relationship between them is, if K avg >C avg , then execute step 3, otherwise, execute step 4;

[0021]

[0022] In formula (1), n ​​is the number of points in the cluster point cloud, V i represents the i-th point in the cluster point cloud, P j represents the centroid of the cluster point cloud, ||V i -P j || indicates V i With Pj The Euclidean distance of

[0023]

[0024] In formula (2), m represents the number of cluster point clouds. represents the average curvature of the i-th cluster point cloud, represents the average curvature of the j-th cluster point cloud;

[0025] Step 3: Use the curvature method to thin the cluster point cloud: First, set the average curvature threshold Then, the mean curvature C avg Less than the mean curvature threshold K thr The points are deleted from the cluster point cloud, and finally, the thinned cluster point cloud is obtained;

[0026] Step 4: Use the bounding box method to thin the cluster point cloud: First, create a bounding box for the cluster point cloud and set the thinning rate D thr = 20%, then multiply the side length of the bounding box by D thr , get the side length of each small cube, and split the bounding box into several small cubes. Secondly, calculate the distance between all points in the small cube and the center of gravity of the small cube. Finally, keep the point closest to the center of gravity and delete the rest of the points to get the thinned cluster point cloud.

[0027] Step 5: Thin out each cluster of point clouds in the internal real-world point cloud data, and merge the thinned point clouds to obtain a point cloud model P;

[0028] Step 6: Select external real-scene image data, and divide the point cloud into several clusters through a clustering algorithm. Repeat steps 2 to 5 to obtain a point cloud model Q.

[0029] The second stage: Use the rotation angle constrained iterative closest point method to align the thinned point cloud, including:

[0030] Step 1: Determine the upper and lower bounds of the three rotation angles using formula (3);

[0031]

[0032] In formula (3), θ x ,θ y ,θ z Respectively represent the rotation angles in the x, y, and z axes, θ xb ,θ yb ,θ zb Respectively represent the mean rotation angles in the three axis directions, Δθ x , Δθ y , Δθ zRespectively represent the rotation angle deviation in the three axis directions; so θ xb -Δθ x ,θ yb -Δθ y ,θ zb -Δθ z Respectively represent the lower bounds of the rotation angles in the three axis directions, θ xb +Δθ x ,θ yb +Δθ y , Δθ z Respectively represent the upper limits of the rotation angles in the three axis directions;

[0033] Step 2: Use the thinned point cloud model P as the reference model, the thinned point cloud model Q as the model to be registered, and set the initial rotation matrix R0=I and the initial translation matrix T0=0;

[0034] Step 3: Set the maximum number of iterations Iter = 100 and the rotation angle change threshold A thr =1°;

[0035] Step 4: Randomly select a point d from the point cloud model Q q , find the distance d in the point cloud P q The closest point to the Euclidean distance d p , add it to the matching point set C k middle;

[0036] Step 5: Update the rotation matrix R using equation (4) k , update the translation matrix T through equation (5) k , so that point d q After rotation and stability, R k ×d q +T k =d p , that is, the points in the point cloud model Q are completely coincident with the points in the point cloud model P after being rotated by a certain angle and translated by a certain distance;

[0037]

[0038] In formula (4), α, β, and γ represent the angles of rotation around the x, y, and z axes respectively;

[0039] T k =m p -VU T ×m q (5)

[0040] In formula (5), U and V are two third-order orthogonal matrices, U T represents the transposed matrix of matrix U;

[0041] Step 6: Update the rotation angle using equation (3) and determine the upper and lower bounds of the three rotation angles;

[0042] Step 7: If the number of iterations does not reach the preset upper limit Iter, or the rotation angle change is less than the preset threshold A thr , go to step 4, otherwise get the optimal rotation matrix R k and the optimal translation matrix T k , then go to step 8;

[0043] Step 8: Transform each point d in the point cloud model Q q , substitute into equation (6) to rotate and translate, and get point d q ', all points d q 'Combined into a new point cloud model Q';

[0044] d q ′ = R k ×d q +T k (6)

[0045] In formula (6), R k is the optimal rotation matrix, T k is the optimal translation matrix;

[0046] Step 9: Merge the point cloud model P and the point cloud model Q' to obtain a registered three-dimensional point cloud model combining the internal real scene and the external real scene;

[0047] The third stage: Use the point cloud filtering method to optimize the registered point cloud, including:

[0048] Step 1: Use the median filter method to remove outliers in the registered point cloud;

[0049] Step 2: Use morphological filtering to fill the holes in the registered point cloud;

[0050] Step 3: Use mean filtering to smooth the registered point cloud model to obtain the 3D model of the building.

[0051] Preferably, reconstructing the inner and outer surfaces of the three-dimensional model of the building includes:

[0052] Reconstruct the internal and external surfaces of the building's three-dimensional model using image texture mapping technology;

[0053] Or, reconstruct the interior and exterior surfaces of a three-dimensional model of a building using lighting and material simulation;

[0054] Alternatively, the interior and exterior surfaces of a 3D building model can be reconstructed using point cloud color mapping.

[0055] Preferably, the real-scene 3D model construction method further includes: optimizing the real-scene 3D model of the building, specifically including:

[0056] Use deep learning models to learn the noise distribution of point cloud data in the real-life 3D model of the building as well as the geometric structure and features of the point cloud;

[0057] According to the noise distribution, redundant noise points in the real-life 3D model of the building are removed;

[0058] According to the geometric structure and characteristics of the point cloud, the missing point cloud data in the real-life 3D model of the building is completed.

[0059] Preferably, the method for constructing a real-scene 3D model of a building further includes performing a lightweight display on the optimized real-scene 3D model of the building, including:

[0060] Use the CDN link to introduce the Three.js library into the HTML document, and use Three.js to set up the scene and camera to complete the initialization of the WebGL renderer;

[0061] Use the Loader function of Three.js to load the 3D model in the corresponding format, configure the camera position to display the model at the appropriate position on the web page, and use the OrbitControls controller to implement interactive operations of the 3D model;

[0062] Use the Three.js animate function to implement the rendering loop and dynamically update and display the scene;

[0063] The model loading speed and rendering performance are improved through texture compression methods. At the same time, the size of the renderer is responsively adjusted according to the parameters of different devices to ensure the display effect of the 3D model on different devices.

[0064] In a second aspect, the present invention provides a system for constructing a real-scene three-dimensional model of a building, comprising:

[0065] A data acquisition module is used to acquire external real-scene image data and internal real-scene cloud data of the building at multiple angles;

[0066] The data acquisition module is used to acquire the external real scene image data and internal real scene cloud data of the building at multiple angles; and is used to process the external real scene image data at multiple angles to obtain the external real scene cloud data.

[0067] The semantic segmentation and feature extraction module is used to perform semantic segmentation and feature extraction on the external real-life point cloud data and the internal real-life point cloud data based on the deep learning model to obtain the internal structure, internal point cloud features, external structure and external point cloud features;

[0068] A learning module is used to learn the internal structure, internal point cloud features, external structure and external point cloud features using a graph pooling method, measure the distance between the external point cloud and the internal point cloud using a triplet loss function, and learn the geometric relationship and hierarchical relationship between the external point cloud and the internal point cloud using a convolutional neural network;

[0069] The point cloud fusion and surface reconstruction module is used to fuse the external real-life point cloud data and the internal real-life point cloud data according to the geometric relationship and hierarchical relationship between the external point cloud and the internal point cloud to obtain a three-dimensional model of the building. The technology reconstructs the inner and outer surfaces of the three-dimensional model of the building to obtain a real-life three-dimensional model of the building.

[0070] In a third aspect, the present invention provides a storage medium storing a computer program for constructing a realistic three-dimensional model of a building, wherein the computer program enables a computer to execute the method for constructing a realistic three-dimensional model of a building as described above.

[0071] In a fourth aspect, the present invention provides an electronic device, comprising:

[0072] one or more processors;

[0073] Memory; and

[0074] One or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including methods for executing the method for constructing a realistic three-dimensional model of a building as described above.

[0075] (III) Beneficial effects

[0076] The present invention provides a method and system for constructing a real-scene three-dimensional model of a building. Compared with the prior art, the present invention has the following beneficial effects:

[0077] The present invention uses a deep learning model to perform feature extraction and semantic segmentation on the two types of point clouds, thereby completing point cloud registration more quickly and accurately, avoiding the error problem caused by manual labeling; at the same time, the deep learning model is used to perform intelligent registration and fusion of the two types of point clouds obtained by oblique photogrammetry and laser scanning, without the need for human intervention, to achieve automatic, accurate and rapid fusion of point clouds inside and outside the building. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0079] Figure 1 The present invention is a block diagram of a method for constructing a real-scene three-dimensional model of a building according to an embodiment of the present invention. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] The embodiments of the present application provide a method and system for constructing a real-life three-dimensional model of a building, which requires manual marking of fusion points and is prone to introducing technical errors. The method utilizes a deep learning model to perform feature extraction and semantic segmentation on point clouds, thereby completing point cloud registration more quickly and accurately and avoiding the error problem caused by manual marking.

[0082] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0083] The existing method for constructing real-life 3D models of buildings requires manual marking of fusion points, which is prone to the following defects: (1) The process of manually marking fusion points is cumbersome and time-consuming, and the operator needs to have high professional knowledge and experience; (2) Manual marking is prone to introduce subjective errors, which will accumulate during the model fusion process, thereby affecting the accuracy of the final model; (3) Due to the limitations of manual marking, the selection of fusion points may not be accurate or comprehensive enough, resulting in insufficient accuracy of the fused real-life 3D model.

[0084] Therefore, the existing methods for constructing real-life three-dimensional models of buildings still need to be improved in terms of accuracy and efficiency. In order to overcome these technical difficulties, it is necessary to study and develop a more efficient and accurate method for constructing real-life three-dimensional models to reduce manual intervention and improve the fusion accuracy and reliability of the model. It is based on this background that the embodiment of the present invention proposes an innovative method for constructing real-life three-dimensional models of buildings, aiming to improve the construction efficiency and accuracy of the model and meet the growing application needs. The embodiment of the present invention first uses a deep learning model to perform feature extraction and semantic segmentation on the point cloud, thereby completing point cloud registration more quickly and accurately, avoiding the error problem caused by manual labeling; then, a deep learning model is used to intelligently align and fuse the two types of point clouds obtained by oblique photogrammetry and laser scanning, without manual intervention, to achieve automatic, accurate and rapid fusion of the internal and external point clouds of the building; finally, the fused point cloud is optimized to improve the quality of the point cloud data.

[0085] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0086] The embodiment of the present invention provides a method for constructing a real-scene three-dimensional model of a building, such as Figure 1 As shown, including:

[0087] S1. Acquire external real scene image data and internal real scene cloud data of a building at multiple angles; and process the external real scene image data at multiple angles to obtain external real scene cloud data;

[0088] S2. Based on the deep learning model, semantic segmentation and feature extraction are performed on the external real-life point cloud data and the internal real-life point cloud data to obtain the internal structure, internal point cloud features, external structure and external point cloud features;

[0089] S3, using the graph pooling method to learn the internal structure, internal point cloud features, external structure and external point cloud features, using the triplet loss function to measure the distance between the external point cloud and the internal point cloud, and using the convolutional neural network to learn the geometric relationship and hierarchical relationship between the external point cloud and the internal point cloud;

[0090] S4. According to the geometric relationship and hierarchical relationship between the external point cloud and the internal point cloud, the external real point cloud data and the internal real point cloud data are merged to obtain a three-dimensional model of the building, and the inner and outer surfaces of the three-dimensional model of the building are reconstructed to obtain a real-scene three-dimensional model of the building.

[0091] The embodiment of the present invention uses a deep learning model to perform feature extraction and semantic segmentation on the two types of point clouds, thereby completing point cloud registration more quickly and accurately, avoiding the error problem caused by manual labeling; at the same time, the deep learning model is used to perform intelligent registration and fusion of the two types of point clouds obtained by oblique photogrammetry and laser scanning without human intervention, thereby realizing automatic, accurate and rapid fusion of point clouds inside and outside the building.

[0092] The following is a detailed description of each step:

[0093] In step S1, external real scene image data and internal real scene cloud data of the building at multiple angles are obtained. The specific implementation process is as follows:

[0094] External real-scene image data is external real-scene image data of buildings from multiple angles collected based on oblique photography technology. In the specific implementation process, the image of the building is captured from multiple angles (including vertical, front view, rear view, left view and right view) through sensors mounted on flying platforms such as drones. These images obtained from multiple perspectives not only provide rich visual information, but also can accurately locate geographic location data. The key parameters of this technology mainly involve image data of each perspective, exposure point coordinates, camera parameters and their relative position relationship.

[0095] The internal real-time point cloud data is collected by laser scanning technology. Laser scanning technology works based on the principle of laser ranging. By recording the laser emission angle and reflection reception time, the three-dimensional point cloud coordinates of the surface of the object being measured can be accurately calculated.

[0096] The computer obtains external real-scene image data and internal real-scene cloud data of the above-mentioned building from multiple angles (ie, imports the above-mentioned data into the computer software).

[0097] Import external real-world point cloud data into professional 3D modeling software, set relevant parameters, and process it through spatial triangulation (aerial triangulation solution) to quickly generate accurate 3D point cloud data, namely external real-world point cloud data.

[0098] It should be noted that, in the specific implementation process, other methods can also be used to process the external real scene image data from multiple angles to obtain the external real scene point cloud data. For example:

[0099] Direct measurement method: By using multiple known reference points (usually GPS-marked control points) for calibration, the reliance on aerial triangulation can be reduced, and point cloud stitching and alignment can be performed directly. This method requires a highly accurate control point layout, but has higher accuracy in small areas.

[0100] Multi-view geometry: point cloud data is derived based on the geometric relationship between multiple image perspectives. Through feature point matching technology, point clouds can be generated directly from image sequences without relying on aerial triangulation.

[0101] Structured light scanning method: Use structured light to project onto the surface of an object, and then use a camera to capture the reflected light to form a point cloud. This method does not rely on the principle of photogrammetry, but is based on optical projection and is suitable for high-precision modeling of small-scale buildings or indoor environments.

[0102] SLAM algorithm (Simultaneous Localization and Mapping): SLAM combines sensor data with point cloud data to build a three-dimensional map of the environment in real time. It is widely used in indoor spaces or areas with weak GPS signals. When the drone is equipped with LiDAR or a camera, SLAM can generate point clouds and perform modeling in real time.

[0103] In step S2, semantic segmentation and feature extraction are performed on the external real-world point cloud data and the internal real-world point cloud data based on the deep learning model to obtain the internal structure, internal point cloud features, external structure and external point cloud features. The specific implementation process is as follows:

[0104] The hypersurface Transformer network is used to perform semantic segmentation on the external and internal point cloud data of the building, and then the graph convolutional network is used to identify the different components of the building, including internal structures and external structures such as walls, roofs, doors and windows, and extract the features of the point clouds of different components, including internal point cloud features and external point cloud features such as geometric shapes, texture information and relative coordinates.

[0105] In step S3, the internal structure, internal point cloud features, external structure and external point cloud features are learned using the graph pooling method, the distance between the external point cloud and the internal point cloud is measured using the triplet loss function, and the geometric relationship and hierarchical relationship between the external point cloud and the internal point cloud are learned using the convolutional neural network. The specific implementation process is as follows:

[0106] The graph pooling method is used to learn the intrinsic structure and features of the two types of point clouds, the triplet loss function is used to measure the distance between point clouds, and the convolutional neural network is used to learn the geometric and hierarchical relationships between point clouds.

[0107] In step S4, according to the geometric relationship and hierarchical relationship between the external point cloud and the internal point cloud, the external real-life point cloud data and the internal real-life point cloud data are fused to obtain a three-dimensional model of the building, and the inner and outer surfaces of the three-dimensional model of the building are reconstructed to obtain a real-life three-dimensional model of the building. The specific implementation process is as follows:

[0108] A three-stage point cloud fusion method is used to fuse the external and internal 3D point cloud data of the building. In the first stage, the curvature-bounding box method is used to thin the point cloud. In the second stage, the rotation angle constrained iterative closest point method is used to align the thinned point cloud. In the third stage, the point cloud filtering method is used to optimize the aligned point cloud. After the above three stages, the fused 3D point cloud can be obtained. Finally, the fused 3D point cloud is converted into a format that can be processed by WebGL, such as OBJ or FBX.

[0109] The specific steps of each stage of the three-stage point cloud fusion method are as follows:

[0110] Stage 1: Use the curvature-bounding box method to thin out the two types of point clouds in turn

[0111] Step 1: Select the point cloud obtained by 3D laser scanning and divide the point cloud into several clusters through clustering algorithm;

[0112] Step 2: Calculate the average curvature C of each point in each cluster point cloud using formula (1): avg , and then calculate the average curvature K of the cluster point cloud through formula (2) avg ; and judge K avg and C avg If K avg >C avg , then execute step 3, otherwise, execute step 4;

[0113]

[0114] Where n is the number of points in the cluster point cloud, V i represents the i-th point in the cluster point cloud, P j represents the centroid of the cluster point cloud, ||V i -P j || indicates V i With P j The Euclidean distance of .

[0115]

[0116] In the formula, m represents the number of cluster point clouds, represents the average curvature of the i-th cluster point cloud, Represents the average curvature of the j-th cluster point cloud.

[0117] Step 3: Use the curvature method to thin the cluster point cloud: First, set the average curvature threshold Then, the mean curvature C avg Less than the mean curvature threshold K thr The points are deleted from the cluster point cloud, and finally, the thinned cluster point cloud is obtained;

[0118] Step 4: Use the bounding box method to thin the cluster point cloud: First, create a bounding box for the cluster point cloud and set the thinning rate D thr = 20%, then multiply the side length of the bounding box by D thr , get the side length of each small cube, and split the bounding box into several small cubes. Secondly, calculate the distance between all points in the small cube and the center of gravity of the small cube. Finally, keep the point closest to the center of gravity and delete the rest of the points to get the thinned cluster point cloud.

[0119] Step 5: Thinning each cluster of point clouds in the point cloud obtained by 3D laser scanning, and merging the thinned point clouds to obtain a point cloud model P;

[0120] Step 6: Select the point cloud obtained by oblique photography, and divide the point cloud into several clusters through clustering algorithm. Repeat steps 2 to 5 to obtain the point cloud model Q.

[0121] Phase 2: Use the rotation angle constrained iterative closest point method to align the thinned point cloud

[0122] Step 1: Calculate the bounded rotation angle using formula (3) and determine the upper and lower bounds of the three rotation angles;

[0123]

[0124] In the formula, θ x ,θ y ,θ z Respectively represent the rotation angles in the x, y, and z axes, θ xb ,θ yb ,θ zb Respectively represent the mean rotation angles in the three axis directions, Δθ x , Δθ y , Δθ z Respectively represent the rotation angle deviation in the three axis directions. So θ xb -Δθ x ,θ yb -Δθ y ,θ zb -Δθ z Respectively represent the lower bounds of the rotation angles in the three axis directions, θ xb +Δθ x ,θ yb +Δθ y , Δθ z Respectively represent the upper bounds of the rotation angles in the three axis directions.

[0125] Step 2: Use the thinned point cloud model P (point cloud obtained by 3D laser scanning) as the reference model, the thinned point cloud model Q (point cloud obtained by oblique photography) as the model to be registered, and set the initial rotation matrix R0 = I and the initial translation matrix T0 = 0;

[0126] Step 3: Set the maximum number of iterations Iter = 100 (i.e., a maximum of 100 cycles), and set the rotation angle change threshold A thr =1°;

[0127] Step 4: Randomly select a point d from the point cloud model Q q , find the distance d in the point cloud P q The closest point to the Euclidean distance d p , add it to the matching point set C k middle;

[0128] Step 5: Update the rotation matrix R using equation (4) k , update the translation matrix T through equation (5) k , so that point d q After rotation and stability, Rk ×d q +T k =d p , that is, the points in the point cloud model Q are completely coincident with the points in the point cloud model P after being rotated by a certain angle and translated by a certain distance;

[0129]

[0130] Where α, β, and γ represent the rotation angles around the x, y, and z axes, respectively.

[0131] T k =m p -VU T ×m q (5)

[0132] Where U and V are two third-order orthogonal matrices, U T Represents the transposed matrix of matrix U.

[0133] Step 6: Update the rotation angle using equation (3) and determine the upper and lower bounds of the three rotation angles;

[0134] Step 7: If the number of iterations does not reach the preset upper limit Iter, or the rotation angle change is less than the preset threshold A thr , go to step 4, otherwise get the optimal rotation matrix R k and the optimal translation matrix T k , then go to step 8.

[0135] Step 8: Transform each point d in the point cloud model Q q , substitute into equation (6) to rotate and translate, and get point d q ', all points d q 'Combined into a new point cloud model Q'.

[0136] d q ′ = R k ×d q +T k (6)

[0137] In formula (7), R k is the optimal rotation matrix, T k is the optimal translation matrix.

[0138] Step 9: Merge the point cloud model P and the point cloud model Q' to obtain a registered three-dimensional point cloud model combining the internal real scene and the external real scene.

[0139] The third stage: Use the point cloud filtering method to optimize the registered point cloud

[0140] Step 1: Use the median filter method to remove outliers in the registered point cloud;

[0141] Step 2: Use morphological filtering to fill the holes in the registered point cloud;

[0142] Step 3: Use mean filtering to smooth the registered point cloud model to obtain the 3D model of the building.

[0143] Image texture mapping technology is used to reconstruct the internal and external three-dimensional surface models of the building, thereby obtaining a highly realistic real-life three-dimensional model of the building.

[0144] It should be noted that in addition to image texture mapping technology, there are several other methods that can be used for surface reconstruction and visualization of building 3D models:

[0145] 1. Procedural texture generation: Textures are generated through algorithms rather than relying on actual images. Procedural textures can generate complex surface details, such as brick walls, concrete and other materials.

[0146] 2. Point cloud color mapping: If the point cloud data already contains color information (for example, obtained by a lidar with an RGB sensor or using an RGBD camera), this color information can be used directly to construct the texture.

[0147] 3. Image segmentation and material classification: Before texture mapping, image segmentation technology can be used to segment different parts of the building (such as walls, roofs, windows) and classify the materials. According to the material properties of different parts of the building (such as glass, concrete, wood), a specific material library is applied.

[0148] 4. Lighting and material simulation: Using physically based rendering technology, by modeling material parameters such as reflectivity, roughness, and normals, the lighting and reflection effects in the real world are simulated.

[0149] 5. Normal mapping and displacement mapping: These methods are used to enhance surface details. Normal maps or displacement maps are superimposed on the generated 3D model to create more detailed surface effects, such as bumps and cracks on the wall.

[0150] The above methods have their own advantages and disadvantages. Image texture mapping is usually used for realistic reproduction of scenes, while other methods focus on increasing material details or improving rendering effects. You can choose according to the needs of the project and available resources.

[0151] In the specific implementation process, the method for constructing a real-scene 3D model of a building further includes S5, optimizing the real-scene 3D model of the building. The specific implementation process is as follows:

[0152] The deep learning model is used to learn the noise distribution of point cloud data, thereby effectively removing redundant noise points. At the same time, the geometric structure and features of the point cloud are also learned, and the missing point cloud data is completed, thereby improving the integrity and clarity of the 3D model of the building.

[0153] In a specific implementation process, the method for constructing a real-scene three-dimensional model of a building further includes S6, performing lightweight display on the optimized real-scene three-dimensional model of the building.

[0154] The lightweight display technology of building 3D models is based on WebGL (Web Graphics Library) technology. WebGL is a JavaScript API for interactively presenting 3D models in compatible web browsers. Users can operate the 3D model of the building without installing additional plug-ins. Through WebGL-based visualization tools, the 3D model of the building can be converted into a web page format and displayed in a lightweight manner in a web browser. The model mesh and texture are optimized to reduce the data transmission load, thereby speeding up the rendering of the model and enabling quick viewing and operation on the browser side. The specific steps are as follows:

[0155] S601: Set up the display environment of the 3D model. Use the CDN link to introduce the Three.js library into the HTML document, and use Three.js to set the scene and camera to complete the initialization of the WebGL renderer;

[0156] S602: Load and display the 3D model. Use the Loader function of Three.js to load the 3D model in the corresponding format, configure the camera position to display the model at the appropriate position on the web page, and use the OrbitControls controller to implement interactive operations such as rotation, scaling, and translation of the 3D model.

[0157] S603: Dynamic rendering and device adaptation. Use the animate function of Three.js to implement the rendering loop, so as to dynamically update and display the scene. Finally, use methods such as texture compression to improve the loading speed and rendering performance of the model. At the same time, adjust the size of the renderer responsively according to the parameters of different devices to ensure the display effect of the 3D model on different devices.

[0158] An embodiment of the present invention provides a system for constructing a real-scene three-dimensional model of a building, comprising:

[0159] The data acquisition module is used to acquire the external real scene image data and internal real scene cloud data of the building at multiple angles; and is used to process the external real scene image data at multiple angles to obtain the external real scene cloud data.

[0160] The semantic segmentation and feature extraction module is used to perform semantic segmentation and feature extraction on the external real-life point cloud data and the internal real-life point cloud data based on the deep learning model to obtain the internal structure, internal point cloud features, external structure and external point cloud features;

[0161] A learning module is used to learn the internal structure, internal point cloud features, external structure and external point cloud features using a graph pooling method, measure the distance between the external point cloud and the internal point cloud using a triplet loss function, and learn the geometric relationship and hierarchical relationship between the external point cloud and the internal point cloud using a convolutional neural network;

[0162] The point cloud fusion and surface reconstruction module is used to fuse the external real-life point cloud data and the internal real-life point cloud data according to the geometric relationship and hierarchical relationship between the external point cloud and the internal point cloud to obtain a three-dimensional model of the building, and to reconstruct the inner and outer surfaces of the three-dimensional model of the building using image texture mapping technology to obtain a real-life three-dimensional model of the building.

[0163] It can be understood that the real-life three-dimensional model construction system for a building provided in an embodiment of the present invention corresponds to the real-life three-dimensional model construction method for the above-mentioned building, and the explanations, examples, beneficial effects, etc. of the relevant contents can refer to the corresponding contents in the real-life three-dimensional model construction method for a building, and will not be repeated here.

[0164] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program for constructing a real-life three-dimensional model of a building, wherein the computer program enables a computer to execute the method for constructing a real-life three-dimensional model of a building as described above.

[0165] An embodiment of the present invention also provides an electronic device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the real-life three-dimensional model construction method of the building as described above.

[0166] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0167] 1. The embodiment of the present invention uses a deep learning model to perform feature extraction and semantic segmentation on two types of point clouds, thereby completing point cloud registration more quickly and accurately, avoiding the error problem caused by manual labeling; at the same time, the deep learning model is used to perform intelligent registration and fusion of the two types of point clouds obtained by oblique photogrammetry and laser scanning, without the need for human intervention, to achieve automatic, accurate and rapid fusion of point clouds inside and outside the building.

[0168] 2. The embodiment of the present invention performs denoising, completion and optimization on the real-scene three-dimensional model to improve the integrity and clarity of the three-dimensional model of the building.

[0169] 3. The embodiment of the present invention performs lightweight display of the real-scene three-dimensional model, optimizes the model mesh and texture to reduce the data transmission load, thereby speeding up the rendering speed of the model and realizing quick viewing and operation on the browser side.

[0170] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0171] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a real-scene three-dimensional model of a building, characterized in that: include: Obtain external real-life image data and internal real-life point cloud data of the building from multiple angles; And process the external real scene image data from multiple angles to obtain external real scene point cloud data; Based on the deep learning model, semantic segmentation and feature extraction are performed on the external real-life point cloud data and the internal real-life point cloud data to obtain the internal structure, internal point cloud features, external structure and external point cloud features; The internal structure, internal point cloud features, external structure and external point cloud features are learned using the graph pooling method. The distance between the external point cloud and the internal point cloud is measured using the triplet loss function. The geometric relationship and hierarchical relationship between the external point cloud and the internal point cloud are learned using the convolutional neural network. According to the geometric relationship and hierarchical relationship between the external point cloud and the internal point cloud, the external real point cloud data and the internal real point cloud data are fused to obtain a three-dimensional model of the building, and the inner and outer surfaces of the three-dimensional model of the building are reconstructed to obtain a real-scene three-dimensional model of the building.

2. The method for constructing a real-scene three-dimensional model of a building according to claim 1, characterized in that: The external real scene image data is external real scene image data of multiple angles of the building collected based on the oblique photography technology.

3. The method for constructing a real-scene three-dimensional model of a building according to claim 1, characterized in that: The processing of the external real scene image data at multiple angles to obtain the external real scene point cloud data includes: processing the external real scene image data at multiple angles by using a spatial triangulation method to obtain the external real scene point cloud data.

4. The method for constructing a real-scene three-dimensional model of a building according to claim 1, characterized in that: The external real-time point cloud data and the internal real-time point cloud data are merged to obtain a three-dimensional model of the building, including: The 3D point cloud data of the exterior and interior of the building are fused using a three-stage point cloud fusion method, which includes: The first stage: Use the curvature-bounding box method to thin out the two types of point clouds in turn, including: Step 1: Select internal real-time point cloud data and divide the point cloud into several clusters through clustering algorithm; Step 2: Calculate the average curvature C of each point in the cluster point cloud using formula (1): avg , and then calculate the average curvature K of the cluster point cloud through formula (2) avg , and determine K avg and C avg The size relationship between them is, if K avg >C avg , then execute step 3, otherwise, execute step 4; In formula (1), n ​​is the number of points in the cluster point cloud, V i represents the i-th point in the cluster point cloud, P j represents the centroid of the cluster point cloud, ||V i -P j || indicates V i With P j The Euclidean distance of In formula (2), m represents the number of cluster point clouds. represents the average curvature of the i-th cluster point cloud, represents the average curvature of the j-th cluster point cloud; Step 3: Use the curvature method to thin the cluster point cloud: First, set the average curvature threshold Then, the mean curvature C avg Less than the mean curvature threshold K thr The points are deleted from the cluster point cloud, and finally, the thinned cluster point cloud is obtained; Step 4: Use the bounding box method to thin the cluster point cloud: First, create a bounding box for the cluster point cloud and set the thinning rate D thr = 20%, then multiply the side length of the bounding box by D thr , get the side length of each small cube, and split the bounding box into several small cubes. Secondly, calculate the distance between all points in the small cube and the center of gravity of the small cube. Finally, keep the point closest to the center of gravity and delete the rest of the points to get the thinned cluster point cloud. Step 5: Thin out each cluster of point clouds in the internal real-world point cloud data, and merge the thinned point clouds to obtain a point cloud model P; Step 6: Select external real-scene image data, and divide the point cloud into several clusters through a clustering algorithm. Repeat steps 2 to 5 to obtain a point cloud model Q. The second stage: Use the rotation angle constrained iterative closest point method to align the thinned point cloud, including: Step 1: Determine the upper and lower bounds of the three rotation angles using formula (3); In formula (3), θ x ,θ y ,θ z Respectively represent the rotation angles in the x, y, and z axes, θ xb ,θ yb ,θ zb Respectively represent the mean rotation angles in the three axis directions, Δθ x , Δθ y , Δθ z Respectively represent the rotation angle deviation in the three axis directions; so θ xb -Δθ x ,θ yb -Δθ y ,θ zb -Δθ z Respectively represent the lower bounds of the rotation angles in the three axis directions, θ xb +Δθ x ,θ yb +Δθ y , Δθ z Respectively represent the upper limits of the rotation angles in the three axis directions; Step 2: Use the thinned point cloud model P as the reference model, the thinned point cloud model Q as the model to be registered, and set the initial rotation matrix R0=I and the initial translation matrix T0=0; Step 3: Set the maximum number of iterations Iter = 100 and the rotation angle change threshold A thr =1°; Step 4: Randomly select a point d from the point cloud model Q q , find the distance d in the point cloud P q The closest point to the Euclidean distance d p , add it to the matching point set C k middle; Step 5: Update the rotation matrix R using equation (4) k , update the translation matrix T through equation (5) k , so that point d q After rotation and stability, R k ×d q +T k =d p , that is, the points in the point cloud model Q completely coincide with the points in the point cloud model P after being rotated by a certain angle and translated by a certain distance; In formula (4), α, β, and γ represent the angles of rotation around the x, y, and z axes respectively; T k =m p -VU T ×m q (5) In formula (5), U and V are two third-order orthogonal matrices, U T represents the transposed matrix of matrix U; Step 6: Update the rotation angle using equation (3) and determine the upper and lower bounds of the three rotation angles; Step 7: If the number of iterations does not reach the preset upper limit Iter, or the rotation angle change is less than the preset threshold A thr , go to step 4, otherwise get the optimal rotation matrix R k and the optimal translation matrix T k , then go to step 8; Step 8: Each point d in the point cloud model Q q , substitute into equation (6) to rotate and translate, and get point d q ', all points d q 'Combined into a new point cloud model Q'; d q ′ = R k ×d q +T k (6) In formula (6), R k is the optimal rotation matrix, T k is the optimal translation matrix; Step 9: Merge the point cloud model P and the point cloud model Q' to obtain a registered three-dimensional point cloud model combining the internal real scene and the external real scene; The third stage: Use the point cloud filtering method to optimize the registered point cloud, including: Step 1: Use the median filter method to remove outliers in the registered point cloud; Step 2: Use morphological filtering to fill the holes in the registered point cloud; Step 3: Use mean filtering to smooth the registered point cloud model to obtain the 3D model of the building.

5. The method for constructing a real-scene three-dimensional model of a building according to claim 1, characterized in that: The reconstructing of the inner and outer surfaces of the three-dimensional model of the building includes: Reconstruct the internal and external surfaces of the building's three-dimensional model using image texture mapping technology; Or, reconstruct the interior and exterior surfaces of a three-dimensional model of a building using lighting and material simulation; Alternatively, the interior and exterior surfaces of a 3D building model can be reconstructed using point cloud color mapping.

6. The method for constructing a real-scene three-dimensional model of a building according to any one of claims 1 to 5, characterized in that: The real-scene 3D model construction method further includes: optimizing the real-scene 3D model of the building, specifically including: Use deep learning models to learn the noise distribution of point cloud data in the real-life 3D model of the building as well as the geometric structure and features of the point cloud; According to the noise distribution, redundant noise points in the real-life 3D model of the building are removed; According to the geometric structure and characteristics of the point cloud, the missing point cloud data in the real-life 3D model of the building is completed.

7. The method for constructing a real-scene three-dimensional model of a building according to claim 6, characterized in that: The method for constructing a real-scene 3D model of a building also includes lightweight display of the optimized real-scene 3D model of the building, including: Use the CDN link to introduce the Three.js library into the HTML document, and use Three.js to set up the scene and camera to complete the initialization of the WebGL renderer; Use the Loader function of Three.js to load the 3D model in the corresponding format, configure the camera position to display the model at the appropriate position on the web page, and use the OrbitControls controller to implement interactive operations of the 3D model; Use the Three.js animate function to implement the rendering loop and dynamically update and display the scene; The model loading speed and rendering performance are improved through texture compression methods. At the same time, the size of the renderer is responsively adjusted according to the parameters of different devices to ensure the display effect of the 3D model on different devices.

8. A system for constructing a real-scene three-dimensional model of a building, characterized in that: include: A data acquisition module is used to acquire external real-scene image data and internal real-scene cloud data of the building at multiple angles; The data acquisition module is used to acquire the external real scene image data and internal real scene cloud data of the building at multiple angles; and is used to process the external real scene image data at multiple angles to obtain the external real scene cloud data. The semantic segmentation and feature extraction module is used to perform semantic segmentation and feature extraction on the external real-life point cloud data and the internal real-life point cloud data based on the deep learning model to obtain the internal structure, internal point cloud features, external structure and external point cloud features; A learning module is used to learn the internal structure, internal point cloud features, external structure and external point cloud features using a graph pooling method, measure the distance between the external point cloud and the internal point cloud using a triplet loss function, and learn the geometric relationship and hierarchical relationship between the external point cloud and the internal point cloud using a convolutional neural network; The point cloud fusion and surface reconstruction module is used to fuse the external real-life point cloud data and the internal real-life point cloud data according to the geometric relationship and hierarchical relationship between the external point cloud and the internal point cloud to obtain a three-dimensional model of the building. The technology reconstructs the inner and outer surfaces of the three-dimensional model of the building to obtain a real-life three-dimensional model of the building.

9. A storage medium, characterized in that: The computer program for constructing a real-scene three-dimensional model of a building is stored therein, wherein the computer program enables a computer to execute the method for constructing a real-scene three-dimensional model of a building as claimed in any one of claims 1 to 7.

10. An electronic device, characterized in that: include: one or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the method for constructing a real-life three-dimensional model of a building as described in any one of claims 1 to 7.

Images