A building fine modeling method and device based on a knowledge graph

Abstract

The application discloses a kind of building fine modeling method and device fusing knowledge graph, belong to landscape modeling and surveying technology field.The method is first obtained target building image and generates point cloud, obtains the mapping relationship between point cloud and image;According to building modeling content determines building modeling knowledge graph architecture, builds knowledge graph by knowledge representation, knowledge storage and knowledge fusion;Under the guidance of knowledge graph, utilize point cloud to build building main structure surface model;Obtain the positioning information of opening component, construct opening component model and modify main structure surface model;Determine the non-main component information outside main structure and opening component, and construct non-main component model by the guidance of knowledge graph;Combine all models, obtain target building model.The application avoids the problems such as model complexity and texture distortion, component monomer reconstruction difficulty of traditional method, and realizes hierarchical, component, fine building modeling.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of landscape modeling and mapping, and relates to a building fine modeling method and device fusing a knowledge graph. BACKGROUND

[0002] Real scene three-dimensional reconstruction involves reproducing the scene of the real world by constructing a high-precision three-dimensional model, and is a key research content in the fields of photogrammetry, remote sensing and computer vision. With the progress of oblique photography technology and the rise of smart city and digital city research, three-dimensional city digital reconstruction based on high-resolution images and point cloud data generated therefrom has become a current research hotspot. Through the generated digital model, combined with virtual reality and augmented reality technology, the surface features and details of the object can be exhibited, and the user can interact with the virtual scene in real time. This technology can be widely applied to automatic driving, personnel training, combat simulation and emergency rescue fields.

[0003] Buildings, as a key component of urban scenes, are always the focus of three-dimensional reconstruction due to their diversity and complexity. Traditional three-dimensional reconstruction of buildings usually has the following shortcomings: first, the entire scene is usually connected as a whole, and the monomerization degree of the building and its components is low, and the grid structure is complex, which leads to difficult and easy-to-distort texture mapping operation. Secondly, these models usually lack semantic information, and the components are mixed together, which is not conducive to splitting, separate data processing and analysis. Finally, there is more manual processing, and the degree of automation is low, and the programmable and parameterized processing is insufficient, and the scalability is poor. SUMMARY

[0004] The application aims at the generation of target building component-level fine three-dimensional model, and provides a building fine modeling method and device fusing a knowledge graph. Based on the obtained target building model, data analysis of the structure and energy of the target building can be further performed, and the application can also serve battlefield simulation, three-dimensional navigation and other applications.

[0005] In a first aspect, the application provides a building fine modeling method fusing a knowledge graph, comprising the following steps:

[0006] S1: obtaining a target modeling building image and generating a point cloud of the target modeling building, and obtaining a mapping relationship between the point cloud and the target modeling building image;

[0007] S2: determining a building modeling knowledge graph architecture according to building modeling content, and building a building modeling knowledge graph through building modeling knowledge representation, building modeling knowledge storage and knowledge fusion;

[0008] S3: Under the guidance of the building modeling knowledge graph, the point cloud of the building is used to construct the main structure surface model of the target modeling building;

[0009] S4: Obtain the positioning information of the opening component in the main structure surface model of the building, and construct the opening component model corresponding to the opening component in the main structure based on the positioning information, and modify the main structure surface model combined with the opening component model;

[0010] S5: Determine the non-main component information outside the main structure and opening component of the target modeling building, and based on the non-main component information, guide the construction of the non-main component model through the building modeling knowledge graph;

[0011] S6: Combine the main structure surface model, the opening component model and the non-main component model to obtain the target modeling building model.

[0012] Further, the algorithm process for obtaining the mapping relationship between the point cloud and the target modeling building image in step S1 is as follows:

[0013] S1-1: Use a shooting device to shoot multiple view images of the target modeling building;

[0014] S1-2: Use a motion inference structure technology (SfM) to generate a sparse point cloud and related files of the target modeling building;

[0015] S1-4: Obtain a dense point cloud of the target modeling building through multi-view stereo technology (MVS) reconstruction;

[0016] S1-5: Perform cropping and denoising processing of the sparse and dense point clouds of the target modeling building.

[0017] At this point, the sparse point cloud and the dense point cloud of the target modeling building have been obtained, and the mapping relationship between the sparse point cloud and the image can be obtained based on the SfM result file.

[0018] Further, the algorithm process for determining the building modeling knowledge graph architecture according to the building modeling content in step S2 is as follows:

[0019] S2-1: Use a crawler technology to crawl building modeling content from multiple building modeling web pages;

[0020] S2-2: Determine the building modeling knowledge graph architecture based on the building modeling content;

[0021] S2-3: Use triplets and attribute graphs to represent building modeling knowledge;

[0022] S2-4: Building modeling knowledge storage using graph database;

[0023] S2-5: Removing repeated content in building modeling through knowledge fusion to build building modeling knowledge graph;

[0024] At this point, the building modeling knowledge graph has been built, and the building modeling knowledge graph is used to guide the modeling process of each level of building components in the subsequent steps.

[0025] Further, step S3 uses the point cloud of the building to construct the main structure surface model of the target modeling building under the guidance of the building modeling knowledge graph, and the algorithm process is as follows:

[0026] S3-1: Using point cloud segmentation method, fitting the dense point cloud in the point cloud of the target modeling building into multiple point cloud planes;

[0027] S3-2: Calculate the shortest Euclidean distance of each pair of point cloud planes in the multiple point cloud planes, compare the shortest Euclidean distance with the preset threshold, and the two point cloud planes corresponding to the shortest Euclidean distance less than the preset threshold are adjacent point cloud planes, and the determined each pair of adjacent point cloud planes is connected to obtain the skeleton line of the modeling building;

[0028] S3-3: Intersecting the skeleton lines belonging to each point cloud plane with each other, taking the line segment obtained by the intersection of the skeleton lines as the unit, and determining the closed loop line around the current point cloud plane;

[0029] S3-4: Triangulation of the closed loop line to obtain the plane model of the current point cloud plane;

[0030] S3-5: Combining all plane models to obtain the main structure surface model of the target modeling building.

[0031] Further, in step S4, the positioning information of the opening components in the main structure surface model is obtained, and the opening component model corresponding to the opening components in the main structure is constructed based on the positioning information, and the algorithm process of combining the opening component model to modify the main structure surface model is as follows:

[0032] S4-1: Determine the texture image set covering each building plane of the target modeling building;

[0033] S4-2: Determine the target texture image from each building plane texture image set;

[0034] S4-3: applying a deep learning algorithm to the target texture image to identify opening component information corresponding to an opening component in the building plane, the opening component at least including a door of the building and a window of the building, the opening component information being determined by building modeling content in the building modeling knowledge graph;

[0035] S4-4: classifying and reconstructing different components in the building main structure in combination with the building modeling knowledge graph, including determining a modeling method of the opening component and a modeling method of removing the main structure surface model of the opening component, and based on the opening component information and the main structure surface model, constructing an opening component model and a modified main structure surface model through the modeling method of the opening component and the modeling method of removing the main structure surface model of the opening component;

[0036] S4-5: using the target texture image to perform model texture mapping on the opening component model and the modified main structure surface model.

[0037] Further, in step S5, the information of the target modeling building main structure and the opening component and the non-main component information outside the opening component are determined, and based on the non-main component information, the algorithm process for guiding the construction of the non-main component model through the building modeling knowledge graph is as follows:

[0038] S5-1: performing data processing and component segmentation processing on the dense point cloud in the point cloud of the target modeling building to obtain multiple cluster non-main component point clouds, the non-main component being a structure independent of the main structure and the opening component;

[0039] S5-2: based on each cluster of segmented non-main component point clouds, performing semantic recognition to obtain the component semantics corresponding to each cluster of component point clouds;

[0040] S5-3: based on the building modeling knowledge graph, determining a template model and a modeling method corresponding to each component semantic to model the components and obtain multiple non-main component structure models;

[0041] S5-4: performing building component model texture mapping on the multiple non-main component structure models to obtain a complete non-main component model.

[0042] Further, in step S6, the algorithm process for combining the main structure surface model, the opening component model and the non-main component model to obtain the target modeling building model is as follows:

[0043] S6-1: combining all building component models to obtain a building model according to the positions of the original point clouds corresponding to all main structure surface models, opening component models and non-main component models.

[0044] Further, after S6, the method further includes:

[0045] The component instance information corresponding to the target modeling building is stored in the building modeling knowledge graph.

[0046] In a second aspect, the application provides a building fine modeling device fusing a knowledge graph, which comprises the following parts:

[0047] An acquisition unit is configured to acquire a target modeling building image and generate a point cloud of the target modeling building, and acquire a mapping relationship between the point cloud and the target modeling building image;

[0048] A building unit is configured to build a building modeling knowledge graph according to a building modeling knowledge graph architecture, through building modeling knowledge representation, building modeling knowledge storage and knowledge fusion.

[0049] A first construction unit is configured to construct a main structure surface model of the target modeling building by using the point cloud of the building under the guidance of the building modeling knowledge graph.

[0050] A second construction unit is configured to acquire positioning information of an opening component in the main structure surface model under the guidance of the building modeling knowledge graph, and construct an opening component model corresponding to the opening component in the main structure based on the positioning information.

[0051] A third construction unit is configured to determine non-main component information of the target modeling building main structure and opening components, and construct a non-main component model based on the non-main component information under the guidance of the building modeling knowledge graph.

[0052] A combination unit is configured to combine the main structure surface model, the opening component model and other component models to obtain a target modeling building model.

[0053] In a third aspect, the application provides an electronic device, which comprises a memory, a processor and an executable program stored in the memory and executable by the processor, wherein the processor can implement the building fine modeling method fusing a knowledge graph when executing the program.

[0054] In a fourth aspect, the application provides a computer readable storage medium, wherein computer executable instructions are used to execute the building fine modeling method fusing a knowledge graph when executed by a computer processor.

[0055] The application provides a building fine modeling method and device fusing a knowledge graph. The three-dimensional building model can be used to perfect the whole scene model and analyze the target building. Compared with the traditional method of generating a three-dimensional building model, the application has the following advantages:

[0056] (1) The model is simple and realistic. The complex grid model is replaced by a simple polygon grid model, which can save model storage space and reduce texture distortion, and is suitable for large-scale scene building modeling;

[0057] (2) The model has stronger hierarchy, semantics and monomer. By analyzing the semantics of building components, different modeling methods are set, the modeling is more targeted, and the model is more realistic. At the same time, different information of different semantic components is stored based on the knowledge graph, the modeling result provides more information, and the subsequent application is facilitated;

[0058] (3) The knowledge graph is used to guide modeling, which is expandable. When a special type of building needs to be modeled, such as Gothic buildings, Chinese ancient buildings, etc., the template model of the knowledge graph can be expanded according to the method of the application, and the corresponding modeling method is given. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below, and the features and advantages of the present application will be more clearly understood by referring to the drawings. The drawings are schematic and should not be understood as any limitation on the present application. For those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings. Among them:

[0060] Figure 1 The building fine modeling method flowchart of the present application is provided;

[0061] Figure 2 The multi-view schematic diagram of the target modeling building part is provided;

[0062] Figure 3 The sparse point cloud schematic diagram of the target modeling building is provided;

[0063] Figure 4 The dense point cloud schematic diagram of the target modeling building is provided;

[0064] Figure 5 The related file schematic diagram of the image generation point cloud is provided;

[0065] Figure 6 The final content schematic diagram of the building modeling knowledge graph is provided;

[0066] Figure 7 The part of the building modeling knowledge graph built by Neo4j is provided;

[0067] Figure 8 The surface model schematic diagram of the main structure of the target modeling building is provided;

[0068] Figure 9 A schematic diagram of a modified target modeling building main body structure surface model;

[0069] Figure 10 A schematic diagram of a target modeling building opening component model;

[0070] Figure 11 A schematic diagram of a target modeling building non-main body component model;

[0071] Figure 12 A schematic diagram of a target modeling building model;

[0072] Figure 13 A schematic diagram of a building fine modeling device provided by an embodiment of the present application is provided;

[0073] Figure 14 A schematic diagram of an electronic device provided by an embodiment of the present application is provided. DETAILED DESCRIPTION

[0074] In the following description, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, persons skilled in the art will understand that the present application can be implemented in other embodiments that do not have these specific details. In other instances, well-known systems, devices, circuits, and methods have not been described in detail so as not to unnecessarily obscure the description of the present application.

[0075] At the present stage, with the continuous development of deep learning, computer vision, knowledge graph and other technologies, various applications have put forward higher requirements for building three-dimensional reconstruction technology, including: constructing a simplified grid model, so that the model has high flatness and is convenient for separate texture mapping operation; being able to identify and extract fine components of the building, such as doors, windows, walls, balconies, etc., and independently model these components; reducing manual intervention, realizing automatic and hierarchical three-dimensional reconstruction of the building, and having certain expandability.

[0076] In view of the hierarchical and parameterized characteristics of buildings as typical artificial structures, in combination with the deficiencies of traditional three-dimensional reconstruction models and new requirements of current technologies, the application provides an innovative technical means for three-dimensional reconstruction of buildings. On the premise of obtaining multi-view images of the building and point clouds generated therefrom, the application guides hierarchical, componentized, programmed and parameterized modeling of the building through a knowledge graph, and can provide various information such as geometry, topology, semantics and texture of the building, thereby avoiding problems such as model complexity and texture distortion, difficulty in component segmentation and reconstruction, and insufficient information display in traditional methods, and not only improving the accuracy of the model and the intelligent level of modeling, but also giving a refined building model result, and providing technical support for subsequent other applications based on the building, such as structure and energy consumption analysis.

[0077] In order to make the technical scheme of the application clearer, the application will be further described below with reference to the drawings.

[0078] A flow chart of a building refined modeling method fusing a knowledge graph provided by the application is shown in Figure 1 .

[0079] Specifically, a building refined modeling method fusing a knowledge graph specifically includes the following steps:

[0080] S1: Obtain target modeling building images and generate point clouds of the target modeling building, and obtain a mapping relationship between the point clouds and the target modeling building images;

[0081] S2: Determine a building modeling knowledge graph architecture according to building modeling content, and build a building modeling knowledge graph through building modeling knowledge representation, building modeling knowledge storage and knowledge fusion;

[0082] S3: Under the guidance of the building modeling knowledge graph, utilize the point clouds of the building to construct a main structure surface model of the target modeling building;

[0083] S4: Obtain positioning information of an opening component in the main structure surface model, and based on the positioning information, construct an opening component model corresponding to the opening component in the main structure, and modify the main structure surface model in combination with the opening component model;

[0084] S5: Determine non-main component information other than the main structure and the opening component of the target modeling building, and based on the non-main component information, guide the construction of a non-main component model through the building modeling knowledge graph;

[0085] S6: Combine the main structure surface model, the opening component model and the non-main component model to obtain a target modeling building model.

[0086] Specifically, step S1 acquires a target modeling building image and generates a point cloud of the target modeling building, and acquires a mapping relationship between the point cloud and the target modeling building image. The specific algorithm process is as follows:

[0087] S1-1: Use a shooting device to shoot the target modeling building from multiple angles, which can cover the entire target modeling building multiple times. It is required that the multi-view of the target modeling building is relatively dense, specifically, the number of images involved in each part is not less than two;

[0088] S1-2: Use the structure from motion (SfM) technology to process multi-view images to estimate the three-dimensional structure of the target modeling building area from two-dimensional images; the SfM technology obtains a sparse point cloud of the area through feature matching, camera pose estimation, triangulation, optimization and verification, etc. and generates related files containing camera pose, sparse point position in image and other information; the SfM technology can be realized by VisualSFM, Colmap, OpenSFM and other open source algorithms;

[0089] S1-3: Based on the camera pose and sparse point cloud obtained by SfM, use multi-view stereo technology (MVS) to reconstruct the dense point cloud of the target modeling building area; the MVS technology can be realized by PMVS, openMVS, Colmap and other open source algorithms; on the basis of having relatively dense multi-view, the present application requires that the obtained building point cloud has no large-scale vacancy;

[0090] S1-4: Use CloudCompare and other software to crop the target modeling building part from the sparse point cloud and the dense point cloud, and use morphological, statistical and other filtering methods for point cloud denoising;

[0091] The final target modeling building part multi-view is shown in Figure 2 , the sparse point cloud is shown in Figure 3 , the dense point cloud is shown in Figure 4 , and the related file is shown in Figure 5 , thus, the multi-view image, point cloud and related file of the target building have been obtained.

[0092] To guide hierarchical and fine-grained building modeling based on a knowledge graph, first, a building modeling knowledge graph needs to be built according to building hierarchical modeling steps, building modeling rule knowledge, and expected data model results and other building modeling contents. Step S2 determines the building modeling knowledge graph architecture according to the building modeling content, and builds the building modeling knowledge graph through building modeling knowledge representation, building modeling knowledge storage and knowledge fusion. The specific algorithm process is as follows:

[0093] S2-1: Using a crawler tool, building modeling related content is crawled from the web page recording the building modeling related standard by writing a script to simulate the way of human browsing the web page; the crawler technology can be realized by using open source crawler tools such as spider, selenium, etc.; the web page recording the building modeling related content includes the web page related to IFC, CityGML, BIM, etc.; the content crawled includes text, table and other forms;

[0094] S2-2: The ontology construction and knowledge extraction are carried out by artificial integration to determine the building modeling knowledge graph architecture. First, according to the building modeling target model and data, the relevant content suitable for building modeling is sorted out from the content obtained by crawling, and the irrelevant or modeling unattended part is removed; second, according to the sorted out building modeling related content, the specific instance information is set to guide the data generation and data storage in the modeling process, which is convenient for serving subsequent other applications;

[0095] Among them, the ontology is the label of the knowledge graph, which records the important concepts and terms in the building modeling process, and can also describe the hierarchical relationship of the building and the corresponding relationship between the parts and the attributes, and define the type of attribute value, so as to determine the final modeling content and instance information;

[0096] Knowledge extraction includes entity, relationship and attribute extraction, entity extraction is named entity, such as door, height, etc., relationship extraction is the specific association between entities, such as door has height, and attribute extraction is the attribute value of the attribute of the entity instance, such as the height of a certain door is two meters;

[0097] Specifically, in the building modeling knowledge graph constructed by the present application, the hierarchy of the building is arranged from top to bottom into three layers, which are building as a whole, building main structure surface and building main structure surface without opening, opening parts and building non-main parts. Different attribute parameters are sorted out for different building levels, and template models are given combining with specific semantic information for subsequent instance construction and storage in the knowledge graph; in addition, different modeling methods are selected for different building parts in each level according to actual modeling requirements;

[0098] S2-3: The present application uses triplets and attribute graphs to form the basic structure of the knowledge graph to realize the building modeling knowledge representation, wherein the triplets contain the subject, predicate and object, the attribute graph is a graph structure containing nodes and edges, and the triplets are a special case or subset of the attribute graph;

[0099] Specifically, the knowledge graph representation is shown in formula (1):

[0100] G={ε,R,A,V,T R ,TR} (1)

[0101] wherein G is a directed property graph, and ε, R, A, V represent entity, relation, attribute and attribute value respectively, and T R ,T A represent relation triple and attribute triple respectively.

[0102] S2-4: The application adopts a graph database for knowledge storage of building modeling, the graph database can store data as nodes in the graph and the relationship between the nodes, has the advantages of fitting the graph structure from the bottom, can be visualized, can be queried by query language, etc. This step can be implemented using a graph database such as Neo4j;

[0103] S2-5: Based on the selected graph database and its corresponding programming language and function, the knowledge fusion is realized by merging the repeated concepts and parameters and the like. In Neo4j, the same keyword can be indexed and de-duplicated by the match function, thereby realizing knowledge fusion.

[0104] At this point, the building modeling knowledge graph has been constructed, and the building modeling knowledge graph finally contains contents such as Figure 6 As shown in the partial building modeling knowledge graph built by Neo4j, as shown in Figure 7 .

[0105] From the S2 step, it is known that the top-down building hierarchy in the application first refines the building as a whole into a building main structure surface, so the S3 step constructs a main structure surface model of the target modeling building using the point cloud of the building under the guidance of the building modeling knowledge graph. The specific algorithm flow of step S3 is as follows:

[0106] S3-1: Adopt a point cloud segmentation method to fit the obtained dense point cloud of the target modeling building into several point cloud planes. The point cloud segmentation can be realized by using typical plane fitting methods such as the random sample consensus (RANSAC) method, the K-means clustering method, the region growing method, etc.

[0107] S3-2: Calculate the shortest Euclidean distance between different point cloud planes, and set a threshold DistPlane. The two planes with the shortest Euclidean distance less than DistPlane are adjacent planes. For each pair of adjacent planes, connect the planes by direct intersection, and the intersection line is the skeleton line of the building.

[0108] S3-3: Intersect the skeleton lines belonging to each plane two by two, and find the closed loop line around the current point cloud plane with the line segment obtained by the intersection of the skeleton lines as the unit. The closed loop line contains multiple line segments, each line segment has two endpoints, and each endpoint connects two line segments.

[0109] S3-4: Triangulate the closed loop to obtain the plane mesh model of the current point cloud plane; the triangulation can be achieved by the delaunay triangulation method, ear cutting method, etc.

[0110] S3-5: Combine all the plane models to obtain the main structure surface mesh model of the target modeling building;

[0111] By now, the main structure surface model of the target modeling building has been constructed, as shown in Figure 9 .

[0112] Further, the component-level model construction of the main structure is realized based on the main structure surface of the target modeling building obtained in the S3 step. The positioning information of the opening components in the main structure surface model is obtained in the S4 step, and the opening component model corresponding to the opening components in the main structure is constructed based on the positioning information. The main structure surface model is modified in combination with the opening component model. The specific algorithm process of step S4 is as follows:

[0113] S4-1: For each plane of the dense point cloud, find the corresponding sparse point cloud within a certain range; use the correspondence file between the sparse point cloud and the image obtained during the point cloud generation to find the image corresponding to the current sparse point cloud; for each vertex of the current main structure surface model, use the image parameters to calculate the position of each vertex in each image with a corresponding relationship. The image containing all the corresponding positions of the vertices is the texture image covering the current building plane, and all the texture images covering the current building plane together form a texture image set covering the current building plane;

[0114] S4-2: For each image in the texture image set, find the number of sparse points corresponding to the current image according to the correspondence file between the sparse point cloud and the image, and select the top five images with the most sparse points as the target texture image candidate images;

[0115] For each target texture image candidate image, calculate the area of the region corresponding to the main structure surface model according to the positions of all the vertices of the main structure surface model in the current image; find the image with the largest area corresponding to the main structure surface model from the five target texture image candidate images, which is the target texture image of the current building plane;

[0116] S4-3: A deep learning algorithm for image semantic segmentation is used to process the target texture image, and the opening components such as doors and windows that need to be modeled are identified from the knowledge graph; the deep learning algorithm for image semantic segmentation that can be used includes traditional deep learning algorithms such as YOLO and Mask R-CNN, or large model deep learning algorithms such as Grounding SAM; the traditional deep learning algorithm needs to be trained using a dataset, and the dataset can be a building component semantic segmentation dataset such as the CMP facade dataset; the large model deep learning algorithm does not need to be trained separately, and the semantic range of the components is given according to the knowledge graph; the opening component information is obtained using the results of semantic segmentation, and the opening component information is determined by the building modeling content in the building modeling knowledge graph;

[0117] S4-4: According to the different modeling methods of different semantic components in the building main structure stored in the building modeling knowledge graph, the classification reconstruction of different semantic components is performed; the present application mainly proposes a classification modeling method for opening components such as doors and windows on the surface of the main structure, and surface components of the main structure after removing the opening components such as walls and roofs in the building modeling knowledge graph;

[0118] Among them, for the opening components such as doors and windows, based on the component results of image semantic segmentation, the minimum circumscribed parallelogram is calculated, and the four vertices of the parallelogram are the four vertex coordinates of the current opening component; the positions of the four vertices of the opening component in the three-dimensional space are determined by using the correspondence between the image and the point cloud, and the four vertices in the three-dimensional space are connected to form a quadrilateral mesh model, which is the modeling result of the opening component;

[0119] For the surface components of the main structure after removing the opening components such as walls and roofs, a method of removing the opening component model belonging to the current plane from the current plane main structure surface model is used for modeling to obtain the modified main structure surface model, which can be realized by programming in open source three-dimensional graphics software such as Blender;

[0120] S4-5: For all opening component models and modified main structure surface models, the positions of each vertex of each component on the target texture image are determined by using the correspondence between the image and the point cloud, and the model texture mapping is performed by using the one-to-one correspondence between the vertex and the texture coordinate;

[0121] At this point, the modified target modeling building main structure surface model has been constructed as shown in Figure 9 , and the target modeling building opening component model is as shown in Figure 11 .

[0122] The model of the component in the target modeling building main structure has been built in the S4 step, so the S5 step determines the non-main component information outside the target modeling building main structure and the opening component, and guides the building of the non-main component model based on the non-main component information through the building modeling knowledge graph. The specific algorithm process of step S5 is as follows:

[0123] S5-1: Calculate the nearest distance of each point in the building dense point cloud to the surface model of the target modeling building main structure, set a point distance threshold, and when the distance is greater than the threshold, it means that the current point is far from the surface model of the target modeling building main structure and belongs to the unmodeled point. The unmodeled points collectively form an unmodeled point cloud;

[0124] Then, remove the point cloud with low height from the unmodeled point cloud, that is, remove the random points existing in the ground position; and perform statistical filtering processing on the other remaining point clouds to remove the point cloud noise;

[0125] Based on the Euclidean distance calculation, the point cloud after noise removal is clustered, the point cloud is divided into different point cloud clusters, the point cloud clusters with a number less than a number threshold are removed, and each remaining point cloud cluster represents a point cloud of a non-building main structure non-main component;

[0126] S5-2: For each non-main component segmented, first calculate the position, shape, size and other feature information of its point cloud in the building, match the feature information with the component features of different semantics stored in the building modeling knowledge graph, and find the corresponding semantics in the knowledge graph that meet the current component feature information;

[0127] For the component that cannot find the corresponding semantics, find the area in the target texture image corresponding to the component point cloud and the point cloud and image through the corresponding relationship, and use a deep learning algorithm to identify the component semantics and obtain the component semantics. The deep learning algorithm includes traditional deep learning algorithms such as YOLO and Mask R-CNN, or large model deep learning algorithms such as Grounding SAM. The traditional deep learning algorithm needs to be trained using a data set, and the data set can use a building component semantic segmentation data set such as the CMP facade data set. The large model deep learning algorithm does not need to be trained separately, and the component semantic range is given according to the knowledge graph;

[0128] S5-3: For each non-main component, read the corresponding component template model and its initial parameters in the building modeling knowledge graph according to its semantics;

[0129] According to the modeling method corresponding to the component semantics in the building modeling knowledge graph, the current component point cloud minimum enclosing box and related data are calculated; specifically, for components without fixed axes such as balconies and stairs, the point cloud minimum enclosing box without axis alignment is calculated, and for components with fixed axes such as columns and chimneys, the point cloud minimum enclosing box with fixed axis alignment is calculated;

[0130] According to the point cloud minimum enclosing box, the size information, angle information and center point position information of the current non-main body component are calculated;

[0131] Based on the proportional relationship between the size of the template model corresponding to the component semantics and the size of the non-main body component, the template model is scaled and adjusted to the size of the non-main body component;

[0132] According to the point cloud minimum enclosing box angle information of the non-main body component, the template model after size adjustment is angle-rotated in combination with the modeling method corresponding to the component semantics in the building modeling knowledge graph; specifically, for components without fixed axes such as balconies and stairs, the angle is adjusted according to the plane normal vector and the plane bottom line of the nearest main body structure surface model, and the rotation method of the building modeling knowledge graph is used for guidance, for example, the template model is rotated to be the same as the plane normal vector, and then the bottom is adjusted to be parallel to the plane bottom line; for components with fixed axes such as columns and chimneys, the template model is rotated to align with the two directions of the point cloud minimum enclosing box non-fixed axis while keeping the fixed axis unchanged;

[0133] The center point position information of the template model after size scaling and angle rotation is determined, the translation vector of the template model is determined using the center point position information and the point cloud minimum enclosing box position information, and the template model is translated based on the translation vector;

[0134] The template model after scaling, rotating and translating is determined as the model of the non-main body component;

[0135] S5-4: Using the correspondence between the point cloud and the image, the image region corresponding to the current component point cloud is found, and the image in the region is used for texture mapping of the model; the texture mapping can be completed by using the shading module in the open source software Blender;

[0136] At this point, the model of the target modeling building non-main body component has been constructed, as shown in Figure 11 .

[0137] The building modeling knowledge graph and the models of various components have been obtained from the S1-S5 steps, so the main body structure surface model, the opening component model and the non-main body component model are combined in the S6 step to obtain the target modeling building model, and the specific algorithm process of the S6 step is:

[0138] S6-1: All building component models are constructed according to the spatial positions of the corresponding point clouds, and are combined to obtain the target modeling building model.

[0139] At this point, the target modeling building model has been constructed, as shown in Figure 12 .

[0140] In addition, considering that the building modeling knowledge graph provides attribute parameters required by various components, different components find attribute parameters of corresponding type components according to semantics, extract attribute parameters of all current components by reading intermediate files in the modeling process or performing parameter calculation on the result model, and store the attribute parameters as building component instance information in the building modeling knowledge graph.

[0141] The application also provides a building fine modeling device fusing a knowledge graph. The device comprises the following parts: an acquisition unit 1301, configured to acquire a target modeling building image and generate a point cloud of the target modeling building, and acquire a mapping relationship between the point cloud and the target modeling building image; a building unit 1302, configured to determine a building modeling knowledge graph architecture according to building modeling content, build the building modeling knowledge graph through building modeling knowledge representation, building modeling knowledge storage and knowledge fusion; a first construction unit 1303, configured to construct a main structure surface model of the target modeling building by using the point cloud of the building under the guidance of the building modeling knowledge graph; a second construction unit 1304, configured to acquire positioning information of an opening component in the main structure surface model, and construct an opening component model corresponding to the opening component in the main structure based on the positioning information, and modify the main structure surface model in combination with the opening component model; a third construction unit 1305, configured to determine non-main component information of the target modeling building main structure and the opening component, and construct a non-main component model based on the non-main component information under the guidance of the building modeling knowledge graph; and a combination unit 1306, configured to combine the main structure surface model, the opening component model and the non-main component model to obtain a target modeling building model. The building fine modeling device fusing a knowledge graph provided by the application is as shown in Figure 13 .

[0142] The application also provides an electronic device 14 comprising a memory, a processor and an executable program stored in the memory and executable by the processor, wherein the processor can implement a building fine modeling method fusing a knowledge graph when executing the program. The electronic device provided by the application is as shown in Figure 14 .

[0143] The application also provides a computer readable storage medium, and computer executable instructions for executing a building fine modeling method fusing a knowledge graph when executed by a computer processor.

Claims

1. A method and apparatus for refined building modeling integrating knowledge graphs, characterized in that, The method specifically comprises the following steps: S1: obtaining a target modeling building image and generating a point cloud of the target modeling building; S2: determining a building modeling knowledge graph architecture according to building modeling content, and building a building modeling knowledge graph through building modeling knowledge representation, building modeling knowledge storage, and knowledge fusion; S3: under the guidance of the building modeling knowledge graph, constructing a main structure surface model of the target modeling building by using the point cloud of the building; S4: obtaining positioning information of building opening components in the main structure surface model, and constructing an opening component model corresponding to the opening components in the main structure based on the positioning information, and combining the opening component model to modify the main structure surface model, wherein the opening components at least include doors and windows of the building; S5: determining non-main component information of the target modeling building other than the main structure and the opening components, and constructing a non-main component model guided by the building modeling knowledge graph based on the non-main component information; S6: combining the main structure surface model, the opening component model, and the non-main component model to obtain a target modeling building model.

2. The method of claim 1, wherein S2 Comprise: S2-1: using a crawler technology to crawl building modeling content from multiple building modeling webpages; S2-2: determining a building modeling knowledge graph architecture based on the building modeling content; S2-3: using triples and attribute graphs for building modeling knowledge representation; S2-4: using a graph database for building modeling knowledge storage; S2-5: removing duplicate content in building modeling through knowledge fusion to build the building modeling knowledge graph.

3. The method of claim 1, wherein S3 Comprise: S3-1: using a point cloud segmentation method to fit dense point clouds in the point cloud of the target modeling building into multiple point cloud planes; S3-2: calculating the shortest Euclidean distance of each pair of point cloud planes in the multiple point cloud planes, comparing the shortest Euclidean distance with a preset threshold, and determining two point cloud planes corresponding to the shortest Euclidean distance less than the preset threshold as adjacent point cloud planes, and connecting each pair of adjacent point cloud planes to obtain a skeleton line of the modeling building; S3-3: intersecting the skeleton lines belonging to each point cloud plane with each other to determine a closed loop line around the current point cloud plane as a unit composed of a line segment obtained by intersecting skeleton lines; S3-4: triangulating the closed loop line to obtain a plane model of the current point cloud plane; S3-5: combining all the plane models to obtain the main structure surface model of the target modeling building.

4. The method according to claim 1, characterized by S4 Comprise: S4-1: determining a texture image set covering each building plane of the target modeling building; S4-2: determining a target texture image from each texture image set of the building plane; S4-3: applying a deep learning algorithm to the target texture image to identify opening component information corresponding to an opening component in the building plane, the opening component at least including a door of the building and a window of the building, the opening component information being determined by the building modeling content in the building modeling knowledge graph; S4-4: classifying and reconstructing different components in the building main structure in combination with the building modeling knowledge graph, including determining a modeling method of the opening component and a modeling method of removing the main structure surface model of the opening component, and based on the opening component information and the main structure surface model, constructing an opening component model and a modified main structure surface model through the modeling method of the opening component and the modeling method of removing the main structure surface model of the opening component; S4-5: using the target texture image to perform model texture mapping on the opening component model and the modified main structure surface model.

5. The method of claim 1, wherein, The specific process of S5 is as follows: S5-1: performing data processing and component segmentation processing on the dense point cloud in the point cloud of the target modeling building to obtain a plurality of cluster non-main component point clouds, the non-main component being a structure independent of the main structure and the opening component; S5-2: based on each cluster of the segmented non-main component point cloud, performing semantic recognition to obtain a component semantic corresponding to each cluster of the component point cloud; S5-3: based on the building modeling knowledge graph, determining a template model and a modeling method corresponding to each of the component semantics for component modeling to obtain a plurality of non-main component structure models; S5-4: performing building component model texture mapping on a plurality of non-main component structure models to obtain a complete non-main component model.

6. The method of claim 1, wherein, After S6, the method further comprises: storing the component instance information corresponding to the target modeling building into the building modeling knowledge graph.

7. The method of claim 5, wherein, S5-3 further comprises: in the case that the component semantic is a non-fixed shaft component, calculating a point cloud minimum circumscribed bounding box without shaft alignment; in the case that the component semantic is a fixed shaft component, calculating a point cloud minimum circumscribed bounding box with fixed shaft axis alignment; based on the minimum circumscribed bounding box, calculating size information, angle information and center point position information corresponding to the current non-main component; based on the proportional relationship between the size of the template model corresponding to the component semantic and the size of the non-main component, scaling the template model to adjust it to the size of the non-main component; based on the angle information of the point cloud minimum circumscribed bounding box corresponding to the non-main component, performing angle rotation on the template model after size scaling in combination with the modeling method corresponding to the component semantic; determining the center point position information of the template model after size scaling and angle rotation, determining a translation vector of the template model using the center point position information and the position information of the point cloud minimum circumscribed bounding box, and translating the template model based on the translation vector; determining the template model after scaling, rotating and translating as the model of the non-main component.

8. A building refined modeling apparatus fused with a knowledge graph, characterized by, The device comprises the following parts: an acquisition unit configured to acquire a target modeling building image and generate a point cloud of the target modeling building, and acquire a mapping relationship between the point cloud and the target modeling building image; A building modeling knowledge graph is built according to a building modeling knowledge graph architecture through building modeling knowledge representation, building modeling knowledge storage and knowledge fusion; A first construction unit is configured to construct a main structure surface model of the target modeling building by using the point cloud of the building under the guidance of the building modeling knowledge graph; A second construction unit is configured to acquire positioning information of a building opening component in the main structure surface model under the guidance of the building modeling knowledge graph, and construct an opening component model corresponding to the opening component in the main structure based on the positioning information, wherein the opening component at least includes a door of the building and a window of the building; A third construction unit is configured to determine non-main component information of the target modeling building other than the main structure and the opening component, and construct a non-main component model based on the non-main component information under the guidance of the building modeling knowledge graph; and a combination unit is configured to combine the main structure surface model, the opening component model and other component models to obtain a target modeling building model.

9. An electronic device comprising a memory, a processor, and an executable program saved in the memory and executable by the processor, characterized by, The processor executes the program to implement the building fine modeling method of the fusion knowledge graph according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer executable instructions are executed by the computer processor to implement the building fine modeling method of the fusion knowledge graph according to any one of claims 1-7.

Images