Building refined modeling method and device fused with knowledge graph

Abstract

The invention discloses a building refined modeling method and device fused with a knowledge graph, and belongs to the technical field of landscape modeling and surveying and mapping. The method comprises the steps of firstly obtaining a target building image and generating a point cloud, and obtaining a mapping relation between the point cloud and the image; determining a building modeling knowledge graph architecture according to the building modeling content, and building a knowledge graph through knowledge representation, knowledge storage and knowledge fusion; constructing a building main body structure surface model by using the point cloud under the guidance of the knowledge graph; positioning information of the opening part is obtained, an opening part model is constructed, and the main body structure surface model is modified; determining information of non-main-body components except the main body structure and the opening component, and constructing a non-main-body component model through guidance of a knowledge graph; and combining all the models to obtain a target building model. According to the method, the problems that a traditional method is complex in model, distorted in texture, difficult in component single body reconstruction and the like are solved, and hierarchical, componentized and refined building modeling is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of landscape modeling and surveying and mapping, and relates to a method and device for fine-grained building modeling integrating knowledge graphs. Background Art

[0002] Real-life 3D reconstruction involves reproducing real-world scenes by building high-precision 3D models. It is a key research topic in fields such as photogrammetry, remote sensing, and computer vision. With the advancement of oblique photography technology and the rise of smart city and digital city research, 3D city digital reconstruction based on high-resolution images and the point cloud data generated by them has become a current research hotspot. The generated digital model, combined with virtual reality and augmented reality technology, can display the surface features and details of objects and allow users to interact with virtual scenes in real time. This technology can be widely used in fields such as autonomous driving, personnel training, combat simulation, and emergency rescue.

[0004] As a key component of urban scenes, buildings have always been the focus of 3D reconstruction due to their diversity and complexity. Traditional 3D reconstruction of buildings usually has the following shortcomings: First, the entire scene is usually connected as a whole, the degree of individualization of buildings and their components is low, and the grid structure is complex, which makes texture mapping difficult and easy to distort. Secondly, these models usually lack semantic information, and the components are mixed together, which is not conducive to splitting, separate data processing and analysis. Finally, there is a lot of manual processing, a low degree of automation, insufficient procedural and parameterized processing, and poor scalability. Summary of the invention

[0005] The present invention aims at the problem of generating a component-level refined three-dimensional model of a target building, and provides a method and device for building refined modeling that integrates a knowledge graph. Based on the obtained target building model, the target building can be further analyzed for data such as structure and energy, and can also be used in applications such as battlefield simulation and three-dimensional navigation.

[0006] A first aspect of the present invention provides a method for fine-grained building modeling integrating knowledge graphs, comprising the following steps:

[0007] S1: Acquire the target modeling building image and generate the point cloud of the target modeling building, and obtain the mapping relationship between the point cloud and the target modeling building image;

[0008] S2: Determine the building modeling knowledge graph architecture based on the building modeling content, and build the building modeling knowledge graph through building modeling knowledge representation, building modeling knowledge storage, and knowledge fusion;

[0009] S3: Under the guidance of the building modeling knowledge graph, the main structure surface model of the target building is constructed using the building’s point cloud;

[0010] S4: obtaining positioning information of the building opening components in the main structure surface model, and constructing an opening component model corresponding to the opening components in the main structure based on the positioning information, and modifying the main structure surface model in combination with the opening component model;

[0011] S5: Determine the non-main component information other than the main structure and opening components of the target building, and based on the non-main component information, guide the construction of the non-main component model through the building modeling knowledge graph;

[0012] S6: Combine the main structure surface model, the opening component model and the non-main component model to obtain the target modeling building model.

[0013] Furthermore, in step S1, the target modeling building image is obtained and the point cloud of the target modeling building is generated. The algorithm flow for obtaining the mapping relationship between the point cloud and the target modeling building image is as follows:

[0014] S1-1: Use a shooting device to capture multi-view images of a target modeling building;

[0015] S1-2: Generate sparse point clouds and related files of the target modeled building using the Structure from Motion (SfM) technique;

[0016] S1-4: The dense point cloud of the target building is obtained through multi-view stereo (MVS) reconstruction;

[0017] S1-5: Perform cropping and denoising of sparse and dense point clouds of target modeling buildings.

[0018] So far, the sparse point cloud and dense point cloud of the target modeling building have been obtained, and the mapping relationship between the sparse point cloud and the image can be obtained based on the SfM result file.

[0019] Furthermore, in step S2, the building modeling knowledge graph architecture is determined according to the building modeling content, and the algorithm flow of constructing the building modeling knowledge graph through building modeling knowledge representation, building modeling knowledge storage and knowledge fusion is as follows:

[0020] S2-1: Use crawler technology to crawl building modeling content from multiple building modeling web pages;

[0021] S2-2: Determine the knowledge graph architecture of building modeling based on the building modeling content;

[0022] S2-3: Using triples and attribute graphs for building modeling knowledge representation;

[0023] S2-4: Using graph database to store building modeling knowledge;

[0024] S2-5: Remove duplicate content in building modeling through knowledge fusion to build a building modeling knowledge graph;

[0025] So far, a building modeling knowledge graph has been constructed, and the various levels of building architecture, attribute parameters and modeling methods stored in it are used to guide the modeling process of building components at various levels in subsequent steps.

[0026] Furthermore, in step S3, under the guidance of the building modeling knowledge graph, the algorithm flow of using the point cloud of the building to construct the main structure surface model of the target modeling building is as follows:

[0027] S3-1: Using the point cloud segmentation method, the dense point cloud in the point cloud of the target modeling building is fitted into multiple point cloud planes;

[0028] S3-2: Calculate the shortest Euclidean distance of each pair of point cloud planes in the multiple point cloud planes, compare the shortest Euclidean distance with a preset threshold, and define two point cloud planes corresponding to the shortest Euclidean distance less than the preset threshold as adjacent point cloud planes, and connect each pair of adjacent point cloud planes to obtain the skeleton line of the modeled building;

[0029] S3-3: intersect the skeleton lines belonging to each point cloud plane in pairs, and determine the closed loop around the current point cloud plane by taking the line segments obtained by the intersection of the skeleton lines as the component units;

[0030] S3-4: triangulate the closed loop to obtain a plane model of the current point cloud plane;

[0031] S3-5: Combine all plane models to obtain the main structure surface model of the target building.

[0032] Furthermore, in step S4, the positioning information of the building opening components in the main structure surface model is obtained, and an opening component model corresponding to the opening component in the main structure is constructed based on the positioning information. The algorithm flow of modifying the main structure surface model in combination with the opening component model is as follows:

[0033] S4-1: Determine a texture image set covering each building plane of the target modeling building;

[0034] S4-2: Determine a target texture image from the texture image set of each building plane;

[0035] S4-3: applying a deep learning algorithm to the target texture image to identify opening component information corresponding to the opening components in the building plane, where the opening components at least include doors and windows of the building, and the opening component information is determined by the building modeling content in the building modeling knowledge graph;

[0036] S4-4: Classify and reconstruct different components in the main structure of the building in combination with the building modeling knowledge graph, including determining a modeling method for opening components and a modeling method for a main structure surface model without opening components, and constructing an opening component model and a modified main structure surface model based on the opening component information and the main structure surface model through the modeling method for the opening components and the modeling method for the main structure surface model without opening components;

[0037] S4-5: For the opening component model and the modified main structure surface model, the target texture image is used to perform model texture mapping.

[0038] Furthermore, in step S5, the non-main component information other than the main structure and opening components of the target modeling building is determined, and based on the non-main component information, the algorithm flow of constructing the non-main component model through the building modeling knowledge graph is guided as follows:

[0039] S5-1: performing data processing and component segmentation processing on the dense point cloud in the point cloud of the target modeling building to obtain multiple clusters of non-main component point clouds, where the non-main component is a structure independent of the main structure and the opening component;

[0040] S5-2: Based on the segmented point cloud of each cluster of non-main component, semantic recognition is performed to obtain the component semantics corresponding to each cluster of component point cloud;

[0041] S5-3: Based on the building modeling knowledge graph, determine the template model and modeling method of each corresponding component semantics to perform component modeling, and obtain multiple non-main component structure models;

[0042] S5-4: Perform building component model texture mapping on multiple non-main component structure models to obtain a complete non-main component model.

[0043] Furthermore, in step S6, the algorithm flow of combining the main structure surface model, the opening component model and the non-main component model to obtain the target modeling building model is:

[0044] S6-1: According to the positions of the original point clouds corresponding to all the main structure surface models, opening component models and non-main component models, all the building component models are combined to obtain the building model.

[0045] Furthermore, after S6, the method further includes:

[0046] The component instance information corresponding to the target modeling building is stored in the building modeling knowledge graph.

[0047] In a second aspect of the present invention, a building fine-scale modeling device integrating knowledge graph is provided, and the device comprises the following parts:

[0048] An acquisition unit, used to acquire an image of a target modeling building and generate a point cloud of the target modeling building, and acquire a mapping relationship between the point cloud and the image of the target modeling building;

[0049] A building unit, used to build a building modeling knowledge graph according to the building modeling knowledge graph architecture through building modeling knowledge representation, building modeling knowledge storage, and knowledge fusion;

[0050] A first construction unit is used to construct a main structure surface model of a target modeling building using a point cloud of the building under the guidance of the building modeling knowledge graph;

[0051] A second construction unit is used to obtain positioning information of the building opening components in the main structure surface model under the guidance of the building modeling knowledge graph, and to construct an opening component model corresponding to the opening component in the main structure based on the positioning information;

[0052] The third construction unit is used to determine the non-main component information other than the main structure and opening components of the target modeling building, and based on the non-main component information, guide the construction of the non-main component model through the building modeling knowledge graph;

[0053] The combination unit is used to combine the main structure surface model, the opening component model and other component models to obtain the target modeling building model.

[0054] According to a third aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and an executable program stored in the memory and executable by the processor, wherein when the processor executes the program, a method for fine-grained building modeling integrating a knowledge graph can be implemented.

[0055] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, and computer-executable instructions are used to execute a method for fine-grained building modeling integrating a knowledge graph when executed by a computer processor.

[0056] The present invention provides a method and device for fine-tuning building modeling by integrating knowledge graph. The three-dimensional building model can be used to improve the overall scene model and analyze the target building. Compared with the traditional method of generating three-dimensional building models, the advantages of the present invention are:

[0057] (1) The model is simple and realistic. Replacing complex mesh models with simple polygonal mesh models can save model storage space and reduce texture distortion, making it suitable for building modeling in large-scale scenes;

[0058] (2) The model is more hierarchical, semantic, and monolithic. By analyzing the semantics of building components and setting different modeling methods, the modeling is more targeted and more realistic. At the same time, based on the knowledge graph to store different information of different semantic components, the modeling results provide richer information, which is convenient for subsequent application.

[0059] (3) Using the knowledge graph to guide modeling is scalable. When there are special types of buildings that need to be modeled, such as Gothic buildings, ancient Chinese buildings, etc., the template model of the knowledge graph can be expanded according to the method of the present invention and the corresponding modeling method can be given. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0061] Figure 1 A flow chart of a method for building refined modeling integrating knowledge graphs provided in an embodiment of the present invention;

[0062] Figure 2 Modeling a multi-view schematic diagram of a building part for the target;

[0063] Figure 3 Schematic diagram of sparse point cloud of building for target modeling;

[0064] Figure 4 Schematic diagram of dense point cloud of building for target modeling;

[0065] Figure 5 Schematic diagram of the relevant files for generating point cloud for an image;

[0066] Figure 6 Modeling the knowledge graph for the building ultimately includes a schematic diagram of the content;

[0067] Figure 7 A schematic diagram of knowledge graph modeling for some buildings built using Neo4j;

[0068] Figure 8 A schematic diagram of the surface model of the main structure of the target building;

[0069] Fig. 9 A schematic diagram of the surface model of the main structure of the building modeled for the modified target;

[0070] Fig.10 Modeling schematic diagram of the building opening parts for the target;

[0071] Fig.11 Modeling schematic diagram of non-main parts of the target building;

[0072] Fig.12 Model a schematic diagram of a building model for the target;

[0073] Fig.13 A schematic diagram of a building refined modeling device integrating a knowledge graph provided in an embodiment of the present invention;

[0074] Fig.14 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0076] At this stage, with the continuous development of various technologies such as deep learning, computer vision, and knowledge graphs, various applications have put forward higher requirements for the three-dimensional reconstruction technology of buildings, including: building a simplified mesh model to make the model have a high degree of flatness and facilitate separate texture mapping operations; being able to identify and extract the fine components of the building, such as doors, windows, walls, balconies, etc., and independently model these components; reducing manual intervention, realizing the automated, hierarchical three-dimensional reconstruction of buildings, and having a certain degree of scalability.

[0077] In view of the hierarchical and parametric characteristics of buildings as typical man-made structures, combined with the shortcomings of traditional three-dimensional reconstruction models and the new needs of current technology, the present invention provides an innovative technical means for the three-dimensional reconstruction of buildings. On the premise of obtaining multi-view images of buildings and generating point clouds, the present invention guides the hierarchical, componentized, programmed, and parametric modeling of buildings through knowledge graphs, and can provide a variety of information such as geometry, topology, semantics, and texture of buildings, avoiding the problems of complex models and texture distortion, difficulty in component segmentation and reconstruction, and insufficient information display in traditional methods. It can not only improve the accuracy of the model and the level of intelligent modeling, and provide refined building model results, but also provide technical support for subsequent other applications based on buildings, such as structural and energy consumption analysis.

[0078] In order to make the technical solution of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings.

[0079] The flowchart of the method for building refined modeling integrating knowledge graph provided by the present invention is as follows: Figure 1 shown.

[0080] Specifically, a building refined modeling method integrating knowledge graph includes the following steps:

[0081] S1: Acquire the target modeling building image and generate the point cloud of the target modeling building, and obtain the mapping relationship between the point cloud and the target modeling building image;

[0082] S2: Determine the building modeling knowledge graph architecture based on the building modeling content, and build the building modeling knowledge graph through building modeling knowledge representation, building modeling knowledge storage, and knowledge fusion;

[0083] S3: Under the guidance of the building modeling knowledge graph, the main structure surface model of the target building is constructed using the building’s point cloud;

[0084] S4: obtaining positioning information of the building opening components in the main structure surface model, and constructing an opening component model corresponding to the opening components in the main structure based on the positioning information, and modifying the main structure surface model in combination with the opening component model;

[0085] S5: Determine the non-main component information other than the main structure and opening components of the target building, and based on the non-main component information, guide the construction of the non-main component model through the building modeling knowledge graph;

[0086] S6: Combine the main structure surface model, the opening component model and the non-main component model to obtain the target modeling building model.

[0087] Specifically, step S1 obtains the target modeling building image and generates the point cloud of the target modeling building, and obtains the mapping relationship between the point cloud and the target modeling building image. The specific algorithm flow is:

[0088] S1-1: Use a shooting device to shoot the target modeling building from multiple angles, and be able to cover the entire target modeling building repeatedly. It is required that the multiple views of the target modeling building are relatively dense, and specifically, each part involves no less than two images;

[0089] S1-2: Use the Structure from Motion (SfM) technique to process multi-view images and estimate the 3D structure of the target modeling building area from the 2D image. The SfM technique obtains the sparse point cloud of the area through feature matching, camera pose estimation, triangulation, optimization and verification, and generates a related file containing information such as the camera pose and the position of the sparse points in the image. The SfM technique can be implemented through open source algorithms such as VisualSFM, Colmap, and OpenSFM.

[0090] S1-3: Based on the camera pose and sparse point cloud obtained by SfM, multi-view stereo technology (MVS) is used to reconstruct a dense point cloud of the target building area. MVS technology can be implemented through open source algorithms such as PMVS, openMVS, and Colmap. The present invention requires that the obtained building point cloud has no large-scale vacancies based on a relatively dense multi-view.

[0091] S1-4: Use software such as CloudCompare to crop the target building modeling parts from the sparse point cloud and dense point cloud, and use morphological, statistical and other filtering methods to denoise the point cloud;

[0092] The ultimate goal is to model multiple views of building parts such as Figure 2 As shown in Figure 3 As shown in Figure 4 As shown, the relevant files are as follows Figure 5 As shown, so far, multi-view images, point clouds and related files of the target building have been obtained.

[0093] To guide hierarchical and refined building modeling based on knowledge graph, it is first necessary to build a building modeling knowledge graph based on building modeling content such as building hierarchical modeling steps, building modeling rule knowledge, and expected data model results. Step S2 determines the building modeling knowledge graph architecture based on the building modeling content, and builds the building modeling knowledge graph through building modeling knowledge representation, building modeling knowledge storage, and knowledge fusion. The specific algorithm flow is as follows:

[0094] S2-1: Use crawler tools to write scripts to simulate the way humans browse web pages, and crawl the building modeling related content from the web pages that record the building modeling related standards; crawler technology can be implemented using open source crawler tools such as spider and selenium; web pages that record the building modeling related content include IFC, CityGML, BIM and other related web pages; crawled content includes text, tables and other forms;

[0095] S2-2: Through manual integration, ontology construction and knowledge extraction are performed to determine the knowledge graph architecture of building modeling. First, based on the building modeling target model and data, relevant content suitable for building modeling is sorted out from the content obtained by the crawler, and parts that are irrelevant or not concerned with modeling are removed; secondly, based on the sorted out building modeling-related content, specific instance information is set to guide data generation and data storage in the modeling process, so as to facilitate subsequent services for other applications;

[0096] Among them, the ontology is the label of the knowledge graph, which records the important concepts and terms in the building modeling process. It can also describe the hierarchical relationship of the building and the correspondence between components and attributes, and define the type of attribute values, so as to determine the final modeling content and instance information;

[0097] Knowledge extraction includes entity, relationship and attribute extraction. Entity extraction refers to named entities, such as door, height, etc. Relation extraction refers to the specific association between entities, such as door has height. Attribute extraction refers to the extraction of attribute values ​​of entity instances, such as the height of a door is two meters.

[0098] Specifically, in the building modeling knowledge graph constructed by the present invention, the building hierarchy is organized into three layers from top to bottom, namely, the building as a whole - the main structure surface of the building - the main structure surface of the building without openings, the opening parts, and the non-main parts of the building. Different attribute parameters are sorted out for different building levels, and template models are given in combination with specific semantic information for subsequent instance construction and storage in the knowledge graph; in addition, different building components in each level are matched to different modeling methods according to actual modeling requirements;

[0099] S2-3: The present invention utilizes triples and attribute graphs to form the basic structure of knowledge graphs to realize building modeling knowledge representation, wherein triples include subjects, predicates and objects, and attribute graphs are graph structures including nodes and edges, and triples are a special case or subset of attribute graphs;

[0100] Specifically, the knowledge graph is represented as shown in formula (1):

[0101] G={ε,R,A,V,T R ,TR} (1)

[0102] Among them, G is a directed attribute graph, ε, R, A, and V represent entities, relations, attributes, and attribute values, respectively, and T R ,T A They represent relation triples and attribute triples respectively.

[0103] S2-4: The present invention uses a graph database to store knowledge of building modeling. The graph database can store data as nodes in a graph and the relationships between nodes. It has the advantages of being able to fit the graph structure from the bottom layer, being able to be visualized, and being able to be queried using a query language. This step can be implemented using a graph database such as Neo4j.

[0104] S2-5: Based on the selected graph database and its corresponding programming language and functions, knowledge fusion is realized, and repeated concepts and parameters are merged. For example, in Neo4j, the match function can be used to index and remove duplicates of the same keywords, thereby realizing knowledge fusion.

[0105] So far, the building modeling knowledge graph has been constructed, and the building modeling knowledge graph finally contains the following contents: Figure 6 As shown in the figure, some building modeling knowledge graphs built using Neo4j are as follows Figure 7 shown.

[0106] From step S2, it is known that the top-down building hierarchy in the present invention first refines the building as a whole into the main structure surface of the building. Therefore, step S3, under the guidance of the building modeling knowledge graph, uses the point cloud of the building to construct the main structure surface model of the target modeling building. The specific algorithm flow of step S3 is:

[0107] S3-1: Using the point cloud segmentation method, the dense point cloud of the target modeling building is fitted into several point cloud planes; point cloud segmentation can be achieved by using typical plane fitting methods such as random sampling consensus (RANSAC) method, K-means clustering method, region growing method, etc.

[0108] S3-2: Calculate the shortest Euclidean distance between different point cloud planes. By setting the threshold DistPlane, two planes whose shortest Euclidean distance is less than DistPlane are adjacent planes. For each pair of adjacent planes, the planes are connected by direct intersection, and the intersection line is the skeleton line of the building.

[0109] S3-3: intersect the skeleton lines belonging to each plane in pairs, and use the line segments obtained by the intersection of the skeleton lines as constituent units to find a closed loop around the current point cloud plane; wherein the closed loop includes multiple line segments, each line segment has two endpoints, and each endpoint connects two line segments;

[0110] S3-4: triangulate the closed loop to obtain a plane mesh model of the current point cloud plane; the triangulation can be achieved by a Delaunay triangulation method, an ear-cut method, or the like;

[0111] S3-5: Combine all plane models to obtain the main structure surface mesh model of the target building;

[0112] So far, the main structure surface model of the target building has been constructed, such as Fig. 9 shown.

[0113] Taking the main structure surface of the target modeled building obtained in step S3 as a unit, further realize the component-level model construction of the main structure. Step S4 is required to obtain the positioning information of the building opening components in the main structure surface model, and build the opening component model corresponding to the opening components in the main structure based on the positioning information. The main structure surface model is modified in combination with the opening component model. The specific algorithm flow of step S4 is as follows:

[0114] S4-1: For each plane's dense point cloud, find the corresponding sparse point cloud within a certain range; use the correspondence file between the sparse point cloud and the image obtained during the point cloud generation process to find the image that has a corresponding relationship with the current sparse point cloud; for each vertex of the current main structure surface model, use the image parameters to calculate the position of each vertex in each image with a corresponding relationship, and the image that can contain the corresponding positions of all vertices is the texture image covering the current building plane, and all texture images covering the current building plane together constitute a texture image set covering the current building plane;

[0115] S4-2: For each image in the texture image set, find the number of sparse points corresponding to the current image according to the correspondence relationship file between the sparse point cloud and the image, and select the top five images with the largest number of sparse points as candidate target texture images;

[0116] For each candidate target texture image, the area corresponding to the main structure surface model is calculated according to the positions corresponding to all vertices of the main structure surface model in the current image; from the five candidate target texture images, the image with the largest area corresponding to the main structure surface model is found, which is the target texture image of the current building plane;

[0117] S4-3: Use the deep learning algorithm of image semantic segmentation to process the target texture image, and identify the opening parts that need to be modeled given in the knowledge graph, such as doors, windows and other parts; the image semantic segmentation deep learning algorithms that can be used include traditional deep learning algorithms such as YOLO and Mask R-CNN or large-model deep learning algorithms such as Grounding SAM; among them, traditional deep learning algorithms need to be trained using data sets, and the data sets can use building component semantic segmentation data sets such as CMP facade data sets; large-model deep learning algorithms do not need to be trained separately, and the semantic range of components is given according to the knowledge graph; the opening component information is obtained using the results of semantic segmentation, and the opening component information is determined by the building modeling content in the building modeling knowledge graph;

[0118] S4-4: According to different modeling methods corresponding to different semantic components in the main structure of the building stored in the building modeling knowledge graph, the different semantic components are classified and reconstructed. The present invention mainly proposes a classification modeling method for the opening components on the surface of the main structure such as doors and windows, and the surface components of the main structure such as walls and roofs after removing the opening components in the building modeling knowledge graph;

[0119] Among them, for opening parts such as doors and windows, the minimum circumscribed parallelogram is calculated based on the component results of image semantic segmentation, and the four vertices of the parallelogram are the coordinates of the four vertices of the current opening part; the corresponding relationship between the image and the point cloud is used to determine the positions of the four vertices of the opening part in the three-dimensional space, and the four vertices in the three-dimensional space are connected into a quadrilateral mesh model, which is the modeling result of the opening part;

[0120] For the main structural surface parts after removing the opening parts such as walls and roofs, the modeling is carried out by removing the opening part model belonging to the current plane from the main structural surface model of the current plane, and the modified main structural surface model is obtained, which can be implemented by programming in open source 3D graphics software such as Blender;

[0121] S4-5: For all opening component models and the modified main structure surface model, the corresponding relationship between the image and the point cloud is used to determine the position of each vertex of each component on the target texture image, and the model texture mapping is performed using the one-to-one correspondence between the vertex and the texture coordinates;

[0122] So far, the modified surface model of the main structure of the target building has been constructed. Fig. 9 As shown, the target building opening component model is as follows Fig.11 shown.

[0123] Since the components in the main structure of the target building have been modeled in step S4, the non-main component information other than the main structure and opening components of the target building is determined in step S5, and based on the non-main component information, the non-main component model is guided by the building modeling knowledge graph. The specific algorithm flow of step S5 is as follows:

[0124] S5-1: Calculate the shortest distance between each point in the dense point cloud of the building and the main structure surface model of the target modeled building, set the point distance threshold, and when the distance is greater than the threshold, it means that the current point is far away from the main structure surface model of the target modeled building and is an unmodeled point. The unmodeled points together constitute an unmodeled point cloud;

[0125] Afterwards, the point clouds with lower heights are removed from the point clouds that have not been modeled, that is, the noise points existing on the ground are removed; the remaining point clouds are statistically filtered to remove the point cloud noise;

[0126] Based on the Euclidean distance calculation, the denoised point cloud is clustered and divided into different point cloud clusters. The point cloud clusters with less points than the threshold are removed. Each of the remaining point cloud clusters represents a point cloud of a non-main component of the main structure of the building.

[0127] S5-2: For each non-main component segmented, first calculate the feature information of its point cloud in the building, such as position, shape, and size, and use the feature information to match the component features of different semantics of the building stored in the building modeling knowledge graph to find the corresponding semantics in the knowledge graph that matches the feature information of the current component;

[0128] For components that cannot find matching semantics, find the corresponding area in the target texture image through the component point cloud and the correspondence between the point cloud and the image, and use the deep learning algorithm to perform component semantic recognition to obtain the component semantics; deep learning algorithms include traditional deep learning algorithms such as YOLO and Mask R-CNN or large-model deep learning algorithms such as Grounding SAM; traditional deep learning algorithms need to be trained using data sets, and the data sets can use building component semantic segmentation data sets such as CMP facade data sets; large-model deep learning algorithms do not need to be trained separately, and the component semantic range is given according to the knowledge graph;

[0129] S5-3: For each non-main component, read the corresponding component template model and its initial parameters given in the building modeling knowledge graph according to its semantics;

[0130] Combined with the modeling method corresponding to the semantics of components in the building modeling knowledge graph, the point cloud external bounding box and related data of the current component are calculated in a targeted manner; specifically, for components without fixed axes such as balconies and stairs, the minimum external bounding box of the point cloud that does not require axis alignment is calculated; for components with fixed axes such as columns and chimneys, the minimum external bounding box of the point cloud that is axis-aligned with the fixed axis is calculated;

[0131] According to the minimum external bounding box of the point cloud, calculate the size information, angle information and center point position information of the current non-main component;

[0132] Based on the proportional relationship between the size of the template model corresponding to the component semantics and the size of the non-main component, the template model is scaled to adjust it to the size of the non-main component;

[0133] Combined with the modeling method in the building modeling knowledge graph corresponding to the semantics of the components, the angle information of the minimum circumscribed bounding box of the point cloud corresponding to the non-main components is used to rotate the template model after the size adjustment; specifically, for components without fixed axes such as balconies and stairs, the angle is adjusted under the guidance of the rotation method of the building modeling knowledge graph according to the plane normal vector of the main structure surface model closest to the component and the plane bottom line and other directions. For example, the template model is rotated to make it the same as the plane normal vector and then its bottom is adjusted to be parallel to the plane bottom line; for components with fixed axes such as columns and chimneys, the template model is rotated to align with the two directions of the non-fixed axis of the minimum circumscribed bounding box of the point cloud while the fixed axis remains unchanged;

[0134] Determine the center point position information of the template model after scaling and angle rotation, determine the translation vector of the template model using the center point position information and the point cloud minimum external bounding box position information, and translate the template model based on the translation vector;

[0135] Determine the template model after scaling, rotation and translation as the model of the non-main component;

[0136] S5-4: Using the correspondence between the point cloud and the image, find the image area corresponding to the point cloud of the current component, and use the image of the area to perform texture mapping on the model; texture mapping can be completed using the shading module in the open source software Blender;

[0137] So far, the non-main component models of the target building have been constructed, such as Fig.11 shown.

[0138] From steps S1 to S5, the building modeling knowledge graph and the models of various components have been obtained. Therefore, step S6 combines the main structure surface model, the opening component model and the non-main component model to obtain the target building model. The specific algorithm flow of step S6 is:

[0139] S6-1: All building component models are constructed according to the spatial positions of the corresponding point clouds and combined to obtain the target building model.

[0140] So far, the target building model has been constructed, such as Fig.12 shown.

[0141] In addition, considering that the building modeling knowledge graph provides the attribute parameters required for various types of components, different components find the attribute parameters of the corresponding type of components according to semantics, and extract the attribute parameters of all current components by reading the intermediate files in the modeling process or performing parameter calculations on the result model. The parameters are stored in the building modeling knowledge graph as building component instance information.

[0142] The present application also provides a device for fine-grained building modeling by integrating knowledge graphs, and the device includes the following parts: an acquisition unit 1301, which is used to acquire a target modeling building image and generate a point cloud of the target modeling building, and to acquire a mapping relationship between the point cloud and the target modeling building image; a construction unit 1302, which is used to determine the building modeling knowledge graph architecture according to the building modeling content, and to build the building modeling knowledge graph through building modeling knowledge representation, building modeling knowledge storage, and knowledge fusion; a first construction unit 1303, which is used to construct the main structure of the target modeling building using the point cloud of the building under the guidance of the building modeling knowledge graph. The second construction unit 1304 is used to obtain the positioning information of the building opening parts in the main structure surface model, and construct the opening part model corresponding to the opening parts in the main structure based on the positioning information, and modify the main structure surface model in combination with the opening part model; the third construction unit 1305 is used to determine the non-main part information other than the main structure and opening parts of the target modeling building, and based on the non-main part information, guide the construction of the non-main part model through the building modeling knowledge graph; the combination unit 1306 is used to combine the main structure surface model, the opening part model and the non-main part model to obtain the target modeling building model. The building refined modeling device integrating the knowledge graph provided by the present invention is as follows: Fig.13 shown.

[0143] The present application also provides an electronic device 14, including a memory, a processor, and an executable program stored in the memory and executable by the processor, wherein the processor can implement a building refined modeling method integrating a knowledge graph when executing the program. Fig.14 shown.

[0144] The present application also provides a computer-readable storage medium, and the computer-executable instructions are used to execute a method for fine-grained building modeling that integrates a knowledge graph when executed by a computer processor.

Claims

1. A method and device for fine-grained building modeling integrating knowledge graph, characterized in that: The method specifically comprises the following steps: S1: Acquire a target modeling building image and generate a point cloud of the target modeling building, and acquire a mapping relationship between the point cloud and the target modeling building image; S2: Determine the building modeling knowledge graph architecture based on the building modeling content, and build the building modeling knowledge graph through building modeling knowledge representation, building modeling knowledge storage, and knowledge fusion; S3: Under the guidance of the building modeling knowledge graph, construct a main structure surface model of the target modeling building using the point cloud of the building; S4: acquiring positioning information of the building opening components in the main structure surface model, and constructing an opening component model corresponding to the opening component in the main structure based on the positioning information, and modifying the main structure surface model in combination with the opening component model; S5: Determine the non-main component information other than the main structure and the opening components of the target modeling building, and based on the non-main component information, guide the construction of a non-main component model through the building modeling knowledge graph; S6: Combine the main structure surface model, the opening component model and the non-main component model to obtain a target modeling building model.

2. The method according to claim 1, characterized in that S2 include: S2-1: Use crawler technology to crawl building modeling content from multiple building modeling web pages; S2-2: Based on the building modeling content, determine the building modeling knowledge graph architecture; S2-3: Using triples and attribute graphs for building modeling knowledge representation; S2-4: Using graph database to store building modeling knowledge; S2-5: Remove duplicate content in building modeling through knowledge fusion to build the building modeling knowledge graph.

3. The method according to claim 1, characterized in that S3 include: S3-1: using a point cloud segmentation method to fit a dense point cloud in the point cloud of the target modeling building into a plurality of point cloud planes; S3-2: Calculate the shortest Euclidean distance of each pair of the point cloud planes in the multiple point cloud planes, compare the shortest Euclidean distance with a preset threshold, and define the two point cloud planes corresponding to the shortest Euclidean distance less than the preset threshold as adjacent point cloud planes, and connect each pair of adjacent point cloud planes to obtain the skeleton line of the modeled building; S3-3: intersecting skeleton lines belonging to each of the point cloud planes in pairs, using line segments obtained by the intersection of the skeleton lines as constituent units, and determining a closed loop around the current point cloud plane; S3-4: triangulate the closed loop to obtain a plane model of the current point cloud plane; S3-5: Combine all the plane models to obtain the main structure surface model of the target modeling building.

4. The method according to claim 1, characterized in that S4 include: S4-1: Determine a texture image set covering each building plane of the target modeling building; S4-2: Determine a target texture image from the texture image set of each of the building planes; S4-3: applying a deep learning algorithm to the target texture image to identify opening component information corresponding to the opening components in the plane of the building, where the opening components at least include doors and windows of the building, and the opening component information is determined by the building modeling content in the building modeling knowledge graph; S4-4: Classify and reconstruct different components in the main structure of the building in combination with the building modeling knowledge graph, including determining a modeling method for the opening component and a modeling method for the main structure surface model without the opening component, and constructing an opening component model and a modified main structure surface model based on the opening component information and the main structure surface model through the modeling method for the opening component and the modeling method for the main structure surface model without the opening component; S4-5: Performing model texture mapping on the opening component model and the modified main structure surface model using the target texture image.

5. The method according to claim 1, characterized in that The specific process of S5 is as follows: S5-1: performing data processing and component segmentation processing on a dense point cloud in the point cloud of the target modeling building to obtain a plurality of clusters of non-main component point clouds, where the non-main component is a structure independent of the main structure and the opening component; S5-2: Based on the segmented point clouds of each cluster of non-main component, semantic recognition is performed to obtain component semantics corresponding to each cluster of component point clouds; S5-3: Based on the building modeling knowledge graph, determine the template model and modeling method of each corresponding component semantics to perform component modeling, and obtain multiple non-main component structure models; S5-4: Perform building component model texture mapping on the plurality of non-main component structure models to obtain a complete non-main component model.

6. The method according to claim 1, characterized in that After S6, the method further includes: The component instance information corresponding to the target modeling building is stored in the building modeling knowledge graph.

7. The method according to claim 5, characterized in that S5-3 also includes: When the semantics of the component is a component without a fixed axis, calculating a minimum circumscribed bounding box of the point cloud that does not require axis alignment; When the component semantics is a fixed-axis component, calculating a minimum circumscribed bounding box of the point cloud aligned with the fixed axis; Based on the minimum external bounding box, calculate the size information, angle information and center point position information corresponding to the current non-main component; Based on the proportional relationship between the size of the template model corresponding to the component semantics and the size of the non-main component, scaling the template model to adjust it to the size of the non-main component; In combination with the modeling method corresponding to the semantics of the component, the angle information of the minimum external bounding box of the point cloud corresponding to the non-main component is used to perform angle rotation on the template model after the size adjustment; Determine the center point position information of the template model after scaling and angle rotation, determine the translation vector of the template model using the center point position information and the position information of the minimum circumscribed bounding box of the point cloud, and translate the template model based on the translation vector; The template model after scaling, rotation and translation is determined as the model of the non-main component.

8. A building refined modeling device integrating knowledge graph, characterized in that: The device comprises the following parts: An acquisition unit, used to acquire a target modeling building image and generate a point cloud of the target modeling building, and acquire a mapping relationship between the point cloud and the target modeling building image; A building unit, used to build a building modeling knowledge graph according to the building modeling knowledge graph architecture through building modeling knowledge representation, building modeling knowledge storage, and knowledge fusion; A first construction unit is used to construct a main structure surface model of the target modeling building using the point cloud of the building under the guidance of the building modeling knowledge graph; A second construction unit is used to obtain positioning information of the building opening components in the main structure surface model under the guidance of the building modeling knowledge graph, and to construct an opening component model corresponding to the opening component in the main structure based on the positioning information; A third construction unit is used to determine non-main component information other than the main structure and opening components of the target modeling building, and based on the non-main component information, guide the construction of a non-main component model through the building modeling knowledge graph; The combining unit is used to combine the main structure surface model, the opening component model and other component models to obtain a target modeling building model.

9. An electronic device comprising a memory, a processor, and an executable program stored in the memory and executable by the processor, wherein: When the processor executes the program, it can implement a building refined modeling method integrating a knowledge graph as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer executable instructions, when executed by a computer processor, are used to execute a building refined modeling method integrating a knowledge graph as described in any one of claims 1-7.

Images