Method and apparatus for searching for building information model, and storage medium

Abstract

The present application relates to a method for searching for a building information model. The method comprises: performing multi-modal feature extraction on building information models in a model library to be searched, and obtaining multi-modal features corresponding to various building information models, the multi-modal features comprising semantic features, topological features, and geometric features; acquiring a search text inputted by a user, parsing the search text, and obtaining search intention information corresponding to the search text, the search intention information comprising search intention semantic features, search intention topological features, and search intention geometric features; on the basis of deep embedding learning, calculating comprehensive similarities between the search intention information and the multi-modal features of the building information models in said model library; and, on the basis of a sorting result of the comprehensive similarities between the search intention information and the multi-modal features of the building information models in said model library, determining a building information model that serves as a search result corresponding to the search text, and recommending the search result to the user.

Description

Building information model search method, device and storage medium Technical Field

[0001] The present application relates to the field of building information technology, and in particular to a building information model search method and device. Background Art

[0002] Building Information Modeling (BIM) technology enables intuitive three-dimensional visualization of design information, providing an efficient solution for cross-disciplinary collaborative design, technical briefing, and full-process engineering management of construction projects.

[0003] However, the research found that existing research lacks efficient search methods for complex building BIM models. Specifically, existing BIM model search methods can only search for building components within the model (such as walls, doors, windows, beams, etc.), but lack the ability to consider the characteristics of the entire building model (for example, a multi-story building, including all components on multiple floors and multiple rooms). Summary of the Invention

[0004] In response to the above problems, the purpose of this application is to provide a building information model search method and device, which can realize the search of building information models at the overall building level or the multi-component combination level.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, the present application provides a building information model search method, comprising:

[0007] Performing multimodal feature extraction on the multi-component combination-level building information models in the model library to be searched to obtain multimodal features corresponding to each building information model, wherein the multimodal features include semantic features, topological features, and geometric features;

[0008] Obtaining a search text input by a user, parsing the search text, and obtaining search intent information corresponding to the search text, wherein the search intent information includes search intent semantic features, search intent topological features, and search intent geometric features;

[0009] Based on deep embedding learning, calculating the comprehensive similarity between the search intent information and the multimodal features of each building information model in the model library to be searched;

[0010] According to the ranking result of the comprehensive similarity between the search intention information and the multimodal features of each building information model in the model library to be searched, the building information model as the search result corresponding to the search text is determined, and the search result is recommended to the user.

[0011] In one implementation of the present application, the multimodal feature extraction of the multi-component combination-level building information model in the model library to be searched includes:

[0012] The semantic information of the attributes of the components at each level in the multi-component combination level building information model is extracted, and then the semantic information of the attributes of the components at each level is summarized and statistically analyzed to obtain the semantic features of the attributes of the multi-component combination level building information model.

[0013] In one implementation of the present application, the components at each level include: a building space, a wall that may be included in the building space, and a door or window that may be included in the wall.

[0014] In one implementation of the present application, the method performs multimodal feature extraction on a multi-component combination-level building information model in a model library to be searched, further comprising:

[0015] According to the properties of components at each level, the spatial adjacency relationship between each building space is determined as the topological feature of the building information model at the multi-component combination level.

[0016] In one implementation of the present application, the spatial adjacency relationship includes three relationships: non-adjacent, adjacent and non-connected, and connected.

[0017] In one implementation of the present application, the multimodal feature extraction of the multi-component combination-level building information model in the model library to be searched further includes:

[0018] The plane outline information of the building information model is extracted as the geometric features of the building information model at the multi-component combination level.

[0019] In one implementation of the present application, parsing the search text to obtain search intent information corresponding to the search text includes:

[0020] Based on natural language processing, text segmentation and regular expression parsing are performed to obtain the search intent semantic features, search intent topological features, and search intent geometric features of the search intent information.

[0021] In one implementation of the present application, the calculation of the comprehensive similarity between the search intent information and the multimodal features of each building information model in the model library to be searched based on deep embedding learning includes:

[0022] According to the extracted semantic features, topological features and geometric features of the building information model, as well as the search intent semantic features, search intent topological features and search intent geometric features of the search intent, they are embedded into a unified vectorized representation for similarity calculation.

[0023] In one implementation of the present application, the similarity calculation is weighted cosine similarity calculation.

[0024] In a second aspect, the present application provides a building information model search device, comprising:

[0025] A feature extraction module is used to extract multimodal features from the multi-component combination-level building information models in the model library to be searched, and obtain multimodal features corresponding to each building information model, wherein the multimodal features include semantic features, topological features, and geometric features;

[0026] A parsing module is used to obtain a search text input by a user, parse the search text, and obtain search intent information corresponding to the search text, wherein the search intent information includes search intent semantic features, search intent topological features, and search intent geometric features;

[0027] A similarity calculation module is used to calculate the comprehensive similarity between the search intent information and the multimodal features of each building information model in the model library to be searched based on deep embedding learning;

[0028] The recommendation module is used to determine the building information model as the search result corresponding to the search text based on the sorting result of the comprehensive similarity between the search intention information and the multimodal features of each building information model in the model library to be searched, and to visually display the search result to the user.

[0029] Due to the adoption of the above technical solution, this application has the following advantages: (1) it realizes the semantic-topological-geometric multimodal feature search of building-level BIM models; (2) it achieves good search results and significantly improves the accuracy; (3) it optimizes the algorithm operation efficiency and ensures the search speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] FIG1 is a flow chart of a building information model search method according to an embodiment of the present application;

[0031] FIG2 is a flowchart of a semantic feature extraction process based on IFC model information analysis;

[0032] Figure 3 is an example of extracting attribute information from a building BIM model based on IFC;

[0033] Figure 4 shows how to handle the topological connectivity between spatial units such as rooms or courtyards.

[0034] FIG5 is a flowchart of a topology connectivity feature extraction program based on adjacency relationships;

[0035] FIG6 is a schematic diagram of a building shape feature extraction method based on geometric contour data;

[0036] FIG7 is a schematic diagram of a method for extracting the plan outline features of each room on each floor of a building BIM model;

[0037] Figure 8 is the algorithm flow for extracting the plane contour features of the building BIM model;

[0038] Figure 9 shows the search intent parsing process based on text segmentation and regular expressions;

[0039] Figure 10 is a flowchart of topological feature embedding in a BIM model based on spatial adjacency and connectivity features;

[0040] FIG11 is a flowchart of a BIM model shape embedding procedure based on contour and plan features;

[0041] Figure 12 is a schematic diagram of the ResNet50 planar shape feature embedding model framework;

[0042] Figure 13 shows the architectural BIM model search process based on comprehensive similarity sorting. DETAILED DESCRIPTION

[0043] To make the purpose, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by ordinary technicians in this field fall within the scope of protection of this application.

[0044] In view of the problem that the existing technology urgently needs to provide a search method for BIM models at the multi-component combination level (typically, such as the overall building level). The technical solution of this application correspondingly provides a building information model search method and device, the method comprising: extracting multimodal features of building information models at the multi-component combination level in the model library to be searched, obtaining multimodal features corresponding to each building information model, the multimodal features including semantic features, topological features and geometric features; obtaining a search text input by the user, parsing the search text, obtaining search intent information corresponding to the search text, the search intent information including search intent semantic features, search intent topological features and search intent geometric features; calculating the comprehensive similarity between the search intent information and the multimodal features of each building information model in the model library to be searched based on deep embedding learning; determining the building information model as the search result corresponding to the search text according to the ranking result of the comprehensive similarity between the search intent information and the multimodal features of each building information model in the model library to be searched, and recommending the search result to the user. This solution realizes the search of building information models at the multi-component combination level.

[0045] Please refer to more drawings of the embodiments of this application, and the method provided by this application will be further explained in more detailed embodiments of this application.

[0046] As shown in FIG1 , the present application provides a building information model search method, the method comprising:

[0047] S1, performing multimodal feature extraction on building information models at the multi-component combination level in the model library to be searched, and obtaining multimodal features corresponding to each building information model, wherein the multimodal features include semantic features, topological features, and geometric features;

[0048] S2, obtaining a search text input by the user, parsing the search text, and obtaining search intent information corresponding to the search text, wherein the search intent information includes search intent semantic features, search intent topological features, and search intent geometric features;

[0049] S3, calculating the comprehensive similarity between the search intent information and the multimodal features of each building information model in the model library to be searched based on deep embedding learning;

[0050] S4, determining a building information model as a search result corresponding to the search text based on a ranking result of comprehensive similarities between the search intent information and multimodal features of each building information model in the model library to be searched, and recommending the search result to the user.

[0051] The specific principles and processes of the above method are described below in more detailed embodiments.

[0052] The model search technology of the prior art can only realize the search of a single component, while the purpose of this application is to realize the search of a model of a multi-component combination. A multi-component combination refers to the combination of components at multiple levels. A typical subset of the multi-component combination model is the building information model at the overall building level. In the application scenario, the overall building, for example, can be a single-family villa in a rural environment, which contains multi-story building space, each floor space can include several different types of rooms, and each room contains different components. The building space can also include open-air courtyards and other spaces. For the sake of convenience of explanation, the subsequent embodiments can be explained by taking the search of the model at the overall building level as an example.

[0053] The above-mentioned processes S1 to S4 of the method of the present application implement two core contents: S1 implements the multimodal feature extraction of the overall building-level BIM model, while S2 to S4 implement BIM model similarity retrieval based on deep embedding learning.

[0054] The two core contents mentioned above are described below respectively.

[0055] Core Content 1: Multimodal feature extraction of BIM models at the overall building level (or other multi-component combination level).

[0056] Multimodal features, including semantic features, topological features, and geometric features.

[0057] Since the existing search algorithms and related deep learning models cannot directly process the file formats of building BIM models such as RVT and IFC. Therefore, the first step in searching the BIM model at the overall building level is to extract information such as attribute text, topology, geometry, etc. in the BIM model. The IFC standard is an open source BIM standard developed by the buildingSMART organization. Various open source parsing tools can be used to extract information from BIM models in IFC format. This application uses Python scripts for data parsing for building BIM models created using Revit software and exported to the IFC4 model format. Specifically, using the IfcOpenShell tool, the "open" and "by_type" methods are used to traverse and obtain components of a specified type in the model. The geometric shape of the building plan, the topological connectivity between rooms, the attribute information of components such as walls, doors and windows, etc. can be extracted from the BIM model, thereby forming an automatic extraction algorithm for multiple types of features of the BIM model.

[0058] For the semantic features of the BIM model at the overall building level or the multi-component combination level, the semantic information of the attributes of the components at each level in the building information model at the overall building level or the multi-component combination level is extracted, and then the semantic information of the attributes of the components at each level is summarized and statistically analyzed to obtain the semantic features of the attributes of the building information model at the overall building level or the multi-component combination level, as shown in Figure 2. Specifically, it includes:

[0059] 1) Extract building space attributes.

[0060] Parse the IfcSpace class that describes each space in the apartment. Get the room name from the base LongName property. Get the specific floor information of the room from the name of the room's associated component. Find the property named "Area" in RelatingPropertyDefinition to obtain the area information. Get the room's location information from IfcSpace.Representation.Representations.SweptArea. Get the room's outline information from BoundedBy. Also, get the partition information between rooms from the ifcRelSpaceBoundary property of BoundedBy.

[0061] 2) Extract the separation properties of walls in space.

[0062] Extract the IfcWall in each room to obtain the wall name and type information in the room. At the same time, search for the properties named "width", "length", "unconnected height", "bottom constraint", and "dimension annotation" from the RelatingPropertyDefinition of IfcWall to obtain all the attribute information of the wall.

[0063] 3) Extract the connectivity properties of door and window components in the wall.

[0064] For each wall, parse the IfcWindow class associated with the wall to obtain the basic information of all windows, and then obtain all the attribute information of the windows through the RelatingPropertyDefinition related to IfcWindow; parse IfcDoor to obtain the basic information of each door, and then read the RelatingPropertyDefinition of IfcDoor to obtain the attribute information of the door.

[0065] 4) Other attribute information that can be extracted when necessary.

[0066] If necessary, the property information of furniture and appliances in the room can be parsed by parsing IfcFurniture in IfcSpace.

[0067] 5) Text processing and summary.

[0068] Preprocess the attribute text, delete invalid information, replace synonyms, standardize text expression, and create multiple dictionaries such as roomSpaceDic, wallDic, doorDic, windowDic, furDic, etc. to store the above attribute semantic information, so as to achieve fast indexing of the corresponding subclass to the IFC original file data.

[0069] 6) Generate attribute semantic feature vector.

[0070] After extracting various attribute information, the BIM model's attributes are summarized according to rules into a specific Python attribute information dictionary, such as "{Province: Beijing, Area: 219, Cost: 600,000, Number of Floors: 3, Number of Rooms: 10, Number of Bedrooms: 4, Number of Bathrooms: 3, Number of Kitchens: 1...}." The information of individual houses can then be processed into feature vectors using agreed parameters for subsequent apartment type queries and matching. Figure 3 shows the overall process.

[0071] For the topological features of the BIM model at the overall building level or the multi-component combination level, the spatial adjacency relationship between each building space is determined according to the attributes of the components at each level, which serves as the topological features of the building information model at the overall building level or the multi-component combination level.

[0072] The topological connectivity feature of the building BIM model refers to the relative position relationship and connection method between the rooms in the building. This application divides the topological relationship of the rooms into two types: whether they are adjacent and whether they are connected, as shown in Figure 4. That is, there are only three possibilities for the relationship between any two rooms: non-adjacent, adjacent and non-connected, and connected. In the BIM model, in addition to walls that can separate rooms, virtual room partitions (Virtual Room Separator) can also separate rooms. This application distinguishes whether the room separation is a physical separation by extracting the PhysicalOrVirtualBoundary attribute in IfcRelSpaceBoundary of IfcSpace. The algorithm flow is shown in Figure 5, and the specific steps are as follows

[0073] 1) Process the spatial relationship between the room IfcSpace and the wall IfcWall and extract the adjacent relationship.

[0074] 1.1) Extract the spatial location of each room

[0075] By extracting the IfcSpace attribute information, we can obtain the spatial information of each room. By extracting the local coordinate position and direction of the room from Representation.SweptArea.Position and the relative coordinates of the contour points, we can calculate the absolute coordinate position of each contour point in each room within the global coordinate system of the entire apartment.

[0076] 1.2) Extract the spatial position outline of each wall in the room space outline.

[0077] By extracting IfcWall, we can obtain the partition information of each room. Based on the IfcSpace.Boundedby of each room, we can extract the IfcWall wall on the room outline. In the ObjectPlacement.RelativePlacement of IfcWall, we can calculate the absolute coordinates of each boundary point of the wall through coordinate transformation.

[0078] 1.3) Determination of room adjacency.

[0079] If two IfcSpace rooms have a common IfcWall boundary and the coordinates of the lines connecting the contour points coincide, then the two IfcSpace rooms can be considered to be adjacent, and the adjacent relationship is stored in the adjacentDic dictionary.

[0080] 2) From the adjacent rooms, process the relationship between the door IfcDoor and the room and wall, and extract the connection information.

[0081] 2.1) Extract the subordination relationship of door components and the corresponding room separation information.

[0082] Extract the names and global numbers of all IfcDoor objects, and then extract the room information separated by the corresponding walls from the ProvidesBoundaries of each IfcDoor object.

[0083] 2.2) Determination of room connection

[0084] If an IfcDoor's ProvidesBoundaries contains two independent IfcSpace rooms, and the RelativePlacement of these two rooms can determine that they have a common IfcWall, and the IfcDoor is located in this IfcWall, then the two IfcSpace rooms connected in the IfcDoor's ProvidesBoundaries are considered to have a connection relationship, and the connection relationship is stored in the accessDic dictionary.

[0085] 3) Verify the connection relationship, organize and store.

[0086] After obtaining the adjacent and connected relationship dictionary, it can be verified that all connected relationships meet the adjacent requirements and that the thickness of walls and doors should meet the specified values, eliminating the influence of information extraction errors and modeling errors. Finally, the dictionary is converted into a Networkx topology map for storage.

[0087] Furthermore, the plane outline information of the building information model is extracted as the geometric features of the building information model at the overall building level or the multi-component combination level.

[0088] Specifically, unlike attribute information and topological relationships, the geometric shape features of a house involve two types of information, two-dimensional and three-dimensional, and it is difficult to directly extract information and abstract features by directly parsing the BIM model file. Due to the complexity of three-dimensional geometric features, this application focuses on analyzing the two-dimensional geometric shape features of the building BIM model. Generally, the geometric features of a house (an instance of the overall building-level BIM model of this application) can be determined by the spatial layout and plane outline information of the building. The former can be directly reflected by the floor plan, while the latter can be obtained by extracting a list of coordinates representing the outline of the building. They both contain rich geometric information, so this application comprehensively considers these two types of information in the task of extracting geometric features, as shown in Figure 6. The specific steps are as follows.

[0089] 1) Extraction of wireframe perspective plan of building BIM model

[0090] 1.1) Wireframe plan capture of building BIM model

[0091] The first-floor plan of the BIM model is intercepted from the Revit software as the overall shape data of the apartment. By directly using the bird's-eye view of the architectural BIM model, fixing the coordinate orientation of the apartment, and intercepting the plan in wireframe format, the floor plan information of the apartment can be obtained.

[0092] 1.2) Preprocessing of wireframe plan images of building BIM models

[0093] Perform image preprocessing on the floor plan, fix the building orientation, and normalize the size by scaling and storing it.

[0094] 2) Extraction of plane contour features of building BIM models

[0095] 2.1) Extracting room outline information from building BIM models

[0096] For the geometric outline information of each room in the building BIM model, IfcOpenShell is used to extract the local coordinates of Representation.SweptArea.Position of each room in the apartment to form a room outline information list.

[0097] 2.2) Conversion of overall outline information of building BIM model

[0098] Through the outline information list of each room and the local coordinate system information, the global coordinates of the overall outline points are calculated, and the overall plane outline data list of the building BIM model is formed to construct the outer contour data of the entire building, as shown in Figure 7.

[0099] 2.3) Normalized storage of contour features

[0100] The coordinate list of the outer contour points of the entire building is fitted into an outer contour polygon with a fixed number of contour points, so that the outer contour point information is uniformly stored using a fixed-length vector. The above overall algorithm flow is shown in Figure 8.

[0101] Core Content 2: BIM model similarity retrieval based on deep embedding learning.

[0102] After extracting multimodal features such as attributes, topology, and geometry from the building BIM model, searching for text in BIM models requires calculating the similarity between the search text and the BIM model. Therefore, this section first uses natural language processing (NLP) text segmentation and regular expression techniques to parse the search text to extract the search intent contained in the natural text. Secondly, using deep learning and special engineering tools, the multimodal features of the BIM model and the search intent are embedded in the search text and converted into a unified feature vector. Finally, a weighted comprehensive similarity calculation is performed based on the feature vector, and intelligent search and recommendation of BIM models is achieved based on similarity sorting. The following three sections describe the detailed steps.

[0103] (1) Search intent analysis based on text segmentation and regular expressions

[0104] The purpose of extracting search intent is to obtain the semantic attributes, room topology, geometric description and other information of the required BIM model from the text information, and then use this information to query the target model. The process is shown in Figure 9, and the specific steps are as follows.

[0105] 1) Define the vocabulary of architectural BIM model domain

[0106] Descriptive nouns and common expressions related to the housing and real estate fields were collected and compiled to create a domain word list for architectural BIM model searches, which helps segment text and define the smallest unit for proper noun subdivision.

[0107] 2) Domain text segmentation based on the Jieba library GRU neural network model

[0108] Jieba is a commonly used open source Chinese segmentation application package in Python. Here, the search text is segmented based on the domain vocabulary and Jieba. Jieba's paddlepaddle mode is used, based on a gated recurrent network, and according to the domain vocabulary, the search sentence is segmented into segments with word attributes and word order to facilitate subsequent attribute and conjunction matching.

[0109] 3) Search intent analysis based on regular expressions

[0110] Manually defined regular expressions are used to extract descriptions of semantic, topological, and geometric features in search intent. Some regular expression styles are shown in Table 1. The specific steps are as follows.

[0111] Table 1

[0112] 3.1) Search intent semantic feature analysis based on regular expressions

[0113] For the attributes and semantic information of the required BIM model, various descriptive clauses in the search statement are extracted, and keywords such as nouns, verbs, and prepositions are matched according to the syntactic structure. For example, when "area" and "is" appear, the number after "is" is extracted as the area of ​​the house for search. Finally, the corresponding attributes and semantics are summarized into a dictionary.

[0114] 3.2) Regular Expression-Based Search Intent Topology Feature Parsing

[0115] Using a method similar to 3.1), we extract the topological connectivity features of the room. For example, for the sentence “living room connected to kitchen”, we can extract the keywords “living room” and “kitchen” representing the room names, as well as the word “connected” representing its connectivity, thereby extracting semantic information related to topological connectivity. Finally, we summarize it into a connectivity dictionary and a NetworkX network diagram.

[0116] 3.3) Regular Expression-Based Geometric Feature Parsing of Search Intent

[0117] For shape features, fuzzy shape descriptions such as "square apartment" and "long and narrow bedroom" are extracted from the search text to generate a list of corresponding contour features, including parameters such as the aspect ratio of the circumscribed rectangle, and finally summarized into a shape attribute dictionary.

[0118] 4) Search intent analysis summary

[0119] Finally, the various feature dictionaries extracted by regular expressions are summarized to facilitate subsequent searches.

[0120] (2) Multimodal feature vectorization representation based on deep embedding learning

[0121] After extracting the semantic, topological, and geometric feature information of the BIM model and the semantic, topological, and geometric features of the search intent, they need to be embedded into a unified vectorized representation for similarity calculation. In general, the semantic information of each attribute is directly organized into a vector consisting of text and data in a unified format for embedding semantic features. Topological information is stored in the form of a Networkx topological graph, and the graph kernel method is used to embed topological features. For the embedding of geometric features, the floor plan image is embedded through a convolutional neural network, the contour information is embedded through the characteristic parameters of the contour, and the geometric shape description in the search text is also embedded as the characteristic parameters of the house contour. The specific methods include:

[0122] 1) Embedding of semantic features based on the combination of artificial rules and Word2Vec neural network

[0123] The multimodal feature extraction method of the BIM model in Section 5.2.1 can extract the semantic attributes of the BIM model as "{Province: Jiangsu, Area: 149 square meters, Cost: US$100,000, Number of floors: 2, Number of bedrooms: 3, Number of bathrooms: 2, Number of living rooms: 1, Number of kitchens: 1...}". The search intent extraction method in this section (1) can also generate an attribute dictionary of the corresponding form. Word2Vec is a type of neural network model commonly used in natural language processing tasks to convert words into word basis vectors. This application uses manually defined conversion rules and combines the Word2Vec neural network method to embed the attribute dictionary into the semantic feature vector. The processing method refers to Table 2. The specific steps are as follows:

[0124] Table 2

[0125] 1.1) Standardization of synonyms

[0126] Specifically, for near synonyms that may exist widely in semantic dictionaries, such as "bathroom", "washroom", "sanitation", "toilet", etc., synonym replacement is performed by manually defining a list of near synonyms and unifying their descriptions into the same descriptive dictionary.

[0127] 1.2) Attribute Data Feature Embedding Based on Artificial Rules

[0128] For data information in attributes, detailed comparisons can be made directly through numerical differences, making embedding relatively simple. After weighting and normalizing the data attributes, manually defined rules are used to embed different types of data according to their type characteristics.

[0129] 1.3) Attribute semantic feature embedding based on Word2Vec

[0130] For semantic information in attributes, the text is typically string attributes and needs to be embedded as string vectors before being embedded using the Word2Vec model. Specifically, the Word2Vec model is implemented using the Gensim package (a popular Python library for natural language processing). Based on the synonym list defined in 1.1), a corresponding text corpus was constructed as a training set. This corpus was used to train the Word2Vec model using the training parameters "vector_size = 100, window = 5, min_count = 1, and workers = 4."

[0131] 1.4) Feature Embedding of Mixed Data and Semantics

[0132] Other types of semantic information are divided into text information and data information according to rules, embedded as string word vectors and data vectors respectively, and then weighted and combined as feature vectors. The corresponding weights are manually determined by experts.

[0133] 2) Topological feature embedding based on graph neural network

[0134] The spatial topological relationships previously extracted from BIM models and search text can be converted into NetworkX topological graphs for storage. Graph kernels are an effective method for embedding graph features in graph neural networks. Therefore, we chose the Deepwalk deep random walk method based on graph kernels to embed topological features. This method is unsupervised, has better transferability, and is more suitable for the graph data used in this application. The program flow is shown in Figure 10, and the specific embedding method is as follows.

[0135] 2.1) Converting topology graphs into network graphs based on Networkx and Grakel

[0136] The NetworkX library is a popular open-source Python library for storing and sharing graph data, while Grakelze is a popular open-source Python library for graph kernel machine learning tasks in graph convolution. This implementation is based on NetworkX and Grakel. Based on the node list, node attribute list, and adjacency matrix of a NetworkX topology, the Grakel.graph method is used to convert a NetworkX topology into a Grakel graph network.

[0137] 2.2) Network Graph Feature Vectorization Based on Random Walk Kernel

[0138] By fixing the same Graph Kernel in the form of random walk, the Grakel graph network is embedded into a topological feature vector based on the Grakel.GraphKernel method. Then, weighted normalization is performed to obtain the BIM model topological feature vector that measures the similarity of the overall connectivity relationship of the apartment types.

[0139] 3) Geometric shape feature embedding based on contour visual features and convolutional neural network

[0140] Previously, the geometric shape information of BIM models was processed separately, dividing it into outline data and floor plan data. Therefore, these two types of features were embedded separately, complementing each other. The geometric outline extracted from the BIM model not only contains the overall unit's external contour information but also the planar location information for each room within the unit. Therefore, shape description features can be directly constructed based on the unit's complete planar contour information, allowing the unit's shape to be vectorized. Simultaneously, a convolutional neural network is used to extract feature vectors of the unit's shape from the floor plan, eliminating the influence of noise and transformations associated with individual outline features and allowing the unit's overall floor plan shape to be vectorized. The embedding process is shown in Figure 11, and the specific embedding method is as follows.

[0141] 3.1) Geometric contour feature extraction based on contour visual features

[0142] The OpenCV computer vision toolkit was used to extract features from the previously extracted BIM model outline information. This outline information was used to calculate indicators describing the apartment layout and its internal room shapes, including the aspect ratio of the outer enveloping rectangle, the ratio of the inner area of ​​the outline to the outer enveloping rectangle, the coordinates of the apartment's centroid, and the coordinates of each room. This data was then normalized to form apartment shape feature description indicators that describe the geometric outline features, as shown in Table 3. Specifically:

[0143] Table 3

[0144] 3.1.1) Fitting and embedding of the outer contour coordinates of the building BIM model

[0145] ApproxPolyDP polygon fitting method is used to fit the polygon contour points to 30, and the outer contour of the apartment is fitted into a 30-sided polygon. The outer contour coordinates are normalized to form a fixed-length contour point vector, thus embedding the outer contour information.

[0146] 3.1.2) Embedding the aspect ratio feature of the building BIM model outline

[0147] The cv2.boundingRect method is used to calculate the circumscribed rectangle of the outline. The aspect ratio of the circumscribed rectangle is used to measure the squareness of the plane shape of the entire building BIM model and is embedded in the form of a floating-point number.

[0148] 3.2.3) Building BIM model outline image data feature embedding

[0149] The image moment of the apartment outline is calculated using the cv2.moments method, so that the center of mass, moment of inertia, third-order moment and other features of the building BIM model can be represented from the image moment.

[0150] 3.2.4) Building BIM model and ellipse feature embedding

[0151] At the same time, considering that there are actually some narrow and long apartments, the shape features of the apartments are embedded by fitting an ellipse. The cv2.fitEllipse method is used to obtain the ellipse that is closest to the building outline. The major and minor axis vector features of the ellipse are extracted and embedded in the form of a fixed-length vector.

[0152] 3.2.5) Embedding of spatial shape features within the building BIM model

[0153] For the shapes of subdivided spaces such as rooms and courtyards within the building BIM model, their squareness can be measured by the average aspect ratio of the circumscribed rectangle of such rooms in the apartment. The average aspect ratio information of each bedroom, living room, kitchen, and courtyard is calculated separately using the cv2.boundingRect method to form the aspect ratio features of the four types of subdivided spaces and embed them in the form of floating-point numbers.

[0154] 3.2) Embedding of Planar Graph Data Information Based on Convolutional Neural Networks

[0155] For the feature extraction of the floor plan of the building BIM model, this application introduces feature engineering technology based on convolutional neural networks, and uses a pre-trained ResNet50 model to extract the overall geometric features of the plane, as follows.

[0156] 3.2.1) Convolutional Neural Network Architecture Selection

[0157] In the selection and construction of the convolutional neural network model, the ResNet50 neural network was selected, and the input and output layer structure of the neural network was adjusted. The input was unified into a three-channel image in the format of 224×224, and the output was unified into a 2048-dimensional image shape feature vector. The neural network structure is shown in Figure 12.

[0158] 3.2.2) Embedding of floor plan data based on pre-trained models

[0159] The widely used ImageNet is used to pre-train the ResNet50 network structure. The model is generated based on the CheckPoint weights pre-trained on the ImageNet dataset. After running and extracting the fully connected layer data, the 2048-dimensional image shape feature vector of the plane image can be output.

[0160] 3.3) Embedding and integration of building BIM model outline and floor plan features

[0161] The above floor plan feature vector is weighted and combined with the building BIM model's outline feature index to form a shape feature vector that integrates both the apartment outline and the floor plan shape features. The weights for embedding these two features were manually determined by experts and manually adjusted three times based on the results.

[0162] (3) Architectural BIM model search based on comprehensive similarity ranking

[0163] Previously, we implemented a unified vectorized embedding method for BIM model semantics, topological and geometric multimodal features, and search intent information. To enable searches from text to BIM models, this section calculates a comprehensive similarity based on the BIM model's feature vectors and search intent, and then retrieves BIM models using weighted cosine similarity sorting. The overall process is shown in Figure 13, and the specific method is as follows.

[0164] 3.1) Determination of Similarity Evaluation Indicators

[0165] By using the weighted cosine similarity of feature vectors, it is easy to compare the similarity between BIM models and models, and between text and BIM models. Then, intelligent retrieval of BIM models can be achieved by sorting the similarity. However, the elements of feature vectors have different compositions, sizes, and meanings. Therefore, it is necessary to use appropriate similarity evaluation indicators to calculate the similarity of different elements of the vectors. Several typical similarity indicator processing methods can be referred to in Table 4, as shown below:

[0166] Table 4

[0167] 3.1.1) Similarity index of string elements

[0168] For string elements, such as locations and room names, character matching rate is used as the similarity metric; for floating-point and integer quantized elements, such as bedroom area and number, relative difference is used to measure similarity.

[0169] 3.1.2) Similarity index of sub-vector elements

[0170] For sub-vector elements with specific meanings, such as word vectors of topological graphs and fitted rectangle features, shape moments, etc., the similarity is also measured by taking the cosine similarity of the sub-vectors.

[0171] 3.1.3) Similarity index of coordinate list elements

[0172] For the fitted outer contour coordinate list in the shape feature, the OpenCV cv2.matchShapes method is introduced to compare the similarity of the contours by the relative difference of the Hu moment of the contours.

[0173] 3.2) Determination of the integrated weights of each detailed indicator

[0174] In addition to similarity metrics, another major focus of calculating comprehensive similarity is to determine the weight of each feature type. Considering the size of the search dataset, this study uses expert manual feedback adjustment to establish the weights, as follows.

[0175] 3.2.1) Determine the initial weight

[0176] The initial weights are first determined manually by experts and then sorted in descending order according to the comprehensive similarity to form a list of BIM model search results.

[0177] 3.2.2) Weight adjustment based on human feedback

[0178] The weight of the comprehensive similarity is adjusted through feedback from experts who manually evaluate the search results. After more than 10 rounds of manual comparison and adjustment, the comprehensive similarity results meet the evaluation requirements of the corresponding experts.

[0179] 3.2.3) Determine the final weight and form a similarity ranking algorithm

[0180] The semantic feature weight percentage is approximately 88%, the topological feature weight is 5%, and the geometric feature weight is 7%. By sorting the weighted feature vectors by cosine similarity, a corresponding BIM library search and sorting algorithm is formed. This has the advantage of fast calculation speed and the search effect can be optimized by adjusting the weights.

[0181] In summary, the method of this application achieves the following effects:

[0182] (1) The semantic-topological-geometric multimodal feature search of BIM models at the overall building level or multi-component combination level is realized, supporting a large model scale and a wide search range. Based on IfcOpenShell, the semantic features, topological connectivity features, and shape features of the overall building BIM model are extracted and deeply embedded. For the first time, the model scale applicable to the BIM model search algorithm is expanded to the overall building level. At the same time, the similarity search of the three types of semantic-topological-geometric features from text to building BIM models is realized for the first time.

[0183] (2) Under the premise of ensuring (1), a good search effect is achieved and the accuracy is significantly improved. Deep embedded learning is introduced, and the "semantic-topological-geometric" multimodal features are embedded to develop an intelligent retrieval algorithm for architectural-level BIM atlases based on comprehensive similarity. The corresponding intelligent search method has been well applied and verified in typical engineering projects. In terms of search accuracy and search ranking effect, the mNDCG1 and mNDCG5 indicators of the algorithm in the test set both reached about 90%, achieving a retrieval quality and retrieval effect that is almost the same as that of expert manual retrieval. Compared with the traditional retrieval algorithm that only considers semantics, the search effect is significantly improved by more than 6.5%, allowing good atlases to be found not only accurately, but also accurately, completely and well.

[0184] (3) Under the premise of ensuring the search quality of (1) and (2), the algorithm operation efficiency is optimized and the search speed is guaranteed. In the test set, under the premise of ensuring the retrieval effect, the average time taken by three experts to retrieve 50 search statements is 1740 seconds, while this application can achieve retrieval quality similar to that of expert retrieval. At the same time, the algorithm can run smoothly on an ordinary civilian notebook, and the algorithm only takes 2.23 seconds (the hardware environment is M1 Pro, 16GB memory). The retrieval efficiency is hundreds of times higher than that of manual retrieval, and the total time is shortened by nearly 3 orders of magnitude. Therefore, under the application environment of ensuring retrieval quality, this application can significantly reduce manual labor, allowing experts and designers to focus more on more difficult manual work.

[0185] Another aspect of the present application provides a building information model search device, comprising:

[0186] A feature extraction module is used to extract multimodal features from the building information models at the overall building level or the multi-component combination level in the model library to be searched, and obtain multimodal features corresponding to each building information model, wherein the multimodal features include semantic features, topological features, and geometric features;

[0187] A parsing module is used to obtain a search text input by a user, parse the search text, and obtain search intent information corresponding to the search text, wherein the search intent information includes search intent semantic features, search intent topological features, and search intent geometric features;

[0188] A similarity calculation module is used to calculate the comprehensive similarity between the search intent information and the multimodal features of each building information model in the model library to be searched based on deep embedding learning;

[0189] The recommendation module is used to determine the building information model as the search result corresponding to the search text based on the sorting result of the comprehensive similarity between the search intention information and the multimodal features of each building information model in the model library to be searched, and recommend the search result to the user.

[0190] The present application also provides a computer-readable storage medium, which includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above method. The specific implementation process will not be repeated here.

[0191] The present application also provides a computer device. This embodiment of the computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor. When the processor executes the computer program, the aforementioned method of the embodiment is implemented. To avoid repetition, a detailed description thereof is omitted here. Alternatively, when the processor executes the computer program, the functions of each model / unit in the apparatus of the embodiment are implemented. To avoid repetition, a detailed description thereof is omitted here.

[0192] A computer device may be a desktop computer, laptop, PDA, server, or cloud server, among other computing devices. A computer device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that a computer device may include more or fewer components than shown, or a combination of certain components, or different components. For example, a computer device may also include input / output devices, network access devices, and buses.

[0193] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0194] Memory can be an internal storage unit of a computer device, such as a computer device's hard drive or memory. It can also be an external storage device, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, or a flash memory card. Furthermore, memory can include both internal storage units and external storage devices. Memory is used to store computer programs and other programs and data required by the computer device. Memory can also be used to temporarily store data that has been output or is about to be output.

[0195] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0197] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform some steps of the above-mentioned method in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0198] The above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A building information model search method, characterized in that: The method comprises: Extracting multimodal features of building information models at the multi-component combination level in the model library to be searched, and obtaining multimodal features corresponding to each building information model, wherein the multimodal features include semantic features, topological features, and geometric features; Obtaining a search text input by a user, parsing the search text, and obtaining search intent information corresponding to the search text, wherein the search intent information includes search intent semantic features, search intent topological features, and search intent geometric features; Based on deep embedding learning, calculating the comprehensive similarity between the search intention information and the multimodal features of each building information model in the model library to be searched; According to the ranking result of the comprehensive similarity between the search intention information and the multimodal features of each building information model in the model library to be searched, the building information model as the search result corresponding to the search text is determined, and the search result is recommended to the user.

2. The building information model search method according to claim 1, characterized in that: The multi-modal feature extraction of the building information model at the multi-component combination level in the model library to be searched includes: The semantic information of the attributes of the components at each level in the multi-component combination level building information model is extracted, and then the semantic information of the attributes of the components at each level is summarized and counted to obtain the semantic features of the attributes of the multi-component combination level building information model.

3. The building information model search method according to claim 2, characterized in that: The components at each level include: building space, walls that may be included in the building space, and doors or windows that may be included in the walls.

4. The building information model search method according to claim 3, characterized in that: The method extracts multimodal features from a building information model at a multi-component combination level in a model library to be searched, and further includes: According to the properties of components at each level, the spatial adjacency relationship between various building spaces is determined as the topological feature of the building information model at the multi-component combination level.

5. The building information model search method according to claim 4, characterized in that: The spatial adjacency relationship includes three relationships: non-adjacent, adjacent and non-connected, and connected.

6. The building information model search method according to claim 2, characterized in that: The multi-modal feature extraction of the building information model at the multi-component combination level in the model library to be searched also includes: The plane outline information of the building information model is extracted as the geometric features of the building information model at the multi-component combination level.

7. The building information model search method according to claim 1, characterized in that: The parsing of the search text to obtain search intent information corresponding to the search text includes: Based on natural language processing text segmentation and regular expression parsing, the search intent semantic features, search intent topological features and search intent geometric features of the search intent information are obtained.

8. The building information model search method according to claim 7, characterized in that: The method of calculating the comprehensive similarity between the search intention information and the multimodal features of each building information model in the model library to be searched based on deep embedding learning includes: According to the extracted semantic features, topological features and geometric features of the building information model, as well as the search intent semantic features, search intent topological features and search intent geometric features of the search intent, they are embedded into a unified vectorized representation for similarity calculation.

9. The building information model search method according to claim 8, characterized in that: The similarity calculation is weighted cosine similarity calculation.

10. A building information model search device, characterized in that: include: A feature extraction module is used to extract multimodal features from the building information models at the multi-component combination level in the model library to be searched, and obtain multimodal features corresponding to each building information model, wherein the multimodal features include semantic features, topological features and geometric features; A parsing module, used to obtain a search text input by a user, parse the search text, and obtain search intent information corresponding to the search text, wherein the search intent information includes search intent semantic features, search intent topological features, and search intent geometric features; A similarity calculation module, used for calculating the comprehensive similarity between the search intention information and the multimodal features of each building information model in the model library to be searched based on deep embedding learning; The recommendation module is used to determine the building information model as the search result corresponding to the search text according to the sorting result of the comprehensive similarity between the search intention information and the multimodal features of each building information model in the model library to be searched, and visually display the search result to the user.

11. A computer-readable storage medium, characterized in that: A computer program is stored, and the computer program is executed by a processor to control the device where the processor is located to implement the following building information model search method: Extracting multimodal features of building information models at the multi-component combination level in the model library to be searched, and obtaining multimodal features corresponding to each building information model, wherein the multimodal features include semantic features, topological features, and geometric features; Obtaining a search text input by a user, parsing the search text, and obtaining search intent information corresponding to the search text, wherein the search intent information includes search intent semantic features, search intent topological features, and search intent geometric features; Based on deep embedding learning, calculating the comprehensive similarity between the search intention information and the multimodal features of each building information model in the model library to be searched; According to the ranking result of the comprehensive similarity between the search intention information and the multimodal features of each building information model in the model library to be searched, the building information model as the search result corresponding to the search text is determined, and the search result is recommended to the user.

Images