Representation learning of model data

Abstract

A method and system provides the ability to perform predictive tasks utilizing a three-dimensional (3D) model. A plurality of 3D models consisting of non-Euclidean data are obtained. Each 3D model is converted into a relational graph having nodes and edges. Each relational graph is processed using a graph neural network (GNN) that computes a node representation for each node. The node representations are aggregated into a structural representation of the 3D model. A plurality of different views of the 3D model are captured and passed through a convolutional neural network (CNN) to compute a view representation for each view. The view representations are aggregated into a single visual representation. The GNN and CNN are trained using a multi-view contrast training objective to maximize consistency between the structural representation and the single visual representation in order to form a final learned representation. The prediction task is performed using the final learned representation.

Description

Background Art

[0001] The present invention relates generally to two-dimensional (2D) and three-dimensional (3D) building information models (BIMs), and more particularly to methods, devices, systems, and articles of manufacture for implementing deep learning of BIMs. Such deep learning includes processing a smaller set of exemplary BIMs to identify and cluster data and information (of such exemplary BIMs) to provide an expressive representation of the BIM model. The expressive representation implements intelligent features such as clustering, classification, recommendation, or search / retrieval of BIM models.

[0002] 2. Related technical description

[0003] (NOTE: This application references a number of different publications, as indicated throughout this specification by reference numbers enclosed in square brackets, such as [x]. A list of these different publications, sorted by these reference numbers, can be found below in the section entitled "References". Each of these publications is incorporated herein by reference.)

[0004] Building Information Modeling (BIM) software, such as REVIT BIM software available from the patentee of the present invention, helps architecture, engineering, and construction (AEC) teams create high-quality buildings and infrastructure. BIMs are very complex and contain information about the structure and appearance of elements in the model (e.g., metadata, etc.). When designing a new building, users search through a database of previously designed BIM models to identify relevant content and / or find similar designs that can charge templates / foundations. This prior art search process is slow and inefficient. What is needed is a fast and efficient mechanism to search and / or identify elements and designs that can be used in the design process. To better understand these issues, a description of prior art BIM applications and limitations may be helpful.

[0005] BIM software can: model shapes, structures, and systems in three dimensions (3D) with parametric accuracy, precision, and ease of use; simplify documentation while instantly revising plans, elevations, schedules, and sections as the project changes; and empower multidisciplinary teams with professional toolsets in a unified project environment. BIM models contain rich information about the semantic, geometric, and visual properties of the building model designed by the architect. In other words, BIM models are complex and contain irregular data structures that can be analyzed from different perspectives such as appearance, structure, semantics, etc.

[0006] It is expected to generate and search BIM using deep learning. Deep learning is a machine learning technique that teaches computers to do what comes naturally to humans: learn by example. In deep learning, data is organized into layers, and the input data (e.g., illustrative images) is transformed into slightly more abstract and complex representations at each layer. The term "depth" in deep learning refers to the number of layers to which the data is transformed. By processing the data at each layer / level, the deep learning process can learn by itself which features are optimally placed at which level. Unfortunately, the irregular data structure and complexity of BIM make BIM incompatible with regular deep learning architectures that expect well-defined input structures. In this regard, it is not an easy task to query information about BIM models or feed such models into deep models for prediction tasks. In addition, BIM models are treated as independent files, and rich patterns across models are ignored.

[0007] To better understand the problems of the prior art, it may be helpful to describe deep learning of images. Suppose it is desired to classify images of cats as cats (e.g., rather than dogs). The system can process cat images through a deep convolutional network (DCN) (also known as a deep convolutional neural network [DCNN] or a convolutional neural network [CNN]). A DCN / DCNN / CNN is a class of artificial neural networks that analyze visual images and use convolution in at least one of the layers. To process new images, several exemplary images of cats and dogs are input into the DCN, which then learns representations. In this regard, based on the DCN processing, a set of numbers (e.g., [0.7, -0.1, 0.4, 0.3, -0.4, -0.1, -0.3]) is assigned to represent the image (e.g., transforming / converting different features / attributes of the image into numerical representations).

[0008] Similar processing can be done for objects / elements other than images (e.g., product reviews). To process such other objects / elements, networks other than DCN can be utilized. For example, deep learning can utilize long / short term memory (LSTM) networks / systems to classify product reviews as good reviews, very good reviews, bad reviews, or very bad reviews.

[0009] Although DCN and LSTM can be used for deep learning for standard images, BIM models are complex. For example, a BIM model may have text, floor plans, structural information, electrical information, mechanical information, etc. It is expected that the BIM model will be classified / classified as residential or commercial, but due to complexity, neither the DCN network nor the LSTM network can be used. In this regard, standard images or texts that can be classified by the DCN or LSTM network include Euclidean / structured data. In contrast, the BIM model contains non-Euclidean or irregular structured data. Non-Euclidean / irregular structured data may exist in a variety of different environments, such as in 3D solid models, chemical structures, graphic structures, 3D building models, etc. However, due to these irregularities, these irregular structured data environments cannot be processed by traditional deep learning models.

[0010] In view of the above, the existing techniques fail to provide the ability to feed BIM models into deep learning models to learn expressive representations of such models. To provide such an ability, the following problems need to be addressed: (1) how to construct BIM models so that they can be fed into deep models; (2) which aspects of BIM models should be considered; and (3) which deep architectures are best suited for BIM models. Summary of the invention

[0011] To overcome the problems of the prior art, embodiments of the present invention use a combination of preprocessing steps (including converting the BIM model into a graph structure and a 2D structure) and the latest advances in deep learning (including self-supervised learning and multimodal learning) to map the BIM model into a dense vector representation using a stack of deep models. Learning such expressive representations on the BIM model enables intelligent features such as clustering, classification, recommendation, or retrieval of the BIM model.

[0012] In other words, one of the goals of embodiments of the present invention is to learn task-independent representations of BIM models (e.g., sets of numbers) from limited input data (i.e., having limited access to illustrative input examples) so that the learned representations can be adapted to new tasks based on a small set of training samples (e.g., the few examples on which the training is based). BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Referring now to the drawings, wherein like reference numerals represent corresponding parts, in which:

[0014] Figure 1 shows a sample building information model (BIM) for training a model according to one or more embodiments of the present invention;

[0015] Figure 2 shows a structural representation of a BIM model according to one or more embodiments of the present invention;

[0016] Figure 3 shows an exemplary structural representation according to one or more embodiments of the present invention;

[0017] Figure 4 An exemplary complex structural representation of a BIM model according to one or more embodiments of the present invention is shown;

[0018] Figure 5 A method for Figure 1 The distribution of the number of nodes and edges of the graph;

[0019] Fig. 6A and Figure 6B Exemplary extracted element species according to one or more embodiments of the present invention are shown ( Fig. 6A ) and relationship types ( Figure 6B ) frequency;

[0020] Figure 7 A process for generating a graph representation according to one or more embodiments of the present invention is shown;

[0021] Figure 8 A logical flow for capturing a visual representation of a BIM model according to one or more embodiments of the present invention is shown;

[0022] Fig. 9 shows a multimodal representation consisting of a combination of a structural representation and a visual representation according to one or more embodiments of the present invention;

[0023] FIG. 10A to FIG. 10C An exemplary process for transfer learning according to one or more embodiments of the present invention is shown;

[0024] Fig.11 An exemplary general implementation of a training pipeline using AMAZON WEB SERVICES according to one or more embodiments of the present invention is shown;

[0025] Fig.12 A more specific implementation of a training pipeline using a graph neural network (GNN) according to one or more embodiments of the present invention is shown;

[0026] Fig.13 An exemplary GNN reasoning pipeline for classifying and reasoning about properties of a BIM model using a GNN according to one or more embodiments of the present invention is shown;

[0027] Fig.14 A logical flow for utilizing a 3D model to perform a prediction task according to one or more embodiments of the present invention is shown;

[0028] Fig.15 is an exemplary hardware and software environment for implementing one or more embodiments of the present invention; and

[0029] Fig.16 A typical distributed / cloud-based computer system according to one or more embodiments of the present invention is schematically illustrated. DETAILED DESCRIPTION

[0030] In the following description, reference is made to the accompanying drawings which form a part hereof and in which are shown by way of illustration several embodiments of the invention. It should be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.

[0031] Overview

[0032] BIM models contain rich information about the semantic, geometric and visual properties of building models designed by architects. However, querying information about them or feeding them into deep models for prediction tasks is not an easy task. In addition, BIM models are treated as independent files, and the rich patterns of cross-models are ignored. To solve these problems, embodiments of the present invention represent BIM models as directed and attributed multigraphs (also called relationship graphs). Once this conversion is completed, the graphs can be fed into graph neural networks (GNNs) to classify them, predict some of the missing attributes, cluster them, or recommend them. Providing such features to BIM models can enhance the user experience.

[0033] Generally speaking, self-supervised learning is used to generate task-independent representations of BIM models (i.e., self-supervision enables learning without human-based annotations / input). Multimodality is then used to augment multi-view data (i.e., using multiple different aspects of the BIM model simultaneously / at the same time). Relational inductive bias is used for sample efficiency (i.e., pre-existing relationships present in the BIM model can be exploited), while transfer learning is used to provide fast adaptation (i.e., learning from one trained model / task can be transferred to another - e.g., from a model representing a particular part to a cost model [e.g., a model for calculating the cost of physical construction of a model in the real world]).

[0034] method

[0035] Overview

[0036] In order to process BIM models and feed them into deep models, embodiments of the present invention consider two modalities to represent the models. The first modality represents the BIM model as a relational structure (i.e., a directed property graph), with the goal of capturing the underlying structure of the BIM model. The second modality captures the appearance of the BIM model. The first modality is fed into a graph neural network (GNN) to compute the structural representation of a given model, and the second modality is fed into a multi-view convolutional neural network (MVCNN) [1] to compute the visual representation of a given model. In order to train the GNN model and the MVCNN model, embodiments of the present invention use a multi-view comparison training objective to maximize the consistency between the structural representation and the visual representation of the same BIM model, and drive the structural representation and the visual representation of different BIM models to separate. Once the training is completed, embodiments of the present invention can treat the learned representation as the representation of the original BIM model and use it for downstream tasks such as classification, regression, or similarity search across BIM models.

[0037] Illustrative Datasets

[0038] Embodiments of the present invention utilize 360 ​​BIM models of Japanese houses. These models are one-story or two-story Japanese toy houses. Some sample models are as follows: Figure 1 shown.

[0039] Structural representation

[0040] Figure 2 A structural representation of a BIM model according to one or more embodiments of the present invention is shown. In order to calculate the structural representation of the BIM model 202, an embodiment of the present invention first converts each BIM model 202 into a relationship graph 204, such that the nodes 206 correspond to elements within the BIM model 204, and the edges 208 correspond to the relationship types between pairs of nodes 206. An embodiment of the present invention attributes each graph by extracting node attributes 210 (e.g., the ID, label, type, kind, bounding box, and orientation of each element) within a given BIM model 202 and assigning dedicated nodes 206 to the elements having these attributes. Similarly, the graph attribute assignment process includes extracting / defining edge attributes 212, which define the relationship between each node in the nodes 206 (e.g., made of...). The resulting graph 204 provides a conceptual view of the BIM model 202.

[0041] Figure 3An illustrative structural representation according to one or more embodiments of the present invention is shown. As shown, two nodes 302A and 302B have their node attributes 304A and 304B extracted / defined. Node 304A has an ID "-352753250424429328", a label "magnetic", a type "instance", a category "material", a bounding box defined by [[0.0,0.0,0.0], [0.0,0.0,0.0]], and an orientation [0.0,0.0,0.0]. Node 304B has the following node attributes: ID - "8454925988329030555"; label - "Bathtub - Rectangular - 1525×762 mm"; type - "Instance"; kind - "Plumbing Fixture"; bounding box - [[-4.46,-7.02,0.0],[-1.96,-2.02,1.25]]"; and orientation - "[-1,0,0]". In addition, an edge attribute 306 has been defined / extracted and provides (via the edge attribute relationship) that node 304B is "made of" node 304A (i.e., the bathtub represented by node 304B is made of the material represented by node 304A). The extracted relationship is visually represented in the structural representation via an arrow pointing from node 302B to node 302A. Thus, Figure 3 A bathtub plumbing arrangement is graphically provided, represented by node 302B, which is connected to node 302A specifying the material the bathtub is made of (ie, magnetic).

[0042] Figure 3 The graphic provides a simple example of a structural representation of a BIM model. Many BIM models may be significantly more complex. Figure 4 An exemplary complex structural representation 404 of a BIM model 402 is shown. As can be seen, the graph 404 is very dense, with nodes and edges overlapping.

[0043] Once all BIM models (e.g. Figure 1 All examples in the BIM model are converted into their graphical representations, and an embodiment of the present invention will have a graphical representation of each BIM model. Figure 5 A method for Figure 1 The distribution of the number of nodes 502 and edges 504 of the graph of FIG. The vertical axis represents the frequency (per graph) and the horizontal axis represents the number of nodes / edges (per graph).

[0044] To define relationships / relationship types (ie, edges 208 / 306), a set of heuristics may be utilized. Fig. 6A and Figure 6B Exemplary extracted element species according to one or more embodiments of the present invention are shown ( Fig. 6A ) and relationship types ( Figure 6B) frequency. In one or more embodiments, a set of heuristic methods is used to derive Figure 6B Eighteen (18) exemplary relationships are specified. Figure 6B The relationship is explained by the relationship obtained:

[0045] ●Element A is made of material M MADE-OF

[0046] ●Element A WITHIN-ROOM R

[0047] ●Element A INVOLVED_IN_PHASE P

[0048] ●Element A INVOLVED_IN_LEVEL L

[0049] ●Element A and Element B IN_SAME_ROOM_AS

[0050] ●Element A is HOSTED_BY element B

[0051] In such relationships, "element" can be replaced by any derived element kind (e.g. Fig. 6A The element type identified.

[0052] Once the BIM model has been converted to a graph structure, GNN (Graph Neural Network) can be used to compute the representation of each node in each graph. To compute this representation, GNN learns representations based on connectivity and node / edge attributes. Strong connections induce bias and produce sample efficiency.

[0053] The node representations can then be aggregated into a graph-level representation, which is essentially a structural representation of the input BIM model. Embodiments of the present invention may utilize a Graph Isomorphic Network (GIN) [2] layer to construct a GNN.

[0054] Figure 7 A process for generating a graph representation according to one or more embodiments of the present invention is shown. As shown, initial attributes 702 are input and used / processed by a GNN 704 (which can be implemented / built using GIN) to generate a node representation 706. The GNN 704 can process the initial attributes 702 based on the following equation:

[0055]

[0056] Where G = (V; E) represents the node feature vector X with the following v : v∈V. There are two tasks of interest: (1) node classification, where each node v∈V has an associated label y v , and the goal is to learn a representation vector h for v v, so that the label of v can be predicted as y v =f(h v ); (2) Graph classification, where a given graph set and its tags Embodiments of the present invention aim to learn a representation vector h G , which helps predict the label of the entire graph, y G =g(h G ). GNN can use graph structure and node features X v To learn the node representation vector h v Or the representation vector h of the entire graph G . In addition, GNN follows neighborhood aggregation, where the representation of a node is iteratively updated by aggregating the representations of its neighbors. After k iterations of aggregation, the representation of a node captures the structural information within its k-hop network neighborhood. In the above equation, is the feature vector of node v at the kth iteration / layer. N(v) is the set of nodes that are adjacent to v.

[0057] The node representations 706 are then pooled 708 to generate a graph representation 710. Pooling can be based on The READOUT function aggregates node features from the final iteration to obtain a representation of the entire graph h G。

[0058] In view of the above, embodiments of the present invention use GNN 704 to attempt to fit a graph into a deep model that learns node representations 706. In this regard, GNN can consist of a message passing technique where a graph node (e.g., starting with initial attributes 702) iteratively updates its representation (producing node representation 706) by exchanging information with its neighbors. Node representations 706 are aggregated and a neural network is used to group similar elements to generate a graph representation 710.

[0059] Visual Representation

[0060] Once the structure of the BIM model has been captured, it is desirable to capture the appearance of the model. In this regard, the appearance of the BIM model is also an important aspect of the model. Figure 8 A logical flow for capturing a visual representation of a BIM model according to one or more embodiments of the present invention is shown.

[0061] To capture visual representation information, embodiments of the present invention take a specified number (e.g., 12) of snapshots / views of a three-dimensional (3D) model of a given BIM model with different camera angles 802, and then feed the snapshots into a pre-trained CNN (convolutional neural network) architecture 804. In this regard, embodiments of the present invention may utilize a pre-trained ResNet backbone 806 trained on ImageNet to compute a visual representation 808 for each snapshot 802. Once all view representations 808 are computed (e.g., one snapshot / view representation 808 for each BIM model view 802), embodiments of the present invention aggregate / pool 810 them into a single visual representation 812 corresponding to the appearance of the input BIM model.

[0062] To create the final (multimodal) representation of the BIM model, information from both the structural view (ie, diagram representation 710 ) and the appearance view (ie, visual representation 812 ) is utilized / aggregated to generate the final representation. Fig. 9 A multimodal representation 902 consisting of a combination of a structural representation 710 and a visual representation 812 is shown in accordance with one or more embodiments of the present invention.

[0063] train

[0064] To train the model, an embodiment of the present invention utilizes contrastive training. Contrastive training utilizes the principle of comparing samples to each other to learn common attributes between data classes and attributes that distinguish one data class from another. In an embodiment of the present invention, the model is trained by: driving the model to detect whether a graphic is from a BIM model and whether an image from a BIM model belongs to the same model. The concept of proximity in vector space is used to drive images to be closer / farther away. For example, if the graph structure is from BIM model 1 and twelve (12) images are from BIM model 2, the model will be trained to drive the images away because they are not from the same model. If the images are from the same BIM model, the system drives the images to be closer. In other words, contrastive training maximizes the consistency between images belonging to a specific model.

[0065] More specifically, it is assumed that the system does not have access to the ground truth labels for any task during training. The framework is trained end-to-end using Deep InfoMax (e.g., with a Jensen-Shannon MI estimator) and maximizes the mutual information (MI) between the visual representation 708 and the structural representation 812 according to the following objective:

[0066]

[0067] Where φ, θ are the parameters of GNN and MVCNN to be learned, Denote the visual representation of BIM model i and the structural representation of BIM model j respectively. I is the mutual information estimator. The embodiment of the present invention can utilize the Jensen-Shannon MI estimator:

[0068]

[0069] in is a discriminator that incorporates node and graph representations and scores the consistency between them and is implemented as Embodiments of the present invention provide positive samples from the joint distribution (p) and from the marginal product negative samples, and use mini-batch stochastic gradient descent to optimize the model parameters for the purpose.

[0070] Transfer Learning

[0071] Once the GNN model and the MVCNN model have been trained end-to-end in a self-supervised manner, the two models can be viewed as encoders that can encode the underlying knowledge across BIM models. Therefore, once the model is equipped with such knowledge, it can be adapted to new tasks by simply providing a few examples to the encoder. These tasks can consist of adapting to classify BIM models, predicting properties of such models, or using the model for clustering and similarity search.

[0072] In other words, using self-supervised learning, the weights can be used and fine-tuned to further train the model and map inputs to outputs. In this regard, rather than starting with a random starting point, the underlying knowledge is transferred to the learning network so that previous classifications can be adapted to new tasks based on only a few examples.

[0073] FIG. 10A to FIG. 10C An illustrative process for transfer learning according to one or more embodiments of the invention is shown. FIG. 10A to FIG. 10C An encoder 1000 is shown, which provides multimodal representation / contrastive training for both the structural representation using GNN 704 and the visual representation using CNN 804, which are combined to generate a multimodal representation 902. In addition, the head 1002 of the multimodal representation 902 is used to identify the goal of the current prediction task. For example, this can be predicting the type of building or its estimated price. In other words, the head 1002 is used to predict different information based on the downstream task.

[0074] exist Fig. 10AIn the example, the multimodal representation 902 is used to classify the BIM model 202 as commercial or residential 1006. The automated self-adjustment automatically fine-tunes 1004 this classification over time to improve the accuracy of the encoder 1000 in classifying future BIM models 202. In this regard, the search space of the automated engineering tool may be reduced. In addition, in Fig. 10A , head 1002 predicts the building type (e.g., whether it is commercial or residential).

[0075] exist Fig. 10B In the embodiment of the present invention, the multimodal representation 902 is used to determine the construction cost 1008 of the BIM model 202 (i.e., predicting the final cost 1008 of building the project in its final form given the initial design of the BIM model 202). Automated self-adjustment automatically fine-tunes 1004 the cost over time to improve the accuracy of the encoder 1000 in costing future BIM models 202. In addition, in Fig. 10B , head 10002 predicts an estimated cost of the building model.

[0076] exist Fig. 10C In the example, the multimodal representation 902 is used to determine clusters for the similarity search 1010 of the BIM model 202. In other words, the pre-trained network is used to cluster the representations (e.g., vectors or numbers) to cluster the BIM models 202 based on similarity. The automated self-adjustment automatically makes minor adjustments to the clusters over time to improve the accuracy of the similarity search 1010 of future BIM models 202. In addition, in Fig. 10C , head 1002 identifies similar architectural elements within a building.

[0077] Implementing and training the pipeline

[0078] In view of the above, various libraries and BIM modeling application add-ons can be created to extract information from BIM models, analyze BIM models (e.g., using neural network models), and visualize the results. Cloud pipelines can also be used to (i) preprocess training and inference data, and (ii) train and serve neural network models.

[0079] The training and reasoning pipelines can be implemented in a variety of ways. An exemplary implementation can utilize AMAZON WEB SERVICES (AWS) CLOUD using the AWS CLI (command line interface). Exemplary training pipeline services can include batch processing and SAGEMAKER. Reasoning pipeline services can include SAGEMAKER, LAMBDA, and an API (application programming interface) gateway. In addition, PYTHON can be used to provide API access for BIM modeling applications.

[0080] Fig.11 An exemplary general implementation of a training pipeline using AWS according to one or more embodiments of the present invention is shown. As shown, code 1102 is written and packaged as a docker image 1104. In this regard, docker 1104 is a software platform that allows users to quickly build, write and deploy applications. Docker 1104 packages software (e.g., code 1102) into standardized units called containers, which have everything needed for the software to run, including libraries, system tools, code, and runtime. Docker 1104 is utilized by AMAZON WEB SERVICES (AWS) ELASTIC CONTAIN REGISTRY (ECR) 1106, which is a fully managed container registry that provides high-performance hosting to enable reliable deployment of application images and artifacts anywhere.

[0081] AWS ECR 1106 is fed into AWS SAGEMAKER 1108, software for building, training, and deploying machine learning (ML) models for any use case with a fully managed architecture, tools, and workflows.

[0082] While the code 1102 is being written and packaged as a docker image 1104, the training data 1110 is provided to an AWS S3 bucket 1112 (an AWS Simple Storage Service (S3) bucket, which is a container for objects stored in AMAZON S3). The data 1110 in the AWS S3 bucket 112 is also provided to an AWS SAGEMAKER 11108.

[0083] AWS SAGEMAKER 1108 is used to generate a trained model 1114 , which is stored in another AWS S3 bucket 1116 .

[0084] Fig.12 A more specific implementation of a training pipeline using a GNN according to one or more embodiments of the present invention is shown. The first part 1202 of the training pipeline is the part that extracts entity information from the BIM / REVIT model data. In this regard, the REVIT / BIM file 1204 is used by a BIM modeling application 1206 (e.g., REVIT desktop application and plug-in) to extract entity information 1208, which is then stored / provided to a data lake 1210, such as an AWS S3 bucket.

[0085] The next part 1212 of the process is to convert the entity information 208 into a graph representation. In this regard, a batch process 1214 (e.g., an AWS batch process) is used to generate a graph representation 1216, which is then stored in a data lake 1218 (e.g., an AWS S3 bucket).

[0086] The next step 1220 is to train a graph neural network (GNN) to infer the model embedding. In this regard, the graph representation 1216 (stored in the data lake 1218) is provided to a model training application 1222 (e.g., AWS SAGEMAKER), which generates a trained graph neural network (GNN) model 1224, which can also be exported / stored in the data lake 1226.

[0087] Regarding the above, Fig.13 An exemplary GNN reasoning pipeline for classifying and reasoning about properties of a BIM model using a GNN is shown. A local machine 1304 performs entity extraction processing 1202 via utilizing a restful API 1302 . The graph representation 1216 / 1314 can be generated in a cloud-based processing system 1306A (e.g., AWS cloud via API gateway 1308, app service function 1310 (e.g., via a serverless event-driven computing service such as AWS LAMBDA), and conversion to graph function 1312 (which can also be implemented via a serverless event-driven computing service). The cloud-based processing 1306B can then process the graph representation 1216 / 1314 via multiple servers / processing functions 1318 (e.g., AWS SAGEMAKER plus TORCHSERVE (a tool for serving PYTHON TORCH models - where TORCH is a framework for deep learning) plus NGINX (a load balancer, web server, and reverse proxy)). The server / processing function 1318 can also receive input of a trained GNN model 1320 stored in a data lake 1322.

[0088] Logical Flow

[0089] Fig.14 A logical flow for utilizing a 3D model to perform a prediction task according to one or more embodiments of the present invention is shown.

[0090] At step 1402 , a plurality of 3D models (eg, BIM models) are obtained, wherein each 3D model is composed of non-Euclidean (eg, complex) data.

[0091] At step 1404, each 3D model is converted into a relationship graph. The relationship graph consists of a plurality of nodes and at least one edge. Each node in the plurality of nodes corresponds to an element within the 3D model, and at least one edge of the relationship graph corresponds to a relationship type between a pair of the plurality of nodes. In one or more embodiments, at least one node in the plurality of nodes has properties of the element to which it corresponds, and the properties may include: identification; label; type; category; bounding box; and orientation.

[0092] At step 1406, each relationship graph is processed using a graph neural network (GNN), which computes a node representation for each node.

[0093] At step 1408, the node representations are aggregated into a structural representation of the 3D model.

[0094] At step 1410, a plurality of different views of the 3D model are captured. In one or more embodiments, each of the plurality of different views is a snapshot of the 3D model from a different camera angle.

[0095] At step 1412, the plurality of different views are passed / processed by a convolutional neural network (CNN) to compute a view representation for each of the plurality of different views.

[0096] At step 1414, the view representations are aggregated into a single visual representation.

[0097] At step 1416, the GNN and CNN are trained using a multi-view contrast training objective to maximize the consistency between the structural representation and the single visual representation to form the final learned representation. In one or more embodiments, the training is performed end-to-end in a self-supervised manner.

[0098] In one or more embodiments, both the structural representation and the single visual representation include sets of numbers that uniquely correspond to the structural representation and the single visual representation, respectively.

[0099] At step 1418, the final learned representation is utilized to perform a prediction task. A variety of different types of prediction tasks are within the scope of embodiments of the present invention. Some exemplary prediction tasks are described below.

[0100] In an exemplary prediction task, a search input is received, and a similar 3D model of one of a plurality of 3D models is identified based on the search input using the final learned representation. Thereafter, the similar 3D model is provided in response to the search input.

[0101] In another exemplary prediction task, a new 3D model is received and clustered with at least one of a plurality of 3D models based on the final learned representation. The clustering is then used to determine properties of the new 3D model.

[0102] In yet another exemplary prediction task, a new 3D model is received and a classification of the new 3D model is determined based on the final learned representation. Based on the classification and the final learned representation, a new attribute of the new 3D model is determined (e.g., cost is determined as an attribute of the classification of commercial vs. residential).

[0103] Hardware Implementation

[0104] Fig.15 15 is an exemplary hardware and software environment 1500 (referred to as a computer-implemented system and / or computer-implemented method) for implementing one or more embodiments of the present invention. The hardware and software environment includes a computer 1502 and may include peripheral devices. The computer 1502 may be a user / client computer, a server computer, or a database computer. The computer 1502 includes a hardware processor 1504A and / or a dedicated hardware processor 1504B (hereinafter collectively referred to as processor 1504 in the alternative) and a memory 1506, such as a random access memory (RAM). The computer 1502 may be coupled to and / or integrated with other devices, including input / output (I / O) devices, such as a keyboard 1514, a cursor control device 1516 (e.g., a mouse, a pointing device, a pen and tablet computer, a touch screen, a multi-touch device, etc.), and a printer 1528. In one or more embodiments, the computer 1502 may be coupled to or may include a portable or media viewing / listening device 1532 (e.g., an MP3 player, an iPod, a NOOK, a portable digital video player, a cellular device, a personal digital assistant, etc.). In yet another embodiment, the computer 1502 may include a multi-touch device, a mobile phone, a gaming system, an Internet-enabled television, a television set-top box, or other Internet-enabled device executing on a variety of platforms and operating systems.

[0105] In one embodiment, computer 1502 operates by hardware processor 1504A executing instructions defined by computer program 1510 (e.g., a computer-aided design [CAD] application) under the control of operating system 1508. Computer program 1510 and / or operating system 1508 may be stored in memory 1506 and may interface with users and / or other devices to accept input and commands and provide output and results based on such input and commands and the instructions defined by computer program 1510 and operating system 1508.

[0106] The output / result may be presented on the display 1522 or provided to another device for presentation or further processing or action. In one embodiment, the display 1522 includes a liquid crystal display (LCD) having a plurality of individually addressable liquid crystals. Alternatively, the display 1522 may include a light emitting diode (LED) display having a cluster of red, green and blue diodes driven together to form full-color pixels. Each liquid crystal or pixel of the display 1522 changes to an opaque or translucent state in response to data or information generated by the processor 1504 according to the application of the instructions of the computer program 1510 and / or the operating system 1508 to the input and command to form a portion of the image on the display. The image may be provided by a graphical user interface (GUI) module 1518. Although the GUI module 1518 is depicted as a separate module, the instructions for executing the GUI functions may reside or be distributed in the operating system 1508, the computer program 1510, or implemented with a dedicated memory and processor.

[0107] In one or more embodiments, the display 1522 is integrated with / integrated into the computer 1502 and includes a multi-touch device having a touch sensing surface (e.g., a track pod or touch screen) having the ability to recognize the presence of two or more points of contact with the surface. Examples of multi-touch devices include mobile devices (e.g., IPHONE, NEXUS S, DROID devices, etc.), tablet computers (e.g., IPAD, HP TOUCHPAD, SURFACE devices, etc.), portable / handheld game / music / video player / console devices (e.g., IPOD TOUCH, MP3 players, NINTENDO SWITCH, PLAYSTATION PORTABLE, etc.), touch tables, and touch walls (e.g., where images are projected through acrylic and / or glass and then illuminated with LED backlights).

[0108] Some or all of the operations performed by computer 1502 according to computer program 1510 instructions can be implemented in special purpose processor 1504B. In this embodiment, some or all of the computer program 1510 instructions can be implemented via firmware instructions stored in read-only memory (ROM), programmable read-only memory (PROM) or flash memory in special purpose processor 1504B or stored in memory 1506. Special purpose processor 1504B can also be hardwired by circuit design to perform some or all of the operations to implement the present invention. In addition, special purpose processor 1504B can be a hybrid processor, which includes a dedicated circuit for performing a subset of functions and other circuits for performing more general functions (such as responding to computer program 1510 instructions). In one embodiment, special purpose processor 1504B is an application specific integrated circuit (ASIC).

[0109] The computer 1502 may also implement a compiler 1512 that allows an application or computer program 1510 written in a programming language (such as C, C++, assembly language, SQL, PYTHON, PROLOG, MATLAB, RUBY, RAILS, HASKELL or other language) to be converted into processor 1504 readable code. Alternatively, the compiler 1512 may be an interpreter that directly executes instructions / source code, converts source code into an intermediate representation that is executed, or executes stored precompiled code. Such source code may be written in a variety of programming languages ​​(such as JAVA, JAVASCRIPT, PERL, BASIC, etc.). Upon completion, the application or computer program 1510 uses the relationships and logic generated using the compiler 1512 to access and manipulate data received from the I / O device and stored in the memory 1506 of the computer 1502.

[0110] Computer 1502 also optionally includes external communications means, such as a modem, satellite link, Ethernet card, or other means for accepting input from other computers 1502 and providing output to other computers.

[0111] In one embodiment, the instructions implementing the operating system 1508, computer program 1510, and compiler 1512 are tangibly embodied in a non-transitory computer-readable medium (e.g., data storage device 1520), which may include one or more fixed or removable data storage devices, such as a zip drive, a floppy disk drive 1524, a hard drive, a CD-ROM drive, a tape drive, etc. In addition, the operating system 1508 and computer program 1510 are composed of computer program 1510 instructions that, when accessed, read, and executed by the computer 1502, cause the computer 1502 to perform the steps required to implement and / or use the present invention, or load the program of instructions into the memory 1506, thereby creating a special data structure that causes the computer 1502 to operate as a specially programmed computer to perform the method steps described herein. The computer program 1510 and / or operating instructions may also be tangibly embodied in the memory 1506 and / or the data communication device 1530, thereby making a computer program product or article of manufacture according to the present invention. Thus, as used herein, the terms "article of manufacture," "program storage device," and "computer program product" are intended to encompass a computer program accessible from any computer-readable device or media.

[0112] Of course, those skilled in the art will recognize that any combination of the above components, or any number of different components, peripherals, and other devices, may be used with the computer 1502 .

[0113] Fig.16 A typical distributed / cloud-based computer system 1600 is schematically shown, which uses a network 1604 to connect a client computer 1602 to a server computer 1606. A typical combination of resources may include: a network 1604, which includes the Internet, a LAN (local area network), a WAN (wide area network), an SNA (system network architecture) network, etc.; a client 1602, which is a personal computer or workstation (such as Fig.15 and server 1606, which is a personal computer, workstation, minicomputer or mainframe computer (such as Fig.15 However, it may be noted that different networks, such as a cellular network (e.g., GSM [Global System for Mobile Communications], etc.), a satellite-based network, or any other type of network, may be used to connect the client 1602 and the server 1606 according to embodiments of the present invention.

[0114] A network 1604 (such as the Internet) connects the client 1602 with the server computer 1606. The network 1604 can utilize Ethernet, coaxial cable, wireless communication, radio frequency (RF), etc. to connect the client 1602 and the server 1606 and provide communication therebetween. In addition, in a cloud-based computing system, resources (e.g., storage devices, processors, applications, memory, architecture, etc.) in the client 1602 and the server computer 1606 can be shared by the client 1602, the server computer 1606, and users across one or more networks. Resources can be shared by multiple users and can be dynamically reallocated as needed. In this regard, cloud computing can be referred to as a model for enabling access to a shared pool of configurable computing resources.

[0115] The client 1602 can execute a client application or a web browser and communicate with a server computer 1606 executing a web server 1610. Such a web browser is generally a program such as MICROSOFT INTERNET EXPLORER / EDGE, MOZILLA FIREFOX, OPERA, APPLE SAFARI, GOOGLE CHROME, etc. In addition, the software executed on the client 1602 can be downloaded from the server computer 1606 to the client computer 1602 and installed as a plug-in or ACTIVEX control of the web browser. Therefore, the client 1602 can use the ACTIVEX component / component object model (COM) or distributed COM (DCOM) component to provide a user interface on the display of the client 1602. The web server 1610 is generally a program such as MICROSOFT'S INTERNET INFORMATION SERVER.

[0116] The web server 1610 can host an active server page (ASP) or Internet server application programming interface (ISAPI) application 1612 that can execute scripts. The script calls the object (called business object) that executes the business logic. Then, the business object manipulates the data in the database 1616 through the database management system (DBMS) 1614. Alternatively, the database 1616 can be a part of the client 1602 or directly connected to the client, rather than transmitting / obtaining information from the database 1616 across the network 1604. When the developer encapsulates the business function into the object, the system can be called a component object model (COM) system. Therefore, the script executed on the web server 1610 (and / or application 1612) calls the COM object that implements the business logic. In addition, the server 1606 can utilize MICROSOFT'S TRANSACTION SERVER (MTS) to access the required data stored in the database 1616 via an interface (such as ADO (active data objects), OLE DB (object linking and embedding database) or ODBC (open database connection)).

[0117] Generally speaking, these components 1600 to 1616 include logic and / or data embedded in and / or retrievable from a device, medium, signal or carrier (e.g., a data storage device, a data communication device, a remote computer or a device coupled to a computer via a network or via another data communication device, etc.) Further, such logic and / or data, when read, executed and / or interpreted, causes the steps required to implement and / or use the present invention to be performed.

[0118] Although the terms "user computer," "client computer," and / or "server computer" are referred to herein, it should be understood that such computers 1602 and 1606 may be interchangeable and may also include thin client devices with limited or full processing capabilities, portable devices (such as cell phones, notebook computers, pocket computers, multi-touch devices), and / or any other device with suitable processing, communication, and input / output capabilities.

[0119] Of course, those skilled in the art will recognize that any combination of the above components, or any number of different components, peripherals, and other devices may be used with computers 1602 and 1606. Embodiments of the present invention are implemented as a software / CAD application on a client 1602 or server computer 1606. Furthermore, as described above, the client 1602 or server computer 1606 may include a thin client device or a portable device with a multi-touch based display.

[0120] in conclusion

[0121] The description of the preferred embodiment of the present invention ends here. In view of the above, embodiments of the present invention provide the ability to streamline representations on complex 3D BIM models (e.g., where the BIM model is represented by / mapped to a digital set). In addition to being used for BIM models, embodiments of the present invention can also be used with computer-aided design (CAD) models or 3D CAD, modeling, manufacturing, industrial design, electronics, and mechanical engineering models / applications. Additional embodiments can be used to reason about the properties / attributes of any type of complex model, such as movie characters, 3D animations, and characters (e.g., in AUTODESKMAYA), etc.

[0122] Furthermore, any type of computer (such as a mainframe, minicomputer, or personal computer) or computer configuration (such as a time-sharing mainframe, a local area network, or a stand-alone personal computer) may be used with the present invention.

[0123] The foregoing description of the preferred embodiments of the present invention has been presented for the purpose of illustration and description. It is not intended to be exhaustive of the present invention or to limit the present invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. It is intended that the scope of the present invention is not limited by this specific embodiment, but rather by the appended claims.

[0124] References

[0125] [1]Su,Hang,et al."Multi-view convolutional neural networks for3dshape recognition."Proceedings of the IEEE international conference oncomputer vision.2015.

[0126] [2]Xu, Keyulu, et al. "How powerful are graph neural networks?." arXivpreprint arXiv:1810.00826(2018).

[0127] [3]Hassani,Kaveh,and Amir Hosein Khasahmadi."Contrastive multi-viewrepresentation learning on graphs."International conference on machinelearning.PMLR,2020.

Claims

1. A computer-implemented method for performing a prediction task using a three-dimensional (3D) model, the method comprising: obtaining a plurality of 3D models, wherein each 3D model includes non-Euclidean data; converting each 3D model into a relationship graph including a plurality of nodes and at least one edge, wherein each node of the plurality of nodes corresponds to an element within the 3D model, and at least one edge of the relationship graph corresponds to a relationship type between a pair of the plurality of nodes; Process each relationship graph using a graph neural network (GNN) that computes a node representation for each node; aggregating the node representations into a structural representation of the 3D model; capturing a plurality of different views of the 3D model; passing the plurality of different views through a convolutional neural network (CNN) to compute a view representation for each of the plurality of different views; aggregating the view representations into a single visual representation; Training the GNN and CNN using a multi-view contrastive training objective to maximize the consistency between the structural representation and the single visual representation to form a final learned representation; as well as The prediction task is performed using the final learned representation.

2. The computer-implemented method of claim 1, wherein the 3D model comprises a Building Information Model (BIM).

3. The computer-implemented method of claim 1 , wherein at least one of the plurality of nodes has properties of the element to which it corresponds, and the properties include: Logo; Label; type; type; Bounding box; as well as orientation. 4 . The computer-implemented method of claim 1 , wherein each of the plurality of different views comprises a snapshot of the 3D model from a different camera angle.

5. The computer-implemented method of claim 1, wherein the training is performed end-to-end in a self-supervised manner.

6. The computer-implemented method of claim 1, wherein the structural representation and the single visual representation both comprise sets of numbers that uniquely correspond to the structural representation and the single visual representation, respectively.

7. The computer-implemented method of claim 1 , wherein said performing said prediction task comprises: Receive search input; identifying a similar 3D model to one of the plurality of 3D models based on the search input using the learned final learned representation; as well as The similar 3D models are provided in response to the search input.

8. The computer-implemented method of claim 1 , wherein said performing said prediction task comprises: Receive new 3D models; clustering the new 3D model with at least one of the plurality of 3D models based on the final learned representation; as well as The clusters are utilized to determine properties of the new 3D model.

9. The computer-implemented method of claim 1 , wherein said performing said prediction task comprises: Receive new 3D models; determining a classification of the new 3D model based on the final learned representation; Based on the classification and the final learned representation, new properties of the new 3D model are determined.

10. A computer-implemented system for performing a prediction task using a three-dimensional (3D) model, the system comprising: (a) a computer having a memory; (b) a processor executing on said computer; (c) the memory storing a set of instructions, wherein the set of instructions, when executed by the processor, causes the processor to perform operations including: (1) obtaining a plurality of 3D models, wherein each 3D model includes non-Euclidean data; (2) converting each 3D model into a relationship graph including a plurality of nodes and at least one edge, wherein each node in the plurality of nodes corresponds to an element in the 3D model, and at least one edge in the relationship graph corresponds to a relationship type between a pair of the plurality of nodes; (3) Using a graph neural network (GNN) to process each relationship graph, the graph neural network calculates the node representation of each node; (4) aggregating the node representations into a structural representation of the 3D model; (5) capturing multiple different views of the 3D model; (6) passing the multiple different views through a convolutional neural network (CNN) to compute a view representation for each of the multiple different views; (7) aggregating the view representations into a single visual representation; (8) training the GNN and CNN using a multi-view contrast training objective to maximize the consistency between the structural representation and the single visual representation to form a final learned representation; as well as (9) Utilizing the final learned representation to perform the prediction task.

11. The computer-implemented system of claim 10, wherein the 3D model comprises a Building Information Model (BIM).

12. The computer-implemented system of claim 10, wherein at least one of the plurality of nodes has properties of the element to which it corresponds, and the properties include: Logo; Label; type; type; Bounding box; as well as orientation.

13. The computer-implemented system of claim 10, wherein each of the plurality of different views comprises a snapshot of the 3D model from a different camera angle.

14. The computer-implemented method of claim 10, wherein the operational training is performed end-to-end in a self-supervised manner.

15. The computer-implemented system of claim 10, wherein the structural representation and the single visual representation both comprise sets of numbers that uniquely correspond to the structural representation and the single visual representation, respectively.

16. The computer-implemented system of claim 10, wherein the operation of performing the prediction task comprises: Receive search input; identifying a similar 3D model to one of the plurality of 3D models based on the search input using the learned final learned representation; as well as The similar 3D models are provided in response to the search input.

17. The computer-implemented system of claim 10, wherein the operation of performing the prediction task comprises: Receive new 3D models; clustering the new 3D model with at least one of the plurality of 3D models based on the final learned representation; as well as The clusters are utilized to determine properties of the new 3D model.

18. The computer-implemented system of claim 10, wherein the operation of performing the prediction task comprises: Receive new 3D models; determining a classification of the new 3D model based on the final learned representation; Based on the classification and the final learned representation, new properties of the new 3D model are determined.

Images