Identifying and matching assets in 3D data

Abstract

The invention relates to identifying and matching assets in 3D data. A computer-implemented method for automatically identifying a representation of an asset in a 3D model includes segmenting the 3D model into a plurality of clustering elements, at least a subset of which includes the representation of the asset; organizing each cluster element into an asset structure comprising a plurality of nodes and edges; and calculating an asset embedding for each node and edge, each embedding representing an asset or a portion thereof, the method further comprising, in particular in real time: obtaining a search template from the user corresponding to the user selected asset or a representation of the asset; generating, based on the obtained search template, a template structure comprising a plurality of nodes and edges and a template insert for each node and edge, each template insert representing an asset selected by the user or a portion thereof; comparing the template embedding with asset embedding of the plurality of asset structures to find a representation of an asset in the model that is similar to the representation of the asset selected by the user; and providing information about the representation of the found asset to the user.

Description

Technical Field

[0001] This invention relates to a method for automatically identifying representations of assets in a 3D dataset such as point clouds. Background Technology

[0002] It is known to use, for example, laser scanners or similar capture devices to capture 3D data of the surrounding environment as point clouds. The captured 3D data can then be visualized to the user on a display. Sometimes, it is necessary to locate a specific asset within the 3D data. For example, a user might want to find a particular machine or a specific type of door in the point cloud data. Depending on the size of the 3D data and the number of assets in the data, this can be a tedious and time-consuming task. Therefore, it is desirable to provide a method that allows users to easily initiate automated searches for assets in 3D data.

[0003] One solution is to run an object detection algorithm in 3D for specific asset types. To distinguish assets, the algorithm would need to be trained on many different asset types. A disadvantage is that the specified assets must be known in advance, i.e., before training and deploying the algorithm. This results in a very large number of categories, thus increasing the necessary size of the neural network. Furthermore, there is little or no asset data available for training. Adding a large number of categories to a limited amount of data for certain classes introduces data imbalance during training. Therefore, the algorithm is prone to classifying unseen objects into the most common categories used during training, or detecting fewer uncommon object categories.

[0004] Alternatively, instead of directly locating assets in 3D space, 2D object detection is used to locate assets on their corresponding images, and once identified, the assets are projected into 3D to highlight them. A drawback is that this solution only works if the image and the accurate projection between the image and the point cloud are available in the data. Other disadvantages include multi-view aggregation, i.e., processing multiple detections of the same object, and poor performance in distinguishing between foreground and background in the point cloud (i.e., between points of an object and points behind it from the image's perspective).

[0005] US2023 / 0028242A1 discloses the generation of unique identifiers for industrial goods, and US11,704,343B2 discloses an artificial intelligence-based method for linking data on physical world assets from multiple databases.

[0006] Point cloud registration (i.e., aligning different scans with a single geometrically consistent point cloud) is a primary application of 3D feature descriptors. For example, K. Fu et al. disclosed a method for 3D point cloud registration: “Robust Point Cloud Registration Framework Based on Deep Graph Matching,” November 9, 2022 (arXiv:2211.04696v1). A. Zeng et al., “3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions,” April 9, 2017 (arXiv:1603.08182v3), disclosed a method for matching local geometric features on real-world depth images. While these methods follow a 2D paradigm—that is, using feature descriptors and matching algorithms to find correspondences between features in 3D space—these features do not represent objects or assets. There is no association between the 3D features and the actual asset type (such as assets from a catalog or library).

[0007] An asset can be viewed as a whole or the sum of its parts. This involves extending the basic principles of 2D matching to 3D space and providing solutions for how to process and compare similar assets or objects. For example, the task of finding assets in 3D space based on a template can be accomplished by: creating a 3D template, calculating the template's embedding, processing point cloud samples with the same window size, calculating a list of 3D embeddings, and comparing the list with the template embeddings to identify potential matches. While this approach is similar to 2D matching methods, a disadvantage in 3D space is that the computational cost of sampling point clouds is very high. Moreover, real-time computation is impossible, and users need to wait several minutes for results. Furthermore, it is difficult to segment edges and subtle details in point clouds compared to images. Summary of the Invention

[0008] Therefore, the object of the present invention is to provide an improved method for identifying assets in a point cloud based on user input.

[0009] The specific objective is to provide a method that allows for reliable and real-time asset identification.

[0010] The claimed invention relates to a computer-implemented method for automatically identifying representations of assets in a 3D model of an environment. The 3D model may be, for example, a point cloud or a mesh, and the environment may include multiple assets of various asset types. The method includes the following steps:

[0011] - The 3D model is segmented into multiple cluster elements, at least a subset of which includes a representation of the assets;

[0012] - Organize each cluster element into an asset structure that includes multiple nodes and edges;

[0013] - Calculate an asset embedding for each of the nodes and edges, where each asset embedding represents an asset or a portion thereof;

[0014] - Obtain a search template from the user, the search template corresponding to an asset selected by the user or a representation of the asset selected by the user;

[0015] - Generate a template structure based on the obtained search template, the template structure including multiple nodes and edges and template embeddings for each of the nodes and edges, each template embedding representing the asset or a portion thereof selected by the user;

[0016] - Compare the template embedding with the asset embeddings of multiple asset structures to find representations of assets in the model that are similar to the representation of the user-selected asset; and

[0017] - Provide the user with information about the representation of the found assets.

[0018] According to some implementations, the asset structure and template structure are graph structures or tree structures, especially octrees.

[0019] According to some implementation methods, a template structure is generated such that the template embedding has the same format as the asset embedding.

[0020] According to some implementations, the comparison between template embeddings and asset embeddings is performed by a trained neural network.

[0021] According to some implementations, the method includes preprocessing of the 3D model, which is performed before obtaining the template from the user, wherein at least the segmentation of the 3D model and the organization of clustered elements are part of the preprocessing. In particular, the computation of node embeddings for each node and edge may also be part of the preprocessing.

[0022] According to some implementations, at least the steps of generating the template structure, comparing the template embedding with the asset embedding, and providing information to the user are performed in real time.

[0023] According to some implementations, the method includes displaying a 3D model to a user. Optionally, the displayed 3D model may be a pre-processed 3D model.

[0024] In some implementations, obtaining a template from a user includes enabling the user to select a representation of an asset from the displayed 3D model.

[0025] According to some implementations, providing users with information about the representation of the found asset includes displaying the representation of the found asset in a 3D model in a highlighted manner.

[0026] According to some implementations, the 3D model is a point cloud, and the obtained search template is or includes a parametric model, particularly a BIM or CAD model, and the template generation structure includes generating a synthetic point cloud from the parametric model, particularly using data augmentation to generate a synthetic point cloud from the parametric model, or comparing one or more existing similar parametric models of a particular asset and using the existing point cloud of that asset.

[0027] According to some implementations, the obtained search template includes at least one of text, video, and image related to the user-selected asset, and the generated template structure includes identifying the user-selected asset from the text, video, or image, particularly where the 3D model is a point cloud, and the generated template structure further includes using an existing point cloud of the identified user-selected asset.

[0028] According to some implementations, the method includes filtering outliers in the representation of the found assets before providing information to the user.

[0029] In some implementations, the method includes enabling a user to provide feedback on the representation of the found assets, wherein the feedback is used to improve filtering, and in particular, wherein the feedback includes examples of selected true positives and false positives.

[0030] According to some implementations, the organization of clustered elements into an asset structure is performed by a trained neural network.

[0031] In some implementations, the generation of the template structure is performed by the same trained neural network.

[0032] In some implementations, the trained neural network is a graph neural network (GNN) that has been trained using symbolic rules to learn relational constraints.

[0033] According to some implementation methods, organizing clustered elements into an asset structure includes:

[0034] - Use a visual foundation model on images to identify and group assets;

[0035] - Enforce multiview Figure One To the point of being compatible;

[0036] - Establish the structure of all assets based on neighborhood;

[0037] - Refine the structure to the desired granularity;

[0038] - Create an embedding vector for each element; and

[0039] - Assign the embedding vector to the structure.

[0040] In some implementations, the asset structure is a layered structure.

[0041] The claimed invention also relates to a computer program product comprising program code having computer-executable instructions for performing the above-described method. Attached Figure Description

[0042] The invention will now be described in detail with reference to exemplary embodiments accompanied by accompanying drawings, in which:

[0043] Figure 1 An exemplary 3D model including a representation of searchable assets is shown;

[0044] Figure 2 A flowchart illustrating an exemplary embodiment of the method according to the present invention is provided; and

[0045] Figure 3 An example is shown where a user-provided search template is compared with a point cloud to identify the asset the user is searching for. Detailed Implementation

[0046] Figure 1 An example of a three-dimensional (3D) model 1 comprising multiple different “assets” as displayed on a computing device's monitor is shown. Specifically, the 3D model can be one of a point cloud and a mesh. In the example shown, the 3D model is a 3D model of a factory workshop. An asset is an object of interest comprising multiple different 3D features. In the example shown, multiple identical (or similar) pumps are located in the workshop, each pump comprising multiple features constituting the pump. The 3D model 1 shown includes a representation of a first pump 20.

[0047] The invention disclosed herein proposes an improved method for effectively identifying and locating identical or similar assets in 3D space. It enables a workflow that allows for the instantaneous comparison of 3D assets. Using an "asset-aware" approach to group / cluster points into structured elements, it provides an efficient method for comparing templates with the "asset-aware" structure. "Asset-aware" means that the structure has information about the "object" level meaning of different assets, which contrasts with, for example, agnostic graph embeddings, which may have some information about objects but lack a process for ordering that information into "objects." Information structures that are well-ordered and organized by "objects" or "assets" are generated because, for example, graph neural networks (GNNs) operate using loss functions or symbolic rules and constraints designed to distinguish assets.

[0048] The key idea is to structure the point cloud to reduce information size while maintaining information about the content and composition of the point cloud, including relationships between assets in the scene, to efficiently locate assets within the point cloud. In summary, this method offers the following advantages over known solutions: searching for identical assets is computationally efficient because the 3D embeddings are pre-computed and finite in size. Asset instance consistency is achieved because individual assets are represented in 3D space, and multi-view aggregation problems are avoided. For example, pre-computation can be performed immediately after the user has uploaded the point cloud to cloud storage. Meshing of the point cloud can be automatically triggered. The creation of an asset-aware structure for the entire data can be triggered in parallel. Robust search and comparison capabilities are achieved by using graphs to abstract the "asset-aware" structure of the point cloud and objects.

[0049] The structured and different embeddings of the asset-aware structure (i.e., the entire scene) and the structured and different embeddings of the template (i.e., the asset) must be compatible for later comparison. In deep learning, this is often referred to as the "joint embedding space" for embeddings. This joint space is typically learned by a neural network.

[0050] Since the user knows in advance what they are looking for, they enter a representation of the asset they are searching for as a search template. For example, a search template could include a parametric model, such as a Building Information Modeling (BIM) or Computer-Aided Design (CAD) model, or a point cloud representing an object. For instance, a point cloud template could be obtained by the user clicking on a visualization of the asset in the displayed point cloud and using image processing to select a set of points of the asset.

[0051] According to the present invention, by selecting one of the assets of a pump (indicated by a white arrow) using, for example, a mouse and cursor or touch on a touchscreen, a user can initiate a search for all identical or similar assets in a 3D model. This not only allows for the identification of all identical or similar assets, but also allows for the identification of all pumps that possess such assets.

[0052] Figure 2 This is a flowchart illustrating an exemplary implementation of a computer-implemented method 100 for comparing objects in a 3D model (particularly in a 3D point cloud).

[0053] In particular, the illustrated method allows for “3D asset matching.” 3D asset matching includes use cases where a user provides details about an asset and wants to accurately and reliably find one or more identical assets in a 3D model. For example, in one such use case, in an industrial facility, the method allows a user to find all pumps from a specified manufacturer based on a template of the pumps (such as a 3D CAD representation of the pump). In an office building with many different types of windows, a user can locate and count the number of windows of a specific type (e.g., all windows that open to the left) by clicking on one of such windows in a point cloud. In an industrial building with several different types of columns, a user can locate structural columns and measure the distances between them by selecting a point of a single column in 3D space. In a transportation environment, where assets such as beacons, signals, or any other infrastructure may be unique to a particular area or location, systems that automatically identify and geolocate assets of interest by simply providing or recognizing these in data, and reference samples, enable or optimize control, maintenance, or planning activities. Other use cases may include identifying “safety”-related assets, such as fire extinguishers or emergency lights, to verify compliance with safety regulations or plan maintenance operations.

[0054] The method may include point cloud preprocessing 105, wherein multiple steps may be performed as part of preprocessing 105.

[0055] In the first step of method 100, the point cloud is segmented 110 into cluster elements. These cluster elements are then organized 120 into an “asset-aware” structure that includes multiple nodes and edges. As shown here, this segmentation 110 and organization 120 can be performed as part of preprocessing 105. Method 100 can be performed on a computing device or multiple connected computing devices. For example, the steps included in preprocessing 105 can be performed on a different computing device than the other steps.

[0056] The “asset-aware” structure can be, for example, a graph structure or a hierarchical structure, such as a tree structure, like an octree. For example, if the clustering elements are windows, organizing the clustering elements in a hierarchical structure involves: grouping all windows, then grouping by typical elements of the windows (such as handles and window frames), and then grouping by the sub-elements of the elements.

[0057] The segmentation 110 and organization 120 are accomplished using an algorithm (clustering algorithm) optimized for asset grouping. These steps can also be learned and executed by a neural network. Thus, the clustering algorithm used to generate the “asset-aware” structure can be a neural network trained to segment objects (i.e., assets). Clustering is then learned. Alternatively, the neural network can be a base model that can cluster and group point clouds into logical and “asset-aware” structures without learning.

[0058] For example, clustering algorithms can be or include the following:

[0059] - Use a visual base model on images to identify and group assets;

[0060] - Implement multi-view Figure One To the point of being compatible;

[0061] - Establish the structure of all assets based on neighborhood;

[0062] - The structure is refined to the desired granularity;

[0063] - Create an embedding vector for each image patch (an element of the object); and

[0064] - An embedding vector is assigned to this structure.

[0065] It can also calculate the embeddings of edges connecting image patches, representing the relationships between adjacent patches.

[0066] Next, 130 3D embeddings are computed for each node of the structure, optionally still as part of preprocessing step 105. Each 3D embedding represents a potential part or object of an asset or sub-asset. A 3D embedding does not necessarily represent a complete object with all its elements (such as a window), but may only represent a portion of it or may even include other asset parts. However, the 3D embedding will represent the asset or a portion thereof very "closely." The embeddings are normalized to a specific point cloud, i.e., they need to be clustered in the same way as the point cloud. Optionally, the 3D embedding may include feature vectors describing local elements.

[0067] The “asset-aware” structure generated by Organization 120 is similar to a scene graph. A scene graph can consist of a graph and / or tree representation with nodes and edges. Each node and edge can store information in the form of features, as in graph theory. In the context of this invention, nodes represent assets or parts of assets, and edges are relationships between assets and parts of assets. Each node and edge has an embedding, i.e., an information vector. These embeddings can be learned from features and / or edges, for example, using a graph neural network (GNN). During such a learning phase, information from neighboring nodes and edges is passed to the target node (message passing). The features from which the embeddings are learned can be geometric, topological, material, or other information. This information can appear in different modalities, such as text descriptions, images, point clouds, or parametric models. Each asset can be viewed as a subgraph composed of “asset-aware” substructures. For example, a window can be represented by a BIM library model. A window is an asset. In geometric representation, a window is a 3D object comprising primitive shapes such as frames, columns, or rectangles. These primitive shapes form substructures. This is similar to CAD models, which are typically geometrically represented in B-rep format. A window can also be abstracted into more logical parts, such as the window frame, multiple panes of glass, seals, and handles, rather than pure geometric primitives. Both representations (i.e., abstract representation and geometric representation) can be combined into a single subgraph or substructure, combining the corresponding information. For example, materials can be attached to each part / node of the window, such as a wooden or plastic window frame. This information is stored as features in the nodes.

[0068] During segmentation 110, the algorithm clusters the entire scene into assets. For example, if the input is a point cloud, the point cloud is clustered into such a scene graph. If the input is a BIM model, the model is clustered into such a scene graph. Clustering involves “asset awareness,” that is, it distinguishes different objects (or “assets”), such as walls, doors, or windows, and sub-parts of objects (or “sub-assets”). During this process, additional information is appended to the graph. For example, if an image is available, it is added to nodes in the asset-aware structure based on spatial information. The initial structure and granularity of the assets may be incorrect. For example, a substructure of the scene structure might be windows, but some nodes and edges come from adjacent walls. To correct the structure, a Generative Neural Network (GNN) can be used. In this case, the GNN uses information stored in the graph to update edge and node embeddings to better represent the real objects, thus representing the scene.

[0069] Importantly, the asset-aware structure has a high granularity. This granularity can be proportional to the BIM's Level of Detail (LOD). For example, a window includes a handle, but the handle has lock-like sub-assets that include pins, screws, and metal parts. Such information is visible in the image and, if the scan resolution is high enough, can also be seen in the point cloud data.

[0070] Optionally, a first version can be assembled before using GNNs to learn an accurate asset-aware structure, for example, by clustering points based on color information. Alternatively, a general object detection method can be deployed to gain a preliminary understanding of the scene and asset-aware structure. For example, object or segmentation neural networks can be used on image or point cloud data to detect assets. In the same way, relation modeling can then be used to compute relationships between assets to create a connectivity graph of the complete scene to generate an initial version of the structure. If the input is another modality, such as BIM, other clustering methods can be applied. In the case of BIM, assets are arranged in a tree structure, and an initial version of the asset-aware structure can be created using simple heuristics.

[0071] In addition to creating asset-aware embeddings for each node and edge, subgraphs / substructures (such as windows) can also have global embeddings. In graph theory, this is called a global representation. However, in the context of this invention, it does not represent the entire scene graph, but rather a subgraph, essentially describing assets within a single embedding. This can also be learned, just like node and edge embeddings.

[0072] The asset embedding can be used later to compare assets (step 170). The learned asset embedding represents the subgraph in a “condensed” way, i.e., a lower / higher dimensional representation or a global / local representation. For example, the learned asset embedding can be a rich low-dimensional representation of the subgraph. Simple similarity comparison methods such as cosine similarity can be deployed to find specific assets in the asset-aware structure. This can be used as a first guess to locate the asset. In the second step, more sophisticated methods can be used, such as graph matching. In this case, the method compares not only with a single embedding like the asset embedding, but also with its complete substructure, including all nodes and edges and their embeddings. This allows for a more reliable comparison compared to a single embedding. For example, in the case of a window, the detailed design on the handle may not be well represented in a single embedding, but it is clearly distinguishable in the substructure. Therefore, identifying specific assets or comparing substructures with each other is an important property. The details of the substructure are limited only by the available information / data. Information can accumulate over time or be completed using GNN messaging methods or prediction methods to fill in missing nodes and edges in the substructure.

[0073] The "granularity" of assets used for matching depends on the desired speed and accuracy for each specific use case. Depending on the use case, the search can follow a top-down or bottom-up approach. The latter may be useful, for example, if a user wants to search for a small part of an object.

[0074] The creation of the asset-aware structure (e.g., preprocessing 105) can occur in a background process. For example, a user can upload a point cloud to a cloud service such as Hexagon Digital Reality (HxDR). This triggers background processes that may include various things, such as generating a mesh, calculating 3D point cloud segmentation, or simply automatically cleaning the point cloud. After a certain amount of time (typically a few minutes), the preprocessed data becomes available. In the same way, and in parallel with other preprocessing, the asset-aware structure can be computed. First, an initial version of the structure is generated, and then in a second step, the initial structure is optimized into a final structure by, for example, applying a pre-trained GNN model. During this process, other modalities, not just the uploaded point cloud, can be integrated into the process. For example, if an image or BIM model is available, information can be projected onto the structure. If no other modalities exist when the point cloud is uploaded, a first version of the asset-aware structure is created, onto which subsequent data can be uploaded and projected, resulting in a single representation of the scene using all available information represented in the asset-aware structure. This ensures the highest reliability in any potential downstream task, such as asset matching, where substructures are provided and need to be identified within the asset-aware structure of the scene.

[0075] An asset-aware structure is a universal container onto which all available information can be projected. This information can come from different modalities. This structure can combine this information into a single "database," whereas otherwise the information is stored unstructured in a single modality (such as a point cloud or image). Therefore, this structure can extract information in a highly optimized way for any downstream task, including asset matching or object classification. An asset-aware structure is a unified representation of a scene that allows storing all information from all sources.

[0076] Any modality can be projected onto such a graph. This includes not only point clouds or images, but also parametric models such as CAD models or BIM library models. In the first iteration, asset embeddings can be generated for such objects, allowing for quick and immediate comparisons. Alternatively, substructures can be generated. While this seems straightforward, since such library models are typically already tree-structured, information such as materials may be missing and must be projected as additional features onto the substructures.

[0077] After the point cloud preprocessing 105 is completed (whether or not it is part of method 100), the point cloud 140 can be displayed to the user on the display of the computing device, for example, as shown below. Figure 1 As shown.

[0078] 150 search templates are obtained from the user. By providing the search templates as input to the computing device, the user can initiate a search for a specific asset in the 3D model (e.g., all identical or similar assets). For example, a user might want to find a specific machine or a specific type of door in point cloud data. The obtained search templates can include parametric models (such as BIM or CAD models), point cloud groups representing assets, or any other representation.

[0079] For example, it can enable users to select assets in a point cloud as search templates by selecting one asset in the point cloud. That is, 150 templates can be obtained by the user clicking (e.g., using a mouse and cursor or a touch on a touchscreen) on the visualization of the assets in the point cloud and using image processing to automatically select a set of points in the point cloud as belonging to the selected assets.

[0080] Then, a structure of 160 templates compatible with the structure of the clustering elements is generated, i.e., having the same format. Similar to the structure generated in step 120, it includes multiple nodes and edges. In other words, a 3D embedding (3D template embedding) is generated for the obtained search template. If the obtained template is a point cloud, the same graph generation functions as in steps 110, 120, and 130 above can be applied to generate the 3D embedding of the template. The template can be hierarchical and / or have a graph representation. In the simplest case, the template can be a structure with a 3D embedding. Typically, the 3D template embedding is an "asset-aware structure" compatible with an "asset-aware scene structure" so that it can be queried and searched. It is also important here that both the template and the scene are in a "unified space," including the structure and all embeddings.

[0081] If the template is a parametric model, such as BIM or CAD, the 3D template embedding can be computed by generating a synthetic point cloud from the parametric model, using data augmentation to create a more generalized embedding. Alternatively, the 3D template embedding can be computed by comparing existing and similar parametric models of a specific asset and ensembling the embeddings using these evaluated real point clouds.

[0082] Instead of using parametric models or point cloud sets, 160 of these 3D template embeddings can be generated from other sources, such as text, video, or images. By providing text-to-embedding functionality, users will be able to query point clouds via text. Different modalities, such as images or text, can be projected onto the asset-aware structure, similar to using point clouds as input to create an initial version of this structure.

[0083] Once the user provides a template, the generated 3D template embedding is compared with the computed 3D embeddings of the nodes.170 This could, for example, include using latent space comparison methods such as dot product or cosine similarity, where the matching algorithm starts at the highest level and determines which pre-computed 3D embedding is the closest. After making the first-level selection, the algorithm searches at any sub-level and validates the hypothesis using the connected nodes in the graph, thus providing an overall similarity score describing the similarity between the 3D embedding and the pre-computed embedding. For sub-parts, a bottom-up approach may be faster. The best match, especially considering the need for structured / hierarchical comparisons, can also be learned and performed by a neural network (NN).

[0084] Comparing a template with a point cloud (structuring element) 170 can be learned via a neural network (NN). In 2D matching, these NN matchers learn on a single feature vector or have positional encoding. According to the invention, the method can be extended to learn to compare “asset-aware” structures and infer correspondences at that level. Alternatively, a neural network can be the underlying model, capable of comparing “asset-aware” structures, their node embeddings, and relationships.

[0085] The “asset-aware” structure maintains explicit relationships between the elements of an object. These relationships are explicitly used for comparison of templates and point clouds, rather than being directly present in the regular 3D embedding of a 3D point cloud matcher. Optionally, the hierarchy not only groups the elements of an asset, but the hierarchy can also represent the level of detail of the asset. This means that if only higher levels are used, a broad description of the object is produced, which can be used to find “similar” assets. Deeper levels represent fine details of the object and can be used to find “identical” assets.

[0086] Optionally, to improve and simplify the matching of templates to the "asset-aware" structure, method 100 may include filtering 180 for outliers in the 3D embedding space. Optionally, in the second stage, the user can provide additional information by selecting examples of true positives and false positives to improve the filtering of the desired target object. These auxiliary templates can be used to calculate a contrast score or a triplet score (selecting two positive samples and one false positive). Filtering outliers 180 in the 3D embedding space can be effectively discarded by providing additional templates from the output of the previous step.

[0087] Optionally, outlier filtering 180 can be extended by a base model (GPT) cue evaluation, which extracts textual cues from the 3D embedding and discards outliers by finding a discriminative prompt.

[0088] Finally, the comparison results are presented to the user. For example, this could include highlighting matches—that is, all identical or similar assets—in the displayed point cloud.

[0089] Advantageously, it's not actually necessary to know what type of asset the requested asset is. Instead, a comparison function can be used to find identical assets in 3D data. However, this comparison function must be very powerful to detect identical assets in 3D data and be able to differentiate them with high granularity.

[0090] Figure 3 An example is shown comparing a search template with a point cloud. Figure 2 The steps of an exemplary method.

[0091] In the upper left corner, the search template obtained from the user is shown as 10 examples of the asset selected by the user, which is a pump.

[0092] On the right, a representation 20 of the asset corresponding to the search template, namely the pump, is shown. This representation is one of multiple clustering elements obtained through point cloud segmentation. Below the pump representation 20, a structure 25 comprising multiple nodes 26, 27 is shown. The clustering elements corresponding to the pump have been organized into this structure 25, where an embedding 28 (node ​​embedding) has been computed for each node. In structure 25, circles represent nodes 26, 27, and the three squares below each node represent the corresponding embedding 28. The node embedding of the top node 26 represents the entire pump, while the node embedding of the child node 27 represents a specific part of the pump.

[0093] When a search template corresponding to the user-selected representation 10 is obtained from the user, a structure 15 is generated for the search template in the same manner as for clustering elements of a point cloud (i.e., in the "joint space"). Furthermore, this structure 15 includes multiple nodes 16, 17 (represented by circles) and embeddings 18 (template embeddings) for each of the nodes. The template embedding of the top node 16 represents a pump, and the embeddings of all other nodes 17 represent specific portions of the pump. The two structures 15, 25 are then compared to each other to determine whether they, and therefore the corresponding representations 10, 20, are identical or at least sufficiently similar. This comparison includes comparing all template embeddings 18 with all node embeddings 28.

[0094] The search template structure 15 can be compared with each of the point cloud structures 25 to identify all assets in the point cloud that are the same as or similar to the searched asset. The identified assets (or their representations 20) are then provided to the user in real time. For example, they can be highlighted in the point cloud.

[0095] Through design, graph neural networks can achieve distinct embedding spaces at both the "node" and "edge" levels. "Node" embeddings tend to learn to encode intrinsic / local properties of objects, while "edge" embeddings capture relational / extrinsic information within the graph. More advanced graph attention networks integrate the concept of "attention" between nodes, enabling the system to learn relationships between nodes with significantly greater scale and reliability than traditional convolutional methods. Graph neural networks can readily integrate symbolic rules (neural symbolic AI) to implicitly learn relational constraints during training. If the semantics associated with each node in the graph are known during training, additional constraints can be designed to inject extra penalties into the loss term during training, thus "guiding" the network to reproduce those constraints during training (i.e., minimizing or eliminating activation / attention on edges connecting floor and ceiling nodes).

[0096] An exemplary method for training a graph neural network (GNN) using neural symbolic AI to minimize connections between certain types of nodes (such as between the “chair” and the “ceiling” in a room diagram) includes the following three steps:

[0097] 1. Graph Definition: First, the asset-aware structure is constructed in such a way that, at a defined level of hierarchy, nodes represent unique objects in the scene (e.g., walls, floors, ceilings, and assets), and edges represent relationships between them (e.g., adjacency and / or relative position).

[0098] 2. Define Symbolic Rules: With the use of neural symbolic AI, it is necessary to enforce some symbolic rules. Here, the rule is that there should be no relationship (e.g., a direct edge) between the "chair" and "ceiling" nodes. This rule is programmed as part of the graph construction, rejecting connections and / or specifying different penalty values ​​representing the probability of connectivity.

[0099] 3. Incorporating Symbolic Rules into GNN Training: There are two approaches to incorporating symbolic rules. On one hand, rules can be incorporated as direct connectivity constraints, selectively deciding not to update certain node relationships by ensuring no message passing occurs between certain types of nodes (in this case, between "chair" and "ceiling" nodes). On the other hand, rules can be indirectly incorporated by implementing connectivity constraints in the loss function, allowing the network to learn when to use attention mechanisms or other automatic selection methods to minimize message passing between nodes.

[0100] Although the invention has been described above with reference to some preferred embodiments, it should be understood that many modifications and combinations of different features of the embodiments can be made. All such modifications are within the scope of the appended claims.

Claims

1. A computer-implemented method (100) for automatically identifying the representation (20) of assets in a 3D model (1) of an environment, the method comprising the steps of: - The 3D model (1) is divided (110) into multiple clustering elements, at least a subset of which includes a representation of the assets (20). - Organize each cluster element (120) into an asset structure (25) comprising multiple nodes (26, 27) and edges; and - Calculate (130) an asset embedding (28) for each of the nodes (26, 27) and edges, where each asset embedding represents an asset or a portion thereof; The method further includes the following steps: - Obtain a search template (150) from the user, the search template corresponding to an asset selected by the user or a representation of the asset selected by the user (10). - Generate a template structure (15) based on the obtained search template (160), the template structure (15) including multiple nodes (16, 17) and edges and a template embedding (18) for each of the nodes (16, 17) and edges, each template embedding (18) representing the asset or a portion thereof selected by the user; - Compare (170) the template embedding (18) with the asset embedding (28) of multiple asset structures (25) to find the representation (20) of an asset in the model (1) that is similar to the representation (10) of the user-selected asset; and - Provide the user (190) with information (20) about the representation of the found asset.

2. The method (100) according to claim 1, wherein, The asset structure (25) and the template structure (15) are graph structures or tree structures, particularly octrees. Specifically, the template structure (15) is generated (160) such that the template embedding (18) has the same format as the asset embedding (28).

3. The method (100) according to claim 1 or claim 2, wherein, The 3D model (1) is a point cloud or a mesh, and in particular, the environment includes multiple assets of various different asset types.

4. The method (100) according to any one of the preceding claims, wherein, The comparison (170) between the template embedding (18) and the asset embedding (28) is performed by a trained neural network.

5. The method (100) according to any one of the preceding claims, the method comprising preprocessing (105) of the 3D model (1), the preprocessing (105) being performed before obtaining (150) the template from the user, wherein, At least the segmentation (110) of the 3D model and the organization (120) of the clustered elements are part of the preprocessing (105), and in particular, the calculation (130) of the node embedding for each of the nodes and edges is also part of the preprocessing (105).

6. The method (100) according to any one of the preceding claims, wherein, The steps of generating at least (160) a template structure (15), comparing the template embedding (18) with the asset embedding (28) (170), and providing (190) information to the user are performed in real time.

7. The method (100) according to any one of the preceding claims, the method comprising displaying (140) the 3D model (1) to a user, particularly wherein, The 3D model displayed is a pre-processed 3D model.

8. The method (100) according to claim 7, wherein, The template obtained from the user (150) includes a representation (20) that enables the user to select the asset from the displayed 3D model (1).

9. The method (100) according to claim 7 or claim 8, wherein, Providing the user (190) with information about the representation (20) of the found asset includes displaying the representation (20) of the found asset in the 3D model (1) in a highlighted manner.

10. The method (100) according to any one of the preceding claims, wherein, The 3D model (1) is a point cloud, and the obtained search template is or includes a parametric model, particularly a BIM or CAD model, and the generated (160) template structure (15) includes: - Generate synthetic point clouds from the parametric model, particularly using data augmentation to generate synthetic point clouds from the parametric model; or - Compare one or more existing similar parametric models of a specific asset and use the existing point cloud of that asset.

11. The method (100) according to any one of the preceding claims, wherein, The obtained search template includes at least one of text, video and image related to the user-selected asset, and the generation (160) template structure (15) includes identifying the user-selected asset from the text, video or image, particularly wherein the 3D model (1) is a point cloud, and the generation (160) template structure (15) further includes using an existing point cloud of the identified user-selected asset.

12. The method (100) according to any one of the preceding claims, the method comprising filtering (180) outliers in the representation (20) of the found assets before providing (190) the information to the user, Specifically, the method includes enabling the user to provide feedback (20) regarding the representation of the found assets provided, wherein, The feedback is used to improve the filtering (180), and in particular, the feedback includes selected examples of true positives and false positives.

13. The method (100) according to any one of the preceding claims, wherein, The organization of clustered elements (130) into an asset structure (25) is performed by a trained neural network, in particular, - The generation of the (160) template structure (15) is performed by the same trained neural network; and / or - The trained neural network is a graph neural network that has been trained using symbolic rules to learn relational constraints.

14. The method (100) according to any one of the preceding claims, wherein, Organizing clustered elements (130) into an asset structure (25) includes: - Use a visual base model on images to identify and group assets; - Implement multi-view consistency; - Establish the structure of all assets based on neighborhood; - Refine the structure to the desired granularity; - Create an embedding vector for each element; and - Assign the embedding vector to the structure. Specifically, the asset structure (25) is a hierarchical structure.

15. A computer program product comprising program code having computer-executable instructions for performing the method (100) according to any one of claims 1 to 14.

Images