A method and device for automatic labeling and verification of embodied simulation scene based on BIM semantics

Abstract

The application discloses a kind of embodied simulation scene automatic labeling and verification method and device fused BIM semantics.The method obtains structured semantic description by parsing BIM model;For each original geometric surface, generate semantic fingerprint, match with centroid nearest neighbor in geometric discretization and impose normal angle constraint to make triangle patch inherit fingerprint, when grid simplification and convex decomposition, vote according to area to decide attribution;Accordingly, establish semantic anchor mapping table supporting bidirectional search, automatically generate segmentation mask, oriented bounding box, semantic point cloud, availability area, joint parameter and finger representation, etc. Multi-modal labeling;Then, execute geometric, semantic and physical accessibility triple consistency verification and fuse into effectiveness index, below threshold value Time positioning failure reason, call repair operator and iterate retest.The application overcomes the problems of BIM semantic loss in geometric conversion, lack of objective truth value in labeling and invalid scene polluting data set.

Description

Technical Field

[0001] This invention relates to the fields of embodied intelligence and building information modeling, specifically to a method and apparatus for automatic annotation and verification of embodied simulation scenes that integrates BIM semantics, and also to corresponding electronic equipment and computer-readable storage media. Background Technology

[0002] Embodied intelligence refers to the technological approach where intelligent agents gain perception and operational capabilities through interaction with their environment via physical entities. Limited by the cost, security, and repeatability of data collection in real-world environments, the industry commonly employs batch generation of training data from 3D simulation scenarios, which is then deployed to physical robots through simulation-to-real-world migration. This type of training relies on large-scale, high-fidelity 3D simulation scenarios with precise semantic annotations. The granularity of the annotations needs to cover multiple levels, including instance segmentation, availability, joint parameters of interactive parts, physical attributes, and natural language representation.

[0003] There are three main technical approaches to existing simulation scene annotation solutions. The first approach is manual 3D annotation, where annotators select instances and specify joints and availability for each component in a 3D editor. This approach offers high accuracy, but the annotation workload for a single medium-sized indoor scene typically reaches several to dozens of person-hours, resulting in high costs and long lead times. Furthermore, different annotators may have varying understandings of availability boundaries and component granularity, making it difficult to guarantee annotation consistency and support large-scale data production.

[0004] The second route involves converting a Building Information Model (BIM) into a simulation scene. BIM itself carries complete component types, attribute sets, spatial hierarchies, and topological relationships, theoretically making it an ideal source of semantics. However, existing conversion tools, when exporting the model to a general 3D exchange format or simulation engine format, require sequential operations such as boundary triangulation, material baking, detail simplification, and convex decomposition for generating collision bodies. These geometric operations prioritize rendering and simulation efficiency, retaining only mesh vertices, faces, and material maps; the globally unique identifiers, types, attribute sets, and topological relationships of components are discarded in the first step of discretization. The converted scene degenerates into an undifferentiated set of meshes, requiring manual annotation again from the first route to obtain annotations; the existing semantic assets in the BIM are not utilized.

[0005] The third approach is pseudo-annotation based on visual models. This involves applying segmentation, detection, or visual language models to the simulated scene rendering or point cloud to infer the annotations. This approach is highly automated, but its output is an estimate rather than the true value, which can lead to problems such as class confusion, instance boundary drift, and missed detections under occlusion conditions. More importantly, due to the lack of independent objective references, it is impossible to determine whether a particular annotation is correct, and erroneous annotations will silently enter the training set.

[0006] Besides the reliability of the annotation itself, existing solutions generally lack a verification step for the usability of the simulation scenario. A simulation scenario may appear geometrically reasonable, but it could be invalid at the physical and task levels: the clear width of a doorway is smaller than the diameter of the robot's circumcircle, making the area inaccessible; furniture arrangement fragments the navigation map into disconnected components; the range of motion of door panels or drawers interferes with adjacent components; and the grasping pose of the target object lacks a feasible inverse kinematic solution. Data collected in such invalid scenarios not only fails to aid training but also introduces incorrect state transition distributions, polluting the dataset. These defects are often only discovered after large-scale data collection or even model training, resulting in extremely high rework costs.

[0007] Furthermore, Building Information Modeling (BIM) undergoes continuous iteration during the design, refinement, and construction phases, with frequent additions, deletions, and modifications to components. Existing solutions cannot pinpoint the affected area after model updates, necessitating a complete re-conversion and re-annotation of the entire scene. This not only wastes computing power but also results in a lack of traceable correspondence between different versions of the same scene.

[0008] In summary, how to maintain the semantic traceability of Building Information Model during geometric transformation, automatically generate multimodal annotations accordingly, and objectively and quantifiablely verify the effectiveness of the annotation results and the simulation scene itself, and perform closed-loop repair, are technical problems that urgently need to be solved in this field. Summary of the Invention

[0009] To address the problems existing in the prior art, the present invention aims to provide an automatic annotation and verification method and apparatus for embodied simulation scenes that integrates BIM semantics, so as to maintain the surface-level traceability of BIM semantics during the geometric discretization process, realize the automatic generation of multimodal annotations, and ensure the effectiveness of annotations and scenes through triple consistency verification and closed-loop repair.

[0010] To achieve the above objectives, the first aspect of this invention provides an automatic annotation and verification method for embodied simulation scenes that integrates BIM semantics, comprising steps S1 to S7. Step S1: Obtain the BIM model of the target building and parse it to obtain a structured semantic description including a spatial hierarchy tree, a set of component entities, a set of component attributes, a set of topological relationships between components, and a set of materials. Step S2: Generate a semantic fingerprint for each original geometric facet of each component entity, including a globally unique identifier, a facet identifier, a material identifier, and a normal clustering identifier. During geometric discretization, each triangular facet inherits the semantic fingerprint by using centroid nearest neighbor matching and applying normal angle constraints. During mesh simplification and convex decomposition, the semantic fingerprint of the merged facet is determined by area-weighted voting results of candidate source facets. Step S3: Establish a semantic anchor mapping table containing a globally unique identifier, an instance-level path, a facet index range, and a version fingerprint, and configured with a bidirectional hash index. Step S4: Automatically generate multimodal annotation data based on the semantic anchor mapping table and the structured semantic description. Step S5: Perform triple verification of geometric consistency, semantic consistency, and physical reachability. Step S6: Weighted fusion is used to obtain the effectiveness index. If the index is lower than the threshold, the root cause of the failure is located and the corresponding repair operator is called for iterative verification. Step S7: Data asset description files are generated and output according to the effectiveness index.

[0011] The second aspect of the present invention provides an automatic annotation and verification device for embodied simulation scenes that integrates BIM semantics, including a BIM semantic parsing module, a semantic anchor mapping module, a multimodal annotation generation module, a validity verification module, a closed-loop repair module, a data asset output module, and a storage module.

[0012] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0013] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0014] Compared with the prior art, the present invention has the following beneficial effects: Firstly, by establishing a face-level semantic fingerprint before geometric discretization, and transferring the fingerprint based on geometric proximity and area dominance principles after each geometric operation such as triangulation, mesh simplification, and convex decomposition, each triangular facet under any degree of simplification can be traced back to a unique BIM component and original face. This fundamentally solves the problem that BIM semantics are discarded in the first step of discretization in the existing conversion process, eliminating the need for re-annotation after conversion.

[0015] Secondly, the semantic anchor mapping table establishes a reversible correspondence between BIM components and simulation instances, enabling the results of the annotation, verification, and repair processes to be located at specific components and specific patch intervals, providing a necessary indexing basis for failure root cause localization and incremental updates.

[0016] Third, multimodal annotation is directly derived from BIM attributes and topological relationships. Its accuracy does not depend on the estimation of the visual model, thus avoiding the inherent category confusion and instance boundary drift of pseudo-annotation schemes in principle. A single parsing can simultaneously generate six types of annotations covering three downstream tasks: perception, operation, and language commands, significantly reducing annotation costs.

[0017] Fourth, the projection mask obtained by backprojection of 3D annotations is compared with the reference mask obtained by the instance identifier buffer output by the simulation renderer. This provides a pixel-level objective reference for the correctness of annotations, making the usability of annotations a quantifiable issue for the first time, rather than relying on manual sampling.

[0018] Fifth, physical accessibility testing can identify defects such as narrow doorways, disconnected areas, joint interference, and unsolvable grasping problems before data collection, intercepting invalid scenarios upstream in the data production pipeline and avoiding rework caused by discovering defects only after large-scale collection or even training.

[0019] Sixth, the effectiveness index formed by the weighted fusion of the three scores and its classification provide a unified criterion for dataset admission, and the weights can be adaptively adjusted according to the downstream embodied task type, so that the same set of scene assets can serve different tasks such as navigation, operation and command following.

[0020] Seventh, the failure root cause localization and repair operator call transforms the verification conclusions into executable repair actions, forming a self-healing data production closed loop and reducing manual intervention; coupled with the incremental update mechanism based on version fingerprint, after the BIM model is iterated, only the affected subtrees need to be recalculated locally, which greatly reduces the overhead of repeated calculations.

[0021] Eighth, the data asset description file records the validity index, three scores, repair records and version fingerprints, supporting the tracing and reproduction of training samples and the auditing of dataset versions, thus meeting the requirements of data asset management. Attached Figure Description

[0022] Figure 1 The overall flowchart of the automatic annotation and verification method for embodied simulation scenes that integrates BIM semantics provided in the embodiments of the present invention; Figure 2 This is a structural block diagram of the embodied simulation scene automatic annotation and verification device integrating BIM semantics provided in an embodiment of the present invention; Figure 3This is a schematic diagram illustrating the bidirectional anchoring relationship between the BIM structured semantic description, the semantic anchor mapping table, and the simulation scene hierarchy diagram in an embodiment of the present invention. Figure 4 This is a schematic diagram illustrating the principle of the patch-level semantic fingerprint inheritance mechanism in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the generation of multimodal labeled data in an embodiment of the present invention; Figure 6 This is a flowchart of the triple consistency verification and hierarchical scoring in an embodiment of the present invention; Figure 7 This is a schematic diagram illustrating the principle of consistency verification between the back projection mask and the instance identifier buffer in an embodiment of the present invention. Figure 8 This is a schematic diagram of physical reachability verification in an embodiment of the present invention; Figure 9 This is a schematic diagram of failure root cause localization and closed-loop repair in an embodiment of the present invention; Figure 10 This is a schematic diagram of incremental re-annotation and data asset description files in an embodiment of the present invention. Detailed Implementation

[0023] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] See Figure 1 This embodiment provides an automatic annotation and verification method for embodied simulation scenes that integrates BIM semantics, including the following steps S1 to S7.

[0025] In this embodiment, the BIM model uses an industrial base class file conforming to the IFC4 standard. After performing entity parsing on this file, the following five types of information are obtained, organized as follows: Figure 3 As shown in (a).

[0026] The first type is a spatial hierarchy tree. With IfcProject as the root node, it sequentially passes through IfcSite, IfcBuilding, IfcBuildingStorey to IfcSpace. Component entities are attached to the IfcSpace node via the IfcRelContainedInSpatialStructure relationship. This tree structure provides the source of floor and spatial constraints for subsequent generation of natural language expressions.

[0027] The second category is a collection of component entities, including types such as IfcWall, IfcSlab, IfcDoor, IfcWindow, IfcFurniture, IfcStair, IfcRailing, and IfcFlowTerminal. Each component entity has a globally unique 22-character identifier, which remains stable throughout the entire lifecycle of the BIM model and forms the basis for establishing semantic anchors in this invention.

[0028] The third category is the component attribute set, provided by IfcPropertySet and IfcElementQuantity, which includes fields such as geometric dimensions, opening method, opening angle, fire rating, load-bearing capacity, and whether it is movable.

[0029] The fourth category is the set of topological relationships between components, including IfcRelContainedInSpatialStructure representing spatial inclusion, IfcRelVoidsElement and IfcRelFillsElement representing openings and fills, IfcRelConnectsElements representing component connections, and IfcRelAggregates representing aggregations. Among them, the opening and fill relationship is used to identify the subordinate relationship between doors / windows and walls, and the connection relationship is used to infer the connection surface where the joint axis is located.

[0030] The fifth category is a material set, obtained by associating IfcRelAssociatesMaterial, which includes the material name, surface texture, and optional physical parameters such as density and coefficient of friction.

[0031] It should be noted that this invention is not limited to the IFC format. Native formats such as Revit and ArchiCAD, or the CityGML format for city information models, are also applicable to this invention, as long as they can export the aforementioned five types of information.

[0032] This step is the core of this invention, distinguishing it from existing BIM conversion processes. Its principle is as follows: Figure 4 As shown. The semantic fingerprint contains four fields: a globally unique identifier G for the component, which is the GlobalId of the IFC entity and remains stable throughout the component's lifecycle, such as 3n1x…7A; an original face identifier f, which is the sequence number of the face in the component's boundary representation, such as 12; a material identifier m, which is the index of the material associated with the face in the material set, such as MAT_OAK; and a normal clustering identifier κ, which is the category number obtained after performing spherical clustering on the unit normal, such as 3.

[0033] Specifically, step S2 includes sub-steps S21 to S24.

[0034] S21. Read the boundary representation of each component entity and generate a semantic fingerprint for each original geometric surface. The normal clustering identifier is obtained by performing spherical k-means clustering on the unit normal of all original surfaces of the component. In this embodiment, the number of clusters is six, corresponding to the six main directions of up, down, front, back, left, and right. The purpose of this identifier is that when multiple surfaces of the same component share the same material, the surfaces with different functions can still be distinguished according to their orientation, such as the tabletop and the side of the table leg.

[0035] S22. The geometric kernel is invoked to triangulate the boundary representation, resulting in a discrete mesh. For each triangular facet, its centroid and unit normal are calculated, and its inherited semantic fingerprint is determined according to the following rules: In the set of original geometric faces of the same component, the original geometric facet that minimizes the distance from the centroid to the facet and whose angle between the facet's normal and the normal of the facet at the nearest point is not greater than θ_max is retrieved, and its semantic fingerprint is assigned to the triangular facet. In this embodiment, θ_max is 15 degrees. The reason for introducing the normal angle constraint is that the front and back faces of a thin-walled component are extremely close in space. Matching only by the nearest distance will misclassify the back facet as the front facet, leading to directional errors in subsequent availability annotations.

[0036] S23. If no original geometric surface satisfies the normal angle constraint, such as a transitional surface generated during triangulation, a ray is projected in the opposite direction of the surface normal to obtain the semantic fingerprint of the first-hit original geometric surface. If the ray projection still fails to find a match, the surface is marked as a surface to be determined, and the mode fingerprint of the determined fingerprint within its connected component is retrieved from the surface adjacency graph and used to fill the gap. Through this three-level determination, no unassigned surfaces appeared in multiple test scenarios in this embodiment.

[0037] S24. Perform mesh simplification and convex decomposition on the discrete mesh. This embodiment uses a quadratic error metric method for mesh simplification to generate different levels of detail, and an approximate convex decomposition method to generate the colliders required by the physics engine. The above operations will merge multiple source patches into a new patch. At this time, the fingerprint of the new patch is determined by area-weighted voting: for the merged patch, the fingerprint of each source patch is voted on using the area as the weight, and the one with the most votes is taken as the fingerprint of the new patch; when the ratio of the maximum vote to the total votes is less than the percentage threshold ρ, it is determined that the patch spans two semantically different original patches, it is marked as a mixed patch, and the fingerprints of the top two votes are retained, and written to the geometric consistency verification queue. In this embodiment, ρ is 0.6.

[0038] The reason for using area weighting instead of vertex counting or simple majority voting is that the simplified mesh faces have significantly different geometric scales, and the area can more accurately reflect the actual contribution of the source faces to the merging result. Setting a percentage threshold and retaining dual fingerprints ensures that faces crossing semantic boundaries are not forcibly assigned to one side, avoiding systematic bias in the instance segmentation mask at object boundaries.

[0039] Through step S2, even after high-intensity mesh simplification and convex decomposition, each triangular facet in the simulation scene can still be traced back to a unique BIM component and original face, and the semantics remain traceable throughout the geometric transformation.

[0040] See Figure 3 In (b), each record in the semantic anchor mapping table contains five fields: a globally unique identifier for the component, the hierarchical path of the corresponding instance in the simulation scene, the patch index range occupied by the instance, the version fingerprint, and an optional remarks field. The hierarchical path is organized according to the scene graph of the simulation engine, for example, " / World / Building_F2 / Room_203 / Door_01".

[0041] The version fingerprint is obtained by concatenating the geometric summary and attribute set summary of the component, hashing the result, and truncating the hash. It is used to quickly determine whether the component has changed during BIM model iteration. This embodiment uses SHA-256 and extracts the first sixteen hexadecimal characters.

[0042] The semantic anchor mapping table simultaneously establishes hash indexes in two directions: a hierarchical path retrieved using a globally unique identifier, used to write BIM-side attributes and topology information into the simulation instance; and a globally unique identifier retrieved using the hierarchical path, used to trace back the failed surfaces discovered during the verification phase to specific components. This bidirectional index is a prerequisite for the localized execution of subsequent root cause localization and incremental updates.

[0043] See Figure 5 This embodiment generates six types of labeled data.

[0044] Firstly, there are two-dimensional instance segmentation masks and panoramic segmentation masks. Each instance is assigned an instance identifier based on the patch index range in the semantic anchor mapping table. This allows the simulation renderer to output pixel-by-pixel instance identifier buffers and category identifier buffers while rendering the color image; these two constitute a pixel-level segmentation mask. Because the identifiers are derived from patch attribution rather than visual inference, this mask remains accurate at occlusion boundaries and small structures.

[0045] Secondly, a 3D directed bounding box. For each instance, principal component analysis is used to determine the principal axis direction of the vertex set. The extreme values ​​along the principal axes are used to construct a directed bounding box, outputting the center coordinates, three-axis dimensions, and yaw angle around the vertical axis. For axis-aligned components such as walls and floors, the positioning grid direction in BIM is directly used to avoid directional jitter in principal component analysis on approximately symmetrical structures.

[0046] Third, semantic point cloud. Points are generated on the grid surface by uniform sampling based on area. Each point inherits the semantic fingerprint of its corresponding facet and is mapped to a semantic label and an instance label accordingly. For regions marked as mixed facests, point labels are randomly assigned according to the area ratio of the two retained fingerprints, and an uncertainty flag is set.

[0047] Fourth, availability zone labeling. Candidate availability categories are determined based on a pre-defined component type-availability mapping table, and then specific areas are filtered according to geometric conditions. The mapping table provides default physical parameters such as semantic category, availability, height range, mass, and coefficient of friction according to component type. An excerpt is as follows: IfcDoor corresponds to a door, with availability as openable / pullable, default mass 35kg, and coefficient of friction 0.55; IfcFurniture / Table corresponds to a table, with availability as a placement surface, height range 0.70–0.90m, default mass 32kg, and coefficient of friction 0.55; IfcFurniture / Chair corresponds to a chair, with availability as a seated surface, height range 0. 40–0.50m, default weight 6kg, coefficient of friction 0.60; IfcFurniture / Cabinet corresponds to a cabinet, available as a pull-out drawer or placement surface, height range 0.30–1.80m, default weight 45kg, coefficient of friction 0.50; IfcFlowTerminal / Switch corresponds to a switch, available as a pressable surface, height range 1.10–1.40m; IfcSlab corresponds to a floor slab, available as a passable surface, height 0.00m, coefficient of friction 0.70. Taking a desktop as an example, its criteria are: the angle between the normal and the opposite direction of gravity is not greater than 10 degrees, the height above the ground is between 0.70m and 0.90m, and the connected area is not less than 0.10 square meters; coplanar areas that meet the criteria are marked as "placement surfaces", and their boundary polygons are output.

[0048] Fifth, the joint types and range of motion parameters of interactive components. The joint type is determined based on the opening method field in the attribute set: hinged doors and windows correspond to revolute joints, sliding doors and drawers to sliding joints, and fixed panels to fixed joints. The position and direction of the joint axis are jointly determined by the connection surface indicated by the relationship between the component's bounding box main axis and the opening fill. The range of motion is taken from the opening angle or travel field in the attribute set; when this field is missing, the median of the distribution of this field among similar components in this model is taken as the inference value, and the source is marked as the inference value and its confidence level in the annotation, so that downstream users can decide whether to accept it. In this embodiment, the default opening angle of the hinged door is 95 degrees.

[0049] Sixth, natural language descriptor annotation. The process involves backtracking from the root node along the spatial hierarchy tree to obtain floor and spatial constraints. Then, based on the instance's coordinates within its space, directional, ordinal, and adjacency constraints are calculated and combined using templates to form a descriptor, such as "the second chair on the window side of meeting room 203 on the east side of the second floor." After generation, a uniqueness search is performed within the scene. If the expression points to more than one instance, the constraint with the highest distinguishability is recursively added until a unique instance is identified or the upper limit of the constraint is reached. This annotation can be directly used for training visual language navigation and language conditional manipulation tasks.

[0050] See Figure 6 This step executes three verification branches in parallel. The verification sub-items and criteria for each branch are as follows: The geometric consistency branch includes the back projection mask intersection-union ratio (IUU) test. In this embodiment, the number of viewpoints N is 24, the occlusion rate threshold is 0.5, the IUU threshold τ is 0.90, and the mesh quality check requires no non-manifold edges, no self-intersections, and consistent normals. The semantic consistency branch performs a similarity weighted comparison between BIM reasoning labels and visual language model recognition labels under a unified label system, with a conflict threshold of 0.60. The physical accessibility branch includes the passage width test, requiring the net width to be no less than the sum of the circumcircle diameter of the body (0.55m) and twice the safety margin (0.05m), the navigation connectivity test, with a grid resolution of 0.05m and expansion according to the body radius to retrieve take-off and landing point pairs, the joint full-stroke sweep collision test, with a step size of 2 degrees and no interference throughout the stroke, and the grab pose inverse kinematics test, sampling 128 poses for each target and requiring the existence of a collision-free feasible solution.

[0051] The principle of geometric consistency verification is as follows: Figure 7As shown, twenty-four virtual viewpoints are uniformly sampled on the sphere centered on the target instance. For each viewpoint, the 3D annotations are projected onto the image plane according to the camera's intrinsic and extrinsic parameters to obtain a projection mask. Simultaneously, the set of pixels with values ​​equal to the instance identifier is extracted from the instance identifier buffer output by the renderer as a reference mask. The intersection-union ratio (IU / U) of these two masks characterizes the degree of consistency between the 3D annotations and the actual rendering result. The occlusion rate under this viewpoint is calculated, and viewpoints with an occlusion rate greater than 0.5 are discarded. The average IU / U for the remaining valid viewpoints is taken as the geometric consistency score. For viewpoints with an IU / U less than a threshold, the differing pixels are back-projected onto the patch index and recorded as failed patches. This record directly serves subsequent root cause localization.

[0052] The key here is that the instance identifier buffer is written directly by the rendering pipeline during the rasterization stage, independent of the 3D annotation generation path, thus forming an objective reference for annotation correctness. If the semantic fingerprint is inherited incorrectly at some point, the projection mask and the reference mask at that point will inevitably not coincide, the intersection-union ratio will decrease accordingly, and the error can be automatically detected without manual sampling.

[0053] Semantic consistency verification maps the labels obtained from BIM attribute reasoning to the labels obtained from zero-sample identification of multi-view renderings by the visual language model, then compares them view-by-view, and weights them according to view confidence. When the degree of conflict between the two exceeds a threshold, the instance is added to the manual review queue and used as an active learning sampling sample. The role of this branch is to identify semantic errors in the BIM model itself, such as mislabeling of component types or incorrect reuse of furniture families, which cannot be detected by geometric consistency verification.

[0054] The four sub-items of physical accessibility verification are as follows: Figure 8 As shown, the steps are as follows: Calculate the net width of each doorway and passage segment and compare it with the diameter of the robot's circumscribed circle plus twice the safety margin; expand the occupied grid according to the robot's radius and retrieve the paths between each pair of take-off and landing points, and calculate the reachability ratio; for each joint, step in segments within the entire stroke at two-degree increments and detect collisions with adjacent components segment by segment; sample the pose around the availability area of ​​the target object and solve the inverse kinematics and collision detection. The physical reachability score is obtained by weighting and summing the scores of the four sub-items according to preset weights. In this embodiment, the sub-item weights are 0.3, 0.3, 0.2, and 0.2 respectively.

[0055] The three scores are weighted and combined to form an effectiveness index. In this embodiment, the weights are set to 0.35, 0.30, and 0.35 respectively, the threshold is set to 0.85, the A-level threshold is set to 0.92, and the maximum number of iterations is set to three.

[0056] When the effectiveness index is below the threshold, according to Figure 9The process shown constructs a failure feature vector and determines the root cause of the failure, then calls the corresponding repair operator. The correspondence between the root cause, typical features, and repair operator is as follows: Root cause R1 is mapping breakage, typical features are semantic anchor hit rate below the threshold and discontinuous patch index intervals, and the corresponding repair operator is semantic anchor reconstruction, i.e., ray projection backtracking and connected component mode fingerprint backfilling; Root cause R2 is attribute missing, typical features are the number of missing attribute fields greater than zero and low semantic consistency score, and the corresponding repair operator is attribute completion reasoning, based on the same type of component attributes. Inferences are made using the distribution of properties, the IFC default value library, and the material-property lookup table. Root cause R3 is a geometric defect, characterized by non-manifold and self-intersection counts greater than zero, and a low geometric consistency score. The corresponding repair operator is geometric repair, including hole filling, normal redirection, remeshing, and convex decomposition parameter renormalization. Root cause R4 is physical inaccessibility, characterized by a low physical accessibility score and failure points clustered at doorways or joints. The corresponding repair operator is layout parameter fine-tuning, including doorway clear width, furniture spacing, joint limits, and collision weight generation.

[0057] After the repair is completed, steps S4 to S5 are re-executed and re-evaluated only on the affected subtrees, while the existing results are reused for the unaffected parts. Figure 9 The right side shows the convergence process for a certain scenario: the initial validity index is 0.58, which rises to 0.92 after three rounds of repair, exceeding the Class A threshold. If the requirement is not met even after reaching the maximum number of iterations, the scenario is marked as unusable and pushed to the manual review queue along with the failure record.

[0058] It should be noted that the feasibility of closed-loop repair is based on the semantic anchor mapping table: since the failed facet can be traced back to the specific component through bidirectional indexing, the repair operator can be applied precisely to the problematic component rather than the entire scene, which is a capability that simulation scenes represented by pure mesh do not have.

[0059] Scenarios with an effectiveness index not less than the threshold are categorized into two levels, A and B, based on the index. Level A scenarios are directly included in the training set, while Level B scenarios are limited in use and subject to proportional sampling. Scenarios with an effectiveness index below the threshold and where repair has not converged are classified as Level C and rejected. A data asset description file is also generated, with fields such as... Figure 10 As shown in (b), this includes scene identifiers, source BIM version and version fingerprint, validity index and grading, three consistency scores, list of labeled items, repair records, manually reviewed items, and audit hashes and timestamps. This file is output along with the dataset, supporting the tracing and reproduction of training samples.

[0060] See Figure 10In section (a), when the BIM model is updated from version to a new version, this embodiment performs incremental re-annotation as follows: First, the version fingerprints of the old and new versions are compared for each component to obtain the set of components whose fingerprints have changed; then, based on the bidirectional hash index of the semantic anchor mapping table, this set is mapped to the corresponding instance and its subtree in the simulation scene; finally, steps S2 to S6 are re-executed only for the subtree and instances within its spatial neighborhood, while instances outside the subtree directly reuse the existing multimodal annotation data and verification conclusions.

[0061] The reason for introducing spatial neighborhood is that changes to components may affect the physical accessibility of their neighboring instances. For example, changes in door size will affect passage width, and furniture relocation will affect navigation connectivity and joint sweep results. Therefore, instances within the neighborhood need to be verified together. In this embodiment, the spatial neighborhood is the area covered by expanding outward by 1.5 meters from the bounding box of the changed component.

[0062] In this embodiment, the fusion weights are adaptively set according to the downstream embodied task type. When the downstream task is a navigation task, the scene's accessibility and connectivity are most critical, so the weight of the physical reachability score is increased to 0.50, while the weights of the geometric consistency and semantic consistency scores are correspondingly decreased. When the downstream task is an operation task, the spatial accuracy of the annotation directly determines the capture success rate, so the weight of the geometric consistency score is increased to 0.50. When the downstream task is a command-following task, the correctness of semantic labels and command expression is most critical, so the weight of the semantic consistency score is increased to 0.50. Thus, the same set of annotated and verified scene assets can be filtered according to different criteria to serve different training tasks.

[0063] See Figure 2 This embodiment provides an automatic annotation and verification device 100 for embodied simulation scenes that integrates BIM semantics, including a BIM semantic parsing module 10, a semantic anchor mapping module 20, a multimodal annotation generation module 30, a validity verification module 40, a closed-loop repair module 50, a data asset output module 60, and a storage module 70.

[0064] The BIM semantic parsing module 10 includes a spatial hierarchy tree construction unit 11, an attribute set and material extraction unit 12, and a component topology relationship parsing unit 13, used to execute step S1. The semantic anchor mapping module 20 includes a semantic fingerprint generation unit 21, a patch semantic inheritance unit 22, and a bidirectional index construction unit 23, used to execute steps S2 and S3. The multimodal annotation generation module 30 includes a segmentation and bounding box generation unit 31, an availability and joint annotation unit 32, and a pointer expression generation unit 33, used to execute step S4.

[0065] The validity verification module 40 includes a geometric consistency verification unit 41, a semantic consistency verification unit 42, a physical reachability verification unit 43, and a weighted fusion and hierarchical scoring unit 44, used to perform the fusion scoring in steps S5 and S6. The closed-loop repair module 50 includes a failure root cause localization unit 51 and a repair operator library and reconstruction unit 52, used to perform root cause localization and repair in step S6, and to send repair feedback back to the multimodal annotation generation module 30 to trigger re-annotation and re-verification. The data asset output module 60 is used to perform step S7. The storage module 70 is used to store the semantic anchor mapping table, the multimodal annotation library, verification records and audit logs, and the repair operator library, and interacts bidirectionally with the above modules.

[0066] The above modules can be implemented in the form of software functional modules, or partially implemented by hardware acceleration units such as field-programmable gate arrays or graphics processors. Each module can be centrally deployed on a single computing node or distributed across a computing cluster. Among them, the geometric consistency verification unit 41 and the physical reachability verification unit 43, due to their involvement in a large amount of rendering and physical simulation, should be deployed on nodes equipped with graphics processors and executed in parallel at the scene granularity.

[0067] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in any of embodiments one to three. The electronic device may also include a display interface and a network interface connected to a graphics processor.

[0068] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium includes, but is not limited to, a read-only memory, a random access memory, a magnetic disk, an optical disk, or a solid-state drive.

[0069] To verify the effectiveness of this invention, it was compared with manual annotation schemes and visual pseudo-annotation schemes in several medium-sized indoor architectural scenarios. The results are as follows: In terms of single-scene annotation time, the manual annotation scheme took 8.5 person-hours, the visual pseudo-annotation scheme took 0.6 person-hours, and this invention took 0.15 person-hours; in terms of the average intersection-union ratio of instance segmentation, the three schemes were 0.96, 0.78, and 0.94, respectively; in terms of availability annotation completeness rate, the three schemes were 0.72, 0.35, and 0.93, respectively; in terms of joint parameter annotation completeness rate, the three schemes were 0.65, 0.12, and 0.91, respectively; in terms of invalid scene detection rate, neither the manual annotation scheme nor the visual pseudo-annotation scheme had this capability, while this invention achieved 0.97; in terms of re-annotation time after BIM update, the manual annotation scheme took 8.5 person-hours, the visual pseudo-annotation scheme took 0.6 person-hours, and this invention took 0.02 person-hours.

[0070] As can be seen from the above comparison results, this invention achieves annotation accuracy close to that of manual annotation schemes while reducing the annotation time per scene by more than an order of magnitude; its completeness rate in fine-grained annotation such as availability and joint parameters is significantly better than the two comparative schemes; and this invention possesses the ability to detect invalid scenes, which the comparative schemes lack. In the scenario of BIM model updates, thanks to the incremental re-annotation mechanism supported by version fingerprint and semantic anchor mapping table, the re-annotation time is further reduced compared to a full redo.

[0071] Those skilled in the art should understand that the specific values ​​given in the above embodiments, such as θ_max being 15 degrees, ρ being 0.6, the number of viewpoints being 24, the intersection-union ratio threshold being 0.90, and the fusion weights being 0.35, 0.30, and 0.35, are all exemplary values ​​and can be adjusted according to the complexity of the scene, the robot's body parameters, and the requirements of the downstream task. Such adjustments do not constitute a limitation on the scope of protection of this invention.

[0072] Similarly, the visual language model, mesh simplification method, convex decomposition method, inverse kinematics solver and path retrieval algorithm described in this invention can all be replaced with other implementations in the field that have the same function, as long as they play the same role in the overall process, they fall within the protection scope of this invention.

[0073] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatic annotation and verification of embodied simulation scenes integrating BIM semantics, characterized in that, Includes the following steps: S1. Obtain the BIM model of the target building and parse the BIM model to obtain a structured semantic description; the structured semantic description includes a spatial hierarchy tree, a set of component entities, a set of component attributes, a set of topological relationships between components, and a set of materials; the spatial hierarchy tree takes the project node as the root node and includes site nodes, building nodes, floor nodes, and space nodes in sequence, and the component entities are attached to the space nodes. S2. Generate a semantic fingerprint for each original geometric face of each component entity in the component entity set. The semantic fingerprint includes the globally unique identifier of the component entity, the face identifier, material identifier, and normal clustering identifier of the original geometric face. Convert the geometric representation of the BIM model into a discrete mesh representation that can be loaded by the simulation engine. During the conversion process, determine the inherited semantic fingerprint of each triangular face in the discrete mesh by matching the centroid nearest neighbors and applying normal angle constraints. When performing mesh simplification and convex decomposition on the discrete mesh, determine the semantic fingerprint of the merged face according to the area-weighted voting result of the candidate source face. S3. Establish a semantic anchor mapping table based on the semantic fingerprint and the structured semantic description; each record in the semantic anchor mapping table includes the globally unique identifier of the component entity, the hierarchical path of the corresponding instance of the component entity in the simulation scene, the face index range occupied by the instance, and the version fingerprint calculated by the structured semantic description; the semantic anchor mapping table is configured with a hash index that supports bidirectional retrieval from the globally unique identifier to the hierarchical path and from the hierarchical path to the globally unique identifier; S4. Based on the semantic anchor mapping table and the structured semantic description, automatically generate multimodal annotation data for the simulation scene; the multimodal annotation data includes two-dimensional instance segmentation mask, three-dimensional directed bounding box, semantic point cloud, availability region annotation, joint type and motion range parameters of interactive components, and natural language index expression annotation generated based on the spatial hierarchy tree and spatial position relationship; S5. Perform triple consistency verification on the simulation scene and the multimodal annotation data to obtain geometric consistency score, semantic consistency score and physical reachability score respectively; wherein, the geometric consistency score is obtained by back-projecting the 3D annotation onto the virtual camera image plane to obtain a projection mask, obtaining the pixel-by-pixel instance identifier buffer output by the simulation renderer and generating a reference mask from it, and calculating the intersection-union ratio of the projection mask and the reference mask; the semantic consistency score is obtained by cross-comparing the semantic tags obtained by reasoning based on the structured semantic description with the semantic tags obtained by zero-shot recognition of the multi-view rendering images of the same instance by the visual language model, and fusing them by weighted fusion according to view confidence; the physical reachability score is obtained by performing passage width test, navigation connectivity test, joint full-stroke sweep collision test and grasping pose inverse kinematics feasibility sampling test on the simulation scene in the physics engine; S6. The geometric consistency score, semantic consistency score, and physical reachability score are weighted and fused according to preset weights to obtain the effectiveness index of the simulation scene. When the effectiveness index is less than a preset threshold, the root cause of failure is determined based on the failure feature vector composed of the three scores and their failure features. The repair operator corresponding to the root cause of failure is called to repair the semantic anchor mapping table, the structured semantic description, the discrete mesh, or the layout parameters of the simulation scene. Then, the execution steps S4 to S5 are returned until the effectiveness index is not less than the preset threshold or the number of iterations reaches the preset upper limit. S7. Output the simulation scenarios and their multimodal annotation data with validity indices not less than the preset threshold according to the validity index, and generate a data asset description file containing the validity index, the three consistency scores, the repair record and the version fingerprint.

2. The automatic annotation and verification method for embodied simulation scenes integrating BIM semantics as described in claim 1, characterized in that, The step S2, which involves determining the inherited semantic fingerprint by centroid nearest neighbor matching and applying normal angle constraints, includes: Calculate the centroid c_t and unit normal n_t of the triangular facet t; in the set of original geometric faces of the component entity to which the triangular facet belongs, search for the original geometric face that minimizes the distance from the centroid c_t to the original geometric face S_i and the angle between the unit normal n_t and the unit normal n_i at the nearest point of the original geometric face S_i is not greater than the preset angle threshold θ_max, and assign the semantic fingerprint of the original geometric facet to the triangular facet t; If there is no original geometric surface that satisfies the normal angle constraint, then a ray is projected onto the triangular facet t in the opposite direction of the unit normal n_t, and the semantic fingerprint of the original geometric surface that the ray first hits is assigned to the triangular facet t. If the ray projection does not hit any original geometric face, the triangular facet t is marked as a facet to be determined, and the mode fingerprint of the semantic fingerprint that has been determined in the connected component to which the triangular facet t belongs is taken and backfilled in the facet adjacency graph of the discrete grid.

3. The automatic annotation and verification method for embodied simulation scenes integrating BIM semantics as described in claim 1, characterized in that, The step S2, which involves determining the semantic fingerprint of the merged facet based on the area-weighted voting results of the candidate source facets, includes: For the merged facet t', obtain its source facet set, and calculate the votes for each candidate semantic fingerprint F according to s(F)=Σ A_t·1[F_t=F], where A_t is the area of ​​the source facet t, 1[·] is the indicator function, and summate and traverse the source facet set; The semantic fingerprint with the largest number of votes is used as the semantic fingerprint of the merged slice t'; When the ratio of the maximum number of votes to the total number of votes is less than the preset percentage threshold ρ, the merged facet t' is marked as a mixed facet, the semantic fingerprints of the top two vote-ranked faces are retained, and the mixed facet is written into the geometric consistency verification queue.

4. The method for automatic annotation and verification of embodied simulation scenes integrating BIM semantics as described in claim 1, characterized in that, The availability region annotations in step S4 are generated as follows: Based on the pre-defined component type-availability mapping table, determine the candidate availability categories of the component entity; Candidate surface regions are obtained by clustering the semantic patches of the component entities according to their normals and coplanarity, and the ground height, area, normal, and support relationship of the candidate surface regions with other component entities are calculated. When the angle between the normal of the candidate surface region and the opposite direction of gravity is not greater than a first preset value, the ground clearance is within the height range corresponding to the candidate availability category, and the area is not less than a second preset value, the candidate surface region is marked as the corresponding availability region, and its boundary polygon and the judgment condition on which it is based are recorded.

5. The method for automatic annotation and verification of embodied simulation scenes integrating BIM semantics according to claim 1, characterized in that, The joint type and range of motion parameters of the interactive component in step S4 are generated as follows: Based on the opening mode field in the component attribute set and the connection relationship in the component topology relationship set, the joint type of the interactive component is determined, and the joint type includes revolute joint, prismatic joint and fixed joint; The position and direction of the joint axis are determined based on the connection surface indicated by the geometric bounding box of the component entity and the connection relationship. The range of motion of the joint is determined based on the opening angle field or travel field in the component attribute set; when the field is missing, the mode or median of the attribute distribution of similar component entities in the BIM model is taken as the inference value, and the source of the parameter and its confidence level are recorded in the multimodal annotation data.

6. The method for automatic annotation and verification of embodied simulation scenes integrating BIM semantics according to claim 1, characterized in that, The natural language representation annotations mentioned in step S4 are generated as follows: By tracing back along the spatial hierarchy tree from the root node to the target instance, the floor constraints and spatial constraints are obtained; Based on the positional relationships of the target instance within its space, calculate the orientation constraint, ordinal constraint, and adjacency constraint; The floor limit, spatial limit, orientation limit, ordinal limit and adjacent object limit are combined into a pointer representation according to the preset template, and a uniqueness search is performed on the pointer representation within the simulation scene; when the number of instances pointed to by the pointer representation is greater than one, the limit with the highest distinguishability is recursively added until it uniquely points to an instance or the preset limit limit is reached.

7. The method for automatic annotation and verification of embodied simulation scenes integrating BIM semantics as described in claim 1, characterized in that, The geometric consistency score mentioned in step S5 is obtained as follows: N virtual viewpoints are uniformly sampled on a sphere centered on the target instance in the simulation scene; For each virtual viewpoint v, a projection mask is obtained by projecting the 3D annotations in the multimodal annotation data according to the camera intrinsic and extrinsic parameters, and a reference mask is obtained by the set of pixels in the instance identifier buffer whose values ​​are equal to the target instance identifier. Calculate the occlusion rate of the target instance under the virtual viewpoint, and remove virtual views with occlusion rates greater than a preset occlusion threshold to obtain the effective viewpoint set; The average value of the intersection-union ratio (IU) of the projected mask and the reference mask under each virtual viewpoint in the effective viewpoint set is taken as the geometric consistency score. The difference pixels corresponding to the virtual viewpoints with IU less than the preset IU threshold τ are back-projected to the patch index and recorded as failed patches.

8. The method for automatic annotation and verification of embodied simulation scenes integrating BIM semantics as described in claim 1, characterized in that, The semantic consistency score in step S5 is obtained as follows: The semantic tags obtained by reasoning based on the structured semantic description and the semantic tags output by the visual language model are mapped to a unified tag system; The semantic consistency score is calculated as C_s=Σ ω_v·sim(l_bim, l_vlm_v), where ω_v is the weight obtained by normalization according to view confidence, and sim(·,·) is the semantic similarity in the unified label system. The summation is performed by traversing each virtual view. When the semantic consistency score is less than the preset conflict threshold, the corresponding instance is written into the manual review queue and used as an active learning sampling sample.

9. The method for automatic annotation and verification of embodied simulation scenes integrating BIM semantics according to claim 1, characterized in that, The physical accessibility score in step S5 is obtained as follows: The passage width test is as follows: calculate the net width W_min for each doorway and passage segment in the simulation scenario. When W_min is not less than the sum of D_robot and 2δ, the doorway or passage segment is determined to be passable. The passability ratio is taken as the passage width sub-score, where D_robot is the diameter of the outer circle of the robot body and δ is the preset safety margin. The navigation connectivity test is as follows: the scene grid is expanded according to the radius of the circumscribed circle of the robot body, the path between each pair of take-off and landing points is retrieved on the expanded grid, and the ratio of the number of reachable pairs of take-off and landing points to the total number of pairs of take-off and landing points is taken as the navigation connectivity sub-score. The full-stroke sweep collision test of the joint is as follows: for each interactive component, the joint moves in segments within its range of motion according to a preset step length, and the collision between the interactive component and the adjacent component is detected segment by segment. The ratio of the number of joints without interference to the total number of joints is taken as the joint sub-score. The feasibility sampling test of the grasping pose inverse kinematics is as follows: the grasping pose is sampled around the availability region of the target instance, the inverse kinematics of the robotic arm is solved for each grasping pose and collisions are detected, and the ratio of the number of grasping poses with feasible solutions to the number of sampled grasping poses is taken as the grasping sub-score. The physical reachability score is obtained by weighting and summing the scores for the passage width, navigation connectivity, joints, and capture according to preset sub-weights.

10. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the automatic annotation and verification method for embodied simulation scenes incorporating BIM semantics as described in any one of claims 1 to 9.

Images