An augmented reality inspection method, device and medium based on a building information model

Abstract

This application discloses an augmented reality inspection method, device, and medium based on Building Information Modeling (BIM). The method includes: converting the BIM-conditional Gaussian point field and semantically signed distance field and collecting multi-source heterogeneous sensor data; constructing a joint optimization factor graph, calculating the current camera pose estimate and covariance matrix, and monitoring the values ​​of projection edge consistency residual and SDF distance residual in real time; dynamically adjusting the transparency and occlusion effect of the augmented reality rendering content according to the uncertainty of the pose estimate, and triggering local non-rigid deformation adjustment based on engineering tolerance constraints when the local area alignment error exceeds a threshold and the pose covariance is lower than a threshold value; generating a signed visual residual proof and performing digital signature and timestamp authentication; mapping the signed visual residual proof to a pre-constructed hash tree to calculate the root hash value, and storing the root hash value and digital signature on the blockchain.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and in particular to an augmented reality inspection method, equipment and medium based on building information modeling. Background Technology

[0002] In architectural and mechanical engineering construction sites, augmented reality technology has been used to overlay building information models onto the real environment to assist in inspection and acceptance. However, existing vision-based registration methods have significant limitations in scenarios with weak textures, highly reflective metal surfaces, and dynamic occlusion. Their pose estimation is prone to drift and realignment occurs frequently, seriously affecting the continuity and reliability of use. These methods typically rely on manual markings or natural environmental features, making it difficult to adapt to the complex geometry and lighting changes in construction scenarios. Furthermore, they lack quantification and visualization mechanisms for the uncertainty of positioning results, making it difficult for operators to detect the risk of false alignment.

[0003] Furthermore, existing solutions struggle to maintain spatial consistency during multi-person collaborative operations. Discrepancies arising from independent tracking by each terminal cannot be effectively eliminated, reducing the accuracy and efficiency of collaborative work. Regarding data trustworthiness, current methods primarily employ full-process video recording or overall model hashing on the blockchain. The former lacks structured, verifiable data support, while the latter, due to its coarse granularity, fails to achieve localized area status auditing. Neither approach addresses the privacy risks associated with the leakage of raw on-site data. Moreover, existing technologies fail to organically integrate physical spatial geometric deviations with engineering constraints, and do not establish a trustworthy closed loop from perception and registration to evidence storage, making it difficult to directly use augmented reality inspection results as the basis for acceptance and settlement. Summary of the Invention

[0004] This application provides an augmented reality inspection method, equipment, and medium based on building information modeling to solve the above-mentioned technical problems.

[0005] On the one hand, embodiments of this application provide an augmented reality inspection method based on building information modeling, including: The building information model is converted into a BIM-conditional Gaussian point field and a semantically signed distance field, and multi-source heterogeneous sensor data is collected to perform timestamp synchronization and alignment on the multi-source heterogeneous sensor data. A joint optimization factor graph is constructed, and an incremental optimization algorithm is used to calculate the pose estimate of the current camera and the corresponding covariance matrix. The values ​​of the projection edge consistency residual and SDF distance residual in the joint optimization factor graph are monitored in real time. The transparency and occlusion effects of augmented reality rendering content are dynamically adjusted based on the degree of uncertainty in pose estimation. When the alignment error of a local area exceeds the threshold and the pose covariance is lower than the threshold value, local non-rigid deformation adjustment based on engineering tolerance constraints is triggered. Based on the optimized residual distribution, the covariance matrix, and the deformation parameters, a signature visual residual proof is generated, and the signature visual residual proof is digitally signed and timestamped for authentication. The signature visual residual proof is mapped to a pre-constructed hash tree, the root hash value of the hash tree is calculated, and the root hash value and the digital signature are stored together on the blockchain to achieve traceability and verifiability of inspection evidence.

[0006] In one implementation of this application, the building information model is converted into a BIM-conditional Gaussian point field and a semantically signed distance field, specifically including: Uniform sampling is performed on the surface mesh and boundary wireframe of the building information model to obtain several three-dimensional points and their corresponding normal vectors, and an anisotropic Gaussian kernel function is assigned to each sampling point; The anisotropic covariance matrix is ​​calculated based on the material properties and construction stage parameters of the sampling points, and a semantic category label and initial uncertainty value are assigned to each Gaussian kernel function to construct a BIM conditional Gaussian point field with multi-dimensional attributes. Based on spatial grid partitioning, a voxelized distance field is generated. The signed distance field from each voxel to the nearest object surface is calculated, and each voxel is assigned a corresponding semantic category weight to generate a semantic signed distance field. Establish geometric consistency constraints between the BIM-conditional Gaussian point field and the semantically signed distance field to form a unified spatial prior representation model.

[0007] In one implementation of this application, data from multiple heterogeneous sensors is collected to perform timestamp synchronization and alignment on the data, specifically including: High-resolution stereo image sequences are acquired using a binocular camera, image feature points and edge information are extracted, and angular velocity and acceleration data are continuously acquired using a nine-axis inertial measurement unit to calculate the initial pose change of the device. By wireless communication between the ultra-wideband anchor point and the tag, ranging information, angle of arrival information, and time difference of arrival information are obtained, and acoustic impulse response data are collected through a microphone array to extract early reflection time characteristics; A precise time protocol is used to time-align the data streams from multiple heterogeneous sensors, establish a unified time series index, and buffer and filter the synchronized data to form a time-consistent multimodal observation dataset.

[0008] In one implementation of this application, constructing a joint optimization factor graph specifically includes: Calculate the grayscale gradient map of the current image frame, and perform a distance transformation operation on the grayscale gradient map to generate an image edge distance transformation map; Based on the current initial pose value, the three-dimensional points in the BIM conditional Gaussian point field are projected onto the two-dimensional pixel plane of the current image frame, and the corresponding distance values ​​of the three-dimensional points in the image edge distance transformation map are obtained to construct the projection edge consistency residual. The feature points in the current image frame are back-projected into the three-dimensional space, and the signed distance values ​​in the semantic signed distance field are queried to construct the SDF distance residual. Adaptive weights based on semantic categories are assigned to the projected edge consistency residual and the SDF distance residual, and robust kernel functions are applied to all types of residuals to suppress the impact of outlier observations on the optimization process.

[0009] In one implementation of this application, an incremental optimization algorithm is used to calculate the current camera pose estimate and the corresponding covariance matrix, and the values ​​of the projection edge consistency residual and SDF distance residual in the joint optimization factor graph are monitored in real time, specifically including: Within a fixed-length sliding window, the weighted sum of squares of all residuals in the joint optimization factor graph is iteratively minimized, and the covariance matrix of the current camera pose estimate is extracted after each iteration of optimization. The statistical distribution of projection edge consistency residual and SDF distance residual in different regions of the image is calculated, and the statistical distribution of the residual is fed back to the joint optimization factor map to dynamically adjust the parameters of the adaptive weights and robust kernel function based on semantic categories.

[0010] In one implementation of this application, when the alignment error of a local region exceeds a threshold and the pose covariance is lower than a threshold value, a local non-rigid deformation adjustment based on engineering tolerance constraints is triggered, specifically including: Real-time monitoring of projection edge consistency residuals and SDF distance residuals in local areas; when the projection edge consistency residuals or the SDF distance residuals exceed a preset threshold, and the trace of the covariance matrix is ​​lower than a preset confidence threshold, the deformation optimization process is activated. Select key electromechanical components as deformation anchor points and add constraints to ensure that the changes in the radius and curvature of the deformed components, as well as their relative positional relationship with adjacent components, meet the predefined engineering tolerance parameters. Solve for the optimal deformation parameters that satisfy the geometric constraints, and record the maximum displacement, total bending angle and affected component segment identifiers during the deformation process to generate a deformation summary report.

[0011] In one implementation of this application, a signature visual residual proof is generated based on the optimized residual distribution, the covariance matrix, and the deformation parameters, and the signature visual residual proof is digitally signed and timestamped for authentication, specifically including: Obtain the optimized projection edge consistency residual, generate a residual distribution heatmap, and statistically analyze the optimized SDF distance residual to generate an error statistics histogram; Record the eigenvalues ​​and condition number of the optimized pose covariance matrix, and organize the adjustment range and constraints of the local deformation parameters; The system aggregates device identification information, timestamp sequences, and environmental context data. It then performs a digital signature on the aggregated visual residual proof through a trusted execution environment and adds timestamp authentication information provided by the Network Time Protocol.

[0012] In one implementation of this application, the signature visual residual proof is mapped to a pre-constructed hash tree, the root hash value of the hash tree is calculated, and the root hash value and the digital signature are stored together on the blockchain to achieve traceability and verifiability of inspection evidence, specifically including: The three-dimensional space containing the building information model is divided into a multi-level octree structure; each leaf node of the octree corresponds to a specific spatial region. The signature visual residual proof is associated with the corresponding spatial region, and the hash value corresponding to each leaf node is calculated, so that the hash value is used as the data of the corresponding leaf node in the octree; The hash of the parent node in the octree is calculated recursively from bottom to top to generate a global root hash value and add timestamp information; The root hash value, the changed node hash value, and the digital signature are submitted to the blockchain network through a smart contract, and a relationship is established between the root hash value and the on-chain data. The complete visual residual proof of the signature is stored in a distributed storage system under the blockchain, and a local verification interface based on spatial regions is provided to allow verification of the integrity of evidence in specific regions.

[0013] On the other hand, embodiments of this application also provide an augmented reality inspection device based on building information modeling, the device comprising: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are then executed by the at least one processor to enable the at least one processor to perform an augmented reality inspection method based on a building information model as described above.

[0014] On the other hand, this application embodiment also provides a non-volatile computer storage medium storing computer-executable instructions, which, when executed, implement the augmented reality inspection method based on building information model as described above.

[0015] This application provides an augmented reality inspection method, device, and medium based on building information modeling, which has at least the following beneficial effects: By converting Building Information Modeling (BIM) into BIM-conditional Gaussian point fields and semantically signed distance fields, a high-precision, differentiable spatial prior representation is provided for registration optimization, solving the problems of difficult feature extraction and registration failure in harsh environments that traditional methods face. By constructing a joint optimization factor graph, even in scenarios where single sensors are prone to failure, the system can utilize stable geometric constraints provided by BIM priors for continuous correction and self-healing, achieving deep fusion and complementarity of multimodal sensor information. Through dynamically adjusting rendering effects and triggering local non-rigid deformation adjustments constrained by engineering tolerances under specific conditions, the system can intelligently absorb unavoidable installation deviations on the construction site, maintaining visual alignment between the virtual model and the real scene while ensuring geometric accuracy and reducing false alarm rates. By constructing a trusted data closed loop from the physical site to digital twins and then to blockchain, a privacy-friendly automated means is provided for project acceptance and settlement, completely changing the traditional inspection methods that rely on coarse-grained and easily disputed manual recording and screenshots, providing directly credible digital evidence for project acceptance and progress payments. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating an augmented reality inspection method based on building information modeling provided in this application embodiment; Figure 2 This is a schematic diagram of the internal structure of an augmented reality inspection device based on building information modeling, provided as an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0019] Figure 1 This is a flowchart illustrating an augmented reality inspection method based on building information modeling, provided in an embodiment of this application.

[0020] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using a server as an example.

[0021] It should be noted that the server can be a single device or a system composed of multiple devices, i.e., a distributed server. This application does not make any specific limitations on this.

[0022] like Figure 1 As shown in the embodiment of this application, an augmented reality inspection method based on building information modeling is provided, including: Step 101: Convert the building information model into a BIM-conditional Gaussian point field and a semantically signed distance field, and collect multi-source heterogeneous sensor data to perform timestamp synchronization and alignment of the multi-source heterogeneous sensor data.

[0023] In this embodiment, the surface geometry and boundary features in the building information model are systematically sampled. The sampling process adopts a curvature-adaptive strategy, increasing the sampling density in areas with large curvature changes to obtain a three-dimensional point set that accurately reflects the geometric features of the building components. In addition to spatial coordinates, each sampling point also has its orientation information obtained by calculating the local surface normal.

[0024] For example, the anisotropic covariance matrix is ​​calculated based on the material properties of the component where the sampling point is located and the parameters of the construction stage. Surfaces made of highly reflective materials are assigned lower confidence weights due to their susceptibility to illumination, while rough surfaces are assigned higher confidence weights. Simultaneously, each Gaussian point is assigned a semantic category label extracted from the BIM model and an initial uncertainty value, thereby constructing a BIM-conditional Gaussian point field with multi-dimensional attributes.

[0025] Understandably, the generation of the semantic signed distance field is based on spatial grid partitioning, which voxels the entire scene. For each voxel, the signed distance to the nearest object surface is calculated, and a differentiated weight is assigned based on the semantic category to which the voxel belongs. Permanent structures are given high weights, while temporary facilities or mobile devices are given low weights. This weighting configuration ensures that different categories of components have different degrees of influence on the overall objective function during subsequent optimization.

[0026] Specifically, the acquisition of multi-source heterogeneous sensor data covers multiple modalities including vision, inertial, radio, and acoustic. High-resolution stereo image sequences are acquired through a binocular camera, and image feature points and edge information are extracted; angular velocity and acceleration data are continuously acquired through an inertial measurement unit; ranging information, angle of arrival information, and time difference of arrival information are obtained through wireless communication between an ultra-wideband anchor point and a mobile tag; and acoustic impulse response data are acquired through a microphone array, and early reflection time characteristics are extracted.

[0027] It should be noted that the timestamp synchronization and alignment adopts a hardware and software combination scheme based on a precise time protocol, establishing a unified high-precision time reference on the main control device. All sensor data streams are timestamped based on this reference during acquisition, and all data are aligned to a unified time series through methods such as time interpolation or data buffer alignment, forming a multimodal observation dataset with strict time consistency.

[0028] Step 102: Construct a joint optimization factor map, use an incremental optimization algorithm to calculate the current camera pose estimate and the corresponding covariance matrix, and monitor the values ​​of the projection edge consistency residual and SDF distance residual in the joint optimization factor map in real time.

[0029] In this embodiment, it should be noted that the joint optimization factor graph is a probabilistic graphical model, where nodes represent state variables to be optimized and edges represent constraint factors established based on observations. Specifically, it includes visual inertial odometry factor, UWB factor, acoustic RIR factor, and two BIM prior factors: projection edge consistency residual and SDF distance residual.

[0030] Specifically, the acquired color image is first converted to a grayscale image. Then, the gradient magnitude of the grayscale image at each pixel is calculated, resulting in a grayscale gradient map that highlights the image edges. It's important to note that the purpose of the distance transformation on the gradient map is to calculate the Euclidean distance from each pixel in the image to the nearest edge pixel. The final generated image edge distance transformation map has values ​​of zero at the edges and monotonically increases with increasing distance from the edge. This image edge distance transformation map allows for soft matching of projection points, meaning that the projection points do not need to fall precisely on the image edges. Instead, the degree of matching is quantified by the distance value, enhancing the algorithm's robustness to noise and local occlusion, and facilitating the subsequent construction of geometric matching residuals with BIM priors.

[0031] For example, when constructing the projection edge consistency residual, the initial estimate of the current camera pose, typically from VIO predictions or optimization results from the previous frame, is used to project 3D points in the BIM conditional Gaussian point field (BGS) onto the 2D pixel coordinate system of the current image frame. Then, the distance values ​​of these projected points in the newly generated image edge distance transformation map are queried. It is understood that if the pose estimation is perfectly accurate and the BIM is perfectly aligned with the real scene, then these 3D points originating from the edges of the BIM model should be projected onto the edges of the real image, meaning their distance values ​​should be zero or close to zero. Therefore, the distance value of each projected point constitutes a projection edge consistency residual term. The sum of the squares of all residuals represents the overall mismatch between the BIM model projection and the real image edges under the current pose, and the goal of the optimization process is to adjust the pose to minimize this residual.

[0032] Specifically, when constructing the SDF distance residual, feature points extracted from the current image frame, such as corner points extracted using FAST or ORB algorithms, are back-projected into 3D space, combining camera intrinsics and depth information (if applicable) or using multi-view geometric methods. Then, for each back-projected 3D point, its corresponding signed distance value in the Semantic Signed Distance Field (SDF) is queried. It should be noted that, ideally, points back-projected from the real scene should lie on the surface of objects, meaning their value in the SDF should be zero. This signed distance value constitutes the SDF distance residual, which directly compares the observed scene geometry with the geometry of the BIM prior model, providing a global, dense geometric constraint.

[0033] Finally, assigning adaptive weights to the residuals and applying robust kernel functions is crucial for ensuring optimization robustness. For example, both the projection edge consistency residual and the SDF distance residual are assigned adaptive weights based on the semantic category of their associated BIM components. For instance, the residual weight associated with a load-bearing wall will be significantly higher than that associated with a temporary toolbox, because load-bearing walls are more stable and reliable. Simultaneously, robust kernel functions such as Huber or Cauchy are applied to all residual terms. Understandably, the role of robust kernel functions is to reduce the influence of particularly large residuals on the optimization objective function, such as those caused by anomalous observations like transient occlusion, strong reflections, or moving objects, thereby preventing these outliers from skewing the optimization process and ensuring the system's stability in complex field environments.

[0034] In this embodiment, when iteratively minimizing the weighted sum of squares within a sliding window, the system maintains a sliding window of fixed time length. This window contains the camera pose state variables, sensor biases, and all relevant observation factors for the most recent period. The optimizer, such as those using the Gauss-Newton or Levenberg-Marquardt methods, iteratively adjusts all state variables within the window to minimize the weighted sum of squares of the residuals of all factors. It should be noted that after each successful iteration, the system not only outputs the optimal estimate of the latest camera pose within the window but also calculates the covariance matrix of that pose. The covariance matrix quantifies the uncertainty of the current pose estimate, providing an objective basis for subsequent confidence rendering and deformation-triggered decisions.

[0035] For example, real-time monitoring of residuals and feeding them back to the factor map constitutes an intelligent closed loop for achieving the system's self-healing function. After each optimization, the system performs a detailed analysis of the statistical distribution of projection edge consistency residuals and SDF distance residuals across different spatial regions of the image. For instance, it checks whether the mean residual value in a certain local area (such as the upper left corner of the image) is higher than that in other areas, or whether the variance of the residuals is abnormal. Understandably, such anomalies in spatial distribution often reveal the problem, which could be due to temporary occlusion in the area, or a deviation between the BIM model and the actual geometry of the area that cannot be compensated for by rigid body pose adjustment, i.e., construction error.

[0036] At this point, the system uses this analysis result as a feedback signal to dynamically adjust the semantic category weights of relevant residuals and the parameters of the robust kernel function in the factor graph. For example, if a region is found to have persistently high residuals and is determined to be due to dynamic occlusion, the system can automatically reduce the weights of all residuals in that region or tighten the threshold of the robust kernel function, making it more proactive in treating these observations as outliers and suppressing their impact. This dynamic adjustment mechanism allows the factor graph to adapt to environmental changes rather than rigidly processing all observation data, thereby significantly improving the stability and reliability of long-term operation.

[0037] Step 103: Dynamically adjust the transparency and occlusion effect of the augmented reality rendering content based on the uncertainty of the pose estimation. When the alignment error of a local area exceeds the threshold and the pose covariance is lower than the threshold value, trigger the local non-rigid deformation adjustment based on engineering tolerance constraints.

[0038] In this embodiment, the calculated pose covariance matrix is ​​mapped to the rendering engine. It is understood that when the covariance is large, indicating high pose uncertainty, the system automatically reduces the opacity of the virtual model or switches to wireframe display mode, and issues a prompt to the user, thereby effectively avoiding misleading overlays when positioning is unreliable.

[0039] Understandably, traditional augmented reality registration methods typically employ rigid transformations, meaning that only the entire model can be rotated and translated. When there are significant deviations between the on-site installation and the design, this rigid overlay can lead to noticeable misalignment between the virtual model and the real object in certain areas, severely impacting the accuracy of inspections and the user experience.

[0040] This application aims to intelligently and controllably adjust the geometry of a BIM model locally to fit the actual scene. For example, the system divides an image or 3D space into multiple local regions, such as based on a grid or semantic instances, and independently calculates the statistical characteristics of the residuals within each region, such as the average and maximum values. It should be noted that triggering deformation requires two key conditions to be met simultaneously: First, in a certain local region, for example, the area surrounding a specific valve, the residuals consistently and significantly exceed a preset threshold, indicating that simple camera pose adjustment cannot eliminate the geometric inconsistencies here; second, simultaneously, the trace of the covariance matrix of the current camera pose estimation must be below a preset confidence threshold. It is understood that the trace of the covariance matrix is ​​a scalar indicator measuring the overall uncertainty of the pose; a trace below the threshold means that the system has high confidence in its positioning results, excluding the possibility that inaccurate positioning itself causes an increase in residuals. The dual-condition triggering mechanism of this application is a safety valve to prevent deformation from masking positioning errors, ensuring that deformation operations are only activated when positioning is reliable and geometric deviations actually exist.

[0041] Specifically, selecting deformation anchor points and adding engineering tolerance constraints are crucial to ensuring that the deformation results conform to engineering practices and acceptance specifications. Once the deformation process is activated, the system first selects key electromechanical components as deformation anchor points. For example, for a pipeline system, key connection points and support points such as valves, flanges, and supports are selected as control points (anchor points) for deformation. It's important to note that deformation is not arbitrary but is solved under strict constraints of preset engineering tolerance parameters. These constraints are explicitly added to the optimization problem, including that the change in the radius of the deformed component must not exceed the design tolerance range; the change in the curvature of the pipeline must meet the minimum bending radius requirement to prevent unrealistic bends; and the deformed component must maintain necessary coaxiality, parallelism, or clearance (i.e., the minimum distance between it and other components or structures) thresholds with adjacent components. These constraints ensure that although the deformed BIM model is geometrically adjusted, it remains completely within the acceptable range of engineering specifications and will not produce results that violate physical laws or safety regulations.

[0042] Finally, the optimal deformation parameters are solved and a deformation summary report is generated, completing the closed loop from calculation to recording. The system obtains a set of optimal deformation parameters, such as the displacement of B-spline control points, by solving a constrained optimization problem. For example, before and after deformation application, the system automatically records key deformation metrics, such as maximum displacement (the maximum distance a point moves within the deformation area), total bending angle (for pipes, describing the change in their overall bending degree), and affected component segment identifiers (clearly recording which BIM components were adjusted).

[0043] All this information is compiled into a deformation summary report. Understandably, this report makes the deformation operation fully traceable and auditable. During the subsequent acceptance phase, relevant personnel can review the report to understand the extent of adjustments made to the design model to align with actual conditions. If necessary, they can easily revert to the original design model for comparison, ensuring transparency and credibility throughout the process and avoiding potential disputes arising from model adjustments.

[0044] Step 104: Based on the optimized residual distribution, covariance matrix, and deformation parameters, generate a signed visual residual proof, and perform digital signature and timestamp authentication on the signed visual residual proof.

[0045] In this embodiment, the system first obtains the optimized projection edge consistency residuals. These residuals are the final mismatch between the BIM projection and the real image edges under the current optimal pose estimation after optimization by the joint factor graph. For example, to visually demonstrate the spatial distribution of this mismatch, the system maps these residual values ​​onto the image plane, generating a residual distribution heatmap. In this heatmap, warmer colors (e.g., red) indicate larger residuals, meaning a greater alignment error between the virtual model and the real scene at that location; cooler colors (e.g., blue) indicate good alignment. It should be noted that the system performs statistical analysis on the optimized SDF distance residuals, calculating their mean, variance, maximum value, and other statistics, and generates an error statistics histogram to describe the overall consistency between the scene's 3D geometry and the BIM prior. These charts transform abstract numerical errors into intuitive and highly readable visual evidence.

[0046] The system also records the eigenvalues ​​and condition number of the optimized pose covariance matrix. Understandably, the magnitude of the eigenvalues ​​reflects the uncertainty of the pose in different directions, while the condition number characterizes the ill-conditioned nature of the covariance matrix. Together, they constitute a professional and quantitative description of the quality of this pose estimation. Furthermore, if local non-rigid deformation is triggered, the system will organize the adjustment range and constraints of the local deformation parameters, such as the specific section where the deformation is applied and the engineering tolerance constraints, such as a pipe diameter change not exceeding ±5 mm. This information proves that the deformation performed is controlled and compliant with regulations, rather than an arbitrary alteration of the model, ensuring the compliance and auditability of the evidence.

[0047] The system aggregates all evidence data (residual heatmap, error histogram, covariance summary, deformation summary), unique device identifiers (to prevent device impersonation), precise timestamp sequences (to form an operation timeline), and environmental context data (such as scene type and lighting condition summary) into a complete signed visual residual proof data package. It should be noted that, to ensure the integrity and non-repudiation of this evidence package, the system digitally signs the signed visual residual proof data package using the device's private key through a Trusted Execution Environment (TEE) or secure chip on the device. For example, simultaneously, the system obtains an authoritative and trusted timestamp authentication information from a Network Time Protocol (NTP) server and appends it to the signed visual residual proof data package. This process is equivalent to giving this inspection evidence a unique, legally valid digital fingerprint and timestamp. Any subsequent tampering with the evidence content will cause signature verification to fail, thus ensuring the credibility of the entire chain from data generation to final submission for audit.

[0048] Step 105: Map the signature visual residual proof to a pre-built hash tree, calculate the root hash value of the hash tree, and store the root hash value and digital signature together on the blockchain to achieve traceability and verifiability of inspection evidence.

[0049] In this embodiment, it is understood that the traditional method of hashing the entire data file onto the blockchain can ensure that the data is not tampered with, but it cannot support efficient and local verification of evidence for a specific area or component in a large BIM model, and may expose unnecessary global information. This application solves this problem by introducing a spatial octree structure.

[0050] Specifically, an octree is a tree-like data structure used to manage three-dimensional space. For example, the construction of an octree begins with a cube (root node) that can enclose the entire BIM model. This cube is then uniformly divided into eight child cubes (child nodes), and this process is repeated recursively until the resulting cubes (leaf nodes) reach a predetermined resolution. It's important to note that in this way, the entire physical space is discretized into a hierarchical set of voxels (3D pixels) of varying sizes. Each leaf node corresponds to a specific, fine-grained spatial region. This structure provides a framework for associating evidence with spatial location.

[0051] Once an SVRP is generated, the system associates it with one or more leaf nodes in the octree based on the actual spatial location of the BIM component or inspection operation associated with the evidence. Understandably, one piece of evidence may cover multiple adjacent leaf nodes; for example, inspection evidence for a long pipeline might be associated with multiple spatial voxels distributed along the pipeline. The system then calculates the hash value of the data for each associated leaf node. If a leaf node is associated with multiple pieces of evidence, the evidence data is typically serialized before the hash is calculated. It should be noted that the hash value serves as a unique and tamper-proof digest of the leaf node within the hash tree.

[0052] Next, starting from all leaf nodes, the system concatenates the hash values ​​of every eight adjacent child nodes to calculate the hash value of their parent node. For example, this process is performed recursively until the hash value of the root node (i.e., the root node) is calculated; this value is called the global root hash. It's important to note that any tiny change to the data of any underlying leaf node will cause a chain reaction of changes to the hash values ​​of all its ancestor nodes, ultimately leading to a change in the root hash value. Therefore, the root hash can be viewed as a global, compact, and complete integrity fingerprint of all related evidence in the entire space at a specific moment. Furthermore, the system also appends an authoritative timestamp to the root hash.

[0053] Instead of directly uploading the massive amount of raw SVRP data to the blockchain, the system writes and invokes a smart contract to submit only the calculated global root hash, the hash values ​​of nodes that changed during the update (used to prove the update path), and the digital signature to the blockchain network as a transaction. Understandably, once this transaction is confirmed and recorded by the blockchain network, it means that the root hash and the entire spatial evidence state it represents are permanently and immutably anchored at that point in time.

[0054] Finally, storing the complete SVRP off-chain and providing a local verification interface achieves a balance between minimizing data on-chain storage and ensuring efficient verifiability. The complete SVRP, containing detailed information, is stored off-chain in a distributed storage system, such as IPFS or an enterprise private cloud. For example, when an auditor needs to verify the authenticity and tamper-proof nature of evidence in a specific area—such as verifying only valve V-101 in the pump room on the east side of the third floor—a request can be initiated through a local verification interface based on a spatial region.

[0055] It's important to note that during verification, the service provider offers the leaf node hash corresponding to that region, as well as the hashes of all sibling nodes along the path from that leaf node to the root hash (referred to as the Merkle path). The verifier can obtain the complete SVRP data corresponding to that leaf node off-chain, recalculate its hash, and then use the provided Merkle path and the root hash stored on-chain to verify the integrity of that local data. This approach ensures global trustworthiness while avoiding the public exposure of all sensitive data on the blockchain, achieving a perfect balance between privacy protection and verifiability.

[0056] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide an augmented reality inspection device based on building information modeling, the structure of which is as follows: Figure 2 As shown.

[0057] Figure 2 This is a schematic diagram of the internal structure of an augmented reality inspection device based on building information modeling, provided as an embodiment of this application. Figure 2 As shown, the device includes: At least one processor; And, a memory that is communicatively connected to at least one processor; The memory stores instructions that can be executed by at least one processor, and the instructions, when executed by at least one processor, enable at least one processor to: The building information model is converted into a BIM-conditional Gaussian point field and a semantically signed distance field, and multi-source heterogeneous sensor data is collected to perform timestamp synchronization and alignment of the multi-source heterogeneous sensor data. A joint optimization factor graph is constructed, and an incremental optimization algorithm is used to calculate the pose estimate of the current camera and the corresponding covariance matrix. The values ​​of the projection edge consistency residual and SDF distance residual in the joint optimization factor graph are monitored in real time. The transparency and occlusion effects of augmented reality rendering content are dynamically adjusted based on the degree of uncertainty in pose estimation. When the alignment error of a local area exceeds the threshold and the pose covariance is lower than the threshold value, local non-rigid deformation adjustment based on engineering tolerance constraints is triggered. Based on the optimized residual distribution, covariance matrix, and deformation parameters, a signature visual residual proof is generated, and the signature visual residual proof is digitally signed and timestamped for authentication. The signature visual residual proof is mapped to a pre-built hash tree, the root hash value of the hash tree is calculated, and the root hash value and digital signature are stored together on the blockchain to achieve traceability and verifiability of inspection evidence.

[0058] This application also provides a non-volatile computer storage medium storing computer-executable instructions, which, when executed, can: The building information model is converted into a BIM-conditional Gaussian point field and a semantically signed distance field, and multi-source heterogeneous sensor data is collected to perform timestamp synchronization and alignment of the multi-source heterogeneous sensor data. A joint optimization factor graph is constructed, and an incremental optimization algorithm is used to calculate the pose estimate of the current camera and the corresponding covariance matrix. The values ​​of the projection edge consistency residual and SDF distance residual in the joint optimization factor graph are monitored in real time. The transparency and occlusion effects of augmented reality rendering content are dynamically adjusted based on the degree of uncertainty in pose estimation. When the alignment error of a local area exceeds the threshold and the pose covariance is lower than the threshold value, local non-rigid deformation adjustment based on engineering tolerance constraints is triggered. Based on the optimized residual distribution, covariance matrix, and deformation parameters, a signature visual residual proof is generated, and the signature visual residual proof is digitally signed and timestamped for authentication. The signature visual residual proof is mapped to a pre-built hash tree, the root hash value of the hash tree is calculated, and the root hash value and digital signature are stored together on the blockchain to achieve traceability and verifiability of inspection evidence.

[0059] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0060] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0061] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0062] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0063] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0064] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0065] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0066] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0067] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0068] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0069] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A building information model based augmented reality inspection method, characterized in that, The method includes: The building information model is converted into a BIM-conditional Gaussian point field and a semantically signed distance field, and multi-source heterogeneous sensor data is collected to perform timestamp synchronization and alignment on the multi-source heterogeneous sensor data. A joint optimization factor graph is constructed, and an incremental optimization algorithm is used to calculate the pose estimate of the current camera and the corresponding covariance matrix. The values ​​of the projection edge consistency residual and SDF distance residual in the joint optimization factor graph are monitored in real time. The transparency and occlusion effects of augmented reality rendering content are dynamically adjusted based on the degree of uncertainty in pose estimation. When the alignment error of a local area exceeds the threshold and the pose covariance is lower than the threshold value, local non-rigid deformation adjustment based on engineering tolerance constraints is triggered. Based on the optimized residual distribution, the covariance matrix, and the deformation parameters, a signature visual residual proof is generated, and the signature visual residual proof is digitally signed and timestamped for authentication. The signature visual residual proof is mapped to a pre-constructed hash tree, the root hash value of the hash tree is calculated, and the root hash value and the digital signature are stored together on the blockchain to achieve traceability and verifiability of inspection evidence. The building information model is converted into a BIM-conditional Gaussian point field and a semantically signed distance field, specifically including: Uniform sampling is performed on the surface mesh and boundary wireframe of the building information model to obtain several three-dimensional points and their corresponding normal vectors, and an anisotropic Gaussian kernel function is assigned to each sampling point; The anisotropic covariance matrix is ​​calculated based on the material properties and construction stage parameters of the sampling points, and a semantic category label and initial uncertainty value are assigned to each Gaussian kernel function to construct a BIM conditional Gaussian point field with multi-dimensional attributes. Based on spatial grid partitioning, a voxelized distance field is generated. The signed distance field from each voxel to the nearest object surface is calculated, and each voxel is assigned a corresponding semantic category weight to generate a semantic signed distance field. Establish geometric consistency constraints between the BIM-conditional Gaussian point field and the semantically signed distance field to form a unified spatial prior representation model.

2. The augmented reality inspection method based on building information model according to claim 1, characterized in that, Collecting data from multiple heterogeneous sensors and synchronizing and aligning the timestamps of the data specifically includes: High-resolution stereo image sequences are acquired using a binocular camera, image feature points and edge information are extracted, and angular velocity and acceleration data are continuously acquired using a nine-axis inertial measurement unit to calculate the initial pose change of the device. By wireless communication between the ultra-wideband anchor point and the tag, ranging information, angle of arrival information, and time difference of arrival information are obtained, and acoustic impulse response data are collected through a microphone array to extract early reflection time characteristics; A precise time protocol is used to time-align the data streams from multiple heterogeneous sensors, establish a unified time series index, and buffer and filter the synchronized data to form a time-consistent multimodal observation dataset. 3.The building information model-based augmented reality inspection method of claim 1, wherein, Constructing a joint optimization factor graph specifically includes: Calculate the grayscale gradient map of the current image frame, and perform a distance transformation operation on the grayscale gradient map to generate an image edge distance transformation map; Based on the current initial pose value, the three-dimensional points in the BIM conditional Gaussian point field are projected onto the two-dimensional pixel plane of the current image frame, and the corresponding distance values ​​of the three-dimensional points in the image edge distance transformation map are obtained to construct the projection edge consistency residual. The feature points in the current image frame are back-projected into the three-dimensional space, and the signed distance values ​​in the semantic signed distance field are queried to construct the SDF distance residual. Adaptive weights based on semantic categories are assigned to the projected edge consistency residual and the SDF distance residual, and robust kernel functions are applied to all types of residuals to suppress the impact of outlier observations on the optimization process.

4. The augmented reality inspection method based on building information model according to claim 3, characterized in that, An incremental optimization algorithm is used to calculate the current camera pose estimate and the corresponding covariance matrix, and the values ​​of the projection edge consistency residual and SDF distance residual in the joint optimization factor graph are monitored in real time. Specifically, this includes: Within a fixed-length sliding window, the weighted sum of squares of all residuals in the joint optimization factor graph is iteratively minimized, and the covariance matrix of the current camera pose estimate is extracted after each iteration of optimization. The statistical distribution of projection edge consistency residual and SDF distance residual in different regions of the image is calculated, and the statistical distribution of the residual is fed back to the joint optimization factor map to dynamically adjust the parameters of the adaptive weights and robust kernel function based on semantic categories.

5. The augmented reality inspection method based on building information modeling according to claim 1, wherein, When the alignment error in a local area exceeds a threshold and the pose covariance is below a threshold value, a local non-rigid deformation adjustment based on engineering tolerance constraints is triggered, specifically including: Real-time monitoring of projection edge consistency residuals and SDF distance residuals in local areas; when the projection edge consistency residuals or the SDF distance residuals exceed a preset threshold, and the trace of the covariance matrix is ​​lower than a preset confidence threshold, the deformation optimization process is activated. Select key electromechanical components as deformation anchor points and add constraints to ensure that the changes in the radius and curvature of the deformed components, as well as their relative positional relationship with adjacent components, meet the predefined engineering tolerance parameters. Solve for the optimal deformation parameters that satisfy the geometric constraints, and record the maximum displacement, total bending angle and affected component segment identifiers during the deformation process to generate a deformation summary report.

6. The augmented reality inspection method based on building information modeling according to claim 1, wherein, Based on the optimized residual distribution, the covariance matrix, and the deformation parameters, a signature visual residual proof is generated, and the signature visual residual proof is digitally signed and timestamped for authentication, specifically including: Obtain the optimized projection edge consistency residual, generate a residual distribution heatmap, and statistically analyze the optimized SDF distance residual to generate an error statistics histogram; Record the eigenvalues ​​and condition number of the optimized pose covariance matrix, and organize the adjustment range and constraints of the local deformation parameters; The system aggregates device identification information, timestamp sequences, and environmental context data. It then performs a digital signature on the aggregated visual residual proof through a trusted execution environment and adds timestamp authentication information provided by the Network Time Protocol.

7. The augmented reality inspection method based on building information modeling according to claim 1, wherein, The signature visual residual proof is mapped to a pre-constructed hash tree, the root hash value of the hash tree is calculated, and the root hash value and the digital signature are stored together on the blockchain to achieve traceability and verifiability of inspection evidence, specifically including: The three-dimensional space containing the building information model is divided into a multi-level octree structure; each leaf node of the octree corresponds to a specific spatial region. The signature visual residual proof is associated with the corresponding spatial region, and the hash value corresponding to each leaf node is calculated, so that the hash value is used as the data of the corresponding leaf node in the octree; The hash of the parent node in the octree is calculated recursively from bottom to top to generate a global root hash value and add timestamp information; The root hash value, the changed node hash value, and the digital signature are submitted to the blockchain network through a smart contract, and a relationship is established between the root hash value and the on-chain data. The complete visual residual proof of the signature is stored in a distributed storage system under the blockchain, and a local verification interface based on spatial regions is provided to allow verification of the integrity of evidence in specific regions.

8. A building information model based augmented reality inspection device, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform an augmented reality inspection method based on a building information model as described in any one of claims 1-7.

9. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, they implement an augmented reality inspection method based on building information model as described in any one of claims 1-7.

Images