Deep learning-based slam-bim augmented reality collaborative positioning method and system

Abstract

The application relates to the fields of building information modeling, augmented reality and simultaneous localization and mapping, and provides a SLAM-BIM augmented reality cooperative positioning method and system based on deep learning, which comprises the following steps: introducing a Transformer time sequence feature extractor and a geometric relationship graph, combining confidence evaluation to dynamically distribute weights, realizing cross-modal closed loop detection, utilizing a lightweight semantic segmentation network and a feature fusion module to construct a geometric and semantic consistent dense map, constructing a space-time error propagation equation, realizing online calibration with the help of BIM scale priori, designing an incremental fusion algorithm, and combining AR interaction to optimize the global pose; the system fuses SLAM visual track features and BIM semantic geometric features through deep learning technology, solves the problems of large accumulated error of traditional SLAM and low fusion precision of BIM, realizes robust positioning and map construction in a complex scene, improves the cooperative precision and real-time performance of SLAM and BIM, and is suitable for AR scenes such as building construction and operation and maintenance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of building information modeling, augmented reality and simultaneous localization and mapping, in particular to a SLAM-BIM augmented reality collaborative positioning method and system based on deep learning. BACKGROUND

[0002] In the field of the intersection of building information modeling (BIM), augmented reality (AR) and simultaneous localization and mapping (SLAM) technology, traditional SLAM technology has significant limitations in complex building scenes:

[0003] Firstly, traditional SLAM relies on adjacent frame matching, resulting in significant accumulation of long-distance trajectory errors, especially in textureless scenes, the accuracy of visual feature matching is greatly reduced, and the success rate of loop closure detection is low;

[0004] Secondly, existing methods lack effective use of BIM model semantics and geometric information, making it difficult to integrate BIM topological constraints into the SLAM positioning process, resulting in a large deviation between the coordinate systems of the SLAM map and the BIM model, and insufficient collaborative accuracy;

[0005] Thirdly, traditional SLAM scale estimation relies on monocular vision or inertial measurement unit (IMU), lacks scale priors in building scenes (such as standard component dimensions), and is prone to scale drift problems;

[0006] Fourthly, in dynamic building environments, traditional visual constraints are easily disturbed by dynamic objects, leading to mismatching, and the difference between the SLAM map and the BIM model is low in updating efficiency, making it difficult to achieve real-time interactive optimization.

[0007] In addition, the cross-modal fusion of SLAM visual features and BIM geometric semantic features in existing technologies lacks an adaptive weight mechanism, which cannot dynamically adjust the constraint strength according to the scene characteristics, limiting the positioning robustness in complex environments.

[0008] Therefore, the present application provides a SLAM-BIM augmented reality collaborative positioning method and system based on deep learning. SUMMARY

[0009] In order to make up for the deficiencies of the prior art and solve at least one technical problem proposed in the background art.

[0010] The technical solution adopted by the present application to solve its technical problems is:

[0011] In a first aspect, the present application provides a SLAM-BIM augmented reality collaborative positioning method based on deep learning, comprising:

[0012] S1: By introducing a Transformer-based time sequence feature extractor, the spatial correlation in the SLAM trajectory is captured, the structural skeleton of the BIM model is extracted to construct a geometric relationship graph, a confidence assessment module is designed, the weights of visual loopback and BIM topological constraints are dynamically assigned according to scene features, and cross-modal loop detection is realized;

[0013] S2: A lightweight semantic segmentation network is used to identify BIM key elements in the scene in real time, high-density point clouds are generated through multi-view stereo matching in high-semantic-value areas, a feature fusion module is designed, visual features, semantic features and geometric features are projected into a unified feature space, and a dense map with geometric and semantic consistency is constructed;

[0014] S3: A three-dimensional error propagation equation of pose-time-space is constructed, the variation law of SLAM pose error in time and space dimensions is tracked in real time, the scale prior constraint error range of the BIM model is introduced, and the standard objects in the BIM are used as scale anchor points to realize online scale calibration;

[0015] S4: An incremental fusion algorithm is designed to update only the difference areas of the SLAM map and the BIM model, an error marking interaction tool is set in the AR interface, and the global pose is automatically optimized based on user marking and the time-space error model.

[0016] As a further solution of the present application, the spatial correlation in the SLAM trajectory is captured by introducing a Transformer-based time sequence feature extractor;

[0017] The Transformer-based time sequence feature extraction and spatial correlation capture collect a key frame sequence of the SLAM trajectory, the key frame sequence includes: an RGB image, a sparse point cloud, a camera pose and structured data of a BIM model, the structured data of the BIM model includes: a beam column and a wall parsed from an IFC format, the collected key frame sequence of the SLAM trajectory is down-sampled to retain representative key frames and reduce computational load, a trajectory visual feature fusion network is constructed, an image feature of a key frame is extracted by a ResNet in a visual branch, a visual similarity of adjacent frames is captured through a SwinTransformer architecture of a space-time attention mechanism, a camera pose is encoded into a time sequence in a trajectory branch, and a time correlation of the pose change is learned and extracted through a Transformer encoder.

[0018] As a further solution of the present application, the specific process of constructing a geometric relationship graph by extracting a structural skeleton of a BIM model is as follows:

[0019] Extract the structural skeleton from the BIM model, parse the beam-column joint and wall boundary through the IFC file, generate a directed acyclic graph DAG, the beam-column joint is a BIM component, and the edge is a spatial adjacency relationship, extract geometric features for each BIM component, the geometric features include: center point coordinates, bounding box size, normal vector, geometric relationship graph construction, node definition, the SLAM key frame and the BIM component are mixed nodes, in the mixed node, the SLAM node contains RGB image features, sparse point cloud, camera pose, the BIM node contains the center point coordinates of the geometric entity parsed by IFC, the bounding box size geometric feature, and the semantic label.

[0020] As a further solution of the present application, the specific process of designing the confidence evaluation module is:

[0021] Design a multi-dimensional confidence index to evaluate the reliability of scene features, the multi-dimensional confidence index includes: visual confidence, geometric confidence, semantic confidence, wherein the visual confidence is an inlier rate descriptor distance distribution based on ORB feature matching; the geometric confidence is the surface fitting degree of the SLAM point cloud and the BIM model, and the fitting degree is measured by Hausdorff distance; the semantic confidence is the matching degree of scene elements and BIM semantics.

[0022] As a further solution of the present application, the specific process of realizing cross-modal closed loop detection is:

[0023] Candidate set frame screening: based on the visual bag of words BoW fast retrieval possible loop candidate set frames, combined with the BIM geometric relationship graph to filter the candidate set of semantic mismatch, set a bidirectional verification mechanism, the bidirectional verification mechanism includes: forward verification mechanism, reverse verification mechanism, the forward verification mechanism is: through the joint optimization of visual feature matching and BIM geometric constraint to optimize the current frame pose, the reverse verification mechanism is: using the joint optimized current pose frame, reconstruct the point cloud of the current frame, and compare the reconstructed point cloud with the BIM model, calculate the global consistency error, if the error is less than the threshold, confirm the closed loop.

[0024] As a further solution of the present application, the specific process of using a lightweight semantic segmentation network to identify BIM key elements in the scene in real time is:

[0025] A semantic segmentation model is established, inputting the SLAM real-time RGB frame and the IFC file of the BIM model, using the dynamic channel pruning BiSeNet, customizing training for the BIM scene, projecting the 3D contour of the BIM model to the 2D image to generate a virtual annotation box, supplementing the boundary supervision of the BIM element in the real scene, introducing the BIM contour constraint loss, calculating the IoU of the segmentation result and the BIM projected contour, and outputting a pixel-by-pixel semantic mask, which includes a class ID and a BIM correlation score ranging from 0 to 1, and the higher the BIM correlation score, the closer it is to the BIM key element.

[0026] As a further solution of the present application, the specific process of generating a high-density point cloud for the high-semantic-value area through multi-view stereo matching is as follows:

[0027] When the BIM correlation score is greater than or equal to 0.8, it is recorded as a high-semantic-value area, and a differentiable stereo matching is used to optimize the depth map for the high-semantic-value area, combined with BIM constraints, to generate a sub-millimeter-level density point cloud.

[0028] As a further solution of the present application, the specific process of using standard objects in the BIM as scale anchors to realize online scale calibration is as follows:

[0029] According to the timestamp and spatial coordinates of the BIM model, the space-time state of the SLA is matched, the BIM model construction timestamp is synchronized with the SLAM trajectory timestamp through linear interpolation, and the conversion matrix of the BIM coordinate system to the SLAM coordinate system is established using the RTK initial positioning, the scale deviation is calculated using the scale calibration algorithm, and the scale factor and space-time error of the SLAM are corrected in real time using the extended Kalman filter (EKF).

[0030] As a further solution of the present application, the specific process of updating the difference area between the SLAM map and the BIM model is as follows:

[0031] Through double-dimensional recognition based on geometric feature difference and semantic feature difference, the input SLAM real-time point cloud and the BIM model triangular mesh are used to divide the scene into voxels, and the overlapping area of the SLAM map and the BIM model is quickly located, and for each voxel, the minimum distance between the SLAM point cloud and the BIM model mesh is calculated.

[0032] When the absolute value of the geometric feature difference is greater than 0.1 and the BIM correlation score is less than 0.8, it is marked as a core difference area, and when the absolute value of the geometric feature difference is less than 0.1 and the BIM correlation score is greater than or equal to 0.8, it is marked as a secondary difference area, and the difference area is divided into hierarchical cubes using an octree algorithm, and the core difference area is updated first, and then the secondary difference area is updated.

[0033] In a second aspect, the present application provides a deep learning-based SLAM-BIM augmented reality collaborative positioning system, comprising:

[0034] Cross-modal loop detection module: by introducing a Transformer-based time series feature extractor, the spatial correlation in the SLAM trajectory is captured, and the structural skeleton of the BIM model is extracted to construct a geometric relationship graph; a confidence assessment module is designed to dynamically assign the weights of visual loop and BIM topological constraints according to scene features, and cross-modal loop detection is realized;

[0035] Geometric semantic dense map construction module: a lightweight semantic segmentation network is used to identify BIM key elements in the scene in real time, high-density point clouds are generated for high-semantic-value areas through multi-view stereo matching, a feature fusion module is designed to project visual features, semantic features and geometric features into a unified feature space, and a geometric semantic consistency dense map is constructed;

[0036] Space-time error calibration module: a three-dimensional error propagation equation of pose time space is constructed to track the change rule of SLAM pose error in time and space dimensions in real time, scale prior constraint error range of the BIM model is introduced, and standard objects in the BIM are used as scale anchor points to realize online scale calibration;

[0037] Incremental fusion and pose optimization module: an incremental fusion algorithm is designed to update only the difference areas of the SLAM map and the BIM model, an error marker interaction tool is set in the AR interface, and the global pose is automatically optimized based on user marking and the space-time error model.

[0038] The beneficial effects of the present application are as follows:

[0039] 1. By introducing the Transformer time feature extractor and the BIM geometric relationship diagram, combined with the confidence dynamic weighting mechanism, the problem of cumulative error caused by the dependence of traditional SLAM on adjacent frame matching is solved, the BIM geometric constraint weight is automatically improved in the texture missing scene, the visual constraint interference is reduced in the dynamic object scene, the closed loop detection success rate is improved, the pose error is reduced, at the same time, through the bidirectional verification mechanism and the graph optimization technology, the deviation of the SLAM trajectory and the BIM model is calibrated synchronously, the cross-modal space consistency is realized, the lightweight semantic segmentation network and the multi-view stereo matching are used to generate sub-millimeter level point cloud in the high semantic value area, and the visual, semantic and geometric features are projected to the unified space through the feature fusion module, which improves the structural consistency of the point cloud and the BIM model, improves the boundary accuracy of semantic segmentation, effectively solves the problem that the traditional SLAM map lacks semantic information and has low fitting degree with the BIM model, provides more accurate spatial semantic representation for the AR scene, constructs the pose space-time error propagation equation, introduces the BIM scale prior constraint and the standard component anchor point, and corrects the scale factor in real time through the extended Kalman filter, solves the monocular scale drift problem of traditional SLAM.

[0040] 2. Design a dual-dimension difference recognition mechanism and an octree incremental update algorithm, only update the core difference area of the SLAM map and the BIM model, improve the calculation efficiency, combine the error marking interaction tool of the AR interface, convert the user feedback into pose optimization constraint, realize the adaptive adjustment of the global pose through the graph optimization, reduce the cooperative positioning delay in complex scenes, meet the real-time AR interaction demand, integrate the Transformer, graph neural network GNN and differentiable stereo matching modules through deep learning technology, form a full-process intelligent scheme from feature extraction, map construction to error calibration, compared with the traditional method, the present application realizes high-precision cooperative positioning of SLAM and BIM in AR scenes such as building construction and operation and maintenance, and provides more reliable spatial information foundation for intelligent construction. BRIEF DESCRIPTION OF DRAWINGS

[0041] The present application will be further described below in conjunction with the drawings.

[0042] Figure 1 is the step flow chart of the SLAM-BIM augmented reality cooperative positioning method based on deep learning of the present application;

[0043] Figure 2 is the system module diagram of the SLAM-BIM augmented reality cooperative positioning system based on deep learning of the present application. DETAILED DESCRIPTION

[0044] In order to make the technical means, creative features, purposes and effects realized by the present application easy to understand, the present application will be further described below in conjunction with specific embodiments.

[0045] Example 1

[0046] like Figure 1 As shown, the SLAM-BIM augmented reality collaborative positioning method based on deep learning according to an embodiment of the present invention includes:

[0047] S1: By introducing a Transformer-based temporal feature extractor, we capture the spatial correlation in SLAM trajectories and simultaneously extract the structural skeleton of the BIM model to construct a geometric relationship graph. We also design a confidence assessment module that dynamically assigns weights to visual loops and BIM topological constraints based on scene features, achieving cross-modal loop closure detection.

[0048] By introducing a Transformer-based temporal feature extractor, we can capture the spatial correlation in SLAM trajectories.

[0049] Based on Transformer-based temporal feature extraction and spatial correlation capture, we collect keyframe sequences of SLAM trajectories. These keyframe sequences include RGB images, sparse point clouds, camera poses, and structured data of BIM models. The structured data of BIM models includes geometric entities such as beams, columns, and walls parsed in IFC format.

[0050] Downsample the key frame sequence of the collected SLAM trajectory to retain representative key frames and reduce the amount of calculation;

[0051] The specific process of Transformer-based temporal feature extraction is as follows:

[0052] Construct a trajectory visual feature fusion network. The visual branch uses ResNet to extract image features of key frames and captures the visual similarity of adjacent frames through the spatiotemporal attention mechanism (SwinTransformer architecture). The trajectory branch encodes the camera pose into a time sequence and uses the Transformer encoder to learn and extract the temporal correlation of pose changes.

[0053] Specifically, SwinTransformer, as the implementation architecture of the spatiotemporal attention mechanism, associates the features of the visual branch and the trajectory branch across frames through hierarchical feature extraction and sliding window mechanism:

[0054] The visual branch uses ResNet to extract image features and then uses the spatiotemporal attention module of SwinTransformer to capture the visual similarities of adjacent frames.

[0055] Trajectory branch encodes camera poses as a time sequence, learns the time correlation of pose changes through a Transformer encoder, and finally fuses the two types of features to model the dependency relationship of distant keyframes, solving the cumulative error problem of traditional SLAM;

[0056] Through the spatio-temporal attention mechanism, the dependency relationship of distant keyframes in the SLAM trajectory is modeled, and the cumulative error problem caused by the dependence of traditional SLAM trajectory on only adjacent frame matching is solved.

[0057] The specific process of capturing the spatial correlation in the SLAM trajectory is as follows:

[0058] Project the SLAM point cloud and the BIM geometric entity into a three-dimensional space, calculate the distance field between the SLAM point cloud and the BIM geometric entity surface, generate a spatial constraint matrix, and represent the spatial consistency of the scene structure.

[0059] The specific process of extracting the structural skeleton of the BIM model to construct the geometric relationship graph is as follows:

[0060] Extract the structural skeleton from the BIM model, parse the beam-column nodes, wall boundaries and other relevant structural topologies through the IFC file, generate a directed acyclic graph (DAG), the beam-column nodes are BIM components such as column C1 and wall W2, and the edges are spatial adjacency relationships, for example, column C1 supports beam B3;

[0061] Extract the geometric features of each BIM component, including center point coordinates, bounding box size, normal vector, etc., as geometric constraints for subsequent loop closure detection;

[0062] Geometric relationship graph construction, node definition, SLAM keyframes and BIM components as hybrid nodes, in the hybrid node, the SLAM node contains RGB image features, sparse point cloud, camera pose, and the BIM node contains the center point coordinates, bounding box size geometric features of the geometric entity parsed by IFC, and semantic labels;

[0063] Establish the association between visual features and geometric features by calculating the ICP distance between SLAM point cloud and BIM component surface;

[0064] Based on the design logic of the BIM model, establish the structural constraints between BIM nodes;

[0065] Connect in the time order of SLAM keyframes to constrain the continuity of the trajectory

[0066] Optimize the geometric relationship graph, use graph neural network (GNN) to propagate features in GRG, enhance the information interaction between cross-modal nodes, for example, transfer the semantic information of BIM components to the corresponding SLAM nodes;

[0067] The specific process of designing the confidence assessment module is as follows:

[0068] Design multi-dimensional confidence indicators to evaluate the reliability of scene features. The multi-dimensional confidence indicators include: visual confidence, geometric confidence, and semantic confidence. Visual confidence is based on the inlier rate descriptor distance distribution based on ORB feature matching; geometric confidence is calculated by calculating the surface fit between the SLAM point cloud and the BIM model, and the fit is measured by Hausdorff distance; semantic confidence is calculated by calculating the match between scene elements and BIM semantics. For example, whether the detected door is consistent with the door component position in the BIM model, a lightweight CNN (MobileNet) network is used to perform real-time semantic segmentation on the RGB image to verify the semantic consistency of the scene elements;

[0069] The specific process of dynamically allocating weights of visual loops and BIM topology constraints based on scene features is as follows:

[0070] Construct a confidence weighting function: ,in, is a learnable parameter obtained by reverse optimization of the historical closed-loop error. , , They are visual confidence, geometric confidence, and semantic confidence respectively;

[0071] In scenes with missing textures, such as white walls, the BIM geometric constraint weight is automatically increased; in scenes with many dynamic objects, such as people walking, the visual constraint weight is reduced to avoid mismatching;

[0072] The specific process of implementing cross-modal closed-loop detection is as follows:

[0073] Candidate frame screening: Based on the bag of visual words (BoW), possible loop candidate frames are quickly retrieved. The BIM geometric relationship graph is combined to filter semantically mismatched candidate frames. For example, if a staircase is detected in the current frame, only historical frames containing staircase components in the BIM model are matched.

[0074] A two-way verification mechanism is set up. The two-way verification mechanism includes a forward verification mechanism and a reverse verification mechanism. The forward verification mechanism is to jointly optimize the current frame pose through visual feature matching and BIM geometric constraints. The reverse verification mechanism is to use the jointly optimized current pose frame to reconstruct the point cloud of the current frame, compare the reconstructed point cloud with the BIM model, calculate the global consistency error, and confirm the closed loop if the error is less than the threshold;

[0075] If the error is greater than or equal to the threshold, the graph optimization (g2o) technology is used to simultaneously optimize the local deviation of the SLAM trajectory and the BIM model;

[0076] It should be noted that the thresholds are set by technicians in this industry;

[0077] Through Transformer and GNN, bidirectional mapping between SLAM vision-trajectory features and BIM semantic-geometric features is achieved to solve the problem of coordinate system differences. The weights of vision and BIM constraints are automatically adjusted based on scene characteristics, the robustness of closed-loop detection is improved in complex environments, and the topological constraints of BIM are introduced into SLAM closed-loop detection. This not only corrects the posture error, but also synchronously calibrates the deviation between the BIM model and the actual scene.

[0078] S2: Utilize a lightweight semantic segmentation network to identify key BIM elements in the scene in real time. Generate high-density point clouds for areas of high semantic value through multi-view stereo matching. Design a feature fusion module to project visual, semantic, and geometric features into a unified feature space to construct a dense map with geometric and semantic consistency.

[0079] The specific process of using a lightweight semantic segmentation network to identify BIM key elements in a scene in real time is as follows:

[0080] A semantic segmentation model is established, which inputs the SLAM real-time RGB frame and the IFC file of the BIM model. The BiSeNet with dynamic channel pruning is used for customized training for BIM scenes. The 3D contour of the BIM model is projected onto the 2D image to generate a virtual annotation box. This supplements the boundary supervision of the BIM elements in the real scene and introduces the BIM contour constraint loss ( ), calculate the IoU between the segmentation result and the BIM projection contour, and output a pixel-by-pixel semantic mask. The pixel-by-pixel semantic mask contains: category ID, BIM relevance score, and the BIM relevance score ranges from 0 to 1. The higher the BIM relevance score, the closer it is to the BIM key elements.

[0081] The specific process of generating a high-density point cloud through multi-view stereo matching for high semantic value areas is as follows:

[0082] When the BIM relevance score is greater than or equal to 0.8, it is recorded as a high semantic value area; when the BIM relevance score is greater than or equal to 0.5 and less than 0.8, it is recorded as a medium semantic value area; when the BIM relevance score is less than 0.5, it is recorded as a low semantic value area;

[0083] Differentiable Stereo Matching is used to optimize the depth map for areas with high semantic value, and combined with BIM constraints to generate sub-millimeter density point clouds;

[0084] Design a feature fusion module to project visual features, semantic features, and geometric features into a unified feature space to construct a dense map with geometric and semantic consistency;

[0085] The specific process of projecting the visual features, semantic features and geometric features into a unified feature space is as follows:

[0086] First, the visual features, semantic features and geometric features are subjected to feature alignment processing. The visual features are 2D deep features of an image extracted by ResNet50, and the 2D visual features are projected into a 3D space by a spatial transformation network combined with a camera pose, so as to realize physical space alignment with point cloud coordinates. Then, the visual features are reduced in dimension to 256 dimensions by a linear layer, so as to unify the feature dimensions. The semantic features are masks output by a semantic segmentation network, which are mapped into 256-dimensional continuous vectors by an embedding layer after one-hot encoding, so as to inject semantic correlation. The normal 3D, curvature 1D and depth 2D attributes of the point cloud are extracted, and are encoded into 256 dimensions by a multilayer perceptron, so as to strengthen the nonlinear correlation between geometric attributes. A multi-head attention module is constructed, with the semantic features as queries and the visual features and geometric features as keys and values. If the semantic label is a cylinder, the attention mechanism will preferentially associate the circular contour region in the visual features and the cylindrical point cloud with radial normal and uniform curvature in the geometric features, so as to realize directional focusing of the semantic features on the visual features and geometric features. Multiple attention heads mine the correlation among the three from different dimensions, and finally output a 256-dimensional unified feature space.

[0087] The specific process of constructing the dense map with geometric and semantic consistency is as follows:

[0088] The input visual features, semantic features and geometric features. The visual features are 2D features extracted from an image by a CNN, the semantic features are semantic segmentation results, and the geometric features are shape attributes of the point cloud.

[0089] The visual features are converted into 3D point cloud corresponding positions by a spatial transformation network (STN), and the semantic features and geometric features are converted into 256-dimensional vectors, respectively. The multi-head attention technology is used to focus on each other between different features and enhance the correlation. The fused features are input into a point cloud optimization module, in which the visual features guide the point cloud texture alignment, the semantic features constrain the point cloud class labels, and the BIM geometric features limit the point cloud spatial distribution. Through geometric constraint, semantic constraint, BIM constraint and dynamic weight optimization, the ADAM optimizer is used to quickly adjust the position and semantics of the point cloud, so that the three constraints are satisfied, and finally the dense map is output.

[0090] Among them, the geometric constraint is that the point cloud is projected back to the image to see if it matches the features in the original image, the semantic constraint is whether the semantic label of the point cloud is consistent with the segmentation result, and the BIM constraint is whether the point cloud meets the BIM design rules.

[0091] The dynamic weight optimization is to prioritize the BIM constraint to ensure the structural correctness, and to prioritize the visual and semantic constraints to ensure the visual correctness.

[0092] The geometric prior of the BIM model is used to correct segmentation errors, improve the segmentation boundary accuracy of high-value areas, and allow semantic features to actively associate visual features with geometric features, thereby improving the structural consistency between the point cloud and the BIM model.

[0093] S3: Construct the three-dimensional error propagation equation of posture, time and space, and track the variation of SLAM posture error in time and space in real time. Introduce the scale prior constraint error range of the BIM model and use the standard objects in the BIM as scale anchor points to achieve online scale calibration.

[0094] The specific process of constructing the three-dimensional error propagation equation of posture time space is as follows:

[0095] Define a 4×4 spatiotemporal error tensor ,in, is the translation error at the spatial position x at time t, is the rotation error at the spatial position x at time t;

[0096] By camera motion speed The time and space dimensions are related, that is, the error changes over time will propagate in space along the trajectory of motion;

[0097] Establish a spatiotemporal evolution model of the error: ,in, is the rotation matrix of the current pose, is the IMU translation noise, is the angular velocity of rotation The antisymmetric matrix of is the spatial gradient operator, is the camera motion speed, is the speed of the camera at time t, is the time rate of change of the translation error, is time t, spatial position The rotation error at is the angular velocity of the camera;

[0098] The white noise and random walk of the IMU are modeled as the driving terms of the error equation, and the reprojection error of feature matching is used as observation noise and integrated into the error propagation;

[0099] The specific process of introducing the scale priori constraint error range of the BIM model is as follows:

[0100] Extract the scale anchor candidate set from the IFC file of the BIM model, standard components with clear dimensions, and mark their design dimensions and spatial extent;

[0101] Error range constraint When the SLAM pose enters the spatial range of the BIM component, the error is forced to satisfy:

[0102] wherein, , are the SLAM measurement coordinates of the two endpoints of the component, the above constraint is converted into a constrained optimization term by the Lagrange multiplier method, and embedded into the solving process of the error propagation equation, in the present application, is 1.

[0103] The lightweight semantic segmentation network of S2 is used to identify the BIM standard object in the scene, and a 2D bounding box and a semantic confidence are output.

[0104] The point cloud in the bounding box is extracted from the SLAM point cloud, and the geometric shape is fitted; the actual size is calculated , and the BIM design size Comparison, , are the SLAM measurement coordinates of the two endpoints of the component;

[0105] The specific process of using the standard object in the BIM as a scale anchor point to realize online scale calibration is:

[0106] According to the timestamp and spatial coordinates of the BIM model, the space-time state of SLA is matched, the BIM model construction timestamp is synchronized with the SLAM trajectory timestamp through linear interpolation, and the conversion matrix of the BIM coordinate system to the SLAM coordinate system is established by using the RTK initial positioning;

[0107] The scale deviation is calculated by using the scale calibration algorithm, and the extended Kalman filter (EKF) is used to correct the scale factor and the space-time error of SLAM in real time;

[0108] S4: Design an incremental fusion algorithm to update only the difference area of the SLAM map and the BIM model, set an error marking interactive tool in the AR interface, and automatically optimize the global pose based on user marking and a space-time error model.

[0109] The specific process of updating the difference area of the SLAM map and the BIM model is:

[0110] Two-dimensional recognition is performed through geometric feature difference and semantic feature difference, the input SLAM real-time point cloud and the BIM model triangular grid are divided into voxels (size 0.1m), the overlapping area of the SLAM map and the BIM model is quickly located, and the minimum distance between the SLAM point cloud and the BIM model grid is calculated for each voxel.

[0111] When the absolute value of the geometric feature difference is greater than 0.1 and the BIM relevance score is less than 0.8, it is marked as a core difference area and updated first;

[0112] When the absolute value of the geometric feature difference is less than 0.1 and the BIM relevance score is greater than or equal to 0.8, it is marked as a minor difference area and the update is delayed;

[0113] Using the octree algorithm, the difference area is divided into hierarchical cubes, with level 0: 1m³ and level 3: 0.125m³. The core difference area is updated in real time first, followed by the secondary difference area.

[0114] The specific process of automatically optimizing the global pose is as follows:

[0115] Using AR interactive feedback, error marking and posture optimization, align the virtual reality deviation point marking position ,time , extract BIM corresponding position design coordinates , construct constraints: , convert the mark into the pose optimization observation equation;

[0116] A robust kernel function (Cauchy) is used to process abnormal labels to avoid interference from incorrect operations, and the global pose is solved through graph optimization.

[0117] The technical solution of the embodiment of the present invention is as follows: by introducing a Transformer-based temporal feature extractor, the spatial correlation in the SLAM trajectory is captured, and the structural skeleton of the BIM model is extracted to construct a geometric relationship diagram. A confidence assessment module is designed to dynamically assign weights to visual loops and BIM topological constraints based on scene features to achieve cross-modal closed loop detection. A lightweight semantic segmentation network is used to identify key BIM elements in the scene in real time. High-density point clouds are generated for areas of high semantic value through multi-view stereo matching. A feature fusion module is designed to project visual features, semantic features, and geometric features into a unified feature space to construct a dense map of geometric semantic consistency. A three-dimensional error propagation equation for pose, time, and space is constructed to track the variation of SLAM pose errors in time and space in real time. The scale prior constraint error range of the BIM model is introduced, and standard objects in the BIM are used as scale anchors to achieve online scale calibration. An incremental fusion algorithm is designed to update only the areas where the SLAM map and the BIM model differ. An error marker interactive tool is provided in the AR interface to automatically optimize the global pose based on user markers combined with the spatiotemporal error model.

[0118] Example 2

[0119] like Figure 2 As shown, based on Example 1, the present invention provides a SLAM-BIM augmented reality collaborative positioning system based on deep learning, including:

[0120] Cross-modal loop detection module: By introducing a Transformer-based temporal feature extractor, the spatial correlation in the SLAM trajectory is captured, while the structural skeleton of the BIM model is extracted to build a geometric relationship graph. A confidence assessment module is designed to dynamically assign weights to visual loops and BIM topological constraints based on scene features, enabling cross-modal loop detection.

[0121] Geometric semantic dense map construction module: A lightweight semantic segmentation network is used to identify BIM key elements in the scene in real time. High-density point clouds are generated for high-semantic-value areas through multi-view stereo matching. A feature fusion module is designed to project visual, semantic, and geometric features into a unified feature space, constructing a geometric and semantic consistent dense map.

[0122] Temporal and spatial error calibration module: A three-dimensional error propagation equation is constructed for the pose time-space, and the variation of SLAM pose error in time and space dimensions is tracked in real time. The scale prior constraint error range of the BIM model is introduced, and standard objects in the BIM are used as scale anchors to achieve online scale calibration.

[0123] Incremental fusion and pose optimization module: An incremental fusion algorithm is designed to update only the difference areas of the SLAM map and the BIM model. An error marker interaction tool is set up in the AR interface, and based on user marking, the global pose is automatically optimized using the temporal and spatial error model.

[0124] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for SLAM-BIM augmented reality co-localization based on deep learning, characterized in that: Comprise: S1: Capture the spatial correlation in the SLAM trajectory by introducing a Transformer-based temporal feature extractor, extract the structural skeleton of the BIM model to build a geometric relationship graph, design a confidence evaluation module, dynamically assign the weights of visual loop closure and BIM topological constraints according to scene features, and realize cross-modal loop detection; The specific process of the Transformer-based temporal feature extraction is: Build a trajectory visual feature fusion network, use ResNet to extract image features of key frames in the visual branch, capture the visual similarity of adjacent frames through the spatio-temporal attention mechanism SwinTransformer architecture, encode the camera pose into a time sequence in the trajectory branch, and learn the time correlation of the pose change through the Transformer encoder and extract; The specific process of capturing the spatial correlation in the SLAM trajectory is: Project the SLAM point cloud and the BIM geometric entity onto the three-dimensional space, calculate the distance field between the SLAM point cloud and the BIM geometric entity surface, generate a spatial constraint matrix, and represent the spatial consistency of the scene structure; The specific process of extracting the structural skeleton of the BIM model to build a geometric relationship graph is: Extract the structural skeleton from the BIM model, parse the beam-column nodes and wall boundaries through the IFC file, generate a directed acyclic graph DAG, the beam-column nodes are BIM components, and the edges are spatial adjacency relationships. Extract geometric features for each BIM component, including: center point coordinates, bounding box size, normal vector; geometric relationship graph construction, node definition, SLAM key frames and BIM components as hybrid nodes, in the hybrid node, the SLAM node contains RGB image features, sparse point cloud, camera pose, and the BIM node contains center point coordinates, bounding box size geometric features of the geometric entity parsed by IFC, and semantic labels; The specific process of designing a confidence evaluation module is: Design multi-dimensional confidence indicators to evaluate the reliability of scene features, including: visual confidence, geometric confidence, and semantic confidence, wherein the visual confidence is described by the inlier rate using ORB feature matching and the distance distribution; the geometric confidence is the surface fitting degree between the SLAM point cloud and the BIM model, which is measured by the Hausdorff distance; the semantic confidence is the matching degree between scene elements and BIM semantics; The specific process of dynamically assigning the weights of visual loop closure and BIM topological constraints according to scene features is: A confidence weighting function is constructed: wherein, are learnable parameters, obtained by backpropagation of historical closed-loop errors, , , are visual confidence, geometric confidence, semantic confidence, respectively. The specific process of realizing cross-modal loop detection is: Candidate set frame screening: based on the visual bag-of-words BoW fast retrieval of possible loop candidate set frames, combined with the BIM geometric relationship graph to filter the semantic mismatched candidates, set a bidirectional verification mechanism, including: forward verification mechanism, reverse verification mechanism, the forward verification mechanism is: optimize the current frame pose through visual feature matching and BIM geometric constraint joint optimization, the reverse verification mechanism is: use the current pose frame optimized by joint optimization to reconstruct the point cloud of the current frame, and compare the reconstructed point cloud with the BIM model to calculate the global consistency error, if the error is less than the threshold, confirm the loop closure; S2: Real-time identification of BIM key elements in the scene using a lightweight semantic segmentation network, generating high-density point clouds in high-semantic-value areas through multi-view stereo matching, designing a feature fusion module to project visual features, semantic features, and geometric features into a unified feature space, and constructing a dense map with geometric and semantic consistency; S3: Constructing a three-dimensional error propagation equation in the pose-time space, tracking the change rule of SLAM pose error in time and space dimensions in real time, introducing the scale prior constraint error range of the BIM model, and using standard objects in the BIM as scale anchors to realize online scale calibration; The specific process of online scale calibration is as follows: According to the timestamp and spatial coordinates of the BIM model, the space-time state of SLAM is matched, the BIM model construction timestamp is synchronized with the SLAM trajectory timestamp through linear interpolation, and the conversion matrix from the BIM coordinate system to the SLAM coordinate system is established using the RTK initial positioning. Using the scale calibration algorithm, the scale deviation is calculated, and the extended Kalman filter (EKF) is used to real-time correct the scale factor and space-time error of SLAM; S4: Designing an incremental fusion algorithm to update only the difference area between the SLAM map and the BIM model, setting an error marking interaction tool in the AR interface, and automatically optimizing the global pose based on user marking and the space-time error model.

2. The deep learning-based SLAM-BIM augmented reality co-location method of claim 1, wherein: The specific process of using a lightweight semantic segmentation network to real-time identify BIM key elements in the scene is as follows: A semantic segmentation model is established, the SLAM real-time RGB frame and the IFC file of the BIM model are input, the BiSeNet with dynamic channel pruning is used, the BIM model is customized for training, the 3D contour of the BIM model is projected onto the 2D image to generate a virtual bounding box, the boundary supervision of the BIM elements in the real scene is supplemented, the BIM contour constraint loss is introduced, the IoU of the segmentation result and the BIM projected contour is calculated, and the pixel-wise semantic mask is output. The pixel-wise semantic mask includes: class ID, BIM correlation score, and BIM correlation score range is 0-1. The higher the BIM correlation score, the closer to the BIM key element.

3. The deep learning-based SLAM-BIM augmented reality co-location method of claim 1, wherein: The specific process of generating high-density point clouds in high-semantic-value areas through multi-view stereo matching is as follows: When the BIM correlation score is greater than or equal to 0.8, it is recorded as a high-semantic-value area. The Differentiable Stereo Matching is used to optimize the depth map in the high-semantic-value area, and the sub-millimeter level density point cloud is generated by combining the BIM constraint.

4. The deep learning-based SLAM-BIM augmented reality co-location method of claim 1, wherein: The specific process of updating the difference area between the SLAM map and the BIM model is as follows: Through double-dimensional identification of geometric feature difference and semantic feature difference, the scene is divided into voxels by calculating the input SLAM real-time point cloud and the BIM model triangular mesh, and the overlapping area of the SLAM map and the BIM model is quickly located. For each voxel, the minimum distance between the SLAM point cloud and the BIM model mesh is calculated; When the absolute value of the geometric feature difference is greater than 0.1 and the BIM correlation degree score is less than 0.8, it is marked as a core difference area, and when the absolute value of the geometric feature difference is less than 0.1 and the BIM correlation degree score is greater than or equal to 0.8, it is marked as a secondary difference area. The difference area is divided into hierarchical cubes by using the octree algorithm. The core difference area is updated in real time, and then the secondary difference area is updated.

5. A deep learning based SLAM-BIM augmented reality co-location system for performing the co-location method of any one of claims 1-4, characterized in that: Comprise: The cross-modal loop detection module: by introducing a time sequence feature extractor based on Transformer, capturing the spatial correlation in the SLAM trajectory, while extracting the structural skeleton of the BIM model to construct a geometric relationship graph, designing a confidence evaluation module, dynamically assigning the weights of visual loop closure and BIM topological constraints according to scene features, realizing cross-modal loop detection; The specific process of extracting time sequence features based on Transformer is: A trajectory visual feature fusion network is constructed. The visual branch uses ResNet to extract the image features of the key frame. The visual similarity of adjacent frames is captured through the spatiotemporal attention mechanism SwinTransformer architecture. The trajectory branch encodes the camera pose into a time sequence. The time correlation of the pose change is learned through the Transformer encoder and extracted; The specific process of capturing the spatial correlation in the SLAM trajectory is: Project the SLAM point cloud and the BIM geometric entity onto the three-dimensional space uniformly, calculate the distance field between the SLAM point cloud and the BIM geometric entity surface, generate a spatial constraint matrix, and represent the spatial consistency of the scene structure; The specific process of extracting the structural skeleton of the BIM model to construct a geometric relationship graph is: Extract the structural skeleton from the BIM model, parse the beam-column nodes and wall boundaries through the IFC file, generate a directed acyclic graph DAG, the beam-column nodes are BIM components, and the edges are spatial adjacency relationships. Extract the geometric features of each BIM component, including: center point coordinates, bounding box size, normal vector; geometric relationship graph construction, node definition, SLAM key frames and BIM components as hybrid nodes, in the hybrid node, the SLAM node contains RGB image features, sparse point cloud, camera pose, and the BIM node contains the center point coordinates, bounding box size geometric features of the geometric entity parsed by IFC, and semantic labels; The specific process of designing a confidence evaluation module is: Design a multi-dimensional confidence index to evaluate the reliability of scene features. The multi-dimensional confidence index includes: visual confidence, geometric confidence, and semantic confidence. The visual confidence is described by the inlier rate using the ORB feature matching distance distribution. The geometric confidence is the surface fitting degree between the SLAM point cloud and the BIM model, which is measured by the Hausdorff distance. The semantic confidence is the matching degree between the scene elements and the BIM semantics; The specific process of dynamically assigning the weights of visual loop closure and BIM topological constraints according to scene features is: A confidence weighting function is constructed: wherein, are learnable parameters, obtained by backpropagation of historical closed-loop errors, , , are visual confidence, geometric confidence, semantic confidence, respectively. The specific process of realizing cross-modal loop detection is: Candidate frame screening: Based on the bag of words (BoW), the possible loop closure candidate frames are quickly retrieved, and the semantic mismatched candidates are filtered by combining the BIM geometric relationship graph. A bidirectional verification mechanism is set, which includes a forward verification mechanism and a backward verification mechanism. The forward verification mechanism is to optimize the current frame pose by combining visual feature matching and BIM geometric constraints. The backward verification mechanism is to reconstruct the point cloud of the current frame using the optimized current pose frame, and compare the reconstructed point cloud with the BIM model to calculate the global consistency error. If the error is less than a threshold, the loop closure is confirmed. Geometric and semantic dense map construction module: A lightweight semantic segmentation network is used to identify BIM key elements in the scene in real time. High-density point clouds are generated in high-semantic-value areas through multi-view stereo matching. A feature fusion module is designed to project visual features, semantic features, and geometric features into a unified feature space to construct a geometric and semantic consistent dense map. Space-time error calibration module: A three-dimensional error propagation equation of the pose time-space is constructed to track the change rule of the SLAM pose error in the time and space dimensions in real time. The scale prior constraint error range of the BIM model is introduced, and the standard objects in the BIM are used as scale anchors to realize online scale calibration. The specific process of online scale calibration is as follows: According to the timestamp and spatial coordinates of the BIM model, the space-time state of SLAM is matched. The BIM model construction timestamp is synchronized with the SLAM trajectory timestamp through linear interpolation, and the conversion matrix from the BIM coordinate system to the SLAM coordinate system is established using the RTK initial positioning. The scale deviation is calculated using the scale calibration algorithm, and the scale factor and space-time error of SLAM are corrected in real time using the extended Kalman filter (EKF). Incremental fusion and pose optimization module: An incremental fusion algorithm is designed to update only the difference areas of the SLAM map and the BIM model. An error marker interaction tool is set in the AR interface to automatically optimize the global pose based on user marking and the space-time error model.

Images