AR-based railway house BIM visualization method and system

Abstract

This application relates to the field of BIM visualization and discloses an AR-based BIM visualization method and system for railway buildings. The method includes: receiving self-inspection task information of the current railway building construction; calling a pre-stored railway building BIM virtual model based on the self-inspection task information; collecting local environmental data of the current railway building at the self-inspection site through a mobile terminal; spatially registering the local environmental data with the railway building BIM virtual model to obtain registration parameters; performing spatial coordinate transformation on the railway building BIM virtual model based on the registration parameters to obtain visualized BIM data aligned with the current field of view; overlaying the visualized BIM data onto the corresponding position of the real-time image of the railway building's current field of view to obtain a fused augmented reality view; and extracting decision support information matching the self-inspection task from a self-inspection knowledge base according to the task type.

Description

Technical Field

[0001] This application relates to the field of BIM visualization, and in particular to an AR-based BIM visualization method and system for railway buildings. Background Technology

[0002] Railway buildings include station buildings, signal towers, and substations. As a major component of the railway system, the safe and stable operation of railway buildings is crucial to railway transportation efficiency and passenger safety. Traditional railway building self-inspection mainly relies on manual inspection and paper drawings.

[0003] Although some systems attempt to combine BIM models with AR, traditional AR systems rely on preset marker points for spatial positioning, which can easily lead to misalignment between the BIM model and the actual structure in scenarios where marker points are difficult to place.

[0004] Solving this technical problem is a technical challenge that needs to be overcome by those skilled in the art. Summary of the Invention

[0005] This application provides an AR-based railway building BIM visualization method and system to at least partially solve the above-mentioned technical problems.

[0006] To achieve the above objectives, according to a first aspect of this application, an AR-based BIM visualization method for railway buildings is provided, comprising:

[0007] Receive self-inspection task information for the current railway building construction; the self-inspection task information includes task type and self-inspection area range; call the pre-stored railway building BIM virtual model based on the self-inspection task information;

[0008] At the self-inspection site, local environmental data of the railway building is collected using a mobile terminal; the local environmental data is then spatially registered with the railway building's BIM virtual model to obtain registration parameters.

[0009] Based on the registration parameters, the spatial coordinate transformation of the railway building BIM virtual model is performed to obtain visualized BIM data aligned with the current field of view; the visualized BIM data includes the location information of railway building components, pipeline systems, and equipment.

[0010] The real-time image of the railway building's current field of view is acquired through a mobile terminal camera; the visualized BIM data is overlaid onto the corresponding position of the real-time image of the railway building's current field of view to obtain a fused augmented reality view;

[0011] Based on the task type, decision support information matching the self-inspection task is extracted from the self-inspection knowledge base; the decision support information is associated with the corresponding components in the railway building BIM model based on spatial coordinates and overlaid in the augmented reality view to generate an interactive self-inspection guidance interface.

[0012] Optionally, the method further includes:

[0013] The deviation value of spatial location matching is calculated based on the registration parameters; it is determined whether the deviation value exceeds a first preset threshold; if it exceeds, the local edge contour information of the railway building is extracted from the local environment data, and the local edge contour information is compared and analyzed with the contour of the railway building BIM model to adjust the current field of view and generate the final registration parameters; if it does not exceed, the current registration parameters are determined as the final registration parameters.

[0014] Optionally, at the self-inspection site, local environmental data of the railway building can be collected using a mobile terminal, including:

[0015] The original point cloud data of a local area of ​​the railway building was collected by the depth camera of the mobile terminal, and the corresponding environmental images were collected simultaneously by the RGB camera.

[0016] Optimized 3D spatial data is obtained by preprocessing the collected raw point cloud data;

[0017] Edge detection and corner feature extraction are performed on the environmental image to identify the outlines of doors and windows, beam and column structures and wall planar features of railway buildings, and to generate visual feature data containing feature descriptors and coordinate information;

[0018] The optimized 3D spatial data and visual feature data are spatially aligned based on device pose data to construct a spatial feature representation of the local environment;

[0019] Based on optimized 3D spatial data, visual feature data, equipment pose data, and spatial feature representation, a local environment dataset of railway buildings containing geometric information, semantic labels, and spatial positional relationships is generated through data fusion.

[0020] Optionally, the local environmental data is spatially registered with the railway building BIM virtual model to obtain registration parameters, including:

[0021] Before the self-inspection task begins, the background system performs offline feature extraction on the railway building BIM virtual model to generate a BIM feature index library containing planar areas, edge segments, and structural corner points; the BIM feature index library is stored in the local cache of the mobile terminal;

[0022] At the self-inspection site, the BIM feature data of the corresponding area is loaded from the locally cached BIM feature index library according to the self-inspection area range;

[0023] Based on the device pose data of the mobile terminal, the BIM feature data is projected onto the local environment coordinate system to obtain the initial registration position;

[0024] Feature extraction is performed on the collected local environmental data; specifically, planar regions are extracted from optimized 3D spatial data and linear edge features are extracted from the environmental image; the planar regions and linear edge features are spatiotemporally aligned to generate an environmental feature set;

[0025] Within the neighborhood of the initial registration location, the environmental feature set is matched with the projected BIM feature data to generate a set of feature correspondences.

[0026] The feature correspondence set is filtered to remove incorrect matches; the correction vector for the initial registration position is calculated based on the correct matches after filtering.

[0027] The correction vector is applied to the initial registration position to generate a spatial transformation matrix, which is then output as the registration parameter.

[0028] Optionally, calculating the spatial location matching deviation value based on the registration parameters includes:

[0029] Extract a list of key components from the railway building BIM virtual model; the key components include load-bearing walls, load-bearing columns, beams, floor slabs, and pipeline interfaces;

[0030] An initial weight coefficient is assigned to each critical component according to the preset component importance classification rules, wherein the weight coefficient of load-bearing components is higher than that of non-load-bearing components.

[0031] Based on the current registration parameters, the theoretical coordinates of each key component in the BIM virtual model are transformed to the local environment coordinate system to obtain the predicted position of each key component in the local environment.

[0032] Using the predicted location as the search center, a local search area is delineated in the local environmental data, and the actual coordinates of the corresponding physical components are extracted;

[0033] Calculate the three-dimensional Euclidean distance between the predicted position of each key component and its corresponding actual coordinates to obtain the position deviation value of each key component;

[0034] The weighted sum of the initial weight coefficients and the positional deviations of the corresponding key components is then divided by the sum of all weight coefficients to obtain the comprehensive deviation value of spatial position matching.

[0035] The comprehensive deviation value is standardized to generate a standardized deviation value, which serves as the deviation value for spatial location matching.

[0036] Optionally, the decision-aid information is associated with corresponding components in the railway building BIM model based on spatial coordinates and overlaid in the augmented reality view, including:

[0037] Based on the self-check task type, load the corresponding decision support information template from the preset decision support template library;

[0038] Bind the text, icons, or operation instructions in the decision support information template to the spatial coordinates of the components in the railway building BIM model;

[0039] In the augmented reality view, the display position of decision support information is adjusted based on the device pose data of the mobile terminal to align the decision support information with the position of the actual component.

[0040] Interactive operations trigger the expansion or collapse of decision support information associated with components.

[0041] Optionally, adjusting the display position of decision support information in the augmented reality view based on the device pose data of the mobile terminal includes:

[0042] Real-time acquisition of rotation matrix and translation vector from the device pose data of mobile terminals;

[0043] The alignment angle between the display direction of the decision-aid information and the actual component is calculated based on the rotation matrix.

[0044] The relative distance between the decision support information and the actual component is adjusted based on the translation vector so that the decision support information fits the surface of the component.

[0045] When decision support information is obscured by a real object, adjust the display position to avoid the obscured area;

[0046] Decision support information that is outside the current field of view can be cropped or hidden.

[0047] Optionally, the text, icons, or operation instructions in the decision support information template are bound to the spatial coordinates of components in the railway building BIM model, including:

[0048] The decision support information template type corresponding to a component is determined by the mapping relationship between BIM model component ID and decision support template component type.

[0049] Load matching template content from the preset decision support template library;

[0050] Bind the template content to the BIM model component ID and calculate the initial display position of the decision support information in the augmented reality view based on the component center coordinates;

[0051] An operation heatmap is generated by recording the frequency of user interaction with components and the duration of their interaction.

[0052] The operational activity index of a component is calculated based on the sum of the number of interactions and dwell time within a preset time period.

[0053] Component priorities are determined by comparing the operational activity index with a preset threshold. Components with an operational activity index greater than a first preset index are classified as high-priority components, while components with an operational activity index lower than a second preset index are classified as low-priority components.

[0054] Render the decision support information of high-priority components to the central area of ​​the field of view and display the decision support information of low-priority components edge-aligned to reduce visual interference.

[0055] Optionally, the optimized 3D spatial data and visual feature data are spatially aligned based on device pose data, including:

[0056] Obtain the rotation matrix and translation vector from the device pose data of the mobile terminal;

[0057] Transform each point in the point cloud data to the local environment coordinate system based on the rotation matrix and translation vector;

[0058] The feature points in the visual feature data are projected onto three-dimensional space to obtain projection points;

[0059] The depth information of the projection point is corrected using depth camera data;

[0060] Spatial registration is performed between point cloud data and projection points.

[0061] Optionally, the decision-aid information is associated with corresponding components in the railway building BIM model based on spatial coordinates and overlaid in the augmented reality view, including:

[0062] Based on the self-check task type, load the corresponding decision support information template from the preset decision support template library;

[0063] Bind the text, icons, or operation instructions in the decision support information template to the spatial coordinates of the components in the railway building BIM model;

[0064] In the augmented reality view, the display position of decision support information is adjusted based on the device pose data of the mobile terminal to align the decision support information with the position of the actual component.

[0065] Interactive operations trigger the expansion or collapse of decision support information associated with components.

[0066] In summary, the embodiments of this application reduce the registration error caused by the reliance on preset marker points in traditional AR systems through the above technical solutions.

[0067] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a flowchart illustrating the steps of an AR-based railway building BIM visualization method provided in an exemplary embodiment of this application.

[0070] Figure 2 This is a system schematic diagram of an AR-based railway building BIM visualization system provided in an exemplary embodiment of this application;

[0071] Explanation of reference numerals in the attached drawings: 01, First processing module; 202, Second processing module; 203, Third processing module; 204, Fourth processing module; 205, Fifth processing module. Detailed Implementation

[0072] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0073] This application provides an AR-based BIM visualization method for railway buildings. Please refer to [link / reference]. Figure 1 The AR-based railway building BIM visualization method provided in this application includes the following steps:

[0074] Step 101: Receive the self-inspection task information for the current railway housing construction; the self-inspection task information includes the task type and the scope of the self-inspection area; call the pre-stored railway housing BIM virtual model based on the self-inspection task information.

[0075] Specifically, railway building self-inspection task information can be sent by the management system via mobile terminal. The self-inspection task information includes task type and self-inspection area. Task type can include pipeline maintenance, equipment installation, and safety inspection. Self-inspection area includes specific floors and area numbers. The BIM virtual model is a three-dimensional model of the railway building that is pre-stored on the mobile terminal or in the cloud. The BIM virtual model contains information on the location of the railway building's components, pipeline systems, and equipment.

[0076] Step 102: Collect local environmental data of the railway building at the self-inspection site using a mobile terminal; perform spatial registration between the local environmental data and the railway building BIM virtual model to obtain registration parameters.

[0077] Specifically, the mobile terminal collects local environmental data of the current railway building through its camera and depth sensor; the mobile terminal can be a smartphone or AR glasses; the local environmental data includes depth images and 3D point cloud data; the spatial registration refers to the process of aligning the local environmental data with the BIM virtual model.

[0078] Step 103: Based on the registration parameters, perform spatial coordinate transformation on the railway building BIM virtual model to obtain visualized BIM data aligned with the current field of view; the visualized BIM data includes the location information of railway building components, pipeline systems, and equipment.

[0079] Specifically, based on the rotation matrix and translation vector obtained from spatial registration, the BIM virtual model of the railway building is transformed in spatial coordinates to align the BIM model with the current field of view of the mobile terminal; the current field of view is the viewpoint captured by the camera; the transformed visualized BIM data includes the location information of railway building components such as walls and beams, pipeline systems such as heating and cables, and equipment such as distribution boxes and signaling equipment.

[0080] Step 104: Acquire a real-time image of the railway building's current field of view using a mobile terminal camera; overlay the visualized BIM data onto the corresponding position of the real-time image of the railway building's current field of view to obtain a fused augmented reality view.

[0081] Specifically, the mobile terminal camera acquires images of the railway building's current field of view in real time, overlays the visualized BIM data onto the corresponding positions of the real-time images, and generates a fused augmented reality view. Through AR technology, the location information of components, pipeline systems, and equipment in the BIM model can be matched with their corresponding positions in the actual environment to form an augmented reality view.

[0082] Step 105: Extract decision support information matching the self-inspection task from the self-inspection knowledge base according to the task type; associate the decision support information with the corresponding components in the railway building BIM model based on spatial coordinates and overlay it in the augmented reality view to generate an interactive self-inspection guidance interface.

[0083] Specifically, the self-inspection knowledge base contains decision support information for different task types, such as operation procedures for pipeline maintenance, safety specifications for equipment installation, and precautions for safety inspections. Based on the task type, matching decision support information is extracted from the self-inspection knowledge base and associated with corresponding components in the BIM model based on spatial coordinates. This information is then overlaid in the augmented reality view to form an interactive self-inspection guidance interface, which allows self-inspection personnel to view relevant guidance information by clicking on the components.

[0084] The following example illustrates this:

[0085] When a self-inspection worker begins a railway building self-inspection task, their mobile terminal receives task information from the management system. This information includes the task type and the inspection area; for example, if the task type is pipeline maintenance, the inspection area could be Area A on the second floor of the station building. Based on this information, the mobile terminal retrieves a pre-stored railway building BIM virtual model from local storage or the cloud. The worker uses the mobile terminal to scan Area A on the second floor of the station building. The terminal's camera and depth sensor collect depth images and 3D point cloud data of the current environment. This environmental data is spatially registered with the railway building BIM virtual model, and registration parameters are calculated to align the BIM model with the actual environment. Based on these registration parameters, the railway building BIM virtual model undergoes a spatial coordinate transformation to align it with the current field of view. The transformed visualized BIM data includes the location information of components and systems such as walls, beams, columns, heating pipes, cables, and electrical boxes in Area A on the second floor of the station building. The mobile terminal camera acquires real-time images of the current field of view in Area A on the second floor of the station building. Visualized BIM data is overlaid onto the corresponding positions in the real-time image to generate a fused augmented reality view. Through AR technology, the location information of components, pipeline systems, and equipment in the BIM model can be matched with their corresponding locations in the actual environment. Based on the task type of pipeline maintenance, decision support information matching pipeline maintenance is extracted from the self-inspection knowledge base. This decision support information may include operating procedures, safety regulations, and common problems. This information is then associated with corresponding components in the BIM model based on spatial coordinates and overlaid in the augmented reality view, forming an interactive self-inspection guidance interface. Self-inspection personnel can see the pipeline maintenance steps and precautions in the AR view, and can click on the pipeline to view detailed guidance information.

[0086] This application obtains registration parameters by collecting local environmental data of railway buildings in real time through a mobile terminal and spatially registering them with the railway building BIM virtual model. Based on the registration parameters, the BIM model is transformed into spatial coordinates to achieve alignment between the BIM model and the actual environment. This reduces the registration error caused by the reliance on preset marker points in traditional AR systems and improves the self-inspection efficiency of railway buildings.

[0087] In some embodiments, the method further includes:

[0088] The deviation value of spatial location matching is calculated based on the registration parameters; it is determined whether the deviation value exceeds a first preset threshold; if it exceeds, the local edge contour information of the railway building is extracted from the local environment data, and the local edge contour information is compared and analyzed with the contour of the railway building BIM model to adjust the current field of view and generate the final registration parameters; if it does not exceed, the current registration parameters are determined as the final registration parameters.

[0089] Specifically, the deviation value of spatial location matching refers to the alignment error between the BIM virtual model and the actual environment after spatial coordinate transformation; this deviation value can be obtained by calculating the Euclidean distance between the key components in the BIM model and the corresponding positions in the actual environment data; the key components mentioned above can be walls, beams and columns; the first preset threshold is a pre-set error range, such as 5cm; if the deviation value exceeds the first preset threshold, it indicates that there is a significant error in the current registration parameters, and further adjustments are needed.

[0090] When the deviation exceeds a first preset threshold, the mobile terminal extracts local edge contour information of the railway building from local environmental data. This local edge contour information refers to the edge features of the railway building structure identified through depth images or 3D point cloud data; examples of these edge features include wall boundaries and beam / column corners. The extracted local edge contour information is compared and analyzed with the contour of the railway building's BIM model. Using an iterative nearest-point algorithm or feature matching method, adjustment parameters for the current field of view are calculated, and final registration parameters are generated. These adjustment parameters include rotation angles and translation vectors. If the deviation does not exceed the first preset threshold, it indicates that the current registration parameters meet the accuracy requirements, and the current registration parameters are directly determined as the final registration parameters without further adjustment.

[0091] In some embodiments, local environmental data of the current railway building is collected at the self-inspection site via a mobile terminal, including:

[0092] The original point cloud data of a local area of ​​the railway building was collected by the depth camera of the mobile terminal, and the corresponding environmental images were collected simultaneously by the RGB camera.

[0093] Optimized 3D spatial data is obtained by preprocessing the collected raw point cloud data;

[0094] Edge detection and corner feature extraction are performed on the environmental image to identify the outlines of doors and windows, beam and column structures and wall planar features of railway buildings, and to generate visual feature data containing feature descriptors and coordinate information;

[0095] The optimized 3D spatial data and visual feature data are spatially aligned based on device pose data to construct a spatial feature representation of the local environment;

[0096] Based on optimized 3D spatial data, visual feature data, equipment pose data, and spatial feature representation, a local environment dataset of railway buildings containing geometric information, semantic labels, and spatial positional relationships is generated through data fusion.

[0097] Specifically, raw point cloud data refers to a set of three-dimensional spatial points acquired by a depth camera, with each point containing X, Y, and Z coordinate information, used to describe the geometric structure of a local area of ​​a railway building; optimized three-dimensional spatial data refers to high-quality point cloud data after denoising, filtering, and registration of the raw point cloud data, retaining the main geometric features and eliminating sensor noise and outliers; feature descriptors refer to local image features extracted by image processing algorithms, used to describe the texture and shape of specific areas in an image; for example, Harris corner detection extracts corner points at beam-column connections, and the Canny algorithm detects the edges of door and window outlines; device pose data refers to the spatial position and attitude information of the mobile terminal when acquiring data, acquired through an IMU (Integrated Device Unit), including translational components (x, y, z) and rotation angles (roll, pitch, yaw). Spatial feature representation refers to the structured description formed by aligning the optimized three-dimensional spatial data and visual feature data in a unified coordinate system, including component type, bounding box coordinates, and semantic labels.

[0098] In some embodiments, spatial registration is performed between the local environmental data and the railway building BIM virtual model to obtain registration parameters, including:

[0099] Before the self-inspection task begins, the background system performs offline feature extraction on the railway building BIM virtual model to generate a BIM feature index library containing planar areas, edge segments, and structural corner points; the BIM feature index library is stored in the local cache of the mobile terminal;

[0100] At the self-inspection site, the BIM feature data of the corresponding area is loaded from the locally cached BIM feature index library according to the self-inspection area range;

[0101] Based on the device pose data of the mobile terminal, the BIM feature data is projected onto the local environment coordinate system to obtain the initial registration position;

[0102] Feature extraction is performed on the collected local environmental data; specifically, planar regions are extracted from optimized 3D spatial data and linear edge features are extracted from the environmental image; the planar regions and linear edge features are spatiotemporally aligned to generate an environmental feature set;

[0103] Within the neighborhood of the initial registration location, the environmental feature set is matched with the projected BIM feature data to generate a set of feature correspondences.

[0104] The feature correspondence set is filtered to remove incorrect matches; the correction vector for the initial registration position is calculated based on the correct matches after filtering.

[0105] The correction vector is applied to the initial registration position to generate a spatial transformation matrix, which is then output as the registration parameter.

[0106] Specifically, before the self-inspection task begins, the background system performs offline feature extraction on the railway building BIM virtual model; the BIM feature index library is a set of geometric features extracted from the BIM model through algorithms, which includes planar areas, edge segments, and structural corners; for example, the RANSAC algorithm is used to detect the planar areas of the wall, the Hough transform is used to extract the edge segments, and the corner detection algorithm is used to identify the structural corners.

[0107] The BIM feature index library is stored in the local cache of the mobile terminal in a compressed binary format; the local cache is designed with an index structure divided by region; for example, railway buildings are divided into multiple grid units, each unit corresponding to an independent BIM feature data block.

[0108] During the self-inspection process, the mobile terminal loads the corresponding BIM feature data from the local or cloud-cached BIM feature index library based on the self-inspection area. During the loading process, the index file is matched with the area ID to load only the feature data related to the current self-inspection area to reduce memory usage.

[0109] Based on the device pose data of the mobile terminal, the BIM feature data is projected onto the local environment coordinate system. The device pose data includes translation vectors (x, y, z) and rotation angles (roll, pitch, yaw). The BIM feature data is transformed from the model coordinate system to the local environment coordinate system using quaternion interpolation to generate the initial registration position.

[0110] Feature extraction is performed on the collected local environmental data; among them, planar region extraction: planar regions are detected from optimized 3D spatial data, and the plane ID and normal vector are calculated by RANSAC algorithm; linear edge feature extraction: linear edge features are extracted from the environmental image by Canny edge detection and Hough transform, and the starting point, ending point and direction vector of the edge are recorded.

[0111] Spatiotemporal alignment of planar regions and linear edge features generates an environmental feature set; the aforementioned spatiotemporal alignment refers to mapping feature data collected at different time points to a unified timestamp and spatial coordinate system; for example, aligning point clouds and image features using the timestamp of device pose data.

[0112] Within the neighborhood of the initial registration location, the environmental feature set is matched with the projected BIM feature data. The feature matching is used to generate a set of feature correspondences by calculating the descriptor similarity. For example, an edge feature in the environment is matched with the corresponding edge line segment in the BIM model to generate a correspondence pair (environmental feature ID, BIM feature ID).

[0113] The feature correspondence set is filtered to remove incorrect matches. Incorrect matches typically exhibit low descriptor similarity or poor geometric consistency; filtering methods include checking whether the spatial location of the matching features is within the allowable error range. Based on the filtered correct matches, a correction vector for the initial registration position is calculated. The correction vector includes a translation vector (…). x, y, z) and rotation angle ( roll, pitch (yaw), by optimizing the geometric deviation of the matching features using the least squares method, the correction parameters are obtained.

[0114] The correction vector is applied to the initial registration position to generate a spatial transformation matrix as the registration parameter output. The spatial transformation matrix includes a 3×3 rotation matrix and a 3×1 translation vector. The BIM model is aligned with the local environment data through the homogeneous coordinate transformation formula.

[0115] The above approach reduces on-site computational burden by generating a BIM feature index library through offline feature extraction.

[0116] In some embodiments, calculating the spatial location matching deviation value based on the registration parameters includes:

[0117] Extract a list of key components from the railway building BIM virtual model; the key components include load-bearing walls, load-bearing columns, beams, floor slabs, and pipeline interfaces;

[0118] An initial weight coefficient is assigned to each critical component according to the preset component importance classification rules, wherein the weight coefficient of load-bearing components is higher than that of non-load-bearing components.

[0119] Based on the current registration parameters, the theoretical coordinates of each key component in the BIM virtual model are transformed to the local environment coordinate system to obtain the predicted position of each key component in the local environment.

[0120] Using the predicted location as the search center, a local search area is delineated in the local environmental data, and the actual coordinates of the corresponding physical components are extracted;

[0121] Calculate the three-dimensional Euclidean distance between the predicted position of each key component and its corresponding actual coordinates to obtain the position deviation value of each key component;

[0122] The weighted sum of the initial weight coefficients and the positional deviations of the corresponding key components is then divided by the sum of all weight coefficients to obtain the comprehensive deviation value of spatial position matching.

[0123] The comprehensive deviation value is standardized to generate a standardized deviation value, which serves as the deviation value for spatial location matching.

[0124] Specifically, key components include load-bearing walls, load-bearing columns, beams, floor slabs, and pipeline interfaces. Among them, load-bearing walls refer to wall structures that bear the main vertical loads of the building and transfer them to the foundation; load-bearing columns refer to vertical load-bearing members used to support the loads of the upper floors or roof; beams refer to horizontal bending members that connect load-bearing columns or walls and bear lateral loads; floor slabs refer to horizontal slab structures that separate the spaces of different floors of the building and bear the loads of use; and pipeline interfaces refer to the access ports or connection nodes of various equipment pipelines in railway buildings.

[0125] The component importance classification rules are based on pre-set safety levels and functional criticality of railway building structures. For example, the weight coefficient for load-bearing walls is set at 0.30, for load-bearing columns at 0.25, for beams at 0.20, for floor slabs at 0.15, and for pipeline interfaces at 0.10. This ensures that the positioning deviation of structural safety-related components accounts for a higher proportion in the overall assessment, thereby guiding the system to prioritize the registration accuracy of critical parts. The local environment coordinate system is a right-handed rectangular coordinate system established with the spatial pose of the mobile terminal at the time of acquisition as the origin.

[0126] Using the predicted location as the search center, a local search area is defined in the local environmental data, and the actual coordinates of the corresponding physical components are extracted. The local search area is a spherical neighborhood with a radius of 1 meter centered on the predicted location. Within this area, point cloud clustering algorithms or image semantic segmentation results are used to identify physical entities that match the target component type, and their geometric centers or feature points are extracted as actual spatial coordinates.

[0127] Calculate the three-dimensional Euclidean distance between the predicted position of each key component and its corresponding actual coordinates to obtain the position deviation value of each key component; perform a weighted sum based on the initial weight coefficient and the position deviation value of the corresponding key component, and then divide by the sum of all weight coefficients to obtain the comprehensive deviation value of spatial position matching.

[0128] The comprehensive deviation value is standardized to generate a standardized deviation value as the deviation value for spatial location matching; the standardization process refers to mapping the comprehensive deviation value to the interval [0, 1].

[0129] The above scheme identifies key components that are crucial to the structural safety and functional operation of railway buildings, assigns them higher weights based on their safety levels, prioritizes ensuring the alignment accuracy of high-risk areas such as load-bearing structures, and improves the reliability of AR-assisted self-inspection of railway buildings.

[0130] In some embodiments, the decision-aid information is associated with corresponding components in the railway building BIM model based on spatial coordinates and overlaid in an augmented reality view, including:

[0131] Based on the self-check task type, load the corresponding decision support information template from the preset decision support template library;

[0132] Bind the text, icons, or operation instructions in the decision support information template to the spatial coordinates of the components in the railway building BIM model;

[0133] In the augmented reality view, the display position of decision support information is adjusted based on the device pose data of the mobile terminal to align the decision support information with the position of the actual component.

[0134] Interactive operations trigger the expansion or collapse of decision support information associated with components.

[0135] Specifically, the decision support template library is a collection of knowledge pre-built and stored in mobile terminals or backend servers, organized according to task types. Each template contains standardized text instructions, operation flowcharts, safety warning icons, tool lists, and typical fault handling guidelines. For example, when the task type is signal equipment wiring inspection, the system loads a template containing wiring diagrams, voltage detection steps, and insulation requirements.

[0136] Spatial coordinate binding refers to associating each information element in the template with the three-dimensional coordinate position of a specific component in the BIM model, so that the information is always attached to the corresponding physical entity in the AR view; the binding process is achieved through the component's unique identifier.

[0137] In the augmented reality view, the display position of decision-making assistance information is adjusted based on the device pose data of the mobile terminal to align the decision-making assistance information with the position of the actual component. The system calculates the projection coordinates of the decision-making assistance information on the screen. When the user moves or rotates the device, the superimposed text or icons remain stably covered on the surface of the real component, avoiding drift or misalignment.

[0138] Interactive operations trigger the expansion or collapse of decision support information associated with components; the interactive operations include user input behaviors such as clicking, long-pressing, double-clicking, or swiping on the mobile terminal screen; for example, when a user clicks on a folded label on a distribution box in the AR view, it expands to display detailed wiring diagrams and operating steps; clicking again collapses the content to reduce visual interference.

[0139] This solution attaches auxiliary content such as text and icons to the corresponding physical components and aligns them in real time as the equipment moves, achieving WYSIWYG guidance; self-inspection personnel only need to look at the target component to obtain the corresponding operation instructions, improving work efficiency.

[0140] In some embodiments, adjusting the display position of decision-aid information based on device pose data of a mobile terminal in an augmented reality view includes:

[0141] Real-time acquisition of rotation matrix and translation vector from the device pose data of mobile terminals;

[0142] The alignment angle between the display direction of the decision-aid information and the actual component is calculated based on the rotation matrix.

[0143] The relative distance between the decision support information and the actual component is adjusted based on the translation vector so that the decision support information fits the surface of the component.

[0144] When decision support information is obscured by a real object, adjust the display position to avoid the obscured area;

[0145] Decision support information that is outside the current field of view can be cropped or hidden.

[0146] Specifically, the system calculates the alignment angle between the display direction of the decision-making assistance information and the actual component based on the rotation matrix. The system transforms the normal vector of the target component in the BIM model to the coordinate system of the device camera through the current rotation matrix, thereby determining the optimal orientation of the decision-making assistance information so that its front is always perpendicular to the user's line of sight or parallel to the surface of the component, avoiding text inversion, distortion or difficulty in recognition due to tilted viewing angle.

[0147] The system adjusts the relative distance between the decision support information and the actual component based on the translation vector so that the decision support information fits the surface of the component. Specifically, the system determines the spatial distance between the device and the component based on the translation vector and shifts the decision support information along the normal direction of the component by a preset distance so that it floats above the surface of the component without penetrating the entity.

[0148] When decision assistance information is occluded by a real object, the display position is adjusted to avoid the occluded area. The occlusion refers to the partial or complete coverage of the superimposed virtual information caused by other physical structures located between the user's line of sight and the target component. Obstacles on the line of sight are detected by depth images or point cloud data, and the decision assistance information is translated along the visible direction to an unobstructed adjacent position.

[0149] The current field of view refers to the cone-shaped visible space corresponding to the current imaging area of ​​the mobile terminal camera. The projection transformation is used to determine whether the decision-making assistance information is completely or partially outside the view cone. If it is completely outside the view cone, it is hidden to save rendering resources. If it is partially outside the view cone, only the visible area is rendered to prevent broken or floating fragmented content from appearing at the edge of the screen and to improve the user experience of the AR interface.

[0150] The rotation matrix is ​​used to correct the display orientation in real time to ensure that the information is always facing the user's line of sight; no matter what observation angle or position the self-inspection personnel are in, the decision support information can be clearly and without interference attached to the corresponding components, improving the smoothness of AR guidance interaction.

[0151] In some embodiments, binding the text, icons, or operation instructions in the decision support information template to the spatial coordinates of components in the railway building BIM model includes:

[0152] The decision support information template type corresponding to a component is determined by the mapping relationship between BIM model component ID and decision support template component type.

[0153] Load matching template content from the preset decision support template library;

[0154] Bind the template content to the BIM model component ID and calculate the initial display position of the decision support information in the augmented reality view based on the component center coordinates;

[0155] An operation heatmap is generated by recording the frequency of user interaction with components and the duration of their interaction.

[0156] The operational activity index of a component is calculated based on the sum of the number of interactions and dwell time within a preset time period.

[0157] Component priorities are determined by comparing the operational activity index with a preset threshold. Components with an operational activity index greater than a first preset index are classified as high-priority components, while components with an operational activity index lower than a second preset index are classified as low-priority components.

[0158] Render the decision support information of high-priority components to the central area of ​​the field of view and display the decision support information of low-priority components edge-aligned to reduce visual interference.

[0159] Specifically, the decision support information template type corresponding to a component is determined by the mapping relationship between the BIM model component ID and the decision support template component type. The BIM model component ID is a unique identifier for each component in the railway building BIM virtual model, which is generated by the modeling software during creation. The mapping relationship refers to the correspondence table preset in the system configuration file, which establishes an association between the semantic category to which the component ID belongs and the decision support template type.

[0160] The system loads matching template content from a pre-set decision support template library. The decision support template library is a knowledge resource categorized and stored according to component type and task scenario. The decision support template includes text descriptions, safety warning icons, operation flowcharts, tool lists, and typical fault handling steps. For example, when a component is identified as a signal equipment terminal block, the system loads a template that includes voltage detection sequence, insulation requirements, and wiring diagrams.

[0161] The template content is bound to the BIM model component ID, and the initial display position of the decision support information in the augmented reality view is calculated based on the component center coordinates. The component center coordinates refer to the three-dimensional coordinates (x, y, z) of the center point of the geometric bounding box of the component in the BIM model. The system anchors each information element in the template to these coordinates and projects them onto the screen pixel position in combination with the current device pose, as the initial reference point for AR overlay.

[0162] An operation heatmap is generated by recording the frequency of user interaction with components and the dwell time. The interaction frequency refers to the number of times a user clicks, long-presses, or stares at a component per unit time. The dwell time refers to the cumulative duration for which the user's gaze or the device's camera is continuously focused on the component. The system collects the above behavioral data in real time through the log module and constructs a two-dimensional heatmap with the component ID as the index. The color intensity reflects the degree of operation density.

[0163] The priority of components is determined by comparing the operation activity index with a preset threshold. When the operation activity index is greater than the first preset index, it is determined to be a high-priority component. When the operation activity index is lower than the second preset index, it is determined to be a low-priority component. For example, if the first preset index is set to 8.0 and the second preset index is set to 3.0, components in between are considered medium-priority.

[0164] The decision support information for high-priority components is rendered in the central area of ​​the field of view, while the decision support information for low-priority components is displayed edge-aligned to reduce visual interference. Specifically, the central area of ​​the field of view refers to the high-attention area of ​​about 30% in the center of the mobile terminal screen, where the complete guidance for high-priority components is displayed first. The information for low-priority components is arranged along the edge of the screen and displayed in a semi-transparent or smaller font size, which preserves accessibility and avoids distracting the user.

[0165] In some embodiments, spatial alignment of optimized 3D spatial data and visual feature data based on device pose data includes:

[0166] Obtain the rotation matrix and translation vector from the device pose data of the mobile terminal;

[0167] Transform each point in the point cloud data to the local environment coordinate system based on the rotation matrix and translation vector;

[0168] The feature points in the visual feature data are projected onto three-dimensional space to obtain projection points;

[0169] The depth information of the projection point is corrected using depth camera data;

[0170] Spatial registration is performed between point cloud data and projection points.

[0171] In some embodiments, constructing a spatial feature representation of the local environment includes:

[0172] Geometric features are extracted from the registered point cloud data; including detecting the planar region of the wall and calculating its normal vector; and identifying the edge segments and corner coordinates of the beam and column structure.

[0173] Structural semantic labels are generated based on geometric features and preset rules; among them, the wall type is determined by the normal vector and curvature of the planar region; door and window openings are matched according to the length and angle of the edge line segments; and the location of equipment mounting holes is identified by the corner point distribution pattern.

[0174] Spatial binding of geometric features and semantic labels is performed; specifically, component bounding boxes are generated based on the projection relationship between point cloud planar regions and semantic labels; and a mapping table between geometric features and semantic labels is created.

[0175] Record the adjacency relationships between components and store the binding relationships between components and spatial coordinates to obtain a local environmental feature expression that includes spatial relationships.

[0176] Reference Figure 2 The second embodiment of the present invention provides an AR-based railway building BIM visualization system, comprising:

[0177] The first processing module 201 is used to: receive self-inspection task information of the current railway housing construction; the self-inspection task information includes task type and self-inspection area range; and call a pre-stored railway housing BIM virtual model based on the self-inspection task information.

[0178] The second processing module 202 is used to: collect local environmental data of the current railway building at the self-inspection site through a mobile terminal; and perform spatial registration between the local environmental data and the railway building BIM virtual model to obtain registration parameters.

[0179] The third processing module 203 is used to: perform spatial coordinate transformation on the railway building BIM virtual model based on the registration parameters to obtain visualized BIM data aligned with the current field of view; the visualized BIM data includes railway building components, pipeline systems and equipment location information;

[0180] The fourth processing module 204 is used to: acquire a real-time image of the current field of view of the railway building through a mobile terminal camera; and overlay the visualized BIM data onto the corresponding position of the real-time image of the current field of view of the railway building to obtain a fused augmented reality view;

[0181] The fifth processing module 205 is used to: extract decision support information matching the self-inspection task from the self-inspection knowledge base according to the task type; associate the decision support information with the corresponding components in the railway building BIM model based on spatial coordinates and overlay it in the augmented reality view to generate an interactive self-inspection guidance interface.

[0182] It should be noted that the AR-based railway building BIM visualization system provided in this embodiment of the invention is used to execute all the process steps of the AR-based railway building BIM visualization method in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0183] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0184] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0185] The embodiments, implementation methods, and related technical features of this application can be combined and substituted for each other without conflict.

[0186] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. An AR-based BIM visualization method for railway buildings, characterized in that, include: Receive self-inspection task information for current railway housing construction; The self-test task information includes the task type and the self-test area range; The pre-stored railway building BIM virtual model is invoked based on the self-inspection task information; At the self-inspection site, local environmental data of the railway building is collected using a mobile terminal; the local environmental data is then spatially registered with the railway building's BIM virtual model to obtain registration parameters. Based on the registration parameters, the spatial coordinate transformation of the railway building BIM virtual model is performed to obtain visualized BIM data aligned with the current field of view; The visualized BIM data includes location information for railway building components, pipeline systems, and equipment. Real-time images of the current field of view of the railway building are obtained through the mobile terminal camera; The visualized BIM data is overlaid onto the corresponding position of the real-time image of the railway building in the current field of view to obtain a fused augmented reality view; Based on the task type, decision support information matching the self-inspection task is extracted from the self-inspection knowledge base; the decision support information is associated with the corresponding components in the railway building BIM model based on spatial coordinates and overlaid in the augmented reality view to generate an interactive self-inspection guidance interface; The method further includes: calculating the deviation value of spatial location matching based on the registration parameters; determining whether the deviation value exceeds a first preset threshold; if it exceeds, extracting the local edge contour information of the railway building from the local environment data, comparing and analyzing the local edge contour information with the contour of the railway building BIM model to adjust the current field of view, and generating the final registration parameters; if it does not exceed, determining the current registration parameters as the final registration parameters. The deviation value of spatial location matching is calculated based on the registration parameters, including: extracting a list of key components from the railway building BIM virtual model; the key components include load-bearing walls, load-bearing columns, beams, floor slabs and pipeline interfaces; An initial weight coefficient is assigned to each critical component according to the preset component importance classification rules, wherein the weight coefficient of load-bearing components is higher than that of non-load-bearing components. Based on the current registration parameters, the theoretical coordinates of each key component in the BIM virtual model are transformed to the local environment coordinate system to obtain the predicted position of each key component in the local environment. Using the predicted location as the search center, a local search area is delineated in the local environmental data, and the actual coordinates of the corresponding physical components are extracted; Calculate the three-dimensional Euclidean distance between the predicted position of each key component and its corresponding actual coordinates to obtain the position deviation value of each key component; The weighted sum of the initial weight coefficients and the positional deviations of the corresponding key components is then divided by the sum of all weight coefficients to obtain the comprehensive deviation value of spatial position matching. The comprehensive deviation value is standardized to generate a standardized deviation value, which serves as the deviation value for spatial location matching.

2. The method according to claim 1, characterized in that, At the self-inspection site, local environmental data of the railway building is collected using a mobile terminal, including: collecting raw point cloud data of the local area of ​​the railway building using the depth camera of the mobile terminal, and simultaneously collecting corresponding environmental images using an RGB camera; Optimized 3D spatial data is obtained by preprocessing the collected raw point cloud data; Edge detection and corner feature extraction are performed on the environmental image to identify the outlines of doors and windows, beam and column structures and wall planar features of railway buildings, and to generate visual feature data containing feature descriptors and coordinate information; The optimized 3D spatial data and visual feature data are spatially aligned based on device pose data to construct a spatial feature representation of the local environment; Based on optimized 3D spatial data, visual feature data, equipment pose data, and spatial feature representation, a local environment dataset of railway buildings containing geometric information, semantic labels, and spatial positional relationships is generated through data fusion.

3. The method according to claim 2, characterized in that, Spatial registration of the local environmental data with the railway building BIM virtual model to obtain registration parameters includes: before the self-inspection task begins, the background system performs offline feature extraction on the railway building BIM virtual model to generate a BIM feature index library containing planar areas, edge segments, and structural corner points; and stores the BIM feature index library in the local cache of the mobile terminal. During the self-inspection process, the BIM feature data for the corresponding area is loaded from the locally cached BIM feature index library based on the scope of the self-inspection area. Based on the device pose data of the mobile terminal, the BIM feature data is projected onto the local environment coordinate system to obtain the initial registration position; Feature extraction is performed on the collected local environmental data; specifically, planar regions are extracted from optimized 3D spatial data and linear edge features are extracted from the environmental image; the planar regions and linear edge features are spatiotemporally aligned to generate an environmental feature set; Within the neighborhood of the initial registration location, the environmental feature set is matched with the projected BIM feature data to generate a set of feature correspondences. The feature correspondence set is filtered to remove incorrect matches; the correction vector for the initial registration position is calculated based on the correct matches after filtering. The correction vector is applied to the initial registration position to generate a spatial transformation matrix, which is then output as the registration parameter.

4. The method according to claim 3, characterized in that, Associating the decision support information with the corresponding components in the railway building BIM model based on spatial coordinates and overlaying it in the augmented reality view includes: loading the corresponding decision support information template from the preset decision support template library based on the self-check task type; Bind the text, icons, or operation instructions in the decision support information template to the spatial coordinates of the components in the railway building BIM model; In the augmented reality view, the display position of decision support information is adjusted based on the device pose data of the mobile terminal to align the decision support information with the position of the actual component. Interactive operations trigger the expansion or collapse of decision support information associated with components.

5. The method according to claim 4, characterized in that, Adjusting the display position of decision support information based on the device pose data of the mobile terminal in the augmented reality view includes: real-time acquisition of rotation matrix and translation vector in the device pose data of the mobile terminal; The alignment angle between the display direction of the decision-aid information and the actual component is calculated based on the rotation matrix. The relative distance between the decision support information and the actual component is adjusted based on the translation vector so that the decision support information fits the surface of the component. When decision support information is obscured by a real object, adjust the display position to avoid the obscured area; Decision support information that is outside the current field of view can be cropped or hidden.

6. The method according to claim 5, characterized in that, Binding the text, icons, or operation instructions in the decision support information template to the spatial coordinates of components in the railway building BIM model includes: determining the decision support information template type corresponding to the component through the mapping relationship between the BIM model component ID and the decision support template component type; Load matching template content from the preset decision support template library; Bind the template content to the BIM model component ID and calculate the initial display position of the decision support information in the augmented reality view based on the component center coordinates; An operation heatmap is generated by recording the frequency of user interaction with components and the duration of their interaction. The operational activity index of a component is calculated based on the sum of the number of interactions and dwell time within a preset time period. Component priorities are determined by comparing the operational activity index with a preset threshold. Components with an operational activity index greater than a first preset index are classified as high-priority components, while components with an operational activity index lower than a second preset index are classified as low-priority components. Render the decision support information of high-priority components to the central area of ​​the field of view and display the decision support information of low-priority components edge-aligned to reduce visual interference.

7. The method according to claim 6, characterized in that, The optimized 3D spatial data and visual feature data are spatially aligned based on the device pose data, including: obtaining the rotation matrix and translation vector in the device pose data of the mobile terminal; Transform each point in the point cloud data to the local environment coordinate system based on the rotation matrix and translation vector; The feature points in the visual feature data are projected onto three-dimensional space to obtain projection points; The depth information of the projection point is corrected using depth camera data; Spatial registration is performed between point cloud data and projection points; Constructing spatial feature representations of local environments, including: Geometric features are extracted from the registered point cloud data; including detecting the planar region of the wall and calculating its normal vector; and identifying the edge segments and corner coordinates of the beam and column structure. Structural semantic labels are generated based on geometric features and preset rules; among them, the wall type is determined by the normal vector and curvature of the planar region; door and window openings are matched according to the length and angle of the edge line segments; and the location of equipment mounting holes is identified by the corner point distribution pattern. Spatial binding of geometric features and semantic labels is performed; specifically, component bounding boxes are generated based on the projection relationship between point cloud planar regions and semantic labels; and a mapping table between geometric features and semantic labels is created. Record the adjacency relationships between components and store the binding relationships between components and spatial coordinates to obtain a local environmental feature expression that includes spatial relationships.

8. An AR-based railway building BIM visualization system, characterized in that, include: The first processing module is used to: receive the self-inspection task information of the current railway housing construction; The self-inspection task information includes the task type and the self-inspection area; based on the self-inspection task information, a pre-stored railway building BIM virtual model is invoked; The second processing module is used to: collect local environmental data of the current railway building at the self-inspection site via a mobile terminal; and perform spatial registration between the local environmental data and the railway building BIM virtual model to obtain registration parameters. The third processing module is used to: perform spatial coordinate transformation on the railway building BIM virtual model based on the registration parameters to obtain visualized BIM data aligned with the current field of view; the visualized BIM data includes railway building components, pipeline systems and equipment location information; The fourth processing module is used to: acquire real-time images of the current field of view of the railway building through the mobile terminal camera; The visualized BIM data is overlaid onto the corresponding position of the real-time image of the railway building in the current field of view to obtain a fused augmented reality view; The fifth processing module is used to: extract decision support information matching the self-inspection task from the self-inspection knowledge base according to the task type; associate the decision support information with the corresponding components in the railway building BIM model based on spatial coordinates and overlay it in the augmented reality view to generate an interactive self-inspection guidance interface; It also includes: calculating the deviation value of spatial location matching based on the registration parameters; determining whether the deviation value exceeds a first preset threshold; if it exceeds, extracting the local edge contour information of the railway building from the local environment data, comparing and analyzing the local edge contour information with the contour of the railway building BIM model to adjust the current field of view, and generating the final registration parameters; if it does not exceed, determining the current registration parameters as the final registration parameters. The deviation value of spatial location matching is calculated based on the registration parameters, including: extracting a list of key components from the railway building BIM virtual model; the key components include load-bearing walls, load-bearing columns, beams, floor slabs and pipeline interfaces; An initial weight coefficient is assigned to each critical component according to the preset component importance classification rules, wherein the weight coefficient of load-bearing components is higher than that of non-load-bearing components. Based on the current registration parameters, the theoretical coordinates of each key component in the BIM virtual model are transformed to the local environment coordinate system to obtain the predicted position of each key component in the local environment. Using the predicted location as the search center, a local search area is delineated in the local environmental data, and the actual coordinates of the corresponding physical components are extracted; Calculate the three-dimensional Euclidean distance between the predicted position of each key component and its corresponding actual coordinates to obtain the position deviation value of each key component; The weighted sum of the initial weight coefficients and the positional deviations of the corresponding key components is then divided by the sum of all weight coefficients to obtain the comprehensive deviation value of spatial position matching. The comprehensive deviation value is standardized to generate a standardized deviation value, which serves as the deviation value for spatial location matching.

Images