An ar positioning system and method for construction period

Abstract

The application discloses an AR positioning system and method for engineering construction period. The application discretely collects construction site fixed feature environment nodes (the distance is not more than 10 meters) through a device with a Lidar system; the collected data grid is reconstructed and introduced into a game engine to build a space reference array which retains original coordinate information; a virtual information position is set in the array; two algorithms of grid dynamic positioning and video flow dynamic positioning are innovatively written and associated, and the starting priority of the former is set to be higher than that of the latter; when there is an environment node in the visual field, grid matching is preferentially adopted to perform accurate pose calculation and correction; when there is no node in the visual field, video flow positioning is enabled, and the grid data of adjacent nodes are used to suppress accumulated errors in time. The application effectively reduces data cost and performance consumption, and especially for complex environment in the engineering construction period, high-precision and anti-drift dynamic positioning in large-scale roaming is realized.

Description

Technical Field

[0001] This invention relates to the field of AR spatial recognition and positioning technology, specifically to an AR positioning system and method for use during the engineering construction phase. Background Technology

[0002] AR technology, also known as "augmented reality technology," is a new technology that seamlessly integrates information from the physical world and the virtual world. Through computer and other scientific technologies, it simulates and then overlays and merges the virtual information with the real environment, applying virtual information to the real world and allowing it to be perceived by human senses, thereby achieving an audiovisual experience that transcends reality.

[0003] As engineering construction and management gradually move towards BIM-based digital information management, a more intuitive visualization platform is needed to effectively utilize this information. AR (Augmented Reality), a technology that embeds relevant digital information into the real-world interface, will effectively fill this gap in visualization management platforms. AR applications during the construction phase are closely linked to spatial positioning technology. Especially in AR scenarios for business functions such as quality inspection and underground space queries, high-precision fitting of virtual information with the real environment within the same scene is required to facilitate subsequent construction and maintenance. Therefore, high accuracy in positioning within the real environment is crucial. Currently, spatial recognition and positioning technologies are limited by insufficient hardware, algorithms, and computing power, resulting in low accuracy and stability, which restricts the depth of AR applications during the construction phase.

[0004] Existing AR positioning technologies can include the following methods:

[0005] The first approach involves incorporating GPS positioning services into AR application development, placing virtual information onto corresponding latitude and longitude coordinates. The drawbacks of this method are: 1. Large positioning errors; 2. Inability to dynamically correct positioning errors; 3. Continuous deviation during the positioning process; 4. Inability to accurately obtain elevation positioning data.

[0006] The second method involves scanning to obtain the grid data of the entire scene, which is then used as a reference to place virtual information at the corresponding real-world locations during development. The disadvantages of this method are: 1. High scanning costs; 2. Complex and variable on-site environments during construction, making it difficult to capture the required environmental features in one go; 3. Huge data volume, inconvenient storage, and high hardware and software performance.

[0007] The third method uses single-point identification positioning, typically using a single QR code or image as the reference for identification and positioning. Virtual information is then placed and aligned in relative positions based on this reference. However, this method has several drawbacks: 1. Poor positioning accuracy; 2. The error increases as the field of view moves away from the identification point; 3. It is difficult to determine the accuracy of the initial positioning.

[0008] In conclusion, improving the accuracy and anti-drift capability of AR positioning in complex, dynamic, and unstable engineering construction sites is an urgent need for applications in this field. Existing single technical approaches, whether based on GPS, full-field scanning, single-point positioning, or even the general visual inertial odometry (VIO) method, are insufficient to effectively address the challenges posed by dynamic environmental changes during construction, especially in providing continuous, stable, and automatically correctable high-precision positioning during long-term, large-scale roaming. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention provides an AR positioning system and method for use during engineering construction. It utilizes grid data of multiple fixed-feature environments to construct a spatial reference array, connects grid dynamic positioning with video stream dynamic positioning, and achieves high-precision positioning in physical space.

[0010] The technical solution adopted in this invention is: an AR positioning system for engineering construction, comprising a data import module, a spatial reference array creation module, a virtual information placement module, a feature point comparison and analysis module, a positioning and correction module, and an output module.

[0011] The data import module includes an FBX format conversion submodule and an information data import submodule. The FBX format conversion submodule is used to convert grid data and external virtual information into FBX format, which is then imported into the game engine through the information data import submodule.

[0012] The spatial reference array creation module integrates the grid data converted by the data import module. It includes a data coordinate information reading submodule, a data self-distribution submodule, and a spatial reference array submodule. The data coordinate information reading submodule reads the coordinate information imported during data import. The data self-distribution submodule automatically distributes the data based on the read coordinate information. The spatial reference array submodule constructs the spatial reference array based on the data self-distribution and stores and displays the spatial reference array. The spatial reference array constructed by the spatial reference array creation module is not a continuous environmental model, but rather consists of grid data of multiple discrete, fixed characteristic environmental nodes (such as transformer boxes and construction signs) during the construction period. These nodes are distributed in space at intervals not exceeding 10 meters, forming a 'positioning anchor point network' covering the work area. Each node retains its original real-world coordinate information collected by the LiDAR system, providing a reference for subsequent high-precision absolute positioning.

[0013] The virtual information placement module is used to place the virtual information transformed by the data import module into the corresponding marked position in the spatial reference array submodule. This position is the position where the virtual information needs to be mapped in the real environment.

[0014] The feature point comparison and analysis module is used to compare and analyze the grid data in the spatial reference array submodule with the real-time image acquired by the mobile device camera. It includes a feature point extraction and comparison submodule, a pose calculation submodule, and a pose calculation result output submodule. The feature point extraction and comparison submodule extracts 2D feature points from the real-time image captured by the camera and compares them with the 3D grid points in the spatial reference array submodule. The pose calculation submodule calculates the real-time pose of the camera based on the comparison and analysis results from the feature point extraction and comparison submodule. The pose calculation result output submodule outputs the pose calculation result. The virtual information placement module employs a hybrid positioning and dynamic priority switching strategy. When the mobile device camera captures at least one pre-established fixed feature environment node within its field of view, the system prioritizes the grid dynamic positioning mode, matching the 2D feature points in the real-time video stream with the precise 3D grid model of the corresponding node in the spatial reference array to achieve high-precision absolute pose calculation and error correction. When the camera moves out of the node's field of view, the system automatically switches to the video stream dynamic positioning mode, using the visual inertial odometry principle for continuous pose estimation. The cumulative error generated in this mode will be corrected in time by the grid dynamic positioning when the camera enters the field of view of the next node (the error is controllable because the distance is <10 meters), thus ensuring the positioning stability throughout the roaming process.

[0015] The positioning and correction module is used to call the pose calculation result output submodule and reconstruct the spatial coordinate system according to the current view of the camera, so as to redraw the virtual information to the correct position in the real environment.

[0016] The output module is used to mix virtual information with the real environment and output it to the end user.

[0017] An AR positioning method for use during the construction phase of an engineering project includes the following steps:

[0018] S1, Data acquisition equipment with LiDAR system is used to collect data on the fixed and distinctive environment at the construction site during the construction period;

[0019] S2, the collected data is reconstructed into a grid, generating grid data, which is then imported into the game engine platform as a reference node for real-world environment nodes in virtual space;

[0020] S3, Establish a spatial reference array;

[0021] S4. Import virtual information and find the marker position where the virtual information needs to be placed in the spatial reference array for position fitting.

[0022] S5, Write the algorithms for grid dynamic positioning and correction methods and video stream dynamic positioning methods, and associate them;

[0023] S6, set the priority for positioning and correction startup;

[0024] S7 matches and renders virtual information with the real environment to achieve highly accurate positioning and display it to the end user.

[0025] Furthermore, in step S1, the Lidar system is a system that integrates three technologies: laser, global positioning system (GPS) and inertial navigation system (INS), used to obtain point cloud data and generate accurate digital three-dimensional models. Each collected environmental object in reality is named as a node, and the distance between each environmental node is ensured not to exceed 10 meters during the acquisition.

[0026] Furthermore, in step S2, the game engine mainly includes Unity3D and Unreal Engine. After importing into the game engine platform, each reference node is named.

[0027] Furthermore, in step S3, the spatial reference array is composed of grid data from each node. Since the data collected by the LiDAR system contains real location information, the spatial reference array retains the original location information. Further, in step S4, the location fitting is based on self-mapped data. Pre-marked areas in the real environment are used as environmental nodes for data collection, and the converted grid data is then imported into the game engine as reference nodes.

[0028] Furthermore, step S5 includes the following sub-steps:

[0029] S51, Write the grid dynamic positioning and correction algorithm into the AR component. For example, when the camera is within the field of view of node N1, record the images captured by the camera at times t and t+1, and name them Pt and Pt+1. Then, extract a large number of 2D feature points in the images, match the extracted 2D feature points with the 3D grid points of M1 in the spatial reference array, and calculate the pose information required by the camera for positioning and correction.

[0030] S52, Write the algorithm of the video stream dynamic positioning method into the AR component. When the camera is outside the field of view of the fixed feature environment node, the motion sensor of the AR device continuously collects the images captured by the camera, extracts feature points from the images in real time, marks each feature point, assigns it an independent ID identity information, and continuously tracks these feature points. When the device moves or rotates, the feature points in the continuous images will form a parallax. Triangulation is performed using the camera pose parallax to calculate the missing information during the movement for positioning and correction.

[0031] Furthermore, in step S6, the priority of setting the positioning and correction start is set so that the priority of grid dynamic positioning and correction is higher than that of video stream dynamic positioning and correction. If the moving distance is too large, video stream dynamic positioning will produce errors. Therefore, it is required that the distance between each node in the spatial reference array does not exceed 10 meters, and the positioning accuracy is ensured by timely correction through grid dynamic positioning.

[0032] The beneficial effects of this invention are:

[0033] This invention can effectively reduce data acquisition and storage costs, greatly reduce the performance consumption of equipment positioning, and can accurately locate the site during the construction period. It solves the adverse effects caused by environmental changes during the construction period. After positioning, it can roam over a large area and perform dynamic correction to ensure the accuracy and stability of the current positioning. Attached Figure Description

[0034] Figure 1 This is a structural block diagram of an AR positioning system for use during the engineering construction phase, provided as an embodiment of the present invention.

[0035] Figure 2 A flowchart illustrating an AR positioning method for engineering construction phase, provided as an embodiment of the present invention;

[0036] In the diagram: 10 - Data import module; 20 - Spatial reference array creation module; 30 - Virtual information placement module; 40 - Feature point comparison and analysis module; 50 - Positioning and correction module; 60 - Output module; 1001 - FBX format conversion submodule; 1002 - Information data import submodule; 2001 - Data coordinate information reading submodule; 2002 - Data self-distribution submodule; 2003 - Spatial reference array submodule; 4001 - Feature point extraction and comparison submodule; 4002 - Pose calculation submodule; 4003 - Pose calculation result output submodule. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the embodiments and accompanying drawings. Obviously, the described examples are merely a part of the embodiments of the present invention, and not all of them.

[0038] Before providing a further detailed description of the embodiments of this application, the names and terms used in the examples of this application are explained. The nouns and terms used in the embodiments of this application are subject to the following interpretations. 1. Three-dimensional modeling software refers to polygon modeling software and parametric modeling software, such as 3DMax, Revit, etc.

[0039] 2. AR SDK components refer to the toolkit components in the game engine platform used for AR development.

[0040] 3. Camera components mainly refer to camera sensors, including visual and inertial sensors.

[0041] This embodiment takes an indoor scene during the construction phase as an example, such as... Figure 1 As shown, the present invention provides an AR positioning system for engineering construction, comprising a data import module 10, a spatial reference array creation module 20, a virtual information placement module 30, a feature point comparison and analysis module 40, a positioning and correction module 50, and an output module 60.

[0042] The data import module 10 includes an FBX format conversion submodule 1001 and an information data import submodule 1002. The FBX format conversion submodule 1001 is used to convert grid data and external virtual information into FBX format, which is then imported into the game engine through the information data import submodule 1002.

[0043] The spatial reference array creation module 20 is used to integrate the grid data converted by the data import module 10. It includes a data coordinate information reading submodule 2001, a data self-distribution submodule 2002, and a spatial reference array submodule 2003. The data coordinate information reading submodule 2001 is used to read the coordinate information brought with it during data import. The data self-distribution submodule 2002 automatically distributes the data according to the read coordinate information. The spatial reference array submodule 2003 constructs the spatial reference array based on the data self-distribution and stores and displays the spatial reference array.

[0044] The virtual information placement module 30 is used to place the virtual information transformed by the data import module 10 into the corresponding marked position in the spatial reference array submodule 2003. This position is the location where the virtual information needs to be mapped in the real environment.

[0045] The feature point comparison and analysis module 40 is used to compare and analyze the grid data in the spatial reference array submodule 2003 with the real-time image acquired by the mobile device camera. It includes a feature point extraction and comparison submodule 4001, a pose calculation submodule 4002, and a pose calculation result output submodule 4003. The feature point extraction and comparison submodule 4001 is used to extract 2D feature points from the real-time image captured by the camera and compare and analyze them with the 3D grid points in the spatial reference array submodule 2003. The pose calculation submodule 4002 calculates the real-time pose of the camera based on the comparison and analysis results of the previous module 4001. The pose calculation result is output through the pose calculation result output submodule 4003.

[0046] The positioning and correction module 50 is used to call the pose calculation result output submodule 4003 mentioned above, and reconstruct the spatial coordinate system according to the current view of the camera, and redraw the virtual information to the correct position in the real environment.

[0047] The output module 60 is used to mix virtual information with the real environment and output it to the end user.

[0048] like Figure 2 As shown, an AR positioning method for use during the construction phase of an engineering project includes the following steps:

[0049] S1. Data acquisition equipment with LiDAR system is used to collect data on the fixed and distinctive environment at the construction site, including transformer boxes, construction signs, etc. Each environmental object collected in reality is named as a node and named N1, N2, N3, etc. During the collection, the distance between each environmental node is ensured to be no more than 10 meters.

[0050] S2, reconstruct the collected data into a grid, generate grid data, and import it into the game engine platform as a reference node for the real environment nodes in the virtual space. Name each reference node M1, M2, M3, etc. in sequence.

[0051] S3, create a dataset in the game engine platform, drag all grid data into the dataset, select the distribution coordinate type, set it to the self coordinate when importing each grid data, confirm to start automatic distribution, and create a spatial reference array;

[0052] S4 converts the virtual information created by the 3D modeling software into FBX format and imports it into the game engine platform. Based on the surveyed plan and the marker points in the real space, and based on the relative positions of the spatial reference array established in S3, it performs placement and position fitting.

[0053] S5 involves writing algorithmic code in C# for both grid-based dynamic positioning and correction, and video stream-based dynamic positioning within the game engine platform. The completed C# file is then attached to the AR SDK component, which is configured by selecting the camera component as the associated object. The specific steps include:

[0054] S51, in the C# file of this example, write the grid dynamic positioning and correction algorithm. When the camera is within the field of view of node N1, record the images captured by the camera at times t and t+1, named Pt and Pt+1. Then, extract a large number of 2D feature points in the images, match the extracted 2D feature points with the 3D grid points of M1 in the spatial reference array, and calculate the pose information required by the camera for positioning and correction.

[0055] S52, In the C# file of this example, a dynamic positioning algorithm for video stream is written. When the camera is outside the field of view of a fixed feature environment node, the motion sensor of the AR device continuously collects the images captured by the camera and extracts feature points from the images in real time. Then, each feature point is marked and given an independent ID identity information, and these feature points are continuously tracked. When the device moves or rotates, the feature points in the continuous images will form a parallax. Triangulation is performed using the camera pose and parallax to calculate the missing information during the movement for positioning and correction.

[0056] S6, in the AR SDK component, set the priority of localization and correction, setting grid dynamic localization and correction to have a higher priority than video stream dynamic localization and correction. In this example, when all or part of the real-world environment node images appear in the camera's field of view, grid dynamic localization and correction are enabled first. When the camera scans node N1, the image feature points of N1 are matched with the point features of M1 in the virtual reference array. When the two match perfectly, the localization is accurate. In this example, when the camera's field of view completely moves out of the real-world environment nodes, video stream dynamic localization and correction are enabled to track the real-world environment feature points. When the device... When the movement distance is large and the movement speed is fast, feature point tracking will produce errors, affecting the positioning accuracy. Therefore, when collecting data from nodes N1, N2, N3, and N4, ensure that the distance between two adjacent nodes does not exceed 10 meters. The error caused by this distance can be ignored. At the same time, a spatial reference array is formed by connecting each node to ensure that when the camera's field of view leaves a certain distance from a real node, there will be grid data from other adjacent nodes to correct the deviation. For example, after the camera's field of view leaves N1, video stream dynamic positioning and deviation correction are enabled. When it enters the range of N2 and N3, grid positioning is enabled, and matching and deviation correction are performed with M1 and M2 to ensure the accuracy of positioning.

[0057] S7, based on the calculation results of S5, uses the game engine's image calculation and lighting rendering to overlay virtual information with the real environment, achieving accurate mapping of virtual information in the real environment within the same screen, and outputting it to the end user.

[0058] The present invention has been described above with reference to the accompanying drawings, but it should not be construed as limiting the scope of protection of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. An AR positioning system for use in an engineering construction period, characterized by: It includes data import module, space reference array creation module, virtual information placement module, feature point comparison and analysis module, positioning and deviation correction module, output module: The data import module includes FBX format conversion submodule and information data import submodule; the FBX format conversion submodule is used for converting mesh data and external virtual information into FBX format, and the information data import submodule is used for importing the mesh data and external virtual information into the game engine; The space reference array creation module is used for integrating the mesh data converted by the data import module, and includes data coordinate information reading submodule, data distribution submodule and space reference array submodule; the data coordinate information reading submodule is used for reading coordinate information carried by the data import module; the data distribution submodule is used for automatically distributing data according to the read coordinate information; and the space reference array submodule is used for constructing a space reference array based on the data distribution, and storing and displaying the space reference array; wherein the space reference array creation module is constructed based on mesh data of a plurality of discrete collected fixed feature environment nodes, the distance between the nodes is not more than 10 meters, and the original coordinate information is retained; The virtual information placement module is used for placing virtual information converted by the data import module into a corresponding mark position in the space reference array submodule, which is a position where the virtual information needs to be mapped in the real environment; The feature point comparison and analysis module is used for comparing and analyzing the mesh data in the space reference array submodule with a real-time image obtained by a camera of a mobile device, and includes feature point extraction and comparison submodule, pose calculation submodule and pose calculation result output submodule; the feature point extraction and comparison submodule is used for extracting 2D feature points from the real-time image captured by the camera and comparing and analyzing the 2D feature points with 3D mesh points in the space reference array submodule; the pose calculation submodule is used for calculating a real-time pose of the camera based on the comparison and analysis result of the feature point extraction and comparison submodule, and the pose calculation result output submodule is used for outputting the pose calculation result; wherein the feature point comparison and analysis module is configured to: when at least one of the fixed feature environment nodes appears in the field of view of the mobile device camera, preferentially start a mesh dynamic positioning and deviation correction algorithm, match 2D feature points in a real-time image with 3D mesh points of corresponding nodes in the space reference array, and calculate a pose; when there is no fixed feature environment node in the field of view of the mobile device camera, start a video stream dynamic positioning algorithm, and perform triangulation and pose estimation by tracking feature points in consecutive images; The positioning and deviation correction module is used for calling the pose calculation result output submodule, reconstructing a space coordinate system according to a current view angle of the camera, and redrawing virtual information to a correct position in the real environment; The output module is used for outputting the mixed virtual information and real environment image to a terminal user.

2. An AR positioning method for construction period, based on the system of claim 1, characterized in that: It includes the following steps: S1, using a data collection device with a Lidar system to collect data of fixed and feature obvious environments during construction period; the distance between the environment nodes is ensured to be not more than 10 meters during collection; S2, the collected data is reconstructed by grid, the grid data is generated, and the grid data is imported into a game engine platform as a reference node of a virtual space in a real environment; S3, a space reference array is established; the space reference array is composed of a plurality of discrete node grid data with real position information; S4, virtual information is imported, a mark position where the virtual information needs to be placed is found in the space reference array, and position fitting is performed; S5, an algorithm of grid dynamic positioning and deviation correction is written, and the algorithm is associated; S6, a positioning and deviation correction starting priority is set; the starting priority of the grid dynamic positioning and deviation correction is higher than that of video stream dynamic positioning; S7, virtual information is matched and rendered with a real environment, high-precision positioning is realized, and the terminal user is displayed; when an environment node appears in a camera view, the grid dynamic positioning is preferentially started for accurate matching and deviation correction; when there is no environment node in the view, the video stream dynamic positioning is started for pose estimation, and deviation correction is performed in time through grid data with a distance between adjacent nodes of no more than 10 meters, so as to suppress the cumulative error of the video stream dynamic positioning.

3. The AR positioning method for construction period according to claim 2, characterized in that: In step S1, the Lidar system is a system integrating laser, global positioning system (GPS) and inertial navigation system (INS) technologies, which is used to obtain point cloud data and generate an accurate digital three-dimensional model, and a single collected environment object in reality is named as a node.

4. The AR positioning method for construction period according to claim 2, wherein: In step S2, the game engine is Unity3D or Unreal Engine, and each reference node is named after being imported into the game engine platform.

5. The AR positioning method for construction period according to claim 2, characterized in that: In step S3, the space reference array is composed of node grid data, and the original position information is still retained in the space reference array because the data collected by the Lidar system has real position information.

6. The AR positioning method for construction period according to claim 2, wherein: In step S4, the position fitting is based on self-mapping data, marks are pre-prepared in a real environment and then collected as environment nodes, and the converted grid data is imported into a game engine as a reference node.

7. The AR positioning method for construction period according to claim 2, wherein: The step S5 includes the following sub-steps: S51, the grid dynamic positioning and deviation correction algorithm is written in an AR component, when a camera is in a view range of N1 node, images captured by the camera at t and t+1 time points are recorded as Pt and Pt+1, a large number of 2D feature points in the images are extracted, the extracted 2D feature points are matched with 3D grid points of M1 in the space reference array, and the required pose information of the camera is calculated to perform positioning and deviation correction. S52, write the video stream dynamic positioning method in the AR component, when the camera is outside the fixed feature environment node view range, use the motion sensor of the AR device to continuously collect the images taken by the camera, and extract feature points from the images in real time, then mark each feature point, give it an independent ID identity information, and continuously track these feature points. When the device moves or rotates, the feature points in the continuous pictures will form a visual difference. Use the camera pose visual difference for triangulation measurement to calculate the missing information in the movement process for positioning and rectification.

8. The AR positioning method for construction period according to claim 2, wherein: In step S6, the setting of the positioning and rectification start priority is that the grid dynamic positioning and rectification priority is higher than the video stream dynamic positioning and rectification. The video stream dynamic positioning may produce errors when the moving distance is too large, so the distance between each node in the space reference array is required to be not more than 10 meters, and the grid dynamic positioning is used for timely rectification to ensure the accuracy of positioning.

Images