Urban underground pipeline three-dimensional visualization method capable of enhancing virtual-real fusion

Abstract

According to the augmented reality fusion urban underground pipeline three-dimensional visualization method, calculation of internal and external parameters of a camera in an augmented reality fusion system and a three-dimensional tracking registration scheme of the augmented reality fusion system are provided, an underground pipeline visualization system based on augmented reality is constructed, virtual underground pipelines are directly displayed on a construction site scene in an overlapping mode, and the construction efficiency is improved. And fusion of the virtual pipeline and the real scene is realized. A user can correspond a construction site to spatial position information, such as position and depth, of the urban three-dimensional underground pipeline, the depth information of the pipeline in a real environment can be clearly sensed while a virtual-real superposition effect is obtained, and the understanding of the user to the pipeline and the surrounding environment is improved; in addition, the enhanced virtual-real fusion system can interact with a user, so that on-site constructors can understand information such as spatial distribution and shielding relation of underground pipelines, and meanwhile, the user can query various data such as attribution, flow and production places of the pipelines and real images of the pipelines.

Description

Technical Field

[0001] The present application relates to a virtual-real fusion underground pipeline visualization method, and in particular to an enhanced virtual-real fusion urban underground pipeline three-dimensional visualization method, belonging to the field of computer vision virtual reality technology.

[0002] Background Methods

[0003] Underground pipelines are the main arteries of the city. The pipeline structure is intricate and consists of water supply, drainage, electricity, gas, heat, broadcasting and communication, and its ancillary facilities. Because the pipelines are buried deep underground, this brings many problems to the field investigation, construction excavation, planning and deployment of underground pipelines. With the development of computer technology, the management of underground pipelines has changed greatly. The management of pipeline data has changed from paper archives and construction drawings to the current GIS system and BIM system. The traditional management system based on CAD maps and two-dimensional maps has gradually transitioned to three-dimensional. The database management system has improved the retrieval efficiency of complex underground pipeline data. The GIS system displays the spatial location information of underground pipelines in detail, realizes the digital management and display of urban underground pipelines such as water supply and drainage, and provides real and reliable underground pipeline network spatial distribution information and pipeline attribute information for relevant functional departments of the city. On this basis, the GIS system also provides the function of spatial statistical analysis, which can scientifically and effectively plan and design pipelines reasonably, and can effectively reduce the occurrence of construction accidents caused by wrong excavation and mis-excavation.

[0004] However, both the database management system and the 3D GIS system have the disadvantage of not being able to combine the virtual underground pipelines with the real scene. It is difficult for construction workers to match the pipeline model in the system to the real scene at the construction site. During the construction process, due to the lack of understanding of the distribution of underground pipelines, construction accidents such as wrong digging or breaking of pipelines may occur, causing economic losses.

[0005] Traditional underground pipeline visualization methods cannot match the virtual pipeline model in the system with the real environment, and users still cannot determine the location of the pipeline and other information well at the construction site. In the planning stage of the pipeline, the relationship between the pipeline and the surrounding environment can only be virtualized through the system, and the renderings rendered by the system cannot well express the positional relationship between the pipeline and its surrounding environment. During the construction excavation process, accidents such as breaking the pipeline and digging the wrong pipeline are still inevitable, which increases the construction period and seriously affects the lives of citizens.

[0006] With the rapid advancement of urbanization, the structure of pipelines will become more and more complex. Pipelines will be more difficult to manage and maintain, and people will have a stronger desire to improve the efficiency of pipeline management. Through the application of three-dimensional methods, the GIS system simulates the three-dimensional scene of the city's underground structure from multiple angles, and has an intuitive understanding of the distribution of pipelines. However, the traditional three-dimensional GIS system still has the disadvantage of not being able to combine virtual pipeline information with real scenes.

[0007] The problems that need to be solved by the existing underground pipeline 3D visualization method and the key difficulties of this application include:

[0008] (1) The existing technology, whether it is a database management system or a three-dimensional GIS system, has the disadvantage of not being able to combine virtual underground pipelines with real scenes. It is difficult for construction workers to match the pipeline model in the system to the real scene of the construction site. During the construction process, due to the lack of understanding of the distribution of underground pipelines, construction accidents such as wrong digging or breaking of pipelines may occur, causing economic losses. The existing technology cannot superimpose the virtual underground pipeline model and the real environment of the construction site on the same screen for display. The system has poor realism and cannot enhance the user's perception of the real environment by enhancing the screen image. It is impossible to see the deep underground pipelines, etc. The vision does not have the superpower of perspective eyes and cannot enrich people's perception of the environment. It cannot enhance the three-dimensional model into the real environment while reading the auxiliary data of the real scene from the database. It is impossible to display the three-dimensional model of the underground pipeline in the real environment. Construction workers cannot match the pipeline with the surrounding environment. It is not convenient to perceive the depth information and attribute information of the pipeline. The existing technology has very low requirements for hardware equipment and cannot be completed by ordinary mobile devices. The accuracy and timeliness are relatively poor.

[0009] (2) The existing underground pipeline visualization method cannot match the virtual pipeline model in the system with the real environment, and users still cannot determine the location and other information of the pipeline well at the construction site. In the planning stage of the pipeline, the relationship between the pipeline and the surrounding environment can only be virtualized through the system, and the renderings rendered by the system cannot well express the positional relationship between the pipeline and its surrounding environment. Accidents of breaking the pipeline and digging the wrong pipeline during the construction excavation process are still inevitable, which increases the construction period and seriously affects the lives of citizens. It is impossible to directly overlay the virtual underground pipeline on the construction site scene. Users cannot match the construction site with the spatial location information of the city's three-dimensional underground pipeline, such as location and depth, and cannot improve users' understanding of the pipeline and the surrounding environment. With the rapid advancement of urbanization, the structure of pipelines is becoming more and more complex, and pipelines will be more difficult to manage and maintain. People will have a stronger desire to improve the efficiency of pipeline management. Therefore, a new display method is urgently needed in the visualization of underground pipelines.

[0010] (3) The existing sensor-based registration method will also cause the model to float due to the problem of GPS accuracy. GPS cannot provide reliable location information in indoor places with obstructions, which will cause the system to be unavailable. There is a lack of a marker-free tracking registration solution based on three-dimensional visual features, and a lack of a reliable feature point detection algorithm to find natural feature points in the environment. It is impossible to determine the matching relationship between natural feature points in the environment and key frame feature points pre-stored by the system, and it is impossible to solve the camera's internal and external parameters through the matching relationship between the feature points of the two images to determine the position and posture of the camera relative to the scene. The existing technology lacks a three-dimensional tracking registration solution based on mobile sensor network tracking, and it is impossible to obtain data provided by sensors such as GPS and gyroscopes through the API provided by the Android system to determine the position and posture of the camera. The existing technology lacks redundant design, lacks the simultaneous use of a marker-free tracking registration solution based on three-dimensional visual features and a sensor-based registration solution, and cannot achieve three-dimensional registration of pipeline models in complex environments. The traditional three-dimensional GIS system has the disadvantage of not being able to combine virtual pipeline information with real scenes. Summary of the invention

[0011] This application proposes the solution of the internal and external parameters of the camera in the enhanced virtual-real fusion system, the enhanced virtual-real fusion system three-dimensional tracking registration scheme, and the construction of an underground pipeline visualization system based on augmented reality. The three-dimensional underground pipeline visualization system based on the enhanced virtual-real fusion directly superimposes and displays the virtual underground pipeline on the construction site scene to achieve the fusion of the virtual pipeline and the real scene. With the help of the underground pipeline visualization system based on the enhanced virtual-real fusion, the user can correspond the spatial location information such as position and depth of the construction site and the three-dimensional underground pipeline in the city, and can have a clear perception of the depth information of the pipeline in the real environment while obtaining the virtual-real superposition effect. In this way, the user's understanding of the pipeline and the surrounding environment can be improved, and the user has a new basis for the evaluation of the on-site environment. In addition, the enhanced virtual-real fusion system can interact with the user, and the user can exchange data with the system through gestures or touch screens. Through real-time interaction, on-site construction personnel can understand the spatial distribution, occlusion relationship and other information of underground pipelines. At the same time, users can also query the ownership, flow, origin and real image of the pipeline and other data. Mobile enhanced virtual-reality fusion inherits the traditional enhanced virtual-reality fusion method. Construction workers can use PAD and mobile phone handheld devices to check underground pipelines at the construction site, realizing the acquisition of enhanced information anytime and anywhere in a new way.

[0012] In order to achieve the above method effects, the method scheme adopted in this application is as follows:

[0013] Enhance the virtual-real fusion 3D visualization method of urban underground pipelines, establish an enhanced virtual-real fusion 3D tracking and registration solution based on 3D visual feature tracking and mobile device sensors, and build an Android-based underground pipeline visualization system to integrate virtual underground pipelines with the real environment and display them on the same screen;

[0014] 1) Enhance the calculation of camera internal and external parameters in the virtual-reality fusion system: Establish the transformation relationship from the world coordinate system to the pixel coordinate system, determine the camera's position, posture and distortion error by calculating the camera's four internal parameters and six external parameters, and provide the correct transformation relationship for the perspective projection of the 3D pipeline model;

[0015] 2) Enhanced three-dimensional tracking and registration of virtual-reality fusion system: First, establish a tracking and registration scheme without markers based on three-dimensional visual features, improve the SIFT feature point detection algorithm to find natural feature points in the environment, and then determine the matching relationship between the natural feature points in the environment and the key frame feature points pre-stored by the system, and eliminate the mismatched points based on RANSAC, and finally solve the intrinsic and extrinsic parameters of the camera through the matching relationship of the feature points of the two images, so as to determine the position and posture of the camera relative to the scene; second, establish a three-dimensional tracking and registration scheme based on mobile sensor network tracking, and obtain the data provided by GPS and gyroscope sensors based on the Android system API to determine the position and posture of the camera; third, adopt a redundant design, integrate the tracking and registration scheme without markers based on three-dimensional visual features and the three-dimensional tracking and registration scheme based on mobile sensor network tracking, and develop an Android-based underground pipeline enhanced virtual-reality fusion system to realize the three-dimensional tracking and registration of pipeline models in complex environments;

[0016] 3) Construct an underground pipeline visualization system based on augmented reality: Based on Android mobile terminals, an enhanced virtual-reality fusion method is adopted. The location and posture information of the device is determined by tracking feature points based on three-dimensional visual features and matching and using hardware sensors of mobile devices. The virtual pipeline is superimposed on the real scene. At the same time, the GPS hardware sensor obtains the user's location information, and then loads the user's location map and pipeline thematic map to prevent users from getting lost when using the system. The attribute data of the pipeline is managed based on SQLite, and the spatial analysis module based on GIS is used to realize the spatial analysis of the pipeline network. The three-dimensional visualization architecture of urban underground pipelines is divided into five parts: three-dimensional underground pipeline modeling, pipeline spatial database management, tracking and registration without markers based on three-dimensional visual features, three-dimensional tracking and registration based on mobile sensor networks, and computational analysis of pipeline data.

[0017] Preferably, the marker-free tracking registration based on three-dimensional visual features: using the natural features in the camera scene as feature points, determining the camera's posture and position relative to the scene by detecting the feature points, establishing the representation, and matching the feature points. In the marker-free enhanced virtual-real fusion system, firstly, capturing the video frame shot in real time by the camera, searching for natural feature points on the video frame, and establishing the representation of the feature points, and then matching the video frame with the pre-selected reference frame, thereby determining the camera's posture information according to the homography, and registering the virtual three-dimensional model to the system scene using the camera's posture information, completing the virtual-real fusion. The processing process includes:

[0018] 1) Offline preprocessing stage: First, the virtual-real fusion 3D model to be enhanced in the camera scene is created on the computer and its coordinates are corrected. Then, multiple key frame images are taken in the scene, and the feature points on these key frame images are extracted and the feature point indicators are calculated. The data is organized into a database and indexed. The relative position relationship between the 3D pipeline model and the natural feature points on the key frame is pre-set for each frame of the image.

[0019] 2) Online matching stage: extract the video frames obtained by the camera in real time, extract feature points and calculate the representation according to the method of the preprocessing stage, match the feature points by calculating the Hamming distance between the feature points and the representation of the feature points in the database, and after obtaining the matched feature points, calculate the position and posture of the camera relative to the scene, calculate the coordinates of the 3D model in the scene, and finally complete the fusion of virtual and real scenes.

[0020] Preferably, the three-dimensional invariant feature tracking factor is:

[0021] 1) Search and track domain: Define a two-dimensional image as I(x, y), detect in multiple scale spaces, and obtain the representation of the image in different scale spaces by convolving the image with a Gaussian kernel. The coordinates of each pixel in the image are represented as (x, y), and the factors of different scale spaces are represented as σ. In the multi-scale space, the operator is represented by the difference of two Gaussian kernels of different scales. The Gaussian difference value of the pixel on the image in the multi-feature space is calculated, and then each point in the Gaussian difference scale space is compared with the adjacent position in the adjacent scale space to obtain the position of the tracking domain point and the corresponding scale space factor;

[0022] 2) Screening the tracking domain: Compare the local tracking domain of the Gaussian difference detected in the discrete space, remove low-contrast feature points and unstable edge response points, and remove low-contrast key points and image noise in the image;

[0023] 3) Calculate the direction of the key domain: Use the direction distribution of the gradient of the adjacent pixels in the key domain to calculate the direction parameter of the feature point, and then use the gradient histogram of the image to find the stable direction of the local structure of the key point, find the feature point, get the spatial scale σ of the feature point, and calculate the gradient modulus m and direction θ of the point (x, y);

[0024] 4) Describe feature points: Use a set of vectors to describe them. First, correct the main rotation direction to ensure that the feature points have rotation invariance. Then generate a feature vector to describe the feature points and normalize the feature point representation.

[0025] Preferably, tracking point matching based on three-dimensional invariant features: after obtaining the representation of the feature point, the quantitative index of the feature point is calculated. The same feature point has similarities in different images. The similarity of the feature points is determined by calculating the Euclidean distance of the feature vectors. A critical value is set. When the Euclidean distance of two feature points is less than the critical value, the two points are one feature point. A preliminary geometric constraint model is established based on the RANSAC robustness algorithm, and the possible tracking points under the constraint are verified to eliminate erroneous tracking points.

[0026] Preferably, three-dimensional tracking registration based on mobile sensor network:

[0027] 1) Get device location information: In Android applications, use the location service API provided by the system to get the device's location. First, get the location service LocationManager and then specify the data source for obtaining location data as GPS. Finally, use the Location object to resolve to the current location.

[0028] 2) Acquisition of device attitude information: The lateral roll angle and pitch angle are obtained through the accelerometer, g x , g y , g z The components representing the acceleration value of the current device in the three coordinate axis directions of the mobile phone coordinate system are used to calculate the tilt of the mobile phone. The electronic compass is used to obtain the orientation of the device, that is, the direction in which the top of the device points when the device is placed horizontally. The parameters describing the rotational motion of the device itself are obtained by detecting the angle between the device and the gyroscope. The gyroscope instantly completes the measurement of the rotation angle, and cooperates with the magnetic sensor to obtain the motion state of the device and the direction of the device at the same time.

[0029] The mobile terminal posture data includes the device's azimuth angle Yaw, pitch, and roll.

[0030] X-axis direction: from left to right along the horizontal direction of the screen. The device supports automatic screen rotation. If the phone screen is not square, the shorter side should be placed horizontally and the longer side should be placed vertically.

[0031] The direction of the Y axis: from the lower left corner of the screen to the upper left corner of the screen;

[0032] The direction of the Z axis: when placed horizontally, it points vertically to the sky;

[0033] In the onSensorChanged callback function of the Android development program, three parameters about the posture are returned: the first return value is the azimuth angle Yaw, which represents the angle of rotation of the mobile phone around the Z axis. 0° indicates that the device is facing due north, that is, the direction of the y axis. Similarly, 90° represents due east, 180° represents due south, and 270° represents due west; the second return value is the pitch angle, which represents the degree of tilt of the mobile phone. When the mobile phone rotates around the x axis, this return value changes. The value range of the pitch angle is (-180°, +180°); the third return value is the roll angle Roll, which represents the rolling angle of the mobile phone along the y axis, and the value range is (-90°, +90°).

[0034] Preferably, the three-dimensional tracking registration process based on the mobile terminal sensor: obtain the position information and posture information of the device through the mobile terminal hardware sensor, determine the projection relationship between the three-dimensional model and the real scene, and register the real three-dimensional model to the real scene. The process is as follows:

[0035] 1) Build a scene space database: Get the CAD drawing of the three-dimensional pipeline distribution and then make a three-dimensional pipeline model with reference to the CAD drawing. The three-dimensional pipeline model is made in meters. The model coordinate system uses the world coordinate system centered on its own center point. Build the scene's spatial database and calibrate the three-dimensional pipeline model coordinates to the correct geographical location. In the virtual-real fusion, obtain the local scene data of a specific location based on the GPS data;

[0036] 2) Obtain scene space data at a specific location: The GPS positioning module obtains the device location in real time, retrieves the 3D pipeline data of the current location from the database based on the GPS location information, and then obtains the posture information of the mobile phone based on the direction sensor and gravity sensor to determine the location of the 3D pipeline model in the real environment;

[0037] 3) Register spatial data into the real environment: After determining the position of the 3D pipeline model in the real pipeline, use OpenGL ES to overlay the 3D model with the camera video stream to obtain a virtual-real fusion scene.

[0038] Preferably, the three-dimensional visualization architecture of urban underground pipelines:

[0039] 1) 3D underground pipeline modeling: The data sources of the system include two aspects: one is the video stream of the camera, and the other is the virtual pipeline model. In the Android system, the onPreviewFrame interface of setPreviewCallback is implemented to obtain each frame of video stream data in real time, thereby obtaining the video stream data of the real scene. The pipeline model needs to be modeled manually. For pipelines with simpler structures, a database is also used to store their basic information, and then it is drawn to the screen in real time through OpenGL ES during rendering. During the modeling process, the size of the model is ensured to be the same as the size of the real pipeline, and the geometric size of the virtual pipeline model is kept consistent with the real scene;

[0040] 2) Pipeline spatial database management: After obtaining the pipeline model data, it is also necessary to calibrate it to the real geographic space coordinate system. During the calibration process, the latitude and longitude of the center of the model is used as the center and added to the spatial database. At the same time, tile data is generated for the pipeline network spatial data to facilitate smooth rendering of the three-dimensional pipeline data on the mobile terminal. The location information of the mobile device is obtained through the GPS positioning module, and then the corresponding pipeline scene data is loaded from the spatial database;

[0041] 3) Markerless tracking and registration based on 3D visual features: extract feature points in the natural environment and realize the registration between feature points, and calculate the position and posture information of the camera relative to the scene;

[0042] 4) Registration based on mobile hardware sensors: This method combines the registration method based on mobile hardware sensors with the registration method for tracking markers without using three-dimensional visual features. The registration method based on mobile hardware sensors obtains sensor information and calculates the position and posture of the device through sensor data.

[0043] 5) Calculation and analysis of pipeline data: The query of pipeline attribute information is based on the SQLite database on the mobile phone, and the spatial analysis calculation of pipelines is based on GIS spatial analysis.

[0044] Preferably, the virtual-reality fusion visualization framework is enhanced:

[0045] 1) Data layer: including camera video stream data, 3D pipeline model data, pipeline attribute data, pipeline thematic map data, and basic vector map data. The camera video stream data is acquired in real time, and other data is stored in the device in the form of files or in the database in the form of two-dimensional tables;

[0046] 2) Component layer: Data communication between the data layer and the application layer is all in this layer. The component layer accesses various data of the data layer, organizes and encapsulates them, and then hands them over to the application layer for use. OpenCV for Android completes the search and matching of feature points, solves the device posture information, ARCGIS API for Android draws basic vector maps and thematic maps of pipeline networks, Android Camera reads the camera's video stream, OpenGL ES draws a 3D pipeline model on the mobile phone screen, and Android LocationManager obtains the location of the mobile phone. The data obtained includes longitude, latitude and altitude;

[0047] 3) Application layer: responsible for interaction with users, system data display and user data input, including the fusion of virtual and real scenes, the drawing of basic vector maps and pipeline network thematic maps, and spatial analysis and calculation of pipeline networks.

[0048] Preferably, the virtual-real fusion underground pipeline visualization module: adopts modular design to divide the system into several independent modules that do not affect each other;

[0049] 1) Human-computer interaction module: a module for data interaction between the system and the user. The user enters data into the system, and the system feeds back the calculation results to the user, including pipeline enhanced virtual-reality fusion system interface design and human-computer interaction event processing. In this module, the system calls other modules for calculation according to the user's instructions on the screen, and then feeds back the data to the user;

[0050] 2) Sensor data acquisition module: obtain the longitude, latitude and altitude information of the camera in real time so as to call the pipeline model data of the current position from the database, and obtain the attitude information including the inclination angle of the camera in various directions in real time;

[0051] 3) Video stream data acquisition module: acquires the video frames of the camera in real time, provides the video frame data to the feature point search module in the feature point search and matching stage, and provides the video frame to the 3D pipeline model drawing module in the virtual-real fusion stage to complete the fusion display of the virtual 3D pipeline scene and the real environment;

[0052] 4) Feature point search and matching module: It realizes the registration without logo according to the video frame, and solves the position of the camera relative to the scene by establishing the relationship between the feature points on the camera image and the feature points of the real scene;

[0053] 5) Pipeline model drawing module: Use OpenGL ES to draw a 3D pipeline model and overlay the virtual 3D pipeline on the real scene to display enhanced information;

[0054] 6) Two-dimensional map auxiliary module: Based on the GIS method, it provides the location information of the pipeline on the map when the user performs virtual-real fusion augmented reality pipeline data, displays the pipeline information from another angle, and prevents the user from getting lost.

[0055] Preferably, the pipeline visualization module based on the mobile sensor network includes two parts: obtaining the mobile phone location information and obtaining the mobile phone posture information;

[0056] 1) Acquisition of mobile phone location information: There are three parts: longitude, latitude and altitude. In the Android system, the system's built-in components are used to complete the reading of location information. The specific process is as follows:

[0057] Configure the required location permissions in the Android Manifest file. There are two permissions that need to be configured, namely ACCESS_COARSE_LOCATION and ACCESS_FINE_LOCATION. Get the user's location through GPS or mobile base stations. Use the getBestProvider method provided by the LocationProvider class to enable the application to select the most accurate location that the current system can obtain. The LocationListener interface is used to monitor network changes, including changes in the status of GPS, base stations, and WIFI data source providers. This interface must be instantiated before obtaining location information. The location information obtained is a Location object, which records the longitude, latitude, and altitude of the device at that moment, as well as acquisition events and speed information;

[0058] 2) Get sensor data: First, configure the UPDATE_DEVICE_STATS permission in the Manifest file, and then get the SensorManager. The AndroidSensorManager is obtained after the system is initialized. By monitoring the SensorManager, the real-time posture information of the terminal is obtained. The posture information obtained is an encapsulated object.

[0059] Compared with the prior art, the innovations and advantages of this application are:

[0060] (1) This application enhances virtual-real fusion by using computer vision theory, computer graphics methods, sensor methods, three-dimensional drawing methods, psychology and other methods to superimpose virtual three-dimensional models with real scenes in the real environment to achieve a virtual-real fusion effect. During the construction of underground pipelines, construction workers can see the underground pipelines on site. This application proposes the solution of the internal and external parameters of the camera in the enhanced virtual-real fusion system, and the enhanced three-dimensional tracking and registration scheme of the virtual-real fusion system, and constructs an underground pipeline visualization system based on augmented reality. The three-dimensional underground pipeline visualization system based on enhanced virtual-real fusion directly superimposes and displays virtual underground pipelines on the construction site scene to achieve the fusion of virtual pipelines and real scenes. With the help of the underground pipeline visualization system based on enhanced virtual-real fusion, users can correspond the spatial location information of the construction site and the city's three-dimensional underground pipelines, such as location and depth, and can also have a clear perception of the depth information of the pipeline in the real environment while obtaining the virtual-real superposition effect. In this way, users can enhance their understanding of the pipeline and the surrounding environment, and provide users with a new basis for evaluating the on-site environment. In addition, the enhanced virtual-reality fusion system can interact with users, who can exchange data with the system through gestures or touch screens. Through real-time interaction, on-site construction personnel can understand the spatial distribution, occlusion relationship and other information of underground pipelines. At the same time, users can also query the ownership, flow, origin and real image of pipelines and other data. Mobile enhanced virtual-reality fusion inherits the traditional enhanced virtual-reality fusion method. Construction personnel can use PAD and mobile phone handheld devices to view underground pipelines at the construction site, realizing the acquisition of enhanced information anytime and anywhere in a new way.

[0061] (2) This application proposes an augmented reality-based underground pipeline visualization solution, which effectively solves the problem that the existing three-dimensional GIS-based visualization system is difficult to match the pipeline to the real scene. This system can superimpose the virtual underground pipeline with the real environment to achieve a virtual-real fusion effect. Compared with the traditional three-dimensional GIS visualization, this pipeline visualization method can give users a stronger sense of reality, and the background of the pipeline is the real environment, which can provide users with a lot of auxiliary information, and can effectively reduce the construction accidents caused by the lack of understanding of the underground pipeline structure during the construction process. This application adopts a redundant design combining two registration methods, including a registration technology based on computer vision and a registration method based on a mobile sensor network. The registration technology based on computer vision needs to ensure that the scene maintains a certain stable state, and the registration technology based on the sensor needs to ensure that the device can receive a certain number of satellite signals. A redundant design is adopted to combine the two technical solutions. When one registration method fails, it can be switched to another registration method to ensure that the three-dimensional pipeline model can be registered in the real environment to complete the virtual-real fusion, thereby increasing the reliability and robustness of the system.

[0062] (3) Based on the enhanced virtual-real fusion method, the present application superimposes the virtual underground pipeline model and the real environment of the construction site on the same screen for display, so that the enhanced virtual-real fusion system has a good sense of reality. The enhanced screen image increases the user's perception of the real environment, and the deep underground pipelines can be seen, just like having the superpower of perspective eyes. The enhanced virtual-real fusion method can enrich people's perception of the environment. While enhancing the three-dimensional model into the real environment, the auxiliary data of the real scene can also be read from the database, providing users with a new interactive visual interface. Compared with the traditional underground pipeline management system based on three-dimensional GIS, enhanced virtual-real fusion has many advantages. First, by directly displaying the three-dimensional model of the underground pipeline in the real environment, the construction personnel can match the pipeline with the surrounding environment and very conveniently perceive the depth information and attribute information of the pipeline. Secondly, the requirements for enhanced virtual-real fusion on hardware equipment are very low. Currently, mainstream mobile devices such as mobile phones and Pads have such capabilities.

[0063] (4) This application uses computer vision image recognition methods, GIS methods, Android development methods, computer network and communication methods, three-dimensional modeling methods, spatial database management methods, etc. to realize the functions of superimposed display, query management, statistical analysis, etc. of underground pipelines and real scenes, forming a fast, accurate and efficient underground pipeline visualization function, providing auxiliary functions for the planning, laying, and emergency repair of urban pipelines, and realizing the informatization, intelligence, and spatialization of pipeline network data in the process of urban underground pipeline management. The system is based on the enhanced virtual-real fusion method, and uses the method of tracking feature points based on three-dimensional visual features and matching and the hardware sensor of mobile devices to determine the location and posture information of the device, and superimpose the virtual pipeline on the real scene. At the same time, the GPS hardware sensor obtains the user's location information, and then loads the user's location map and pipeline thematic map, which can prevent the user from getting lost when using the system. SQLite is used to manage the attribute data of the pipeline. The user can check the origin information, ownership information, flow information, etc. of the pipeline by clicking on the virtual pipeline. At the same time, with the help of the spatial analysis module of GIS, the spatial analysis function of the pipeline network is realized, providing a strong method support for the management of the pipeline network. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a relationship diagram between the natural feature points in the key frame and the three-dimensional model.

[0065] Figure 2 It is a schematic diagram of the coordinate system of the mobile device.

[0066] Figure 3 It is a visualization framework diagram for enhancing the fusion of virtual and real.

[0067] Figure 4It is the structure diagram of the human-computer interaction module.

[0068] Figure 5 This is a diagram showing the effect of this application system running on a grassland.

[0069] Figure 6 This is the effect diagram of the system of this application running on the road surface.

[0070] Figure 7 This is the effect diagram of the system of this application running on the wall. DETAILED DESCRIPTION

[0071] In conjunction with the accompanying drawings, the method scheme for enhancing the three-dimensional visualization method of urban underground pipelines by integrating virtual and real elements provided in the present application is further described below, so that method personnel in this field can better understand the present application and implement it.

[0072] This application establishes an enhanced virtual-real fusion 3D tracking and registration solution based on 3D visual feature tracking and mobile device sensors, and builds an Android-based underground pipeline visualization system to integrate virtual underground pipelines with the real environment and display them on the same screen;

[0073] 1) Enhance the calculation of camera internal and external parameters in the virtual-reality fusion system: Establish the transformation relationship from the world coordinate system to the pixel coordinate system, determine the camera's position, posture and distortion error by calculating the camera's four internal parameters and six external parameters, and provide the correct transformation relationship for the perspective projection of the 3D pipeline model;

[0074] 2) Enhanced three-dimensional tracking and registration of virtual-reality fusion system: First, establish a tracking and registration scheme without markers based on three-dimensional visual features, improve the SIFT feature point detection algorithm to find natural feature points in the environment, and then determine the matching relationship between the natural feature points in the environment and the key frame feature points pre-stored by the system, and eliminate the mismatched points based on RANSAC, and finally solve the intrinsic and extrinsic parameters of the camera through the matching relationship of the feature points of the two images, so as to determine the position and posture of the camera relative to the scene; second, establish a three-dimensional tracking and registration scheme based on mobile sensor network tracking, and obtain the data provided by GPS and gyroscope sensors based on the Android system API to determine the position and posture of the camera; third, adopt a redundant design, integrate the tracking and registration scheme without markers based on three-dimensional visual features and the three-dimensional tracking and registration scheme based on mobile sensor network tracking, and develop an Android-based underground pipeline enhanced virtual-reality fusion system to realize the three-dimensional tracking and registration of pipeline models in complex environments;

[0075] 3) Construct an underground pipeline visualization system based on augmented reality: Based on Android mobile terminals, an enhanced virtual-reality fusion method is adopted. The location and posture information of the device is determined by tracking feature points based on three-dimensional visual features and matching and using hardware sensors of mobile devices. The virtual pipeline is superimposed on the real scene. At the same time, the GPS hardware sensor obtains the user's location information, and then loads the user's location map and pipeline thematic map to prevent users from getting lost when using the system. The attribute data of the pipeline is managed based on SQLite, and the spatial analysis module based on GIS is used to realize the spatial analysis of the pipeline network. The three-dimensional visualization architecture of urban underground pipelines is divided into five parts: three-dimensional underground pipeline modeling, pipeline spatial database management, tracking and registration without markers based on three-dimensional visual features, three-dimensional tracking and registration based on mobile sensor networks, and computational analysis of pipeline data.

[0076] 1. Enhanced 3D tracking registration of virtual-reality fusion system

[0077] The registration method for tracking without markers based on three-dimensional visual features and the registration method based on device hardware sensors have their own advantages and disadvantages. The existing technology of augmented reality based on three-dimensional visual feature tracking includes two parts: the client and the server. Since the complex operations related to image processing are completed on the server, this augmented reality does not require very high performance of the mobile device, but the system is greatly affected by the network bandwidth. When the network speed is slow, the video data cannot be uploaded to the server in time, and the processing results on the server cannot be returned to the client in time, causing the virtual and real fusion picture to drop frames and freeze, and even cause system abnormalities and system crashes. The registration method based on the hardware sensor of the mobile device uses hardware such as GPS and gyroscope to obtain the position and posture information of the device, so as to determine the projection relationship between the virtual pipeline model and the camera screen. The GPS positioning accuracy of the mobile phone is relatively low and must be calibrated before it can be put into use.

[0078] (I) Markerless tracking and registration based on 3D visual features

[0079] The natural features in the camera scene are used as feature points. The camera's posture and position relative to the scene are determined by detecting feature points, establishing indicators, and matching feature points. In the virtual-reality fusion system without marker points, the video frame shot by the camera in real time is first captured, and the natural feature points are searched for in the video frame, and the indicator of the feature points is established. Then, the video frame is matched with the pre-selected reference frame, so as to determine the camera's posture information according to the homography, and the virtual 3D model is registered to the system scene using the camera's posture information to complete the virtual-reality fusion. The processing process includes:

[0080] 1) Offline preprocessing stage: First, a 3D model that will be enhanced in the camera scene is created on the computer and its coordinates are corrected. Then, multiple key frame images are taken in the scene, and the feature points on these key frame images are extracted and the feature point indicators are calculated. The data is organized into a database and indexed. For each frame of the image, the relative position relationship between the 3D pipeline model and the natural feature points on the key frame is pre-set. Figure 1 Shows the relationship between keyframes and 3D models.

[0081] 2) Online matching stage: extract the video frames obtained by the camera in real time, extract feature points and calculate the representation according to the method of the preprocessing stage, match the feature points by calculating the Hamming distance between the feature points and the representation of the feature points in the database, and after obtaining the matched feature points, calculate the position and posture of the camera relative to the scene, calculate the coordinates of the 3D model in the scene, and finally complete the fusion of virtual and real scenes.

[0082] (1) Three-dimensional invariant feature tracking factor

[0083] 1) Search and tracking domain: Define a two-dimensional image as I(x, y) and detect it in multiple scale spaces. The representation of the image in different scale spaces is obtained by convolving the image with a Gaussian kernel. The coordinates of each pixel in the image are represented as (x, y), and the factors of different scale spaces are represented as σ. In the multi-scale space, the operator is represented by the difference of two Gaussian kernels of different scales. The Gaussian difference value of the pixel on the image in the multi-feature space is calculated, and then each point in the Gaussian difference scale space is compared with the adjacent position in the adjacent scale space to obtain the position of the tracking domain point and the corresponding scale space factor.

[0084] 2) Screening tracking domain: Compare the local tracking domain of Gaussian difference detected in discrete space, remove low-contrast feature points and unstable edge response points, and eliminate low-contrast key points and image noise in the image.

[0085] 3) Calculate the direction of the key domain: Use the directional distribution of the gradient of the adjacent pixels in the key domain to calculate the directional parameters of the feature point, and then use the gradient histogram of the image to obtain the stable direction of the local structure of the key point, find the feature point, obtain the spatial scale σ of the feature point, and calculate the gradient modulus m and direction θ of the point (x, y).

[0086] 4) Describe feature points: Use a set of vectors to describe them. First, correct the main rotation direction to ensure that the feature points have rotation invariance. Then generate a feature vector to describe the feature points and normalize the feature point representation.

[0087] (2) Tracking point matching based on 3D invariant features

[0088] After obtaining the representation of the feature point, the quantitative index of the feature point is calculated. The same feature point has similarities in different images. The similarity of the feature point is determined by calculating the Euclidean distance of the feature vector. A critical value is set. When the Euclidean distance of two feature points is less than the critical value, the two points may be a feature point. Based on the RANSAC robustness algorithm, a preliminary geometric constraint model is established to verify the possible tracking points under the constraint, thereby eliminating the wrong tracking points.

[0089] 2. 3D tracking and registration based on mobile sensor network

[0090] Determine the position and attitude of the device in real time, establish the projection relationship between the virtual 3D model and the real scene, and obtain the device's position information and attitude information through the Android API or iOS API. The acquired position information includes the device's longitude, latitude and altitude, and the attitude information includes the device's rotation angle in all directions. Obtaining these position and attitude information only requires a small amount of system resources, and the device obtains gyroscope data almost instantly. But the problem is that the positioning accuracy of GPS is not high enough, and it is not applicable to more demanding systems.

[0091] 1) Get device location information

[0092] In Android applications, the location of the device is obtained through the location service API provided by the system. First, obtain the location service LocationManager and then specify the data source for obtaining location data as GPS. Finally, use the Location object to resolve the current location.

[0093] 2) Obtaining device posture information

[0094] The lateral roll angle and pitch angle are obtained by the accelerometer, g x , g y , g z Represents the components of the current device's acceleration value in the three coordinate axes of the mobile phone coordinate system, and calculates the tilt of the mobile phone.

[0095] Use an electronic compass to obtain the orientation of the device, that is, the direction the top of the device points to when the device is placed horizontally;

[0096] By detecting the angle between the device and the gyroscope, the parameters describing the device's own rotational motion are obtained. The gyroscope instantly completes the measurement of the rotation angle, and cooperates with the magnetic sensor to obtain the device's motion state and direction at the same time.

[0097] The mobile terminal posture data includes the device's azimuth angle Yaw, pitch, and roll. Figure 2 This is a schematic diagram of the coordinate system of the mobile device.

[0098] X-axis direction: from left to right along the horizontal direction of the screen. The device supports automatic screen rotation. If the phone screen is not square, the shorter side should be placed horizontally and the longer side should be placed vertically.

[0099] The direction of the Y axis: from the lower left corner of the screen to the upper left corner of the screen;

[0100] The direction of the Z axis: when placed horizontally, it points vertically to the sky;

[0101] In the onSensorChanged callback function of the Android development program, three parameters about the posture are returned: the first return value is the azimuth angle Yaw, which represents the angle of rotation of the phone around the Z axis. 0° means that the device is facing due north, that is, the direction of the y axis. Similarly, 90° means due east, 180° means due south, and 270° means due west; the second return value is the tilt angle Pitch, which represents the tilt degree of the phone. When the phone rotates around the x axis, this return value changes. The tilt angle range is (-180°, +180°); the third return value is the roll angle Roll, which represents the roll angle of the phone along the y axis, and the range is (-90°, +90°).

[0102] (1) 3D tracking registration process based on mobile terminal sensors

[0103] The location and posture information of the device is obtained through the hardware sensor of the mobile terminal, the projection relationship between the 3D model and the real scene is determined, and the real 3D model is registered to the real scene. The specific process is as follows:

[0104] 1) Build a scene space database: Get the CAD drawing of the three-dimensional pipeline distribution and then make a three-dimensional pipeline model with reference to the CAD drawing. The three-dimensional pipeline model is made in meters. The model coordinate system uses the world coordinate system centered on its own center point. Build the scene's spatial database and calibrate the three-dimensional pipeline model coordinates to the correct geographical location. In the virtual-real fusion, obtain the local scene data of a specific location based on the GPS data;

[0105] 2) Obtain scene space data at a specific location: The GPS positioning module obtains the device location in real time, retrieves the 3D pipeline data of the current location from the database based on the GPS location information, and then obtains the posture information of the mobile phone based on the direction sensor and gravity sensor to determine the location of the 3D pipeline model in the real environment;

[0106] 3) Register spatial data into the real environment: After determining the position of the 3D pipeline model in the real pipeline, use OpenGL ES to overlay the 3D model with the camera video stream to obtain a virtual-real fusion scene.

[0107] 2. Construction of mobile-enhanced virtual-reality fusion system

[0108] Underground pipelines are distributed in almost every corner of the city, and the environment they are in is also complex and changeable. In terms of pipeline model registration, this application adopts a redundant design, that is, a fusion of a marker-free tracking registration scheme based on three-dimensional visual features and a three-dimensional tracking registration scheme based on a mobile sensor network.

[0109] The unmarked tracking registration scheme based on three-dimensional visual features requires the environment to maintain a certain stability. The growth of grass and trees, the erosion of rainwater, etc. will cause changes in the natural feature points in the environment, resulting in the feature points in the new environment cannot match the feature points in the key frames pre-stored by the system, and ultimately lead to registration failure. The three-dimensional tracking registration scheme based on the mobile sensor network is highly dependent on GPS. The occlusion of buildings will cause GPS signal loss and positioning accuracy changes, resulting in pipeline model registration failure. Therefore, in order to ensure that the virtual underground pipeline model can be three-dimensionally tracked and registered in a complex environment, the above two registration schemes are integrated. When one of the registration schemes fails, it can switch to another registration mode in time to enhance the robustness of the system and ensure the normal operation of the system.

[0110] 1.3D Visualization Architecture of Urban Underground Pipelines

[0111] It is divided into five parts: three-dimensional underground pipeline modeling, pipeline space database management, marker-free tracking and registration based on three-dimensional visual features, three-dimensional tracking and registration based on mobile sensor networks, and computational analysis of pipeline data.

[0112] 1) 3D underground pipeline modeling: The system data sources include two aspects: one is the video stream of the camera, and the other is the virtual pipeline model. In the Android system, the onPreviewFrame interface of setPreviewCallback is implemented to obtain each frame of video stream data in real time, thereby realizing the acquisition of real scene video stream data. The pipeline model needs to be modeled manually. For pipelines with simpler structures, a database is also used to store their basic information, and then it is drawn to the screen in real time through OpenGL ES during rendering. During the modeling process, the size of the model is ensured to be the same as the size of the real pipeline, and the geometric size of the virtual pipeline model is kept consistent with the real scene.

[0113] 2) Pipeline spatial database management: After obtaining the pipeline model data, it is also necessary to calibrate it to the real geographic space coordinate system. During the calibration process, the latitude and longitude of the center of the model is used as the center and added to the spatial database. At the same time, tile data is generated for the pipeline network spatial data to facilitate smooth rendering of three-dimensional pipeline data on the mobile terminal and improve loading efficiency. The location information of the mobile device is obtained through the GPS positioning module, and then the corresponding pipeline scene data is loaded from the spatial database.

[0114] 3) Markerless tracking and registration based on 3D visual features: Extract feature points in the natural environment and realize the alignment between feature points, and calculate the position and posture information of the camera relative to the scene.

[0115] 4) Registration based on mobile hardware sensors: It integrates the registration method based on mobile hardware sensors and the registration method for markerless tracking based on three-dimensional visual features. The registration method based on mobile hardware sensors obtains sensor information and calculates the position and posture of the device through sensor data.

[0116] 5) Calculation and analysis of pipeline data: The query of pipeline attribute information is based on the SQLite database on the mobile phone, and the spatial analysis calculation of pipelines is based on GIS spatial analysis.

[0117] 2. Enhanced virtual-reality fusion visualization framework

[0118] like Figure 3 As shown, 1) Data layer: including camera video stream data, three-dimensional pipeline model data, pipeline attribute data, pipeline thematic map data, and basic vector map data, wherein the camera video stream data is acquired in real time, and other data is stored in the device in the form of files or in the database in the form of two-dimensional tables.

[0119] 2) Component layer: Data communication between the data layer and the application layer is all in this layer. The component layer accesses various data in the data layer, organizes and encapsulates them, and then hands them over to the application layer for use. OpenCV for Android completes the search and matching of feature points and solves the device posture information. ARCGIS API for Android draws basic vector maps and thematic maps of pipeline networks. Android Camera reads the camera's video stream. OpenGL ES draws a 3D pipeline model on the mobile phone screen to achieve the fusion of virtual and real scenes. Android LocationManager obtains the location of the mobile phone, and the data obtained includes longitude, latitude and altitude.

[0120] 3) Application layer: responsible for interaction with users, system data display and user data input, including the fusion of virtual and real scenes, the drawing of basic vector maps and pipeline network thematic maps, and spatial analysis and calculation of pipeline networks.

[0121] (III) Virtual and real fusion underground pipeline visualization module

[0122] The modular design is used to divide the system into several independent modules that do not affect each other, reducing the coupling of each module;

[0123] 1) Human-computer interaction module: a module for data interaction between the system and the user. The user enters data into the system, and the system feeds back the calculation results to the user, including pipeline enhanced virtual-reality fusion system interface design and human-computer interaction event processing. In this module, the system calls other modules for calculation according to the user's instructions on the screen, and then feeds back the data to the user;

[0124] 2) Sensor data acquisition module: obtain the longitude, latitude and altitude information of the camera in real time so as to call the pipeline model data of the current position from the database, and obtain the attitude information including the inclination angle of the camera in various directions in real time;

[0125] 3) Video stream data acquisition module: acquires the video frames of the camera in real time, provides the video frame data to the feature point search module in the feature point search and matching stage, and provides the video frame to the 3D pipeline model drawing module in the virtual-real fusion stage to complete the fusion display of the virtual 3D pipeline scene and the real environment;

[0126] 4) Feature point search and matching module: It realizes the registration without logo according to the video frame, and solves the position of the camera relative to the scene by establishing the relationship between the feature points on the camera image and the feature points of the real scene;

[0127] 5) Pipeline model drawing module: Use OpenGL ES to draw a 3D pipeline model and overlay the virtual 3D pipeline on the real scene to display enhanced information;

[0128] 6) Two-dimensional map auxiliary module: Based on the GIS method, it provides the location information of the pipeline on the map when the user performs virtual-real fusion augmented reality pipeline data, displays the pipeline information from another angle, and prevents the user from getting lost.

[0129] 1. Human-computer interaction module

[0130] like Figure 4 After the user enters the system, the system automatically turns on the camera and loads the current environment image onto the screen. At this time, the user can only see the real scene image and cannot see the enhanced information. By opening the layer management interface, the user selects the layer to be loaded. The layers include water supply pipeline layer, gas heat layer, and cable communication layer. After checking the corresponding layer switch, the enhanced pipeline information is superimposed on the real scene. The virtual-real fusion interface is the main interface of the system. The user can switch to the two-dimensional map interface to view the basic vector map, thematic map of the pipeline network and the user's location information.

[0131] 2. Get video stream data

[0132] First, register permissions in the Manifest file so that the application has the permission to call the camera hardware and software. The permission declarations are Android.permission.camere and Android.hardware.camera. The first permission declaration is the permission to use the camera software API, and the second is the permission to use the camera hardware. Get the video stream data, use SurfaceView to preview the camera image in real time, change the size and format of SurfaceView, and implement the SurfaceHolder.Callback interface during the development process. Before drawing the video frame returned by the camera, Surface must be created first. The content written to Surface is read directly from the video memory, so the speed is very fast.

[0133] 3. Drawing of pipeline model

[0134] After obtaining the camera position and posture information, the virtual pipeline model data is superimposed on the real scene to achieve an enhanced effect of the real scene. The enhanced information is the pipeline model, and the three-dimensional rendering is implemented based on OpenGL ES.

[0135] 4. 2D map auxiliary module

[0136] The system sets up the drawing and spatial query modules of pipeline thematic maps, allowing users to obtain pipeline attribute information and current geographical environment information. The pipeline two-dimensional map module is a GIS system that integrates navigation, positioning and query. The required basic map data are water systems, streets, residential areas, and pipeline distribution maps. The two-dimensional map auxiliary modules include: drawing of basic vector maps, drawing of basic image maps, drawing of pipeline thematic maps, and spatial analysis of pipeline networks.

[0137] (IV) Pipeline visualization module based on 3D visual feature tracking

[0138] 1. Feature point search and matching module

[0139] When the scene is relatively stable, the feature point search and matching method based on three-dimensional visual feature tracking is used to complete the estimation of the camera posture and relative position to the scene. In the image matching process, the video frame and the real scene are always undergoing scale transformation and rotation transformation when the smart terminal acquires the image. Therefore, based on the SIFT image feature point search and matching algorithm, the specific process of the feature point search and matching module is as follows: first, the current video frame is obtained through the Android Camera API, and then the feature points are searched through OpenCv for Android, and the feature point identifier is established. After the feature point identifier is obtained, the feature point is compared with the key frame in the database to match the closest key frame to the video frame. Finally, the position and posture information of the camera relative to the real environment is calculated by matching the key frames.

[0140] 2. System operation effect

[0141] The enhanced virtual-real fusion registration method based on 3D visual feature tracking has strict requirements on the environment and needs to select a relatively stable scene. The objects in the scene cannot change significantly, especially at intersections with large traffic. The lighting conditions cannot change significantly. In addition, in rainy and snowy weather, the road conditions will also change significantly. In addition, the scene needs to have relatively obvious features, and symmetrical patterns should be avoided as much as possible.

[0142] In order to verify the feasibility of this method, the applicant conducted tests on grass and a road paved with tiles. The following figure shows the system operation effect. Figure 5 , Figure 6 , Figure 7 .

[0143] (V) Pipeline visualization module based on mobile sensor network

[0144] It includes two parts: obtaining the mobile phone location information and obtaining the mobile phone posture information.

[0145] 1) Acquisition of mobile phone location information: There are three parts: longitude, latitude and altitude. In the Android system, the system's built-in components are used to complete the reading of location information. The specific process is as follows:

[0146] Configure the required location permissions in the Android Manifest file. There are two permissions that need to be configured, namely ACCESS_COARSE_LOCATION and ACCESS_FINE_LOCATION. Get the user's location through GPS or mobile base stations. Use the getBestProvider method provided by the LocationProvider class to enable the application to select the most accurate location that the current system can obtain. The LocationListener interface is used to monitor network changes, including changes in the status of GPS, base stations, and WIFI data source providers. The interface must be instantiated before obtaining location information. The location information obtained is a Location object, which records the longitude, latitude, and altitude of the device at that moment, as well as acquisition events and speed information.

[0147] 2) Get sensor data

[0148] First, configure the permission UPDATE_DEVICE_STATS in the Manifest file, and then obtain the SensorManager. The AndroidSensorManager is obtained after the system is initialized, and the real-time posture information of the terminal is obtained by monitoring the SensorManager. The posture information obtained is an encapsulated object.

[0149] Both 3D pipeline registration methods have their own advantages and disadvantages. Using a single registration method alone cannot complete pipeline registration in all scenarios. Therefore, in the actual application of the system, different methods should be used for different scenarios. If necessary, redundant design should be used for key scenarios to ensure the smooth operation of the system.

Claims

1. Enhanced virtual-real fusion urban underground pipeline three-dimensional visualization method, characterized in that: Establish an enhanced virtual-real fusion 3D tracking and registration solution based on 3D visual feature tracking and mobile device sensors, and build an Android-based underground pipeline visualization system to integrate virtual underground pipelines with the real environment and display them on the same screen; 1) Enhance the calculation of camera internal and external parameters in the virtual-reality fusion system: Establish the transformation relationship from the world coordinate system to the pixel coordinate system, determine the camera's position, posture and distortion error by calculating the camera's four internal parameters and six external parameters, and provide the correct transformation relationship for the perspective projection of the 3D pipeline model; 2) Enhance the 3D tracking registration of the virtual-reality fusion system: First, establish a tracking registration scheme without markers based on 3D visual features, improve the SIFT feature point detection algorithm to find natural feature points in the environment, and then determine the matching relationship between the natural feature points in the environment and the key frame feature points pre-stored by the system, and remove the mismatched points based on RANSAC, and finally solve the intrinsic and extrinsic parameters of the camera through the matching relationship between the feature points of the two images, so as to determine the position and posture of the camera relative to the scene; The second is to establish a three-dimensional tracking registration solution based on mobile sensor network tracking, and obtain the data provided by GPS and gyroscope sensors based on the Android system API to determine the position and posture of the camera; Third, adopt redundant design, integrate the tracking and registration scheme without markers based on 3D visual features and the 3D tracking and registration scheme based on mobile sensor network tracking, develop an Android-based underground pipeline enhanced virtual-real fusion system, and realize the 3D tracking and registration of pipeline models in complex environments; 3) Construct an underground pipeline visualization system based on augmented reality: Based on Android mobile terminals, an enhanced virtual-reality fusion method is adopted. The location and posture information of the device is determined by tracking feature points based on three-dimensional visual features and matching and using hardware sensors of mobile devices. The virtual pipeline is superimposed on the real scene. At the same time, the GPS hardware sensor obtains the user's location information, and then loads the user's location map and pipeline thematic map to prevent users from getting lost when using the system. The attribute data of the pipeline is managed based on SQLite, and the spatial analysis module based on GIS is used to realize the spatial analysis of the pipeline network. The three-dimensional visualization architecture of urban underground pipelines is divided into five parts: three-dimensional underground pipeline modeling, pipeline spatial database management, tracking and registration without markers based on three-dimensional visual features, three-dimensional tracking and registration based on mobile sensor networks, and computational analysis of pipeline data.

2. The method for enhancing virtual-real fusion 3D visualization of urban underground pipelines according to claim 1 is characterized in that: Unmarked object tracking and registration based on 3D visual features: Using natural features in the camera scene as feature points, the camera's posture and position relative to the scene are determined by detecting feature points, establishing identifiers, and matching feature points. In the unmarked point enhanced virtual-real fusion system, first capture the video frame shot by the camera in real time, search for natural feature points on the video frame, and establish identifiers for feature points. Then match the video frame with the pre-selected reference frame to determine the camera's posture information based on homography. Use the camera's posture information to register the virtual 3D model to the system scene to complete virtual-real fusion. The processing process includes: 1) Offline preprocessing stage: First, the virtual-real fusion 3D model to be enhanced in the camera scene is created on the computer and its coordinates are corrected. Then, multiple key frame images are taken in the scene, and the feature points on these key frame images are extracted and the feature point indicators are calculated. The data is organized into a database and indexed. The relative position relationship between the 3D pipeline model and the natural feature points on the key frame is pre-set for each frame of the image. 2) Online matching stage: extract the video frames obtained by the camera in real time, extract feature points and calculate the representation according to the method of the preprocessing stage, match the feature points by calculating the Hamming distance between the feature points and the representation of the feature points in the database, and after obtaining the matched feature points, calculate the position and posture of the camera relative to the scene, calculate the coordinates of the 3D model in the scene, and finally complete the fusion of virtual and real scenes.

3. The method for enhancing virtual-real fusion 3D visualization of urban underground pipelines according to claim 1 is characterized in that: 3D invariant feature tracking factor: 1) Search and track domain: Define a two-dimensional image as I(x, y), detect in multiple scale spaces, and obtain the representation of the image in different scale spaces by convolving the image with a Gaussian kernel. The coordinates of each pixel in the image are represented as (x, y), and the factors of different scale spaces are represented as σ. In the multi-scale space, the operator is represented by the difference of two Gaussian kernels of different scales. The Gaussian difference value of the pixel on the image in the multi-feature space is calculated, and then each point in the Gaussian difference scale space is compared with the adjacent position in the adjacent scale space to obtain the position of the tracking domain point and the corresponding scale space factor; 2) Screening the tracking domain: Compare the local tracking domain of the Gaussian difference detected in the discrete space, remove low-contrast feature points and unstable edge response points, and remove low-contrast key points and image noise in the image; 3) Calculate the direction of the key domain: Use the direction distribution of the gradient of the adjacent pixels in the key domain to calculate the direction parameter of the feature point, and then use the gradient histogram of the image to find the stable direction of the local structure of the key point, find the feature point, get the spatial scale σ of the feature point, and calculate the gradient modulus m and direction θ of the point (x, y); 4) Describe feature points: Use a set of vectors to describe them. First, correct the main rotation direction to ensure that the feature points have rotation invariance. Then generate a feature vector to describe the feature points and normalize the feature point representation.

4. The method for enhancing virtual-real fusion 3D visualization of urban underground pipelines according to claim 1 is characterized in that: Tracking point matching based on three-dimensional invariant features: After obtaining the representation of the feature point, the quantitative index of the feature point is calculated. The same feature point has similarities in different images. The similarity of the feature points is determined by calculating the Euclidean distance of the feature vectors. A critical value is set. When the Euclidean distance of two feature points is less than the critical value, the two points are a feature point. A preliminary geometric constraint model is established based on the RANSAC robustness algorithm, and the possible tracking points under the constraint are verified to eliminate the erroneous tracking points.

5. The method for enhancing virtual-real fusion 3D visualization of urban underground pipelines according to claim 1 is characterized in that: 3D tracking registration based on mobile sensor network: 1) Get device location information: In Android applications, use the location service API provided by the system to get the device's location. First, get the location service LocationManager and then specify the data source for obtaining location data as GPS. Finally, use the Location object to resolve to the current location. 2) Acquisition of device attitude information: The lateral roll angle and pitch angle are obtained through the accelerometer, g x , g y , g z The components representing the acceleration value of the current device in the three coordinate axis directions of the mobile phone coordinate system are used to calculate the tilt of the mobile phone. The electronic compass is used to obtain the orientation of the device, that is, the direction in which the top of the device points when the device is placed horizontally. The parameters describing the rotational motion of the device itself are obtained by detecting the angle between the device and the gyroscope. The gyroscope instantly completes the measurement of the rotation angle, and cooperates with the magnetic sensor to obtain the motion state of the device and the direction of the device at the same time. The mobile terminal posture data includes the device's azimuth angle Yaw, pitch, and roll. X-axis direction: from left to right along the horizontal direction of the screen. The device supports automatic screen rotation. If the phone screen is not square, the shorter side should be placed horizontally and the longer side should be placed vertically. The direction of the Y axis: from the lower left corner of the screen to the upper left corner of the screen; The direction of the Z axis: when placed horizontally, it points vertically to the sky; In the onSensorChanged callback function of the Android development program, three parameters about the posture are returned: the first return value is the azimuth angle Yaw, which represents the angle of rotation of the mobile phone around the Z axis. 0° indicates that the device is facing due north, that is, the direction of the y axis. Similarly, 90° represents due east, 180° represents due south, and 270° represents due west; the second return value is the pitch angle, which represents the degree of tilt of the mobile phone. When the mobile phone rotates around the x axis, this return value changes. The value range of the pitch angle is (-180°, +180°); the third return value is the roll angle Roll, which represents the rolling angle of the mobile phone along the y axis, and the value range is (-90°, +90°).

6. The method for enhancing virtual-real fusion 3D visualization of urban underground pipelines according to claim 1 is characterized in that: The 3D tracking registration process based on mobile terminal sensors: The location information and posture information of the device are obtained through the mobile terminal hardware sensor, the projection relationship between the 3D model and the real scene is determined, and the real 3D model is registered with the real scene. The process is as follows: 1) Build a scene space database: Get the CAD drawing of the three-dimensional pipeline distribution and then make a three-dimensional pipeline model with reference to the CAD drawing. The three-dimensional pipeline model is made in meters. The model coordinate system uses the world coordinate system centered on its own center point. Build the scene's spatial database and calibrate the three-dimensional pipeline model coordinates to the correct geographical location. In the virtual-real fusion, obtain the local scene data of a specific location based on the GPS data; 2) Obtain scene space data at a specific location: The GPS positioning module obtains the device location in real time, retrieves the 3D pipeline data of the current location from the database based on the GPS location information, and then obtains the posture information of the mobile phone based on the direction sensor and gravity sensor to determine the location of the 3D pipeline model in the real environment; 3) Register spatial data into the real environment: After determining the position of the 3D pipeline model in the real pipeline, use OpenGL ES to overlay the 3D model with the camera video stream to obtain a virtual-real fusion scene.

7. The method for enhancing virtual-real fusion 3D visualization of urban underground pipelines according to claim 1 is characterized in that: 3D visualization architecture of urban underground pipelines: 1) 3D underground pipeline modeling: The data sources of the system include two aspects: one is the video stream of the camera, and the other is the virtual pipeline model. In the Android system, the onPreviewFrame interface of setPreviewCallback is implemented to obtain each frame of video stream data in real time, thereby obtaining the video stream data of the real scene. The pipeline model needs to be modeled manually. For pipelines with simpler structures, a database is also used to store their basic information, and then it is drawn to the screen in real time through OpenGL ES during rendering. During the modeling process, the size of the model is ensured to be the same as the size of the real pipeline, and the geometric size of the virtual pipeline model is kept consistent with the real scene; 2) Pipeline spatial database management: After obtaining the pipeline model data, it is also necessary to calibrate it to the real geographic space coordinate system. During the calibration process, the latitude and longitude of the center of the model is used as the center and added to the spatial database. At the same time, tile data is generated for the pipeline network spatial data to facilitate smooth rendering of the three-dimensional pipeline data on the mobile terminal. The location information of the mobile device is obtained through the GPS positioning module, and then the corresponding pipeline scene data is loaded from the spatial database; 3) Markerless tracking and registration based on 3D visual features: extract feature points in the natural environment and realize the registration between feature points, and calculate the position and posture information of the camera relative to the scene; 4) Registration based on mobile hardware sensors: The registration method based on mobile hardware sensors and the registration method for tracking markers without markers based on three-dimensional visual features are integrated. The registration method based on mobile hardware sensors obtains sensor information and calculates the position and posture of the device through sensor data. 5) Calculation and analysis of pipeline data: The query of pipeline attribute information is based on the SQLite database on the mobile phone, and the spatial analysis calculation of pipelines is based on GIS spatial analysis.

8. The method for enhancing virtual-real fusion 3D visualization of urban underground pipelines according to claim 1 is characterized in that: Enhanced virtual-reality fusion visualization framework: 1) Data layer: including camera video stream data, 3D pipeline model data, pipeline attribute data, pipeline thematic map data, and basic vector map data. The camera video stream data is acquired in real time, and other data is stored in the device in the form of files or in the database in the form of two-dimensional tables; 2) Component layer: Data communication between the data layer and the application layer is all in this layer. The component layer accesses various data of the data layer, organizes and encapsulates them, and then hands them over to the application layer for use. OpenCV for Android completes the search and matching of feature points, solves the device posture information, ARCGIS API for Android draws basic vector maps and thematic maps of pipeline networks, Android Camera reads the camera's video stream, OpenGL ES draws a 3D pipeline model on the mobile phone screen, and Android LocationManager obtains the location of the mobile phone. The data obtained includes longitude, latitude and altitude; 3) Application layer: responsible for interaction with users, system data display and user data input, including the fusion of virtual and real scenes, the drawing of basic vector maps and pipeline network thematic maps, and spatial analysis and calculation of pipeline networks.

9. The method for enhancing virtual-real fusion 3D visualization of urban underground pipelines according to claim 1 is characterized in that: Virtual-real fusion underground pipeline visualization module: modular design is used to divide the system into several independent modules that do not affect each other; 1) Human-computer interaction module: a module for data interaction between the system and the user. The user enters data into the system, and the system feeds back the calculation results to the user, including pipeline enhanced virtual-reality fusion system interface design and human-computer interaction event processing. In this module, the system calls other modules for calculation according to the user's instructions on the screen, and then feeds back the data to the user; 2) Sensor data acquisition module: obtain the longitude, latitude and altitude information of the camera in real time so as to call the pipeline model data of the current position from the database, and obtain the attitude information including the inclination angle of the camera in various directions in real time; 3) Video stream data acquisition module: acquires the video frames of the camera in real time, provides the video frame data to the feature point search module in the feature point search and matching stage, and provides the video frame to the 3D pipeline model drawing module in the virtual-real fusion stage to complete the fusion display of the virtual 3D pipeline scene and the real environment; 4) Feature point search and matching module: It realizes the registration without logo according to the video frame, and solves the position of the camera relative to the scene by establishing the relationship between the feature points on the camera image and the feature points of the real scene; 5) Pipeline model drawing module: Use OpenGL ES to draw a 3D pipeline model and overlay the virtual 3D pipeline on the real scene to display enhanced information; 6) Two-dimensional map auxiliary module: Based on the GIS method, it provides the location information of the pipeline on the map when the user performs virtual-real fusion augmented reality pipeline data, displays the pipeline information from another angle, and prevents the user from getting lost.

10. The method for enhancing virtual-real fusion 3D visualization of urban underground pipelines according to claim 1 is characterized in that: Pipeline visualization module based on mobile sensor network: It includes two parts: acquisition of mobile phone location information and acquisition of mobile phone posture information; 1) Acquisition of mobile phone location information: There are three parts: longitude, latitude and altitude. In the Android system, the system's built-in components are used to complete the reading of location information. The specific process is as follows: Configure the required location permissions in the Android Manifest file. There are two permissions that need to be configured, namely ACCESS_COARSE_LOCATION and ACCESS_FINE_LOCATION. Get the user's location through GPS or mobile base stations. Use the getBestProvider method provided by the LocationProvider class to enable the application to select the most accurate location that the current system can obtain. The LocationListener interface is used to monitor network changes, including changes in the status of GPS, base stations, and WIFI data source providers. This interface must be instantiated before obtaining location information. The location information obtained is a Location object, which records the longitude, latitude, and altitude of the device at that moment, as well as acquisition events and speed information; 2) Get sensor data: First, configure the UPDATE_DEVICE_STATS permission in the Manifest file, and then get the SensorManager. The AndroidSensorManager is obtained after the system is initialized. By monitoring the SensorManager, the real-time posture information of the terminal is obtained. The posture information obtained is an encapsulated object.

Images