A semantic point cloud-based real scene three-dimensional visualization method and system

Abstract

The application belongs to the technical field of computer vision, and discloses a kind of based on semantic point cloud's real scene three-dimensional visualization method and system, including step 1: point cloud processing and publishing;Step 2: visualization based on mass point cloud;Step 3: the fusion display of large scene real scene three-dimensional data and scene roaming.The application adopts multi-level slice+WebGL display technology, realizes the integration of ground and underground, indoor and outdoor in cloud environment, with colored point cloud as the base, attaching image, tilt model, panorama, vector and other data, forms a platform of independent intellectual property rights, breaks through the limitation of commercial software, reduces the threshold for use of each department, and provides support for improving the holographic display effect of basic geographic information, which can be used in natural resource monitoring, geological disaster prevention and control, city CT, forestry investigation, underground space census, power line patrol and other scene applications.The display engine of the application has great advantages in the beauty and expansibility of point cloud display.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a method and system for real-scene 3D visualization based on semantic point clouds. Background Technology

[0002] Currently, with the development of computer technology and surveying science and technology, 2D GIS can no longer meet the development needs of various industries. In recent years, UAV oblique photogrammetry technology and point cloud technology have developed rapidly, and their application fields are becoming increasingly wide. They have become one of the main means of high-precision 3D modeling data acquisition; 3D modeling using point cloud data acquired by 3D scanning equipment can produce detailed 3D reality models, which are indispensable map data for the development of smart cities.

[0003] Research on 3D display systems began relatively early abroad, but 3D display on the Web originates from the WebGL standard. WebGL is a Web graphics API, a cross-platform, open, free, low-level JavaScript API based on OpenGL. It allows developers to accelerate the rendering of HTML5 canvas elements within the browser using the GPU. Since the World Wide Web Consortium (W3C) finalized the HTML5 standard in October 2014 and included WebGL, WebGL-based 3D display systems have rapidly developed in the research field. As an effective global information dissemination channel, it has become one of the most important forms of geographic information on the internet.

[0004] Currently, commonly used 3D visualization platforms abroad include Three.js, MapBox.gl, Cesium, Skyline, and ArcGIS. Three.js is an open-source WebGL framework that encapsulates commonly used objects in 3D graphics programming in an easy-to-use, simple, and intuitive way. However, Three.js is relatively weak in large-scale geographic information data visualization. Cesium is also an open-source 3D globe framework that supports loading massive amounts of data and has good data visualization capabilities, but it is relatively weak in 3D spatial analysis. Skyline and ArcGIS have strong spatial information visualization capabilities, supporting point clouds, tilt model loading, dynamic measurement, and ground measurement, but they are not open-source, making them difficult for developers to extend, and are commercial software with high prices. MapBox.gl is based on MapBox's open-source map engine and uses WebGL to render interactive vector tile maps, raster tile maps, and topographic maps, with functions such as 3D map loading, roaming, positioning, and measurement.

[0005] Of the platforms mentioned above, MapBox and Cesium are open-source WebGIS map engines, meaning they are free. Other platforms have limited application scenarios or require usage fees. However, Cesium's vector tiles are rendered as images, resulting in blurry styles and text, and its display speed for massive point cloud data is generally slow. MapBox lacks the ability to load point clouds, tilted 3D layers, etc.; it lacks a concept of the Earth; and its camera uses a third-person controller.

[0006] In summary, research on web-based 3D scene visualization has made some progress, but it still falls short in terms of display speed and aesthetics, especially for large-scale and complex 3D scenes. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for real-world 3D visualization based on semantic point clouds to solve the above-mentioned technical problems.

[0008] To address the aforementioned technical problems, the specific technical solution of the semantic point cloud-based real-scene 3D visualization method and system of the present invention is as follows:

[0009] A real-scene 3D visualization method based on semantic point clouds includes the following steps:

[0010] Step 1: Point Cloud Processing and Publishing: Based on data collection and analysis, a general workflow is established, including aerial triangulation, point cloud coloring, color transformation, coordinate transformation, and tiling. Airborne LiDAR point clouds undergo aerial triangulation, point cloud coloring, color transformation, coordinate transformation, and tiling. Ground / indoor point clouds + panoramic imagery, in addition to the processing of airborne LiDAR point clouds, also take into account the registration with panoramic imagery data. Underwater point cloud data does not require coloring or color transformation processing. InSAR point clouds are processed in a conventional manner while expanding the number of attributes to accommodate the serial numbers of long-term monitoring points and annual average settlement values.

[0011] Step 2: Visualization based on massive point clouds: Considering the different characteristics of raw 3D point cloud data acquired by airborne lasers, ground lasers, and underwater sonar, this study investigates the organization and management mode of data in 3D space based on point cloud models. By establishing quadtree and octree indexes and reasonable scheduling methods, rapid visualization of massive point clouds is achieved.

[0012] Step 3: Fusion display and scene roaming of large-scale real-scene 3D data: A hierarchical retrieval strategy is adopted from the provincial scene to the detailed scene to achieve a smooth user experience. Orthophotos, 3D terrain, point clouds and tilt models are automatically selected at different landscape heights and visual ranges, and indoor, underwater, underground and panoramic data are displayed together.

[0013] Further, step 1 includes the following steps:

[0014] Step 1.1: Collect and process data;

[0015] Step 1.2: Generate colorized point cloud data on a province-wide scale;

[0016] Step 1.3: Spatial matching processing of panoramic and point cloud data.

[0017] Further, step 1.1 includes the following steps:

[0018] The system collected and published a total of 2352.3 GB of data, equivalent to 2.3 TB, including provincial-level colored point cloud data, provincial-level remote sensing image slice data, provincial-level 3D topographic slice data (levels 9-14), provincial-level InSAR subsidence point cloud data, forestry colored point cloud data, power application colored point cloud data, underground facility colored point cloud data, underwater sonar point cloud data, tilt models, highway point cloud + panoramic view, indoor point cloud + panoramic view, historical building colored point cloud + panoramic view, and artificial modeling results. This demonstrates that the system can meet the requirements for publishing and displaying TB-level data.

[0019] Further, step 1.2 includes the following steps:

[0020] The province-wide DOM and DEM data are used to generate colored point cloud data covering the entire province. The generation of province-wide colored point cloud data is processed by a self-developed program. The self-developed program processing includes: generation and block processing of latitude and longitude projection point clouds.

[0021] The generation of the latitude and longitude projection point cloud: By studying the LAS format standard, a CGCS2000 latitude and longitude coordinate system that matches the display system is customized in the file header, and the X and Y coordinate precision is set to 7 decimal places for processing. In this way, the colorized point cloud of the whole province produced meets the publishing standard.

[0022] The block processing: when the generated color point cloud block is less than 100GB, normal slicing is performed, and the block is divided into 14 3DTiles with a scale of 1:50000 as the basis. Finally, the color point cloud of the whole province is divided into 14 3DTiles with a point spacing of 2 meters for storage.

[0023] Further, step 1.3 includes the following steps:

[0024] This solution fuses 3D point cloud data with 2D image data to achieve a complementary combination of precise spatial geometric information and realistic color and texture information. Computer technology is then used to render the colored point cloud, creating a novel 3D visualization solution. This solution includes: spatial positioning of the point cloud + panorama within the display system, as well as panoramic image slicing and rapid display. The spatial positioning of the point cloud + panorama within the display system is achieved by reading the interior and exterior orientation elements inherent in the panoramic acquisition data to reconstruct the position and orientation of the panoramic camera within the point cloud scene during capture. The solution also records the vertical scanning angle and panoramic image size for each camera, and compares it with commonly available point cloud + panorama integrated scanning devices. The data is collected and summarized to form a panoramic camera interior orientation element table, which is recorded in the panoramic camera interior orientation element parameter file of this system. Then, by reading the exterior orientation elements of the scanning station as the panoramic image exterior orientation element parameter file, the six exterior orientation elements of the panorama are located in the three-dimensional scene. A block attribute column is added, which can distinguish between floors and areas. The panoramic image tiling and quick display adopt the method of spherical projection + UV mapping of panoramic photos to form six cube tiles. A panoramic tiling tool is developed based on OpenCV, and the sphere2cube function of the tool is called to realize spherical panoramic tiling and generate thumbnails.

[0025] Further, step 2 includes the following steps:

[0026] Step 2.1: Selecting the indexing method;

[0027] Point cloud data is mainly divided into two types: one is a large-area point cloud of the whole province or a local area that is directly overlaid with other data on the display interface; the other is a small-area point cloud that is combined with small scene data such as panoramas. Since it does not involve point cloud editing and query functions, the optimal method is to choose a quadtree for displaying large-scene point clouds and an octree for displaying small-scene point clouds.

[0028] Step 2.2: Parse and load the quadtree point cloud using mapbox.gl;

[0029] By using an external program and rendering with three.js, the quadtree point cloud is loaded. The data adopts a geocentric coordinate system. The logic is as follows: parse the PNTS format point cloud file, convert it into Points in Three.js, and realize the geographic coordinate positioning based on the Bounding and Offset properties.

[0030] Step 2.3: Memory scheduling for point cloud rendering;

[0031] Resource scheduling optimization control: By optimizing the line-of-sight distance, the display scale of point cloud data is controlled. The logic is as follows: during roaming, the tiles to be loaded are placed in the requestTiles queue, and the tiles to be unloaded are placed in the unloadTiles queue. Memory scheduling is performed based on the line of sight, distance, and cache size. The loading control logic is based on the line of sight and distance.

[0032] Step 2.4: mapbox.gl loads and displays the octree point cloud;

[0033] The engine can load point cloud data in a relative coordinate system and perform coordinate rotation, scaling, and translation through algorithms to match it to the real coordinate system. The logic is as follows: the octree point cloud is parsed using Potree.loadPointCloud and loaded into the 3D scene.

[0034] Furthermore, step 3 includes the following steps:

[0035] Step 3.1: A strategy for seamlessly integrating and displaying multi-source data;

[0036] A hierarchical retrieval strategy is adopted from the overall provincial scene to detailed scenes, achieving seamless switching from orthophotos to 3D terrain, oblique models / colored point clouds / white films, and indoor / underground / underwater / panoramic views. The integrated display of provincial, municipal, county, and township levels is controlled by the map layer rendering mechanism in mapbox.gl and the minzoom and maxzoom parameters. The layer rendering control of street and community level and building level detailed scenes is controlled by the renderOrder property of the drawing object in WebGL, which controls the display priority. The switching between large scenes and building level western scenes is implemented by placing icons in the large scene, and clicking the icon will enter the indoor or underground detailed scene.

[0037] Step 3.2: A method for measurement based on point clouds in panoramic space;

[0038] The mapping of panoramic and point cloud data from the free coordinate system to the real coordinate system. In street view and indoor scenes, if panoramic data is available, the system will prioritize and automatically display the 3D scene in panoramic form. The mapping from panoramic space to real geographic space will be based on the spatial registration and mapping relationship of the point cloud-panorama preprocessing stage, transforming the position of the panoramic space into the real geographic space to obtain its real geographic location.

[0039] The imaging model of a spherical panoramic image is centered on the optical center of the panoramic camera. A photographic beam is formed by connecting the object point to the center of the sphere. The projection point of this photographic beam onto the sphere is the image of the object point. Assuming that the coordinates of object point A in the coordinate system of the point cloud model are [Xp Yp Zp], and the corresponding 3D point coordinates of the panoramic sphere are [Xs Ys Zs], the expression for converting the object coordinates of point A to spherical coordinates is as follows:

[0040]

[0041] The rotation matrix R in the formula is obtained by projecting the camera's viewpoint onto the panoramic viewpoint in the 3D scene. The implementation logic is as follows:

[0042] Project the clicked point on the plane in the current panoramic scene onto the world coordinate system;

[0043] Calculate the ray from the panoramic shooting point to a point in the world coordinate system;

[0044] The intersection of the ray and the point cloud model is the actual coordinate of the clicked point in the panoramic scene.

[0045] The present invention also discloses a real-scene 3D visualization system based on semantic point cloud. The system page function modules include scene display, layer management, scene topics, search bar, toolbox and display mode switching.

[0046] The scene display is used to showcase visualized scenes;

[0047] The layer management is categorized by data type into point cloud, image, map, and 3D as the primary classification, and the secondary classification focuses on the specific acquisition method.

[0048] The scenario topics are quickly located and displayed according to the application direction of the system and specific demonstration cases;

[0049] The search bar allows users to search and locate places by name.

[0050] The toolbox includes basic functions such as distance and area measurement, and acquisition and export of point, line and surface vector elements. When there is point cloud data, it automatically selects to capture point cloud data and retains the original point cloud measurement accuracy.

[0051] The display mode switching allows switching between image, map, terrain, and blended browsing modes.

[0052] Furthermore, the system's functional modules include a scene point cloud display module and a spatial analysis module;

[0053] The scene point cloud display module includes point cloud display settings: the point cloud rendering method is set by setting the point cloud quality, viewing angle range, number of points, point size, and rendering method;

[0054] The spatial analysis module includes point cloud segmentation, point cloud plotting, 3D distance measurement, 3D area measurement, importing and exporting results, clearing measurements, and flight path.

[0055] The point cloud slicing includes horizontal slicing and vertical slicing. The former slices horizontally according to height and thickness, while the latter slices vertically according to the cutting lines and thickness drawn by the user. The slicing results are highlighted.

[0056] The point cloud display: The selected point cloud after being sectioned is displayed as an image with a transparent background in .png format;

[0057] The 3D distance measurement is based on point clouds. In a small scene's free coordinate system, the true coordinates of the point cloud are directly obtained based on its own XYZ coordinates. Within the small scene, the true coordinates of the point cloud are captured, and the measurement and export are visualized. For large scenes covering a provincial area, coordinate transformation is used to project the small scene's own coordinates onto the true coordinates.

[0058] The 3D distance measurement tool calculates the 3D distance between two points on an image. The calculation formula is as follows:

[0059]

[0060]

[0061] Where p is the Euclidean distance between point (x2,y2,z2) and point (x1,y1,z1); X is the Euclidean distance from point (x2,y2,z2) to the origin;

[0062] The three-dimensional area measurement: The three-dimensional area measurement calculates the total area of ​​the spatial surface formed by each node of the surface element. It is independent of the terrain but related to the XYZ values ​​of each point in the point cloud. The surface is drawn by continuously clicking on the surface point cloud or the indoor point cloud to measure the three-dimensional area of ​​the space. This area is the surface area.

[0063] The import and export of results: The export results tool exports the measurement results as files to formats such as geojson, kml, and dxf. These formats can be imported and displayed again in this system, or they can be opened and displayed on third-party tool websites such as AutoCAD, MapShaper, and Geojson.io.

[0064] The measurement clearing tool removes all measurement information drawn on the image.

[0065] The flight path is defined as follows: add a flight path, add a viewpoint, set the playback time of the path, and then fly and roam along the path.

[0066] The real-scene 3D visualization method and system based on semantic point clouds of the present invention have the following advantages:

[0067] 1. This invention addresses the heterogeneous characteristics of point cloud data, including airborne / vehicle-mounted lidar point clouds, ground / indoor point clouds, underwater sonar point clouds, and InSAR settling data. It develops a semantically comprehensive point cloud data processing and publishing tool that integrates modular processing such as aerial triangulation, point cloud coloring, color transformation, coordinate transformation, and slicing. Simultaneously, it performs coordinate space and panoramic space matching and linkage construction on point cloud and panoramic image data, achieving pixel-by-pixel mapping of images to coordinate space voxels. Furthermore, it overcomes the attribute size limitations of 3dtiles format point cloud data storage by utilizing machine numbers, solving the problem of exponentially increasing volume caused by attaching massive attributes to point cloud data.

[0068] 2. By using self-developed external code to parse 3dtiles / pnts formats, the lack of point cloud layer loading capability in MapBox.gl was resolved. A lightweight hybrid organization and publishing technology for TB-level point cloud models was developed. For large-scale scene (province, city, county) point clouds, geocentric coordinates and quadtree rendering are adopted. The display scale of point cloud data is controlled by the view distance, which solves the problems of slow rendering and crashes at the user's eye level. For small-scale scene (village level, block, indoor, etc.) point clouds, the limitation of Potree engine can only use relative coordinates is broken. Octree rendering is adopted, and coordinate rotation, scaling and translation are automatically performed to match the real coordinates in real time, ensuring the display effect of high-density point cloud scenes such as indoor / underground.

[0069] 3. Considering user experience and hardware limitations, a fusion display mode from province-wide scenes to detailed scenes was developed. A hierarchical retrieval strategy was adopted, using view distance control and field of view constraints to achieve seamless switching from orthophotos to 3D terrain, tilted models / colored point clouds / white films, and indoor / underground / underwater / panoramic views. While ensuring a smooth user experience, memory allocation was optimized, resulting in rendering efficiency 2-3 times higher than other open-source platforms. This addressed the shortcomings in displaying 3D scenes in non-tilted 3D data coverage areas within a large provincial scope. Furthermore, a technology for acquiring 3D model space and panoramic spatial elements based on background point clouds was researched. By automatically capturing background point clouds for measurement, the accuracy loss after 3D model construction and the inability to measure panoramic images were avoided, thus improving measurement accuracy. Attached Figure Description

[0070] Figure 1 This is a flowchart of the point cloud processing of the present invention;

[0071] Figure 2 This is a flowchart of the massive color point cloud generation process of the present invention;

[0072] Figure 3This is a flowchart of the point cloud and panoramic fusion 3D visualization process of the present invention;

[0073] Figure 4 The test results are shown in the image below, which is a display of the onboard point cloud using the Cesium.js engine included in Cesiumlab V3.0.

[0074] Figure 5 This is a test result diagram of the airborne point cloud of the engine of this invention;

[0075] Figure 6 This is a test image showing the point cloud display effect of the engine in a small scene of this invention. Detailed Implementation

[0076] To better understand the purpose, structure, and function of this invention, the following detailed description of a real-scene 3D visualization method based on semantic point clouds is provided in conjunction with the accompanying drawings.

[0077] This invention employs multi-level slicing + WebGL display technology to achieve integrated display of above-ground and underground, indoor and outdoor data in a cloud environment, based on colored point clouds and attached with images, tilted models, panoramas, vectors, and other data. It forms an independent intellectual property platform, providing support for overcoming the limitations of commercial software, lowering the usage threshold for various departments, and improving the holographic display effect of basic geographic information. It can be used in scenarios such as natural resource monitoring, geological disaster prevention and control, urban CT, forestry surveys, underground space surveys, and power line inspection.

[0078] The present invention provides a real-scene 3D visualization method based on semantic point clouds, comprising the following steps:

[0079] Step 1: Point Cloud Processing and Publishing: Based on data collection and analysis, a general technical workflow is developed, including aerial triangulation, point cloud coloring, color transformation, coordinate transformation, and slicing. Detailed processing methods are then developed for airborne point clouds, panoramic point clouds, and InSAR point clouds, along with corresponding processing tools. For example... Figure 1 As shown, large-scale point cloud data are categorized by acquisition method into airborne LiDAR point clouds, ground / indoor point clouds + panoramic imagery, underwater point clouds, and InSAR point clouds. Airborne LiDAR point clouds undergo aerial triangulation, point cloud coloring, color transformation, coordinate transformation, and slicing processes. Ground / indoor point clouds + panoramic imagery, in addition to the above processing, also require registration with panoramic imagery data. Underwater point cloud data does not require coloring or color transformation. InSAR point clouds, while undergoing conventional processing, require expansion of the number of attributes to accommodate the serial numbers of long-term monitoring points and annual average settlement values.

[0080] Step 1.1: Collect and process data;

[0081] This invention collects, processes, and ultimately publishes the following data in the system:

[0082] Table 1. Data Collection and Processing for the Project

[0083]

[0084] The project collected and published a total of 2352.3GB of data, which is approximately 2.3TB, indicating that the system can meet the requirements for publishing and displaying TB-level data.

[0085] Step 1.2: Generate colorized point cloud data on a province-wide scale;

[0086] To stress test the system under massive data conditions, this invention specifically generated province-wide shaded point cloud data using provincial DOM and DEM data. The generation of this province-wide shaded point cloud data was processed using a self-developed program, which is also applicable to batch production of shaded point clouds from other DOM and DEM data. The batch processing flow is as follows: Figure 2 As shown.

[0087] The self-developed program processing includes: generation and block processing of latitude and longitude projection point clouds.

[0088] Generation of Latitude and Longitude Projected Point Clouds: Since point cloud data is typically used and processed in planar projection coordinates, latitude and longitude projection coordinates are not available. Furthermore, point cloud coloring can only be performed in planar projection coordinates. However, to publish the data in the display system of this invention and avoid data discontinuity caused by upgrades, latitude and longitude projected point clouds are required. This invention, through research on the LAS format standard, defines a custom CGCS2000 latitude and longitude coordinate system in the file header that matches the display system, and sets the X and Y coordinate precision to 7 decimal places. This results in a province-wide colored point cloud that meets the publishing standards.

[0089] Chunk Processing: Due to the sheer volume of data—800GB for the province's DEM and 2TB for the DOM—processing in bulk was impossible. Furthermore, to maintain smooth display, excessive data cropping was contraindicated. This invention conducted multiple tests on the maximum capacity of the CesiumLab slicing tool, ultimately finding that normal slicing was possible when a single block of the generated colored point cloud was less than 100GB. Based on calculations using a 1:50000 scale as a foundation, the province's colored point cloud was ultimately divided into 14 3DTiles with a 2-meter dot spacing, ensuring a good browsing experience.

[0090] Step 1.3: Spatial matching processing of panoramic and point cloud data;

[0091] The 3D visualization method based on panoramic images involves taking a series of photos in a 360-degree circle with a camera, and then synthesizing a 3D panoramic image of the target scene using 2D image stitching technology. It is a virtual reality technology based on static images. Table 2 summarizes the characteristics and advantages and disadvantages of point cloud and panoramic visualization technologies:

[0092] Table 2 Comparison of Point Cloud and Panoramic Visualization

[0093]

[0094] Considering that 3D point clouds acquired by 3D laser scanners can provide accurate 3D spatial geometric information, and 2D images acquired by CCD cameras can provide rich color and texture information, such as... Figure 3 As shown, by fusing 3D point cloud with 2D image data, accurate spatial geometric information and realistic color and texture information can be complemented. Then, by rendering the colored point cloud with computer technology, a brand-new 3D visualization solution can be formed.

[0095] Currently mature ground-based laser scanning equipment can directly and simultaneously acquire panoramic images and point cloud data. Since these data have already undergone rigorous calibration, the built-in processing software can complete the registration of the point cloud with the image and perform point cloud coloring. This invention, considering the perspective of data dissemination, primarily addresses the following issues:

[0096] 1) Spatial positioning of point cloud + panorama in the display system

[0097] This invention reconstructs the position and orientation of the panoramic camera within the point cloud scene during panoramic image capture by reading the interior and exterior orientation elements inherent in the panoramic acquisition process. Panoramic images differ from ordinary frame-type images; they possess an ultra-wide-angle field of view, typically 360° horizontally and nearly 180° vertically. Therefore, focal length is not a factor when recording interior orientation elements. Instead, the vertical scanning angle and panoramic image size for each camera need to be recorded. We have compiled and summarized the parameters of commonly used point cloud + panoramic integrated scanning devices to create a panoramic camera interior orientation element table, which is recorded in the panoramic camera interior orientation element parameter file (JSON format) of this system for future expansion and use.

[0098] Subsequently, since the vehicle-mounted and ground-based laser scanning systems generate exterior orientation elements for each scanning station during the stitching stage (equipment such as Rigel and Faro supports exporting to Excel format), this invention reads the exterior orientation elements of the scanning stations as the panoramic image exterior orientation element parameter file, and positions the six exterior orientation elements (three positions and three rotation angles) of the panorama into the three-dimensional scene.

[0099] In addition, to meet the needs of roaming and POI search of panoramic scenes by floor and region during later 3D display, a block (partition) attribute column has been added, which can distinguish between floors and regions.

[0100] Table 3. Example of import format for exterior orientation elements from panoramic cameras.

[0101]

[0102]

[0103] 2) Panoramic image slicing and rapid display

[0104] Panoramic photos, typically exceeding 20 million pixels and with individual files larger than 5MB, load slowly on the web. They must be loaded into 3D scenes using a combination of tiered tiling and caching.

[0105] This invention uses spherical projection and UV mapping of panoramic photos to form six cubic slices. Because loading textured photos in WebGL requires a photo size of 2 to the power of N, this invention develops a panoramic tiling tool based on OpenCV. By calling the tool's sphere2cube function, spherical panoramic tiling and thumbnail generation are achieved. Example parameters are as follows:

[0106] sphere2cube.exe -i"sphere jpg file" -o"out dir" -l 0 -itemSize 2048 -previewSize 512.

[0107] Step 2: Visualization based on massive point clouds: Considering the different characteristics of raw 3D point cloud data collected by airborne lasers, ground lasers, and underwater sonar, this study investigates the organization and management mode of data in 3D space based on point cloud models. By establishing quadtree and octree indexes and reasonable scheduling methods, rapid visualization of massive point clouds can be achieved to make up for the lack of 3D scene display in non-tilted 3D data coverage areas within a large provincial scene.

[0108] Step 2.1: Selecting the indexing method;

[0109] Like other spatial data in GIS, point cloud data presents its own organizational challenges. Extensive research has been conducted on spatial indexing in GIS, and various indexing methods from Geographic Information Systems (GIS) can be applied to organize 3D point cloud data. Common spatial indexes employ a top-down, hierarchical partitioning approach, with representative examples including regular grids, quadtrees, octrees, KD trees, KDB trees, BSP trees, R trees, and R+ trees. Different spatial indexing methods have varying applications, advantages, and disadvantages. Current GIS software often utilizes different indexing mechanisms, employing a strategy that leverages the strengths of each. Since each index has its own advantages and disadvantages, and a single indexing method cannot meet the needs of point cloud organization and spatial indexing, it is necessary to explore an indexing method suitable for managing massive point clouds, specifically tailored to the characteristics of point clouds themselves.

[0110] The point cloud data involved in this invention is mainly divided into two types: one is a large-area point cloud of the entire province or a local area that is directly overlaid with other data on the display interface; the other is a small-area point cloud that is combined with small scene data such as panoramas for display. Since this invention does not involve point cloud editing and querying functions, the optimal method is to choose a quadtree for displaying large-scene point clouds and an octree for displaying small-scene point clouds.

[0111] Step 2.2: Parse and load the quadtree point cloud using mapbox.gl;

[0112] MapBox.gl itself does not support loading point cloud layers. This invention uses an external program, three.js, to render quadtree point clouds, and the data uses a geocentric coordinate system.

[0113] Implementation logic: Parse the PNTS format point cloud file and convert it into Points in Three.js.

[0114] Geographic coordinate positioning is achieved based on the Bounding and Offset properties.

[0115] Step 2.3: Memory scheduling for point cloud rendering;

[0116] To address the weaknesses of slow rendering and susceptibility to crashes in large-scale quadtree point cloud data, the system has implemented optimized resource scheduling. By optimizing the viewing distance, the display scale of the point cloud data is controlled to avoid excessive loading of point cloud data when the user is at eye level, which would otherwise consume a large amount of bandwidth and memory.

[0117] The implementation logic is as follows: during roaming, the tiles to be loaded are placed in the requestTiles queue, and the tiles to be unloaded are placed in the unloadTiles queue. Memory scheduling is performed based on the field of view, distance, and cache size.

[0118] Control logic that controls loading based on field of view and distance.

[0119] Step 2.4: mapbox.gl loads and displays the octree point cloud;

[0120] The loading of octree point clouds is also achieved by extending the layer method of MapBox.gl. Its loading and rendering scheduling are achieved by integrating the Potree rendering engine. This engine can load point cloud data in a relative coordinate system. The system uses algorithms to rotate, scale and translate the coordinates to match the real coordinate system. This method is suitable for high-density point cloud display in indoor / underground scenes.

[0121] Implementation logic:

[0122] The octree point cloud is parsed using Potree.loadPointCloud and loaded into the 3D scene.

[0123] Step 3: Fusion and display of large-scale real-scene 3D data and scene roaming. To ensure a smooth user experience and optimized memory usage, a hierarchical retrieval strategy is adopted from the provincial scene to detailed scenes to achieve a smooth user experience. Orthophotos, 3D terrain, point clouds, and tilted models are automatically selected at different landscape heights and visual ranges, and displayed in conjunction with indoor, underwater, underground, and panoramic data.

[0124] Step 3.1: A strategy for seamlessly integrating and displaying multi-source data;

[0125] In terms of integrated display, in order to ensure a smooth user experience and optimized memory usage, a hierarchical retrieval strategy is adopted from the provincial scene to the detailed scene to achieve seamless switching from orthophotos to 3D terrain, tilted model / colored point cloud / white film, and indoor / underground / underwater / panoramic views. The specific strategy is shown in Table 4.

[0126] Table 4 Integration Display Strategy

[0127]

[0128] The integrated display of provincial, municipal, county, and township levels is controlled by the map layer drawing mechanism in mapbox.gl and the minzoom and maxzoom parameters; the layer rendering control of street community and building detail scenes is controlled by the renderOrder property of the drawing object in WebGL, which controls the display priority.

[0129] In addition, the switching between large scenes and architectural-level western scenes is achieved by placing icons in the large scenes, and clicking the icons to enter the indoor or underground detailed scenes.

[0130] Step 3.2: A method for measurement based on point clouds in panoramic space;

[0131] The mapping of panoramic and point cloud data from a free coordinate system to a real coordinate system. In street view and indoor scenes, if panoramic data is available, the system will prioritize automatically displaying the 3D scene in panoramic form. The mapping from panoramic space to real geographic space will be based on the spatial registration and mapping relationship in the point cloud-panorama preprocessing stage, transforming the position of the panoramic space into the real geographic space to obtain its true geographic location.

[0132] Principle: The imaging model of a spherical panoramic image has the optical center of the panoramic camera at its center. A photographic beam is formed by connecting the object point to the center of the sphere. The projection point of this photographic beam onto the sphere is the image of that object point. Assuming the coordinates of object point A in the point cloud model's coordinate system are [Xp Yp Zp], and the corresponding 3D point coordinates of the panoramic sphere are [Xs Ys Zs], the expression for converting the object coordinates of point A to spherical coordinates is as follows:

[0133]

[0134] The rotation matrix R in the formula can be obtained from the camera's projection matrix when the viewpoint is at the panoramic viewpoint in the 3D scene. Therefore, the main implementation logic is as follows:

[0135] / / Project the current planar click point in the panoramic scene onto the world coordinate system

[0136] var worldpt=screenpt.unproject(this.camera);

[0137] / / Calculate the ray from the panorama shooting point to the world coordinate system point.

[0138] this.rayDir.set(mS,mR,this.camera.near).sub(this.camera.position);

[0139] raycaster.ray.set(this.camera.position,this.rayDir.normalize());

[0140] / / The intersection of the ray and the point cloud model is the true coordinate of the clicked point in the panoramic scene.

[0141] var intersects=pointstileset.Octree.intersectLODTree(raycaster).

[0142] The page function modules of the real-scene 3D visualization system based on semantic point cloud of the present invention include scene display, layer management, scene topics, search bar, toolbox and display mode switching.

[0143] Scene display: Used to showcase visualized scenes;

[0144] Layer management: The layers are classified into primary categories based on data type, such as point cloud, image, map, and 3D. Secondary categories focus on the specific acquisition methods. For example, point cloud layers include airborne point clouds, vehicle-mounted point clouds, and InSAR point clouds.

[0145] Scene-based displays: Based on the system's application direction, quick location and display are provided according to specific demonstration cases, such as urban, indoor 3D scenes, underwater 3D scenes, power 3D scenes, etc.

[0146] Search bar: Search and locate locations by place name;

[0147] Toolbox: Basic functions include distance and area measurement, and acquisition and export of point, line and area vector elements. When there is point cloud data, it automatically selects to capture point cloud data and retains the original point cloud measurement accuracy.

[0148] Display mode switching: Switch between image, map, terrain, and blended browsing modes.

[0149] The system's functional modules include a scene point cloud display module and a spatial analysis module.

[0150] Scene point cloud display module: Point cloud display settings: The rendering method of the point cloud can be set by adjusting the point cloud quality, viewing angle, number of points, point size, and rendering method.

[0151] The spatial analysis module includes point cloud sectioning, point cloud plotting, 3D distance measurement, 3D area measurement, importing and exporting results, clearing measurements, and flight path.

[0152] Point cloud slicing includes horizontal slicing and vertical slicing. The former slices horizontally according to height and thickness, while the latter slices vertically according to the cutting lines and thickness drawn by the user. The slicing results are highlighted.

[0153] Point Cloud Display: Displays the selected point cloud after sectioning as a transparent image in .png format. This function can be used to generate plan, elevation, and section views from point clouds of historical buildings.

[0154] 3D distance measurement: Based on point clouds, in a small scene's free coordinate system, the true coordinates of the point cloud can be directly obtained based on its own XYZ coordinates. In small scenes, by capturing the true coordinates of the point cloud, measurements can be visualized and exported. However, large scenes at the provincial level use CGCS2000 latitude and longitude coordinates, therefore, coordinate transformation is required to project the small scene's own coordinates onto the true coordinates.

[0155] The 3D distance measurement tool can calculate the 3D distance between two points on an image. The calculation formula is as follows:

[0156]

[0157]

[0158] Where p is the Euclidean distance between point (x2,y2,z2) and point (x1,y1,z1); X is the Euclidean distance from point (x2,y2,z2) to the origin.

[0159] 3D area measurement: 3D area measurement calculates the total area of ​​the spatial surface formed by each node of the surface element. It is independent of the terrain but related to the XYZ values ​​of each point in the point cloud. By continuously clicking and drawing surfaces on the ground point cloud or indoor point cloud, the 3D area of ​​the space is measured. This area is the surface area.

[0160] Import and export results: The results export tool can export measurement results as files to formats such as geojson, kml, and dxf. These formats can be imported and displayed again in this system, or opened and displayed on third-party tool websites such as AutoCAD, MapShaper, and Geojson.io.

[0161] Clear Measurements: The Clear Measurements tool can completely erase all measurement information drawn on an image.

[0162] Flight path: Add flight path, add viewpoint, set the playback time of the path, and you can fly and roam along the path after setting.

[0163] Display effect comparison

[0164] The point cloud visualization engine of this invention is compared with the display effect based on the conventional Cesium.js, as detailed in Table 5.

[0165] Table 5 Comparison of point cloud display between Cesium.js and this system

[0166]

[0167] like Figure 4As shown, this is a test result of using the Cesium.js engine included in Cesiumlab V3.0 to display the airborne point cloud. Figure 5 The image shows the test results of the airborne point cloud display of the engine of this invention, as follows: Figure 6 The image shows the test results of point cloud display in a small scene using the engine of this invention. The comparison reveals that the Cesium.js engine itself is smaller, while the display engine of this invention has significant advantages in terms of the aesthetics and scalability of point cloud display.

[0168] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A method for live three-dimensional visualization based on semantic point clouds, characterized in that, Comprising the following steps: Step 1: Point cloud processing and publishing: on the basis of data collection and analysis, forming the general flow of point cloud coloring, color transformation, coordinate transformation, slicing processing, airborne LiDAR point cloud through point cloud coloring, color transformation, coordinate transformation, slicing processing flow; ground / indoor point cloud + panoramic image in addition to the processing process of airborne LiDAR point cloud, also take into account the registration with panoramic image data; underwater point cloud data does not need to be colored and color transformed; InSAR point cloud is expanded in attribute number while being processed conventionally to accommodate the serial number and annual average subsidence value of long-time series monitoring points; Step 1.1: Collect and process data; Step 1.2: Generate provincial-level colored point cloud data; Provincial DOM data and DEM data are used to produce colored point cloud data covering the whole province, and provincial-level colored point cloud data is generated by using self-developed programs for processing, and the processing process of the self-developed programs includes: generation and blocking processing of latitude and longitude projection point cloud; The generation of latitude and longitude projection point cloud: through the study of LAS format standard, the CGCS2000 latitude and longitude coordinate system matching the display system is defined and displayed in the file header, and the X and Y coordinate precision is set to 7 decimal places for processing, so that the produced provincial colored point cloud meets the publishing standard; The blocking processing: when the single blocking of color point cloud is less than 100GB, normal slicing is performed, and the final provincial colored point cloud is divided into 14 3DTiles with a point spacing of 2 meters for storage based on the 1:50,000 scale framing; Step 1.3: Spatial matching processing of panoramic and point cloud data; Step 2: Visualization based on massive point cloud: according to the different characteristics of the original three-dimensional point cloud data collected by airborne laser, ground laser and underwater sonar, the organization and management mode of data in three-dimensional space based on point cloud model is studied, and through the establishment of quadtree and octree index and reasonable scheduling method, the rapid visualization of massive point cloud is realized; Step 3: Fusion display and scene roaming of large scene real scene three-dimensional data: from provincial scene to detail scene, the hierarchical retrieval strategy is adopted to realize smooth user experience, and orthographic image, terrain three-dimensional, point cloud and tilt model are automatically selected in different landscape height and visual range, and indoor, underwater, underground, panoramic data display is associated.

2. The method for live three-dimensional visualization based on semantic point clouds according to claim 1, characterized in that, The step 1.1 comprises the following steps: Collecting provincial-level colored point cloud data, provincial remote sensing image slice data, provincial terrain three-dimensional slice data 9-14, provincial InSAR subsidence point cloud, forestry colored point cloud data, power application colored point cloud, underground facility colored point cloud, underwater sonar point cloud, tilt model, highway point cloud + panorama, indoor point cloud + panorama, historical building colored point cloud + panorama, artificial modeling results, a total of 2352.3GB of data is collected and published, which is equivalent to 2.3TB, indicating that the system can meet the release and display of TB level data.

3. The method of claim 1, wherein, The step 1.3 comprises the following steps: The three-dimensional point cloud is fused with the two-dimensional image data to realize the complementation of accurate spatial geometric information and real color texture information, and a new three-dimensional visualization solution is formed by rendering the color point cloud through computer technology, and the solution comprises: point cloud+panorama spatial positioning in a display system and panorama image slicing and rapid display, The point cloud+panorama spatial positioning in the display system reconstructs the position and posture of the panorama camera in the point cloud scene during shooting by reading the internal and external orientation elements of the panorama camera during shooting; the vertical scanning angle of each camera and the panorama photo size are recorded, the parameters of the common point cloud+panorama integrated scanning equipment on the market are collected and sorted, the internal orientation element table of the panorama camera is formed, and the internal orientation element parameters of the panorama camera are recorded in the internal orientation element parameter file of the system; then, the external orientation elements of the scanning station are read as the external orientation element parameter file of the panorama image, and the six external orientation elements of the panorama are positioned in the three-dimensional scene; a block attribute column is added, which can distinguish floors and areas; The panorama image slicing and rapid display adopts the spherical projection+uv mapping of the panorama photo to form six cubic slices; the panorama slicing tool is developed based on OpenCV, and the sphere2cube function of the tool is called to realize the spherical panorama slicing and generate a thumbnail.

4. The method of claim 1, wherein, The step 2 comprises the following steps: Step 2.1: selection of an index mode; Point cloud data is mainly divided into two types, one is large-area point cloud of a whole province or a local area which is directly superimposed and displayed on a display interface, and the other is small-area point cloud which is combined with small scene data for display; since no editing and query functions of point cloud are involved, the optimal method is to select a quadtree for display of large scene point cloud and an octree for display of small scene point cloud; Step 2.2: loading of quadtree point cloud by mapbox.gl; The loading of quadtree point cloud is realized by using an external program and rendering by three.js; the data adopts a geocentric coordinate system, and the implementation logic is as follows: the PNTS format point cloud file is parsed and converted into Points in Three.js; the geographic coordinate positioning is realized according to the Bounding and Offset attributes; Step 2.3: memory scheduling of point cloud rendering; Optimized control of resource scheduling: the display scale of point cloud data is controlled in a manner of optimized sight distance, and the implementation logic is as follows: the tiles to be loaded during roaming are placed in the requestTiles queue, and the tiles to be unloaded are placed in the unloadTiles, the memory scheduling is performed according to the field of view, distance and cache size, and the control logic of the loading is controlled according to the field of view and distance; Step 2.4: display of octree point cloud loaded by mapbox.gl; The engine data can load point cloud data of a relative coordinate system, and the coordinate rotation, scaling and translation are performed through an algorithm to make it match to a real coordinate system, and the implementation logic is as follows: the octree point cloud is parsed through Potree.loadPointCloud and loaded into a three-dimensional scene.

5. The method of claim 1, wherein, The step 3 comprises the following steps: Step 3.1: Fusion display strategy for seamless connection of multi-source data; The hierarchical retrieval strategy is adopted from the provincial scene to the detailed scene to realize seamless switching from orthographic image-terrain three-dimensional-tilt model / colorized point cloud / white membrane-indoor / underground / underwater / panorama, fusion display of provincial, municipal, county, and township levels, control by the map layer drawing mechanism and minzoom, maxzoom parameters in mapbox.gl; layer rendering control of street community and building level detailed scenes, control of display priority by the renderOrder attribute of the drawing object in WebGL; switching of large scenes and building level western scenes, implementation by placing icons in large scenes, clicking the icons to enter indoor or underground detailed scenes; Step 3.2: Method for measurement based on point cloud in panoramic space; Mapping of panoramic and point cloud data from a free coordinate system to a real coordinate system, in street view and indoor scenes, if there is panoramic data, the system automatically displays the three-dimensional scene in the form of panorama first, and the mapping from panoramic space to real geographic space is based on the spatial registration and mapping relationship in the point cloud-panorama preprocessing stage, to transform the position of the panoramic space to the real geographic space to obtain its real geographic position; The imaging model of spherical panoramic image has the following characteristics: the center of the imaging model is the connecting line of the optical center of the panoramic camera and the object point, which forms a photographic beam, and the projection point of the photographic beam on the sphere is the imaging of the object point. Assuming that the coordinates of the object point A in the coordinate system of the point cloud model are [Xp Yp Zp], and the corresponding three-dimensional point coordinates of the panoramic sphere are [Xs Ys Zs], the expression for converting the object coordinates of point A to the spherical coordinates is as follows: ， The rotation matrix R in the formula is obtained by the projection matrix of the camera in the three-dimensional scene when the view angle is at the panoramic point, and the implementation logic is as follows: Project the plane click point in the current panoramic scene to the world coordinate system; Calculate the ray from the panoramic shooting point to the world coordinate system point; The intersection of the ray and the point cloud model is the real coordinates of the click point in the panoramic scene.

6. A live three-dimensional visualization system based on semantic point clouds for performing a live three-dimensional visualization method based on semantic point clouds according to any one of claims 1 to 5, characterized in that The system page function modules include scene display, layer management, scene special topic, search bar, toolbox, and display mode switching; The scene display is used to display the visual scene; The layer management is classified into point cloud, image, map, and three-dimensional according to data types, and the second classification focuses on specific collection means; The scene special topic is quickly positioned and displayed according to specific display cases according to the application direction of the system; The search bar is used for location search and positioning according to place names; The toolbox includes basic functions such as distance, area measurement, and point, line, and face vector element collection and export, and automatically selects and captures point cloud data when there is point cloud data, and retains the original point cloud measurement accuracy; The display mode switching switches between image, map, terrain, and fusion browsing modes.

7. The semantic point cloud based live 3D visualization system of claim 6, wherein, The system function modules include scene point cloud display module and spatial analysis module; The scene point cloud display module includes display settings of point cloud: setting the quality, view angle range, point quantity, point size, and rendering mode of point cloud to set the rendering mode of point cloud. The spatial analysis module includes point cloud sectioning, point cloud plotting, three-dimensional distance measurement, three-dimensional area measurement, import and export results, measurement clearing, flight path; The point cloud sectioning includes horizontal sectioning and vertical sectioning. The former is sectioned in the horizontal direction according to height and thickness, and the latter is sectioned in the vertical direction according to the cutting line drawn by the user and the thickness. The results of the sectioning are highlighted. The point cloud plotting displays the selected point cloud after sectioning as a background transparent picture in.png format. The three-dimensional distance measurement is based on point cloud. In a small scene free coordinate system, the real coordinates of the point cloud are directly obtained based on the XYZ of the point cloud itself. In a small scene, the real coordinates of the point cloud are captured, visualized and exported. In a large scene in the provincial range, the coordinates of the small scene are projected to the real coordinates through coordinate conversion. The three-dimensional distance measurement tool calculates the three-dimensional distance between two points on the image. The calculation formula is as follows: ， ， Where p is the Euclidean distance between point (x2, y2, z2) and point (x1, y1, z1); X is the Euclidean distance from point (x2, y2, z2) to the origin. The three-dimensional area measurement calculates the total area of the space formed by the nodes of the two-dimensional element, which is independent of the terrain and related to the XYZ values of the points of the point cloud. On the ground point cloud or indoor point cloud, the three-dimensional area of the space is measured by continuously clicking to draw a face. The area is the surface area. The import and export results tool exports the measurement results to geojson, kml, dxf format in file form. These formats can be imported and displayed again in the system, and can also be displayed in AutoCad, MapShaper, Geojson.io third-party tool websites. The measurement clearing tool clears all the measurement information drawn on the image. The flight path adds a flight path, adds a viewing angle, sets the playback time of the path, and then flies along the path for roaming.

Images