3DTiles data storage method and map data query method

Abstract

The invention discloses a 3DTiles data storage method and a map data query method.The 3DTiles data storage method comprises the following steps that (1) an LOD hierarchy pyramid type hierarchy table is established, and an LOD range index is created; 2) generating a three-dimensional object into an agent point, and segmenting a tile according to a KD-tree structure to construct a three-dimensional KD-tree index tree; and 3) storing the data of the KD-tree leaf node into a GridFS of MongoDB multi-mode storage, mapping a file path into a unique hash value, writing the unique hash value into a corresponding node according to a KD-tree node storage structure, and creating a geographic spatial index. The method creatively solves the problems that file system management is difficult due to the fact that the number of files is large, a hierarchical structure based on a file directory is difficult to support efficient spatial range retrieval, and the spatial query response time is reduced to the millisecond level in combination with a three-dimensional KD-tree index and a MongoDB geographic spatial index.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional geospatial data storage and management, and particularly relates to a method for storing 3DTiles data and a method for querying map data. Background Art

[0002] 3DTiles is an open standard three-dimensional geospatial data format, which is widely used in fields such as digital twins and smart cities. Traditional 3DTiles data is stored in the form of files, and there are the following problems: A large number of files: A single scene may contain millions of files, resulting in difficult management of the file system.

[0003] Low transmission efficiency: When a large number of small files are transmitted, the delay is significant due to the large network request overhead.

[0004] Poor spatial query performance: It is difficult to support efficient spatial range retrieval based on the hierarchical structure of the file directory.

[0005] In the prior art, folder management is generally adopted, and there are also solutions based on the HBASE database management, or a combination of the file system and the database, which still have problems of storage redundancy and query performance bottlenecks. Therefore, there is an urgent need for a solution that supports efficient storage, fast retrieval, and transmission of massive 3DTiles data. Summary of the Invention

[0006] The purpose of the present invention is to overcome the deficiencies of the prior art, and the present invention proposes a method for storing 3DTiles data and a method for querying map data, aiming to solve the efficiency problems in the transmission, storage, and management of massive three-dimensional tile data. This solution provides an efficient and scalable management paradigm for massive 3DTiles data by deeply integrating spatial indexing and database storage.

[0007] A method for storing 3DTiles data includes the following steps: 1) Data preprocessing step: Sort the tile data at different LOD levels according to the corresponding scales to establish a LOD-level pyramid-shaped hierarchical table and create a LOD range index; 2) Generate proxy points for three-dimensional objects, divide the tiles according to the KD-tree structure to construct a three-dimensional KD-tree index tree, store the tiles in the corresponding spatial range and the references to the child nodes in each node, and merge adjacent low-detail tiles. At the same time, perform multi-object clustering on the tiles in this spatial range and map the clustering results back to the proxy points; 3) Store the data of the KD-tree leaf nodes into GridFS of MongoDB multi-modal storage, map the file path to a unique hash value, generate a record of the path mapping in the file_index collection, and write it into the corresponding node according to the KD-tree node storage structure and create a geospatial index.

[0008] Preferably, step 1) further includes the steps of obtaining three-dimensional data, converting all data into the same geodetic coordinate system and projection, aligning the Z-axis reference, and then cleaning the data.

[0009] Preferably, step 3) further includes a dynamic KD-tree adjustment step, which includes the following steps: 31) If the maximum LOD level value of the current node > global threshold, or the data point density > density threshold, then trigger segmentation; 32) Adjust the position of the segmentation point according to the LOD level distribution gradient of the current node; 33) Verify again whether the maximum LOD level value of the current node ≤ global threshold and the data point density ≤ density threshold. If so, stop segmentation, otherwise continue segmentation verification; 34) Mark the nodes that meet the stop segmentation condition as leaf nodes, store the aggregation information of their covered areas, do not merge high LOD level nodes, and maintain their original fine-grained KD-tree structure to achieve high-precision query.

[0010] Preferably, step 32) is to obtain the distribution trend of the LOD level values on the segmentation axis to obtain the distribution gradient, and convert the distribution gradient into the segmentation point offset.

[0011] Preferably, preload the KD-tree nodes in the high LOD level area into memory, store the node pointers using an efficient in-memory data structure, and persistently store the leaf nodes in the low LOD level area to disk.

[0012] Preferably, it further includes a dynamic update step. When new high LOD level data is added, only update the affected KD-tree branches, and use version control to mark the modification status of the nodes to avoid global reconstruction.

[0013] A method for querying map data, the map data is stored by the data storage method, the map data is deployed in a cluster storage, and the query method includes: if directly accessed from the map page, first call the LOD range index, and then the three-dimensional KD-tree index; if spatial operation, first the three-dimensional KD-tree index, and then the geospatial index; If a specified administrative region is specified, first the three-dimensional KD-tree index, and then the LOD range index.

[0014] The advantages and beneficial effects of the present invention are as follows: The present invention creatively solves the problem of difficult file system management caused by a large number of files. By storing a large number of small files through GridFS, the number of files is reduced by more than 90%. The hierarchical structure based on file directories is difficult to support efficient spatial range retrieval. Combining a three-dimensional KD-tree index with a MongoDB geospatial index reduces the spatial query response time to the millisecond level. Detailed implementation manners

[0015] To enable those skilled in the art to better understand the solution of the present invention, the technical solution of the present invention will be further described below in conjunction with specific embodiments.

[0016] A 3DTiles data storage method of the present invention is specifically a 3DTiles data storage method based on MongoDB and KD-tree, which includes the following steps: 1) Data preprocessing step: Sort the tile data of different LOD levels according to the corresponding scale to establish an LOD level pyramid-type hierarchical table and build an LOD range index, and establish a spatial hash grid for the LOD level data. Specifically, it includes: 1. Structured annotation of LOD level information, which includes the definition of LOD levels, that is, clarifying the correspondence between LOD levels and map scales (such as LOD level 12 corresponding to a scale of 1:500), and establishing an LOD level table. Assign a unique LOD level identifier to each tile or geographical area to ensure that the LOD level values within the same area are consistent; it also includes the integration of LOD level metadata, that is, embedding the LOD level information into the spatial data structure (such as the attribute fields of a vector layer or the hierarchical naming rules of raster data), and constructing an LOD level metadata index (such as an R-tree or a range tree) to support fast query of the spatial range corresponding to a certain LOD level.

[0017] 2. Hierarchical organization of spatial data, which includes: Multi-scale data integration: Sort the tile data of different LOD levels according to the scale and establish an LOD level pyramid.

[0018] Ensure that the high-LOD level data covers the detailed parts of the low-LOD level data (such as the LOD level 12 tile contains the complete area of the LOD level 5 tile).

[0019] Spatial index optimization: Construct an LOD range index (such as an interval tree) to support fast query of areas where "LOD level ≥ N", and establish a spatial hash grid for the LOD level data to accelerate local area queries.

[0020] 2) Generate proxy points for 3D objects, segment the tiles of 3DTiles according to the KD-tree structure to construct a 3D KD-tree index tree. Each node stores the tile data and child node references within the corresponding spatial range. At the same time, perform multi-object clustering division on the tiles within this spatial range and map the clustering results back to the proxy points. Among them, multi-object clustering division mainly fuses attributes such as geometric features (volume, surface area), semantic labels (building type, land use), and environmental parameters (sunshine duration, wind speed), etc. The implementation method includes first extracting the proxy points and attribute matrices of 3D objects, then running the K-means algorithm to generate clusters (such as by functional partition), and finally mapping the clustering results back to the original 3D entities.

[0021] Among them, proxy point generation includes the geometric center method and the surface sampling method. The geometric center method takes the centroid of each 3D object (such as a building) as the proxy point; the surface sampling method uniformly samples points on the surface of the 3D model (applicable to mesh models). The process of constructing a 3D KD-tree index tree includes recursively dividing the space, and each time the dimension with the largest span in the current region is selected for cutting. The stopping condition is that the number of points within the node is less than the threshold (such as 100) or the maximum depth is reached.

[0022] 3) Store the data of the KD-tree leaf nodes into GridFS of MongoDB multi-modal storage, map the file path to a unique hash value, generate a file_index collection to record the path mapping, write it into the corresponding node according to the KD-tree node storage structure, and create a geospatial index. Store the 3DTiles files (such as.b3dm,.pnts) of the KD-tree leaf nodes (a single 3DTiles file in.b3dm format is usually in the range of 1KB - 2MB) in binary form into MongoDB GridFS (16M) to solve the problem of storing a large number of small files. The file path is mapped to a unique hash value, and the association between the logical path and physical storage is maintained through the file_index collection, realizing enhanced spatial indexing. Add a MongoDB geospatial index (2dsphere) to the KD-tree nodes to support fast spatial range queries. Combine the KD-tree hierarchy and LOD hierarchy information to achieve double filtering of "space + level of detail".

[0023] Through the storage method of the present invention, the storage efficiency is improved: by storing a large number of small files through GridFS, the number of files is reduced by more than 90%. The query performance is optimized, the KD-tree is combined with the geospatial index, and the spatial query response time is reduced to the millisecond level. The transmission bandwidth is saved, and data chunk compression reduces the network transmission volume by 20% - 36%. Taking the smart city project as an example, the original 3DTiles data contains 2.2 million files with a total size of 1.3TB. After being processed by the method of the present invention: the storage occupancy of MongoDB is reduced to 1TB (compression rate 23%); the tile loading time is shortened from an average of 8s to 1.5s; the server-side concurrent support capacity is increased to 10,000 QPS. This solution provides an efficient and scalable management paradigm for a large amount of 3DTiles data through the deep integration of spatial index and database storage, and has significant industrial application value.

[0024] Specifically, the following displays can be quickly achieved: (a) Range-based division display Define a three-dimensional query window: text [xmin, xmax] × [ymin, ymax] × [zmin, zmax] Example: Range query of a certain urban development zone (x = 113° ~ 116°, y = 22° ~ 25°, z = 0 ~ 150m) Execute the query: Use the three-dimensional KD-tree index tree to quickly locate all points / objects within the area.

[0025] Result output: A list of three-dimensional entities that meet the conditions and their attributes.

[0026] (b) Division under administrative region constraints Constraint condition design: 1. Spatial overlap principle: Only three-dimensional objects that are completely within the same administrative region are allowed to participate in the division.

[0027] 2. Vertical stratification rule: Vertically cut the same administrative region according to height intervals (such as 0 ~ 50m / 50 ~ 100m).

[0028] Implementation method: Find the intersection of the three-dimensional boundary of the administrative region and the query window of the three-dimensional KD-tree index tree.

[0029] Recursively subdivide the data within the intersection area.

[0030] Typical application scenarios of the above - mentioned mode include smart city management, such as demarcating priority areas for fire rescue (combining building height and road network), monitoring the distribution of urban heat island effect (analyzing the relationship between surface temperature and building density); cultural heritage protection, such as three - dimensional zoning of ancient building groups (by historical period or architectural style), setting virtual tour routes to avoid sensitive areas; disaster simulation analysis, such as predicting the inundation range of flood disasters (combining terrain elevation and administrative division), evaluating the threat of wildfire spread path to residential areas, etc.

[0031] Furthermore, the data pre - processing step also includes three - dimensional data acquisition, which includes generating point cloud data through lidar scanning or photogrammetry, such as using GIS tools (such as QGIS) to export the three - dimensional boundaries of administrative divisions (such as converting Shp files into three - dimensional polyhedra). It also includes coordinate system unification, such as converting all data into the same geographic coordinate system (such as CGCS2000) and projection (such as UTM), and aligning the Z - axis datum (such as using the same datum plane for altitude: Huanghai Elevation System). At the same time, clean the data, remove noise points (such as outlier height values) and redundant blank areas, and perform interpolation filling on sparse three - dimensional areas (such as Kriging interpolation).

[0032] As a further optimization scheme, it also includes the dynamic hierarchical construction of KD - tree, that is, dynamically adjusting the depth of KD - tree according to the tile level of detail (LOD level) to ensure fine - grained division of nodes in high - detail areas and merged storage in low - detail areas. It includes the following steps: 31) If the maximum LOD level value of the current node > global threshold, or the data point density > density threshold, then trigger segmentation; 32) Adjust the segmentation point position according to the LOD level distribution gradient of the current node, that is, statistically obtain the distribution trend of LOD level values on the segmentation axis to get the distribution gradient, and convert the distribution gradient into the segmentation point offset.

[0033] 33) Verify the above condition 31) before segmentation until the maximum LOD level value of the current node ≤ global threshold and the data point density ≤ density threshold. If so, stop segmentation; otherwise, continue segmentation; 34) Mark the nodes that meet the stop condition as leaf nodes, store the aggregated information of their covered areas, do not merge high - LOD level nodes, and keep their original fine - grained KD - tree structure to achieve high - precision query.

[0034] Specifically, the refinement of dynamic KD - tree construction includes the following steps: 1. Dynamic calculation of segmentation conditions First, LOD level threshold determination: Set the global LOD level threshold (e.g., LOD level_THRESHOLD = 10) as a hard condition for whether to continue splitting. If not satisfied, directly do not perform splitting.

[0035] Calculate the maximum LOD level value of the current node's coverage area (obtained through the preprocessed LOD range index).

[0036] Secondly, perform data density evaluation: Define the density threshold (e.g., density threshold (DENSITY_THRESHOLD) = 100 points / km²). When the data point density within the node exceeds the threshold, trigger splitting.

[0037] Density calculation method: The number of data points within the node ÷ the coverage area of the node.

[0038] 2. Dynamic selection of the splitting axis The traditional method is to select the dimension with the largest variance (x→y→z cyclic priority), while the present invention adopts a dynamic expansion scheme, that is, combining the LOD level gradient to adjust the axis priority. For example, split preferentially in the dimension with a large LOD level difference. The splitting point adjustment mechanism is the basic splitting point, that is, the median or mean calculated according to the traditional KD-tree algorithm.

[0039] At the same time, it can also be adjusted according to the LOD level weighting, that is, adjust the splitting point position according to the LOD level distribution gradient of the current node. Specifically, it includes gradient calculation, statistical analysis of the distribution trend of LOD level values on the splitting axis (such as the slope of linear regression), that is, to analyze the distribution of LOD level values on the splitting axis to obtain the distribution gradient. The method of numerical differentiation can be used, such as calculating the difference between adjacent LOD level values to approximate the gradient; The offset calculation is to convert the gradient into a splitting point offset (e.g., the larger the gradient, the larger the offset), which can be achieved through linear transformation or non-linear transformation. The proportional coefficient in the linear transformation formula is used to control the influence degree of the gradient on the offset, and it is adjusted according to specific application scenarios and requirements.

[0040] 3. The recursive splitting termination conditions include, The maximum LOD level value of the current node ≤ the global threshold, and, the data point density ≤ the density threshold, and the node coverage area completely belongs to the same LOD level and cannot be further split, then forcefully terminate the splitting. At the same time, verify the above conditions before each splitting to avoid unnecessary recursive calls.

[0041] Among them, parallel splitting is adopted in the process of constructing the KD-tree structure, that is, multi-threading or distributed computing frameworks (such as Apache Spark) are used to parallelly process the splitting tasks of child nodes, and a task allocation strategy of dynamically allocating computing resources according to the region size or LOD level complexity is adopted. At the same time, approximate splitting is adopted, that is, an error range (such as a slight deformation of the geometric shape) is allowed to be introduced in the low LOD level region to reduce the number of splits. The error control mechanism is to set the maximum allowable deviation (such as 5% shape similarity).

[0042] Furthermore, the refinement steps for complex scene processing are also included during splitting, which include 1. The coordination of horizontal LOD levels and vertical LOD levels, which mainly considers the adaptation of 3D scenes, that is, combining the horizontal LOD level (map scale) with the vertical LOD level (height accuracy). For example: high horizontal LOD level (city center) + high vertical LOD level (building facade details). Low horizontal LOD level (suburbs) + low vertical LOD level (simplified terrain model). The hybrid LOD level strategy is to dynamically adjust the splitting priority of the 3D space according to different 3D scenes (such as splitting first according to the horizontal LOD level and then subdividing according to the vertical LOD level).

[0043] 2. Dynamic LOD level threshold adjustment, that is, threshold modification during runtime. For example, providing an API interface allows users to adjust the LOD level threshold during runtime (such as according to the current network bandwidth or hardware performance). The threshold adjustment can automatically trigger the local reconstruction of the KD-tree (such as merging or splitting nodes).

[0044] Furthermore, the refinement steps for the dynamic storage strategy of the KD-tree are also included The aggregation steps of leaf nodes include Merging of low LOD level regions, that is, marking the nodes that meet the termination conditions as leaf nodes and storing the aggregation information of their covered regions (such as LOD level range, maximum height, minimum bounding box). The aggregation method can use a quadtree or a grid strategy to perform secondary indexing on the internal data of leaf nodes. At the same time, high LOD level regions are retained, and high LOD level nodes are not merged, maintaining their original fine-grained KD-tree structure to achieve high-precision queries.

[0045] 2. Design of a hybrid storage architecture, including Memory layer storage: Preloading KD-tree nodes in high LOD level regions into memory (such as using object pool technology to manage the node lifecycle), and using an efficient in-memory data structure (such as array-based or linked list-based) to store node pointers.

[0046] Disk layer storage: Persistently store the leaf nodes of the low LOD level regions to the disk (e.g., manage node files using the file system directory tree). At the same time, optimize disk I / O performance: adopt block compression (such as Snappy) or columnar storage format (such as Parquet).

[0047] 3. Dynamic update mechanism The incremental construction strategy is that when new high LOD level data is added, only the affected KD-tree branches are updated, and version control is used to mark the modification status of the nodes to avoid global reconstruction. At the same time, lazy deletion technology is adopted, and the deletion operation is delayed. The list of nodes to be cleaned is recorded and batch processed when the query is idle.

[0048] Meanwhile, the present invention also discloses a method for querying map data. The map data is stored by the above data storage method, and the map data is stored and deployed in a cluster. The query method includes: if directly accessed from the map page, first call the LOD range index, and then the three-dimensional KD-tree index tree; if performing a spatial operation, first the three-dimensional KD-tree index tree, and then the mongodb geospatial index; if specifying an administrative region, first the three-dimensional KD-tree index tree, and then the LOD range index. If it is a composite query method or an unclear query method, the index mode transmitted last time is adopted, and at the same time, one of the other two modes is selected, and the transmission is carried out for a predetermined time or a predetermined data volume. The transmission speeds in the two modes are compared and the better one is selected as the index mode. When encountering a composite query or an unclear query mode next time, directly select the index mode finally selected last time and the third mode not participating in the comparison last time for comparison, and select the better one as the index mode. By repeating this way, fast decision-making can be ensured and the optimal transmission method can be guaranteed as much as possible.

[0049] The on-demand transmission and caching mechanism of the present invention optimizes transmission according to requests. The client sends a spatial query request to the server according to the view range and zoom level. The server quickly locates the required tile nodes through the KD-tree index tree and only returns the necessary data. When the user queries data, for example, when the user views the map and queries according to the viewing field range, the server will quickly query the data required by the user through MongoDB GridFS, quickly locate the data within the viewing field range according to the KD-tree index tree, and return it to the user.

[0050] Meanwhile, LOD level sensitive queries can be added, that is, a priority traversal strategy that preferentially accesses child nodes that may contain high LOD level data during the query, and uses the maximum LOD level pre-stored in the nodes for pruning. A traversal order adjustment strategy that dynamically adjusts the access order of child nodes according to the LOD level requirements of the query area (such as giving priority to high LOD level child nodes). A range query optimization strategy that combines an LOD spatial index (such as an R-tree) to quickly locate candidate nodes, and performs LOD level conditional filtering on each candidate node (only retaining nodes with an LOD level ≥ the target LOD level).

[0051] Furthermore, a result cache for the LOD level and area of high-frequency queries (such as Redis in-memory cache) is also set up, with a cache invalidation strategy that invalidates dynamically based on the data update time or query popularity; and a prefetching mechanism that predicts the areas that may be accessed next based on the query history and preloads relevant nodes into memory. The memory cache can also be enabled for KD-tree nodes with high-frequency access to avoid repeated calculations. Preload KD-tree nodes in high LOD level areas into memory, use an efficient in-memory data structure to store node pointers, and persistently store the leaf nodes in low LOD level areas to disk.

[0052] Furthermore, the present invention also adopts a data chunk compression strategy, which merges the tiles of the same KD-tree node into a single data chunk. For example, the LZ4 compression algorithm is used to reduce the transmission volume. Merging and then compressing the batch of data queried by the user and then sending it to the user can improve efficiency.

[0053] The present invention creatively solves the following technical problems: 1. Difficult file system management caused by a large number of files: Store a large number of small files through GridFS, reducing the number of files by more than 90%. 2. Network latency caused by the transmission of a large number of small files: Adopt data chunk compression technology to reduce the network transmission volume by 20% - 36%. 3. The hierarchical structure based on file directories is difficult to support efficient spatial range retrieval: Combine a three-dimensional KD-tree index tree with a MongoDB geospatial index to reduce the spatial query response time to the millisecond level.

[0054] The innovation of the present invention lies in the deep combination of three-dimensional tile data and spatial index, realizing efficient data management and transmission optimization. Through the dynamic construction of the KD-tree and the multi-modal storage design, the efficiency bottleneck in the traditional 3DTiles data storage and management is solved, significantly improving the data access performance and system scalability.

[0055] The above has made an exemplary description of the present invention. It should be noted that without departing from the core of the present invention, any simple deformation, modification or equivalent replacement that can be made by those skilled in the art without creative efforts shall fall within the protection scope of the present invention.

Claims

1. A 3D Tiles data storage method, characterized in that: The following steps are included: 1) Data preprocessing step: sort the tile data of different LOD levels according to the corresponding scale to establish the LOD level pyramid level table and create the LOD range index; 2) Generate proxy points for 3D objects, segment tiles according to the KD-tree structure to construct a 3D KD-tree index tree. Each node stores tiles and sub-node references in the corresponding spatial range and merges adjacent low-detail tiles. At the same time, multi-target clustering is performed on the tiles in the spatial range and the clustering results are mapped back to the proxy points. 3) Store the data of the KD-tree leaf nodes into the GridFS of MongoDB multimodal storage, map the file path to a unique hash value, generate a file_index collection to record the path mapping, write it into the corresponding node according to the KD-tree node storage structure, and create a geospatial index.

2. The 3D Tiles data storage method according to claim 1, characterized in that: The step 1) also includes the steps of acquiring three-dimensional data and converting all data into the same geographic coordinate system and projection, aligning the Z-axis reference, and then cleaning the data.

3. The 3D Tiles data storage method according to claim 1, characterized in that: Step 3) also includes a dynamic KD-tree adjustment step, which includes the following steps: 31) If the maximum LOD level value of the current node is greater than the global threshold, or the data point density is greater than the density threshold, the segmentation is triggered; 32) Adjust the split point position according to the LOD level distribution gradient of the current node; 33) Split again to verify whether the maximum LOD level value of the current node is ≤ the global threshold, and the data point density is ≤ the density threshold. If so, stop splitting, otherwise continue splitting verification; 34) Mark the nodes that meet the stop splitting conditions as leaf nodes, store the aggregation information of their coverage areas, do not merge high LOD level nodes, and maintain their original fine-grained KD-tree structure to achieve high-precision queries.

4. The 3D Tiles data storage method according to claim 3, characterized in that: The step 32) is to obtain a distribution gradient by counting the distribution trend of the LOD level value on the segmentation axis, and convert the distribution gradient into a segmentation point offset.

5. The 3D Tiles data storage method according to claim 3, characterized in that: Preload the KD-tree nodes of the high LOD level area into memory, use efficient memory data structure to store node pointers, and persist the leaf nodes of the low LOD level area to disk.

6. The 3D Tiles data storage method according to claim 1, characterized in that: It also includes a dynamic update step. When new high-LOD level data is added, only the affected KD-tree branches are updated, and the modified status of the node is marked with version control to avoid global reconstruction.

7. A map data query method, characterized in that: The map data is stored by the 3D Tiles data storage method according to any one of claims 1 to 6, the map data is deployed in cluster storage, and the query method includes: If you access it directly from the map page, first call the LOD range index, and then the 3D KD-tree index; For spatial operations, first use the 3D KD-tree index, then the geographic spatial index; If a partition is specified, first perform a 3D KD-tree index, then perform a LOD range index.