A mass spatial vector data dynamic storage method and system
By preprocessing, hierarchizing, indexing, and compressing massive spatial vector data, the problems of storage redundancy and low retrieval efficiency are solved, achieving efficient data storage and retrieval.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INSTITUTE OF SURVEYING AND MAPPING
- Filing Date
- 2026-04-22
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies suffer from significant storage redundancy and low retrieval efficiency in storing massive spatial vector data. They fail to effectively adapt to data accuracy characteristics and application scenarios, resulting in resource waste and excessively long retrieval times.
By preprocessing and multi-scale layering of the original massive spatial vector data, an improved quadtree spatial index is constructed, and geohashing and adaptive compression are performed to generate multiple compressed data blocks, which are managed using a dynamic storage strategy.
It reduces storage redundancy, improves data retrieval efficiency, and achieves efficient data storage and retrieval.
Smart Images
Figure CN122432260A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data storage, and in particular to a method and system for dynamic storage of massive spatial vector data. Background Technology
[0002] Efficient storage of massive spatial vector data is a crucial foundation for the practical application of geographic information-related industries. Its storage cost and ease of data retrieval directly impact project efficiency and resource allocation in fields such as remote sensing data processing and smart city planning. Current mainstream technologies for addressing this type of data storage problem often employ single-format, indiscriminate data storage such as Shapefile and GeoJSON, combined with linear or simple grid indexes for data location and retrieval. Existing methods, lacking differentiated storage design based on data precision characteristics and actual application scenarios, and with index structures failing to adapt to the growing demands of massive data volumes, suffer from severe waste of storage resources, excessively long data retrieval times, and poor scalability for large-scale data storage.
[0003] At present, the storage of massive spatial vector data suffers from significant storage redundancy and low retrieval efficiency. Summary of the Invention
[0004] This application provides a method and system for dynamic storage of massive spatial vector data. It employs preprocessing and multi-scale layering of the original massive spatial vector data to generate multi-scale layered datasets adapted to different precision requirements. Based on the base layer data of this dataset, an improved quadtree spatial index is constructed. The geographic space covered by the original data is divided into multiple data blocks. These data blocks are geo-hash encoded and a mapping table is established. Based on this mapping table, an adaptive compression strategy is used to compress the multi-scale layered data corresponding to each data block, resulting in multiple compressed data blocks. All compressed data blocks are dynamically stored according to a preset directory structure. These techniques solve the technical problems of prominent storage redundancy and low retrieval efficiency in existing massive spatial vector data storage, achieving the technical effects of reducing storage redundancy and improving data retrieval efficiency.
[0005] This application provides a method for dynamically storing massive spatial vector data, comprising: preprocessing and multi-scale layering the original massive spatial vector data to obtain a multi-scale layered dataset; constructing an improved quadtree spatial index based on the base layer data in the multi-scale layered dataset to divide the geographic space covered by the original data into multiple data blocks; performing geo-hash encoding on the multiple data blocks to establish a geo-hash encoding mapping table; adaptively compressing the multi-scale layered data corresponding to each data block according to the geo-hash encoding mapping table to obtain multiple compressed data blocks; and dynamically storing the multiple compressed data blocks according to a preset directory structure.
[0006] In a possible implementation, the original massive spatial vector data is preprocessed and multi-scale layered to obtain a multi-scale layered dataset. The following processing is then performed: the original massive spatial vector data is cleaned and format-standardized to obtain a standardized massive spatial vector dataset; based on a preset precision grading threshold, the standardized massive spatial vector dataset is subjected to layered simplification processing to generate the multi-scale layered dataset containing multiple levels from low precision to high precision; wherein, the low-level data is a simplified subset of the high-level data, and during the layered simplification process, a correlation relationship is established for the same geographic feature in different levels.
[0007] In a possible implementation, based on the base layer data in the multi-scale hierarchical dataset, an improved quadtree spatial index is constructed to divide the geographic space covered by the original data into multiple data blocks. The following processing is performed: a quadtree root node is constructed based on the geographic range corresponding to the base layer data; a dynamic partitioning threshold is calculated, and the amount of data in the current node of the quadtree root node is recursively determined. If it is greater than the dynamic partitioning threshold, child nodes are partitioned; if it is less than or equal to the dynamic partitioning threshold, the current node is marked as a leaf node of the data block storing data; based on the leaf nodes of the data blocks, the geographic space covered by the original data is divided into the multiple data blocks; wherein, the dynamic partitioning threshold is calculated based on the total data size, geographic area, and average data density.
[0008] In a possible implementation, the following processing is performed: during the construction of the improved quadtree spatial index, an empty node merging operation is also performed on the parent node containing multiple empty child nodes.
[0009] In a possible implementation, geohashing is performed on the plurality of data blocks to establish a geohashing mapping table, and the following processes are performed: extracting a first data block from the plurality of data blocks; determining the length of a first geohashing code based on the precision level of the first data block; calculating the first spatial location of the first data block and generating a first geohashing code by combining it with the length of the first geohashing code; and establishing a mapping relationship between the first data block and the first geohashing code to obtain the geohashing mapping table.
[0010] In a possible implementation, based on the geohash encoding mapping table, adaptive compression is performed on the multi-scale hierarchical data corresponding to each data block to obtain multiple compressed data blocks. The following processing is then performed: a first compression algorithm is used to compress the polygon feature data in the first data block; a second compression algorithm is used to compress the line feature and point feature data in the first data block; and dictionary encoding is used to compress the attribute information of the geographic features in the first data block to obtain a first compressed data block. The multiple data blocks are then traversed, and the compression steps of the first compressed data block are repeatedly executed to obtain the multiple compressed data blocks.
[0011] In a possible implementation, the following processing is also performed: in response to a data update request, based on the association between the geohash encoding mapping table and the features in the multi-scale hierarchical dataset, the data blocks associated with the changed geographic features and their corresponding compressed data blocks are located and updated incrementally.
[0012] In possible implementations, the following processing is also performed: establishing a multi-level caching mechanism, which includes: a client-side cache established based on the least recently used cleanup principle, used to store compressed data blocks that are frequently accessed by users; and a server-side cache established based on a distributed architecture, used to store compressed data blocks that are frequently accessed globally.
[0013] This application also provides a dynamic storage system for massive spatial vector data, comprising: a multi-scale layering module for preprocessing and multi-scale layering of the original massive spatial vector data to obtain a multi-scale layered dataset; an improved quadtree spatial index construction module for constructing an improved quadtree spatial index based on the base layer data in the multi-scale layered dataset, dividing the geographic space covered by the original data into multiple data blocks; a geohash encoding module for performing geohash encoding on the multiple data blocks to establish a geohash encoding mapping table; an adaptive compression module for adaptively compressing the multi-scale layered data corresponding to each data block according to the geohash encoding mapping table to obtain multiple compressed data blocks; and a dynamic storage module for dynamically storing the multiple compressed data blocks according to a preset directory structure.
[0014] This application proposes a method and system for dynamically storing massive spatial vector data. First, the original massive spatial vector data is preprocessed and layered at multiple scales to obtain a multi-scale layered dataset. Then, based on the base layer data in the multi-scale layered dataset, an improved quadtree spatial index is constructed to divide the geographic space covered by the original data into multiple data blocks. Next, these multiple data blocks are geo-hash encoded to establish a geo-hash encoding mapping table. Then, according to the geo-hash encoding mapping table, the multi-scale layered data corresponding to each data block is adaptively compressed to obtain multiple compressed data blocks. Finally, these compressed data blocks are dynamically stored according to a preset directory structure. Through the above process, the method and system proposed in this application achieve the technical effects of reducing storage redundancy and improving data retrieval efficiency. Attached Figure Description
[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from these processes.
[0016] Figure 1 This is a flowchart illustrating a method for dynamically storing massive spatial vector data, as provided in an embodiment of this application.
[0017] Figure 2 This is a schematic diagram of the structure of a dynamic storage system for massive spatial vector data provided in an embodiment of this application.
[0018] Figure labeling: Multi-scale hierarchical module 10, improved quadtree spatial index construction module 20, geohashing encoding module 30, adaptive compression module 40, dynamic storage module 50. Detailed Implementation
[0019] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.
[0020] This application provides a method for dynamically storing massive spatial vector data, such as... Figure 1 As shown, the method includes: Step S100: Preprocess and multi-scale hierarchical layering of the original massive spatial vector data to obtain a multi-scale hierarchical dataset.
[0021] Specifically, multiple sources of raw data are collected, including remote sensing interpretation data, manual surveying data, and data exported from GIS systems. These are first uniformly converted to GeoJSON format for standardization. Then, data cleaning is performed to remove duplicate geographic features, correct topological errors, and supplement missing attribute information, resulting in a standardized dataset. Based on preset precision grading thresholds, an improved Douglas-Peucker algorithm combined with the Spark distributed computing framework is used to perform hierarchical simplification processing on the standardized dataset, generating multiple tiers of datasets from low to high precision. Lower-level data are simplified subsets of higher-level data. Simultaneously, a unique ID is established for the same geographic feature across different tiers to ensure feature consistency.
[0022] For example, 5TB of spatial vector data covering 16,000 square kilometers in a first-tier city is preprocessed and divided into three scale levels: basic layer (1:100,000), intermediate layer (1:10,000), and detail layer (1:500). A road with ID RD001 in the detail layer retains this unique ID in both the intermediate layer and the basic layer.
[0023] In one possible implementation, the raw massive spatial vector data is preprocessed and multi-scale layered to obtain a multi-scale layered dataset. Step S100 further includes step S110, which cleans and standardizes the format of the raw massive spatial vector data to obtain a standardized massive spatial vector dataset. Specifically, in terms of format standardization, data conversion tools are used to uniformly convert raw data from different sources, such as geographic data in Shapefile and CSV formats, into GeoJSON format to ensure consistent data structure. In terms of data cleaning, duplicate geographic features are removed through topology checks, such as records of the same building being entered multiple times. Topology errors are corrected using geometric repair tools, such as breakpoints in line features and self-intersection problems in polygon features. Missing attribute information is supplemented using attribute completion rules, such as missing road grade and width information in road data.
[0024] Step S120: Based on preset precision grading thresholds, perform hierarchical simplification processing on the standardized massive spatial vector dataset to generate a multi-scale hierarchical dataset containing multiple levels from low to high precision. The low-level data is a simplified subset of the high-level data. During the hierarchical simplification process, relationships are established for the same geographic feature in different levels. Specifically, based on the precision requirements of geographic information application scenarios, a quantitative precision grading threshold system is established. For example, the basic layer sets a minimum retained area threshold of 1000 square meters for polygon features and a simplified node spacing threshold of 50 meters for line features. The intermediate layer adjusts the minimum retained area for polygon features to 500 square meters and the simplified node spacing for line features to 10 meters. At the detail level, there is no area filtering threshold for features, and the node spacing for line features does not exceed 1 meter. The hierarchical simplification uses an improved Douglas-Peucker algorithm. By calculating the curvature of line segments, nodes with curvature not less than 0.1 are marked as feature points. Feature points are retained first, and then non-feature points are filtered using distance thresholds. Curvature = 1 / turning radius. This improvement reduces the data volume by 60%-70% while maintaining over 90% morphological similarity. Simultaneously, based on the Spark distributed computing framework, the standardized massive spatial vector dataset is partitioned into 100km x 100km geographical areas. Each partition performs hierarchical simplification operations in parallel, and a shared feature ID mapping table ensures cross-partition feature hierarchy consistency. Through this parallel hierarchical processing mechanism, processing efficiency is improved by 8-10 times compared to single-threaded processing, supporting the completion of hierarchical processing for petabyte-scale data within 24 hours. Finally, unique IDs are assigned to the same geographic feature in different levels to establish associations, ensuring that lower-level data is a simplified subset of higher-level data.
[0025] For example, the road line elements of a city are processed in layers. A road with 2,000 nodes in the detail layer is simplified to 300 feature points and key nodes in the intermediate layer, and further simplified to 50 core nodes in the base layer. The road in all three layers is associated with the same ID: RD002, and the road data in the base layer and intermediate layer are simplified versions of the data in the detail layer.
[0026] Step S200: Based on the base layer data in the multi-scale hierarchical dataset, construct an improved quadtree spatial index to divide the geographic space covered by the original data into multiple data blocks.
[0027] Specifically, the entire geographic region corresponding to the base layer data in the multi-scale hierarchical dataset is used as the root node of the quadtree. First, a dynamic partitioning threshold is calculated, based on the total data size, geographic region area, and average data density. The amount of data within the root node and its child nodes is recursively checked. If it exceeds the dynamic partitioning threshold, quadrants are further divided to generate child nodes; otherwise, the node is marked as a leaf node of a data block storing data. Simultaneously, an empty node merging mechanism is used during index construction. If three of the four child nodes of a parent node are empty nodes and one is a data node, the empty nodes are merged into a virtual node, storing only the data node information. Finally, the geographic space covered by the original data is divided into multiple data blocks through the data block leaf nodes.
[0028] For example, if the dynamic partitioning threshold is 12GB, when constructing a quadtree, if the data volume in a certain quadrant is 15GB, exceeding 12GB, the partitioning continues; if the data volume in a certain quadrant is 8GB, the partitioning stops and is marked as a leaf node of the data block.
[0029] In one possible implementation, based on the base layer data in the multi-scale hierarchical dataset, an improved quadtree spatial index is constructed, dividing the geographic space covered by the original data into multiple data blocks. Step S200 further includes step S210, constructing the quadtree root node based on the geographic range corresponding to the base layer data. Specifically, the geographic coordinate range of the base layer data in the multi-scale hierarchical dataset is extracted, that is, the rectangular geographic area enclosed by the minimum longitude, maximum longitude, minimum latitude, and maximum latitude covered by the base layer data. This rectangular geographic area is used as the root node of the quadtree. The root node contains the coordinate boundary information of the geographic area, the corresponding total data statistics, and the initial state of the child nodes. When not divided, the child nodes are empty.
[0030] For example, the geographical range covered by the basic layer data of a certain city is from 116.0 degrees east longitude to 117.0 degrees east longitude and from 39.0 degrees north latitude to 40.0 degrees north latitude. This rectangular area is used as the root node of a quadtree. The root node records the coordinate range and also counts the total amount of basic layer data in this area.
[0031] Step S220: Calculate the dynamic partitioning threshold. Recursively determine the amount of data within the current node of the quadtree root node. If it is greater than the dynamic partitioning threshold, perform child node partitioning. If it is less than or equal to the dynamic partitioning threshold, mark the current node as a leaf node of the data block storing data. The dynamic partitioning threshold is calculated based on the total data size, geographical area, and average data density. Specifically, the calculation logic for the dynamic partitioning threshold is as follows: First, calculate the total size of the original data and the geographical area covered by the original data, then calculate the average data density: Average data density = Total data size / Geographical area. Next, determine the scenario coefficient based on the application scenario. The scenario coefficient for urban core areas is 1.2, for suburban areas it is 0.8, and for ordinary areas it is 1.0. Finally, substitute the values into the formula to calculate the dynamic partitioning threshold: Dynamic partitioning threshold = Scenario coefficient × (Total data size) × (Data size × 100%). 1 / 3 ) × (Average data density) 2 / 3 During the recursive judgment process, the amount of data in the geographic area corresponding to the current node is first counted. If the amount of data is greater than the dynamic partitioning threshold, the current node is divided into 4 equal child nodes along the longitude and latitude midline. Then, the amount of data in each child node is counted and the judgment process is repeated. If the amount of data in the current node is less than or equal to the dynamic partitioning threshold, the partitioning is stopped and the node is marked as a data block leaf node to store the actual data in the geographic area.
[0032] Step S230: Based on the leaf nodes of the data blocks, the geographic space covered by the original data is divided into the multiple data blocks. Specifically, all nodes marked as leaf nodes of data blocks in the improved quadtree are traversed, and the geographic coordinate boundaries corresponding to each leaf node are extracted, i.e., the minimum longitude, maximum longitude, minimum latitude, and maximum latitude covered by the leaf node. Using the geographic coordinate boundaries of each leaf node as the range, all multi-scale hierarchical data in the original data falling within this range, including the data of the corresponding areas of the basic layer, intermediate layer, and detail layer, are divided into the dataset corresponding to the leaf node. Each leaf node corresponds to an independent data block, and the data block contains vector data within the geographic range and corresponding attribute information and scale level identifiers.
[0033] For example, the geographic range corresponding to a certain leaf node is 116.2 degrees to 116.3 degrees east longitude and 39.1 degrees to 39.2 degrees north latitude. All building surface features, road line features, POI point features and their corresponding attribute information that fall within this rectangular range in the original data are organized into an independent data block according to different scale levels. This data block only contains relevant data for this geographic range.
[0034] In one possible implementation, step S200 further includes step S240, which, in the process of constructing the improved quadtree spatial index, also includes performing an empty node merging operation on the parent node containing multiple empty child nodes.
[0035] Specifically, during the quadtree node partitioning process, the status of the four child nodes of each parent node is monitored in real time. When it is detected that three of the four child nodes of a parent node are empty nodes and one is a data node, the empty node merging mechanism is triggered to merge the three empty nodes into a virtual node. The virtual node only records the status identifier as empty and does not occupy additional storage space. The parent node only associates the information of the virtual node and the data node and does not store the complete boundary and status data of the three empty nodes separately.
[0036] For example, in the geographic region corresponding to a parent node in the suburbs of a city, most areas have no valid geographic data. Of its four child nodes, three are empty nodes and one is a data node. In this case, the three empty nodes are merged into a virtual node, and the parent node only stores the virtual node identifier and the association information of the data node. Compared with the unmerged state, the storage overhead of the three empty nodes is reduced, and the overall index storage usage is reduced by more than 30%.
[0037] Step S300: Perform geo-hash encoding on the multiple data blocks and establish a geo-hash encoding mapping table.
[0038] Specifically, all pre-defined data blocks are traversed, and the geohash code length is dynamically determined based on the scale level corresponding to each data block. For example, the base layer uses a 10-bit code, corresponding to a geographic range of approximately 1.2 km × 0.6 km; the intermediate layer uses a 12-bit code, approximately 150 m × 75 m; and the detail layer uses a 14-bit code, approximately 19 m × 9 m. The longitude and latitude center coordinates of each data block are calculated, and these two-dimensional longitude and latitude coordinates are converted into a one-dimensional string of the corresponding length using Base32 encoding, which serves as the geohash code for that data block. Based on the prefix matching characteristic of geohash codes, a prefix-region index table is constructed. The geohash codes are grouped according to the first 4-8 bits of the prefix, recording all data blocks corresponding to each prefix group and adjacent prefix groups. The edit distance between adjacent prefixes is ≤1. For example, the prefix group corresponding to the code wx4g5 has adjacent prefix groups such as wx4g4, wx4g6, and wx4f3. Finally, a mapping table between geohash codes and data blocks is established. This table records information such as the storage path, data block ID, scale level, size, prefix group, and adjacent prefix groups for each geohash code, and is stored in the HBase distributed database. This mechanism enables rapid location of data blocks in adjacent regions, improving the speed of data retrieval in those regions.
[0039] In one possible implementation, geohashing is performed on the multiple data blocks to establish a geohashing mapping table. Step S300 further includes step S310, which involves traversing and extracting a first data block from the multiple data blocks. Specifically, a distributed traversal algorithm is used, based on the storage path or data block ID order of the data blocks, to sequentially extract individual data blocks as the first data block from the multiple data blocks. During the traversal, the identifiers of the processed data blocks are recorded to avoid duplicate processing. At the same time, multi-threaded parallel traversal improves processing efficiency. The traversal order can be from east to west or from north to south according to the geographical region corresponding to the data block, or arranged in ascending order by data block ID.
[0040] Step S320: Determine the first geohash code length based on the precision level of the first data block. Specifically, a correspondence table between precision levels and geohash code lengths is pre-established, for example, specifying that the basic layer corresponds to a 10-bit code, the intermediate layer to a 12-bit code, and the detail layer to a 14-bit code. Extract the scale level identifier of the first data block, such as scale_1 representing the basic layer, scale_2 representing the intermediate layer, and scale_3 representing the detail layer. Query the correspondence table based on the identifier to determine the geohash code length corresponding to the data block.
[0041] Step S330: Calculate the first spatial location of the first data block and generate a first geohash code by combining it with the first geohash code length. Specifically, calculate the first spatial location of the first data block, i.e., the coordinates of the center point of the data block. This is done by extracting the minimum longitude, maximum longitude, minimum latitude, and maximum latitude of the data block's geographical range, and calculating the midpoint of longitude and the midpoint of latitude respectively. These two coordinates form the center point coordinates. Using the two-dimensional longitude and latitude coordinates of the center point as input, the Base32 encoding algorithm is used to encode and convert the coordinates according to the first geohash code length. During the encoding process, the longitude and latitude coordinates are first mapped to the intervals of [-90, 90] (latitude) and [-180, 180] (longitude). Then, the intervals are continuously subdivided using a binary search method. Each subdivision step corresponds to one character of the Base32 encoding until the preset encoding length is reached, generating the first geohash code.
[0042] For example, the geographical range of the first data block is 116.2° to 116.3° east longitude and 39.1° to 39.2° north latitude. The calculated center point coordinates are 116.25° east longitude and 39.15° north latitude. The length of the first geohash code is 12 bits. After Base32 encoding conversion, wx4g56789abc is generated as the first geohash code.
[0043] Step S340: Establish a mapping relationship between the first data block and the first geohash code to obtain the geohash code mapping table. Specifically, construct a mapping table data structure, including fields such as geohash code, data block ID, data block storage path, scale level of the data block, data block size, data block center point coordinates, data block update time, prefix group, and adjacent prefix groups. Fill the relevant information of the first geohash code and the corresponding first data block into the corresponding fields of the mapping table. The prefix group is the first 4-8 characters of the geohash code, and the adjacent prefix groups are determined by calculating the prefix edit distance. Prefix groups with an edit distance ≤ 1 are selected as adjacent prefix groups. Traverse all data blocks and repeat the above steps, recording the geohash code and corresponding information of each data block into the mapping table one by one, finally forming a complete geohash code mapping table, which is stored in the HBase distributed database. When querying adjacent area data blocks, obtain its adjacent prefix group through the geohash code of the target data block, and then traverse all data blocks under the adjacent prefix group to quickly locate adjacent area data blocks, which is 5-8 times faster than the adjacent query speed of traditional spatial index.
[0044] For example, if the geohash code of the target data block is wx4g56789abc, its prefix group is wx4g56, and its adjacent prefix groups are wx4g55, wx4g57, wx4g46, and wx4g66, when querying the adjacent area data of this data block, all data blocks under these four adjacent prefix groups can be directly extracted from the mapping table. There is no need to traverse all data blocks, and the location of adjacent area data blocks can be completed with only one query.
[0045] Step S400: Based on the geo-hash encoding mapping table, adaptive compression is performed on the multi-scale hierarchical data corresponding to each data block to obtain multiple compressed data blocks.
[0046] Specifically, the geohashing mapping table is queried to obtain the feature type composition and attribute information characteristics of each data block. Different compression strategies are adopted for different types of vector data within the data block. For example, the ZIP compression algorithm is used for polygon feature data, while the LZ77 compression algorithm is used for line and point feature data. Attribute information is encoded using dictionary encoding, and repeated attribute values are replaced with fixed codes, such as 001 for urban roads. During the compression process, the compression level is dynamically adjusted according to the data volume. For example, a high compression level is used when the data volume is greater than 1GB, with a compression rate of approximately 40%; a medium compression level is used when the data volume is between 500MB and 1GB, with a compression rate of approximately 35%; and a low compression level is used when the data volume is less than 500MB, with a compression rate of approximately 30%.
[0047] In one possible implementation, based on the geohashing encoding mapping table, adaptive compression is performed on the multi-scale hierarchical data corresponding to each data block to obtain multiple compressed data blocks. Step S400 further includes step S410, which involves compressing the polygon feature data within the first data block using a first compression algorithm, compressing the line and point feature data within the first data block using a second compression algorithm, and compressing the attribute information of the geographic features within the first data block using dictionary encoding to obtain the first compressed data block. Specifically, the first compression algorithm uses the ZIP compression algorithm, which is suitable for batch compression of polygon feature data. By differentially encoding the coordinate data of adjacent polygon features and then performing Huffman encoding, a compression rate of 30%-40% is achieved. The second compression algorithm uses the LZ77 compression algorithm, which finds repeated byte sequences in the data and replaces the repeated sequences with pointers to those sequences. This algorithm is suitable for coordinate data of line and point features and can achieve a compression rate of 40%-50%. During dictionary-based encoding compression, the process begins by identifying all recurring attribute values in the geographic feature attribute information within the first data block. A dictionary mapping table of attribute values and their corresponding codes is then established (e.g., highways are coded as 003, hospitals as 004, and rivers as 005). Next, all attribute values in the attribute information are replaced with their corresponding codes, reducing the storage volume of duplicate data. Finally, the compressed polygon, line, and point feature data, along with the coded attribute information, are integrated, and a data header is added, containing information such as the compression algorithm identifier, data block ID, and original data size, to generate the first compressed data block.
[0048] Step S420: Traverse the multiple data blocks and repeatedly execute the compression steps of the first compressed data block to obtain the multiple compressed data blocks. Specifically, a distributed batch processing framework is adopted, and the data blocks are fragmented according to their geohash encoding prefix or storage path. Each fragment contains several data blocks, and the compression operation is performed in parallel on each fragment. During the traversal, the compression logic of step S410 is repeated for each data block, that is, according to the feature type in the data block, the corresponding compression algorithm is used to process polygon features, line features, point features, and attribute information to generate the corresponding compressed data block. During the compression process, the compression status of each data block is recorded, including uncompressed, compressing, compression completed, and compression failed. Data blocks that fail to compress are retried to ensure that all data blocks can be successfully compressed.
[0049] Step S500: Dynamically store the multiple compressed data blocks according to the preset directory structure.
[0050] Specifically, the preset directory structure adopts a multi-level directory structure of scale level-geohash prefix. The first-level directory is named with the scale level identifier, such as scale_1 corresponding to the basic layer, scale_2 corresponding to the intermediate layer, and scale_3 corresponding to the detail layer. The second-level directory is named with the first 4-6 digits of the geohash code as a prefix, such as wx4g56 for the geohash code wx4g56789abcde. The third-level directory stores all compressed data blocks corresponding to this prefix. The filename of the compressed data block consists of the complete geohash code string and the data block ID, such as wx4g56789abcde_DB0006.zip. The storage adopts a distributed file storage system, which dynamically allocates storage nodes according to the size and access frequency of the compressed data blocks. Data blocks with high access frequency are stored in high-performance storage nodes, and data blocks with low access frequency are stored in ordinary storage nodes. At the same time, the storage system records metadata information such as the storage path, size, modification time, and associated geohash code of each compressed data block, supporting fast retrieval by metadata.
[0051] In one possible implementation, the method further includes step S600, in response to a data update request, locating and updating only the data blocks associated with the changed geographic features and their corresponding compressed data blocks based on the association between the geohash encoding mapping table and the features in the multi-scale hierarchical dataset, and performing incremental updates.
[0052] Specifically, when a data update request is received, such as adding a road, modifying building attributes, or deleting a POI, the ID of the changed geographic feature and its corresponding scale level are first determined. The geohash mapping table is queried using the geographic feature's coordinate information to locate all data blocks containing that feature, including data blocks at the feature's corresponding scale level and related levels. The original data corresponding to these data blocks is extracted, and only the changed geographic features are modified, including adding, deleting, or updating attributes or coordinates; unchanged features remain unchanged. The adaptive compression process of step S400 is re-executed on the modified original data to generate new compressed data blocks, replacing the original compressed data blocks. Simultaneously, the metadata information of the corresponding data block in the geohash mapping table is updated, such as update time and data block size. Finally, a version number verification mechanism is used to synchronize the updated version number to the client, ensuring that the client receives the latest data.
[0053] For example, a city adds a new urban expressway with the ID RD007. This road has corresponding data in both the detail layer and the intermediate layer. By querying the geohash mapping table using its coordinates, the data is located in the detail layer (DB0010) and the intermediate layer (DB0015). The original data of these two data blocks is extracted, and vector data and attribute information for road RD007 are added. After recompression, new compressed data blocks are generated, replacing the original compressed files corresponding to DB0010 and DB0015. The update time of these two data blocks in the geohash mapping table is modified, the data block size is adjusted accordingly, and the version number is updated. By updating only the changed data, the high time consumption and resource waste caused by a full update are avoided, thus improving data update efficiency.
[0054] In one possible implementation, the method further includes step S700, establishing a multi-level caching mechanism, which includes: a client-side cache established based on the least recently used cleanup principle, used to store compressed data blocks that are frequently accessed by users; and a server-side cache established based on a distributed architecture, used to store compressed data blocks that are frequently accessed globally.
[0055] Specifically, the client-side cache employs the Least Recently Used (LRU) principle for cleanup, dividing the cache into a high-frequency data area and a temporary data area. The high-frequency data area stores data accessed at least three times within the past hour and uses local IndexedDB for persistent storage. The temporary data area stores data accessed less than three times and uses SessionStorage for in-memory caching. The cache size is dynamically allocated based on device storage space; for example, 500MB for mobile devices and 2GB for PCs. When device storage space is insufficient, the temporary data area is cleaned up first, followed by the LRU algorithm to clean up data in the high-frequency data area that has not been accessed within the past 72 hours. The client-side cache also uses a version number verification mechanism. Each time data is requested, the version number of the locally cached data is compared with the latest version number on the server, and only data blocks with inconsistent versions are updated.
[0056] The server-side cache employs a distributed architecture, dividing cache nodes into 16 hash slots based on a consistent hashing algorithm. Each hash slot corresponds to a fixed geographical region, such as hash slots 1-4 for North China and 5-8 for East China. During data storage, the hash value is calculated based on the geographic hash code of the compressed data block and mapped to the corresponding hash slot node. Each cache node is configured with one standby node, monitoring the primary node's status via a heartbeat check every 10 seconds. If the primary node fails, the standby node takes over the hash slot service within 3 seconds, simultaneously synchronizing missing data from the HBase database, keeping service interruption time within 5 seconds. The server-side cache stores the top 50% of compressed data blocks by global access frequency, such as data at various levels in the city's core area and detailed data in popular areas. It continuously monitors the access frequency of each data block, deleting data accessed less than 10 times per day and replenishing newly added high-frequency access data. Through collaborative caching between the client and server, the transmission and retrieval of duplicate data are reduced, improving data access speed.
[0057] This application employs preprocessing and multi-scale layering of raw massive spatial vector data to generate multi-scale layered datasets adapted to different precision requirements. Based on the base layer data of this dataset, an improved quadtree spatial index is constructed. The geographic space covered by the original data is divided into multiple data blocks. These data blocks are geo-hash encoded and a mapping table is established. Based on this mapping table, an adaptive compression strategy is used to compress the multi-scale layered data corresponding to each data block, resulting in multiple compressed data blocks. According to a preset directory structure, all compressed data blocks are dynamically stored. These techniques solve the technical problems of prominent storage redundancy and low retrieval efficiency in existing massive spatial vector data storage, achieving the technical effect of reducing storage redundancy and improving data retrieval efficiency.
[0058] In the above text, refer to Figure 1 A method for dynamically storing massive spatial vector data according to an embodiment of the present invention is described in detail. Next, reference will be made to... Figure 2 This invention describes a dynamic storage system for massive spatial vector data according to an embodiment of the present invention.
[0059] A dynamic storage system for massive spatial vector data according to an embodiment of the present invention addresses the technical problems of prominent storage redundancy and low retrieval efficiency in existing massive spatial vector data storage, achieving the technical effects of reducing storage redundancy and improving data retrieval efficiency. The dynamic storage system for massive spatial vector data includes: a multi-scale hierarchical module 10, an improved quadtree spatial index construction module 20, a geohashing encoding module 30, an adaptive compression module 40, and a dynamic storage module 50.
[0060] The multi-scale layering module 10 is used to preprocess and layer the original massive spatial vector data at multiple scales to obtain a multi-scale layered dataset; the improved quadtree spatial index construction module 20 is used to construct an improved quadtree spatial index based on the base layer data in the multi-scale layered dataset, dividing the geographic space covered by the original data into multiple data blocks; the geohashing encoding module 30 is used to perform geohashing encoding on the multiple data blocks and establish a geohashing encoding mapping table; the adaptive compression module 40 is used to adaptively compress the multi-scale layered data corresponding to each data block according to the geohashing encoding mapping table to obtain multiple compressed data blocks; and the dynamic storage module 50 is used to dynamically store the multiple compressed data blocks according to a preset directory structure.
[0061] The detailed description of the specific configuration of the multi-scale layering module 10 is explained as follows: As mentioned above, the original massive spatial vector data is preprocessed and layered at multiple scales to obtain a multi-scale layered dataset. The multi-scale layering module 10 may further include: a preprocessing unit for cleaning and format standardizing the original massive spatial vector data to obtain a standardized massive spatial vector dataset; and a layering simplification processing unit for performing layering simplification processing on the standardized massive spatial vector dataset based on a preset precision grading threshold to generate the multi-scale layered dataset containing multiple levels from low precision to high precision. Among these, the low-level data is a simplified subset of the high-level data. During the layering simplification process, a correlation relationship is established for the same geographic element in different levels.
[0062] The detailed configuration of the improved quadtree spatial index construction module 20 is explained below: As mentioned above, based on the base layer data in the multi-scale hierarchical dataset, an improved quadtree spatial index is constructed to divide the geographic space covered by the original data into multiple data blocks. The improved quadtree spatial index construction module 20 may further include: a quadtree root node construction unit for constructing a quadtree root node based on the geographic range corresponding to the base layer data; a child node partitioning unit for calculating a dynamic partitioning threshold, recursively judging the amount of data in the current node of the quadtree root node, if it is greater than the dynamic partitioning threshold, then child node partitioning is performed, if it is less than or equal to the dynamic partitioning threshold, then the current node is marked as a data block leaf node storing data; and multiple data block partitioning units for dividing the geographic space covered by the original data into the multiple data blocks based on the data block leaf nodes; wherein, the dynamic partitioning threshold is calculated based on the total data size, geographic area, and average data density.
[0063] The improved quadtree spatial index construction module 20 may further include: in the process of constructing the improved quadtree spatial index, it also includes performing an empty node merging operation on the parent node containing multiple empty child nodes.
[0064] The specific configuration of the geohash encoding module 30 is described in detail below: As mentioned above, geohash encoding is performed on the multiple data blocks to establish a geohash encoding mapping table. The geohash encoding module 30 may further include: a first data block extraction unit for extracting a first data block from the multiple data blocks; a first geohash encoding length determination unit for determining the first geohash encoding length according to the precision level of the first data block; a first geohash encoding generation unit for calculating the first spatial location of the first data block and generating a first geohash encoding by combining it with the first geohash encoding length; and a mapping relationship establishment unit for establishing a mapping relationship between the first data block and the first geohash encoding to obtain the geohash encoding mapping table.
[0065] The specific configuration of the adaptive compression module 40 is described in detail below: As mentioned above, based on the geohashing encoding mapping table, adaptive compression is performed on the multi-scale hierarchical data corresponding to each data block to obtain multiple compressed data blocks. The adaptive compression module 40 may further include: an adaptive compression unit used to compress the polygon feature data in the first data block using a first compression algorithm, to compress the line feature and point feature data in the first data block using a second compression algorithm, and to compress the attribute information of the geographic features in the first data block using dictionary encoding to obtain a first compressed data block; and a traversal unit used to traverse the multiple data blocks and repeatedly execute the compression steps of the first compressed data block to obtain the multiple compressed data blocks.
[0066] The system may further include: an incremental update module for responding to a data update request, based on the association between the geohash encoding mapping table and the central elements of the multi-scale hierarchical dataset, locating and updating only the data blocks associated with the changed geographic elements and their corresponding compressed data blocks, and performing incremental updates.
[0067] The system may further include: a multi-level caching mechanism establishment module for establishing a multi-level caching mechanism, the multi-level caching mechanism including: a client cache established based on the least recently used cleanup principle for storing compressed data blocks frequently accessed by users; and a server cache established based on a distributed architecture for storing compressed data blocks frequently accessed globally.
[0068] The massive spatial vector data dynamic storage system provided in this embodiment of the invention can execute the massive spatial vector data dynamic storage method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.
[0069] Although this application makes various references to certain modules in the system according to the embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of this invention.
[0070] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.
Claims
1. A method for dynamically storing massive spatial vector data, characterized in that, The method includes: The original massive spatial vector data is preprocessed and multi-scale layered to obtain a multi-scale layered dataset. Based on the base layer data in the multi-scale hierarchical dataset, an improved quadtree spatial index is constructed to divide the geographic space covered by the original data into multiple data blocks. Geo-hash encoding is performed on the multiple data blocks to establish a geo-hash encoding mapping table; Based on the geohash encoding mapping table, the multi-scale hierarchical data corresponding to each data block is adaptively compressed to obtain multiple compressed data blocks. The multiple compressed data blocks are dynamically stored according to a preset directory structure.
2. The method for dynamic storage of massive spatial vector data as described in claim 1, characterized in that, Preprocessing and multi-scale hierarchical layering of the original massive spatial vector data yields a multi-scale hierarchical dataset, including: The original massive spatial vector data is cleaned and standardized to obtain a standardized massive spatial vector dataset. Based on a preset precision grading threshold, the standardized massive spatial vector dataset is subjected to hierarchical simplification processing to generate the multi-scale hierarchical dataset containing multiple levels from low precision to high precision. Among them, low-level data is a simplified subset of high-level data. In the process of hierarchical simplification, the relationship between the same geographic element in different levels is established.
3. The method for dynamic storage of massive spatial vector data as described in claim 1, characterized in that, Based on the base layer data in the multi-scale hierarchical dataset, an improved quadtree spatial index is constructed, dividing the geographic space covered by the original data into multiple data blocks, including: Construct a quadtree root node based on the geographical range corresponding to the base layer data; Calculate the dynamic partitioning threshold, recursively determine the amount of data in the current node of the root node of the quadtree. If it is greater than the dynamic partitioning threshold, then perform child node partitioning. If it is less than or equal to the dynamic partitioning threshold, then mark the current node as the leaf node of the data block storing data. Based on the leaf nodes of the data blocks, the geographic space covered by the original data is divided into the multiple data blocks; The dynamic partitioning threshold is calculated based on the total data size, geographical area, and average data density.
4. The method for dynamic storage of massive spatial vector data as described in claim 3, characterized in that, The process of constructing the improved quadtree spatial index also includes performing an empty node merging operation on the parent node containing multiple empty child nodes.
5. The method for dynamic storage of massive spatial vector data as described in claim 1, characterized in that, Geohashing is performed on the multiple data blocks to establish a geohashing mapping table, including: Extract the first data block by traversing through the plurality of data blocks; The length of the first geohash code is determined based on the precision level of the first data block; Calculate the first spatial location of the first data block, and combine it with the length of the first geohash code to generate the first geohash code; Establish a mapping relationship between the first data block and the first geohash code to obtain the geohash code mapping table.
6. The method for dynamic storage of massive spatial vector data as described in claim 5, characterized in that, Based on the geohash encoding mapping table, adaptive compression is performed on the multi-scale hierarchical data corresponding to each data block to obtain multiple compressed data blocks, including: The polygon feature data in the first data block is compressed using a first compression algorithm, the line feature and point feature data in the first data block are compressed using a second compression algorithm, and the attribute information of the geographic features in the first data block is compressed using dictionary encoding to obtain the first compressed data block. The compression steps of the first compressed data block are repeatedly executed by traversing the plurality of data blocks to obtain the plurality of compressed data blocks.
7. The method for dynamic storage of massive spatial vector data as described in claim 1, characterized in that, Also includes: In response to a data update request, based on the association between the geohash encoding mapping table and the features in the multi-scale hierarchical dataset, the data blocks associated with the changed geographic features and their corresponding compressed data blocks are located and updated incrementally.
8. The method for dynamic storage of massive spatial vector data as described in claim 1, characterized in that, Also includes: Establish a multi-level caching mechanism, which includes: A client-side cache, built on the Least Recently Used (LRU) principle, is used to store compressed data blocks that are frequently accessed by users. A server-side cache built on a distributed architecture is used to store compressed data blocks that are accessed frequently globally.
9. A dynamic storage system for massive spatial vector data, characterized in that, The system is used to implement the dynamic storage method for massive spatial vector data according to any one of claims 1-8, and the system includes: The multi-scale layering module is used to preprocess and layer the raw massive spatial vector data at multiple scales to obtain a multi-scale layered dataset. An improved quadtree spatial index construction module is used to construct an improved quadtree spatial index based on the base layer data in the multi-scale hierarchical dataset, dividing the geographic space covered by the original data into multiple data blocks; The geo-hash encoding module is used to perform geo-hash encoding on the multiple data blocks and establish a geo-hash encoding mapping table; An adaptive compression module is used to adaptively compress the multi-scale hierarchical data corresponding to each data block according to the geo-hash encoding mapping table to obtain multiple compressed data blocks; The dynamic storage module is used to dynamically store the multiple compressed data blocks according to a preset directory structure.