A high-performance map storage updating method
By employing hierarchical data organization, adaptive compression, and distributed version control technologies, the storage and query efficiency issues of high-precision map data have been resolved, achieving efficient storage, low-latency updates, and fast queries, thereby improving the performance of map data management.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TIANJIN YUNJI AVIONICS INTELLIGENT TECHNOLOGY CO LTD
- Filing Date
- 2026-03-30
- Publication Date
- 2026-06-23
AI Technical Summary
Traditional map storage methods face challenges in managing high-precision map data, including the pressure of massive data storage, the need for real-time updates, and bottlenecks in query performance. Existing technologies struggle to achieve efficient storage, low-latency updates, and fast queries.
We employ hierarchical data organization, adaptive compression algorithms, and distributed version control technology, combined with Z-Order curve encoding, Delta compression, Snappy compression, MVCC mechanism, and spatial-temporal joint indexing, to construct an efficient map storage and update method.
It achieves a 40% reduction in storage overhead for data volumes of hundreds of millions, incremental update latency ≤200ms, version rollback time ≤50ms, spatiotemporal query response time ≤10ms, and improves query performance by more than 10 times.
Smart Images
Figure CN122262243A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of geographic information and data storage technology, specifically a high-performance map storage and update method. Background Technology
[0002] With the rapid development of fields such as autonomous driving and smart cities, the amount of data in high-precision maps is growing exponentially, and traditional map storage methods face the following challenges:
[0003] Massive data storage pressure: Map data contains multi-level information (such as roads, buildings, and dynamic targets), which traditional relational databases struggle to support for efficient storage and retrieval.
[0004] Real-time update requirements: Maps in dynamic environments need to be updated frequently (such as traffic conditions, temporary obstacles), and existing incremental update technologies suffer from problems such as high latency and poor consistency.
[0005] Query performance bottleneck: Complex spatial queries (such as range search and nearest neighbor retrieval) respond slowly with hundreds of millions of data points, affecting real-time decision-making.
[0006] Current technologies, such as distributed storage and spatial indexes (e.g., R-trees, GeoHash), partially alleviate the aforementioned problems, but still suffer from drawbacks such as storage redundancy, update conflicts, and high index maintenance costs. Therefore, a solution that balances efficient storage, low-latency updates, and fast querying is urgently needed. Summary of the Invention
[0007] To address the aforementioned problems, this invention provides the following technical solution: a high-performance map storage and update method, which achieves efficient storage, second-level updates, and millisecond-level queries of map data through hierarchical data organization, adaptive compression algorithms, and distributed version control technology.
[0008] To achieve the above objectives, the present invention provides a high-performance map storage and update method, comprising the following steps:
[0009] S1: Divide the map data into a static layer (roads, buildings) and a dynamic layer (traffic flow, events). The static layer uses columnar storage (such as Parquet), and the dynamic layer uses memory-mapped files. The data is then divided into blocks according to spatial levels, with the block ID = GeoHash(Lat, Lon, Level) ⊕ Timestamp, where Level represents the block granularity (levels 1-12), and Timestamp is the data version identifier.
[0010] S2: For static layer data, spatial coordinates are encoded using Z-Order curves, and Delta compression is used to reduce redundancy. For dynamic layer data, lightweight Snappy compression is used, and consecutive frames are merged based on temporal similarity. The compression ratio is calculated as CR = |data size after compression| / |data size before compression|. Compression is triggered when CR > 0.7.
[0011] S3: Employs an MVCC mechanism, where each data block is associated with a version chain. Update operations generate a new version and append it to the end of the chain, while older versions are marked as read-only. Conflict resolution strategy: If multiple nodes modify the same block simultaneously, consensus is reached through hash voting, prioritizing higher-priority sources (e.g., vehicle sensors > roadside units).
[0012] S4: Construct a spatial-temporal composite index, embedding the time dimension on top of an R* tree, with index keys of (MBR, Tstart, Tend), supporting spatiotemporal range queries. Index partitions are stored on distributed nodes, achieving load balancing through consistent hashing.
[0013] S5: Hot data (such as frequently accessed areas) is cached in the memory pool, and cold data is evicted using the LRU-K strategy. The parallel query engine decomposes complex requests into subtasks, schedules them to be executed on multiple nodes, and returns the aggregated results.
[0014] Experiments have verified that, under the scale of hundreds of millions of data points, the present invention reduces storage overhead by 40%, compression time is <5ms / GB, incremental update latency is ≤200ms, version rollback time is ≤50ms, and spatiotemporal query response time is ≤10ms (P99), which is more than 10 times better than traditional methods.
[0015] Compared with the prior art, the beneficial effects of the present invention are: Attached Figure Description
[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0017] Figure 1 This is a system architecture diagram of the present invention;
[0018] Figure 2 This is a data storage process and strategy diagram for the present invention. Detailed Implementation
[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0020] Example 1
[0021] like Figure 1 As shown, the present invention provides a high-performance map storage and update method, comprising the following steps:
[0022] S1: Divide the map data into static and dynamic layers. The static layer contains data with low change frequency, such as roads and buildings, and uses columnar storage (e.g., Parquet) to improve compression and query efficiency. The dynamic layer contains real-time changing data such as traffic flow and events, and uses memory-mapped files to support low-latency access. The data is divided into blocks according to spatial levels, with the block division rule defined as BlockID=GeoHash(Lat,Lon,Level)⊕Timestamp, where Level represents the block granularity (levels 1-12), and Timestamp is the data version identifier. Spatial and temporal information are fused through an XOR operation to ensure the uniqueness and traceability of data blocks.
[0023] S2: For static layer data, Z-Order curves are used to encode spatial coordinates, mapping multi-dimensional spatial coordinates to one-dimensional values. Delta compression is used to store the difference between adjacent data blocks, reducing redundancy. For dynamic layer data, lightweight Snappy compression is used, and consecutive frames are merged based on temporal similarity. The compression ratio is calculated as CR = |data size after compression| / |data size before compression|. Compression is triggered when CR > 0.7 to balance compression efficiency and computational overhead.
[0024] S3: Employs an MVCC mechanism, where each data block is associated with a version chain. Update operations generate a new version and append it to the end of the chain, while older versions are marked as read-only. Historical version rollback and concurrent read / write are supported. Conflict resolution strategy: If multiple nodes modify the same block simultaneously, consensus is reached through hash voting, prioritizing data from higher-priority sources (e.g., vehicle-mounted sensors have higher priority than roadside units) to ensure data consistency and reliability.
[0025] S4: Construct a spatial-temporal composite index. An R* tree is built with an embedded time dimension, and the index key is (MBR, Tstart, Tend). Queries such as "retrieve map data for a specific spatial region within a specific time period" are supported. Index partitions are stored on distributed nodes, and load balancing is achieved through consistent hashing to avoid single-point bottlenecks.
[0026] S5: Hot data (such as frequently accessed areas) is cached in a memory pool, and cold data is evicted using an LRU-K strategy, balancing access frequency and timeliness. The parallel query engine decomposes complex requests into subtasks, schedules them to multiple nodes for execution, and returns aggregated results, significantly improving query throughput.
[0027] like Figure 2 As shown, the data storage process of this invention is as follows: after data collection, it is processed by hierarchical classification and then enters the static storage engine and the dynamic storage engine respectively. After compression encoding, version control and index construction, it is uniformly provided by the query interface.
[0028] Tests have shown that this invention reduces storage overhead by 40% and compression time by less than 5ms / GB for data volumes of hundreds of millions; incremental update latency is no more than 200ms and version rollback time is no more than 50ms; and spatiotemporal query P99 response time is no more than 10ms, which is more than 10 times faster than traditional methods.
[0029] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0030] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A high-performance map storage and update method, characterized in that, Includes the following steps: S1: Divide the map data into a static layer and a dynamic layer. The static layer uses columnar storage, and the dynamic layer uses memory-mapped files. S2: For the static layer data, spatial coordinates are encoded using Z-Order curves and combined with Delta compression; for the dynamic layer data, lightweight Snappy compression is used, and consecutive frames are merged based on temporal similarity. S3: Employs a multi-version concurrency control mechanism, associating each data block with a version chain. Update operations generate new versions and append them to the end of the chain, while old versions are marked as read-only. S4: Construct a spatial-temporal composite index, where the key of the index contains the minimum bounding rectangle and the time interval, and store the index partitions on distributed nodes.
2. The method according to claim 1, characterized in that, Step S1 also includes dividing the space into blocks according to spatial hierarchy, with the following block division rules: BlockID = GeoHash(Lat, Lon, Level) ⊕ Timestamp Where Level represents the granularity of the data segmentation, and Timestamp is the data version identifier.
3. The method according to claim 1, characterized in that, In step S3, the conflict resolution strategy is as follows: if multiple nodes modify the same block at the same time, consensus is reached based on hash voting, and data from the higher priority source is retained first.
4. The method according to claim 1, characterized in that, In step S4, the spatial-temporal joint index is an RT tree built on top of an R* tree by embedding a time dimension, which is used to accelerate spatiotemporal range queries.
5. The method according to claim 1, characterized in that, It also includes step S5: caching hot data in a memory pool, using the LRU-K strategy to evict cold data, and using a parallel query engine to decompose complex requests into subtasks and schedule them to be executed on multiple nodes.
6. The method according to claim 1, characterized in that, In step S2, the compression ratio calculation formula for the dynamic layer data is as follows: CR = |Data size after compression| / |Data size before compression| Compression is triggered when the compression ratio CR > 0.7.