Vector tile cutting and publishing system

Abstract

The invention relates to the field of geographic information system data processing, in particular to a vector tile cutting and publishing system which comprises a heterogeneous data self-adaptive access and normalization subsystem used for receiving multi-mode heterogeneous input; the spatial-temporal feature analysis and index construction middleware is used for generating a metadata index reflecting data spatial distribution features; the vector tile streaming production engine drives a tile generation process based on the metadata index, converts the intermediate state data into a vector tile data packet by adopting a parallel computing architecture, and writes the vector tile data packet into a tile storage warehouse; and the standardized service publishing bus is used for monitoring the change state of the tile storage warehouse in real time and externally providing a vector tile network service interface which accords with a standard protocol. According to the method, the structure barriers of heterogeneous data sources are eliminated, the stability of data production is ensured through topology self-healing, the defect that one set of rules cut all is overcome through a density-based parameter inversion mechanism, and the tile size and the visualization precision are balanced.

Description

Technical Field

[0001] This application relates to the field of geographic information system data processing, specifically to a vector tile cutting and publishing system. Background Technology

[0002] Geographic Information Systems (GIS), as a core engineering platform for processing spatial data, are widely used in urban planning, real-time traffic navigation, and complex spatial analysis. With the development of internet map service technology, vector tile technology, due to its significant advantages in client-side rendering flexibility, interactivity, and network transmission efficiency, has gradually replaced traditional raster tiles and become the mainstream technical standard for online map publishing. In engineering practice, vector tile processing and publishing systems typically include key steps such as the access of multi-source heterogeneous spatial data, coordinate projection transformation, geometric simplification, tile encoding, and network service distribution.

[0003] Existing technologies still have significant limitations in practical large-scale geographic information application scenarios. First, when faced with multi-source heterogeneous raw geographic data (such as Shapefile, GeoJSON, PostGIS database, etc.), existing systems usually lack efficient automated cleaning and topology consistency maintenance mechanisms. Insufficient compatibility of heterogeneous data sources leads to lengthy preprocessing links, which significantly increases the time cost of data access.

[0004] Secondly, traditional tile cutting processes largely rely on static, pre-defined configuration files, such as uniform simplified tolerances or buffer sizes. They cannot dynamically adjust cutting rules based on the density of spatial data distribution, resulting in excessively large tiles in densely populated areas that impact network transmission efficiency, while wasting computing resources in sparsely populated areas. Furthermore, existing deployment architectures often employ a strictly sequential model of data processing and service deployment, or over-rely on the real-time computing capabilities of databases under large-scale concurrent requests. This leads to high latency from data changes to service effectiveness, and makes it difficult to guarantee service scalability and response stability under high load environments. Summary of the Invention

[0005] The purpose of this application is to provide a vector tile cutting and publishing system, comprising: a heterogeneous data adaptive access and normalization subsystem, used to receive multimodal heterogeneous inputs from spatial databases, file-type data, and streaming data sources, with a built-in multimodal data cleaning engine and a unified intermediate format conversion unit, dynamically mapping nested attributes to a columnar storage structure through a streaming flattening mapping algorithm, performing coordinate system standardization, and outputting topologically healthy and structurally unified intermediate spatiotemporal data; a spatiotemporal feature analysis and index building middleware, connected to the heterogeneous data adaptive access and normalization subsystem, used to build a spatiotemporal quadtree index on the intermediate data in memory, perform spatial aggregation statistics on the full data, and generate a metadata index reflecting the spatial distribution characteristics of the data; a vector tile streaming production engine, which drives the tile generation process based on the metadata index, uses a parallel computing architecture to convert intermediate data into vector tile data packets, and writes them to a tile storage warehouse; and a standardized service publishing bus, used to monitor the change status of the tile storage warehouse in real time and provide a vector tile network service interface conforming to standard protocols.

[0006] Optionally, the spatiotemporal feature analysis and index construction middleware further includes a dynamic slicing rule adaptive generation module. This module performs feature density analysis based on a columnar analysis engine, specifically including: spatial sampling at the leaf node level of the quadtree index to calculate the number of nodes, line segment curvature, and polygon overlap in each grid region, thereby generating spatial feature density distribution entropy; establishing a density-parameter mapping model to invert the spatial feature density distribution entropy into tile cutting parameters; automatically increasing the simplification tolerance and enabling the on-demand discard strategy for dense regions with high entropy values; and automatically increasing the detail precision for sparse regions with low entropy values, generating partitioned heterogeneous slicing configuration files to drive the vector tile streaming production engine to perform differentiated processing.

[0007] Optionally, the multimodal data cleaning engine further includes a topology self-healing and geometry repair unit, configured to: automatically detect the topological integrity of geometric features during the data access phase, identify self-intersection, dangling nodes, and non-closed polygon errors; and reconstruct or remove abnormal geometric objects using a preset geometry repair algorithm library to ensure that the data input to the production engine meets the geometric specifications of vector tile encoding, and prevent rendering artifacts or production process interruptions caused by topology errors.

[0008] Optionally, the heterogeneous data adaptive access and normalization subsystem further includes a streaming attribute structure flattening unit, configured to: perform recursive attribute scanning for complex geographic entity objects including multi-layer nested structures; dynamically flatten the nested key-value pair structure into a single-layer attribute table, and standardize the field names to eliminate attribute structure differences between different data sources, providing a consistent data view for subsequent columnar storage analysis.

[0009] Optionally, the vector tile streaming production engine executes a geofence-based partitioning and parallel slicing mechanism: based on the tile configuration file output by the adaptive generation module according to the dynamic slicing rules, the full data is divided into three logical processing units: high-density area, medium-density area, and low-density area; independent computing resources are allocated to execute the tile generation tasks of each logical processing unit in parallel; finally, the tile data packets of different density partitions are aggregated into a single tile dataset through a tile merging tool, achieving a balance between data quality and volume across the entire domain.

[0010] Optionally, the standardized service publishing bus includes: a viewport-aware clipping unit, configured to parse the protocol buffer data stream in the tile data packet in real time when the server receives a client request, remove visual noise elements according to the client's viewport parameters, and repackage the lightweight data packet for return; and an incremental hot update notification mechanism, configured to trigger the server's memory index refresh through a semaphore mechanism when the production engine generates new tile data, and calculate the differential data packet of the changed elements, supporting the client to perform local rendering updates only for the changed area.

[0011] Optionally, the viewport-aware clipping unit has a built-in end-side capability adaptation filter configured to: parse the client device type identifier and device pixel ratio parameter in the HTTP request header; calculate the projected area of ​​geometric elements on the screen at the current zoom level based on the effective visual information content algorithm; and filter out polygonal elements with a projected area less than a preset pixel threshold or line segment elements with a length less than a preset pixel threshold in real time, thereby achieving real-time compression of the transmission load for mobile devices.

[0012] Optionally, the incremental hot update notification mechanism further includes a differential tile calculation engine: configured to cache fingerprint features of the previous version of tile data; when a change in source data is detected that causes tile regeneration, the binary encoding differences between the old and new tile data are compared; the changed geometric features and attribute information are extracted, an independent differential vector tile data packet is generated, and a response message including an update instruction is sent to the client, instructing the client to merge the local cache and the differential data.

[0013] Optionally, the spatiotemporal feature analysis and index building middleware uses a memory-resident columnar analysis database as its core component, configured to: perform high-speed memory-level scanning and statistics on the intermediate data accessed via streaming without persisting it to disk; and serve as a read / write buffer between the production engine and the data source, using a vectorized execution engine to accelerate the aggregation query of large-scale spatial data and reduce disk input / output overhead.

[0014] Optionally, it also includes a global resource adaptive scheduling module, configured to: predict the computational load of tile production in different regions based on the density heatmap generated by the spatiotemporal quadtree index; dynamically adjust the process concurrency and memory quota of the vector tile streaming production engine, prioritize the supply of computational resources in high-density regions, and automatically trigger the load balancing strategy of the viewport-aware pruning unit when the server detects high-concurrency requests.

[0015] This application introduces a spatiotemporal feature analysis and index construction middleware. By building a quadtree index in memory and performing feature density analysis, it quantifies and calculates the entropy of spatial feature density distribution. Based on this, a density-parameter mapping model is established, generating heterogeneous tile configuration files for different partitions. This drives the production engine to perform differentiated parallel cutting on regions with different densities. On the service publishing side, a viewport-aware load optimization and incremental update module is designed. By parsing client viewport parameters and device type in real time, it dynamically removes visual noise elements and compresses the load. Combined with a differential tile calculation engine, it transmits only differential packets of changed data, achieving efficient updates under low bandwidth. This eliminates the structural barriers of heterogeneous data sources, ensures the stability of data production through topology self-healing, and solves the drawback of "one set of rules for all" by using a density-based parameter inversion mechanism. It balances tile volume and visualization accuracy, and the viewport-aware pruning and incremental update mechanism significantly reduces network transmission overhead and improves mobile rendering performance and service response speed. Attached Figure Description

[0016] Figure 1 This is a block diagram of the vector tile cutting and publishing system of this application. Detailed Implementation

[0017] like Figure 1 As shown in the figure, this application provides a vector tile cutting and publishing system, including a heterogeneous data adaptive access and normalization subsystem, a spatiotemporal feature analysis and index construction middleware, a vector tile streaming production engine, and a standardized service publishing bus.

[0018] Understandably, this vector tile cutting and publishing system aims to address the bottleneck of map publishing delays caused by the low processing efficiency of heterogeneous data sources. The heterogeneous data adaptive access and normalization subsystem receives multimodal heterogeneous inputs from spatial databases, file-based data, and streaming data sources. It incorporates a multimodal data cleaning engine and a unified intermediate format conversion unit, dynamically mapping nested attributes to a columnar storage structure through a streaming flattening mapping algorithm, and performing coordinate system standardization to output topologically healthy and structurally uniform intermediate spatiotemporal data. The spatiotemporal feature analysis and index construction middleware adapts to the heterogeneous data... The system should be connected to the normalization subsystem to build a spatiotemporal quadtree index on intermediate data in memory, perform spatial aggregation statistics on the full data, and generate a metadata index reflecting the spatial distribution characteristics of the data. A vector tile streaming production engine drives the tile generation process based on the metadata index, uses a parallel computing architecture to convert intermediate data into vector tile data packets, and writes them to the tile storage warehouse. A standardized service publishing bus is used to monitor the changes in the tile storage warehouse in real time and provide a vector tile network service interface conforming to standard protocols. The specific implementation details of this embodiment are as follows.

[0019] Specifically, in the engineering implementation of the heterogeneous data adaptive access and normalization subsystem, the physical access layer of the system can be designed as a throughput architecture supporting multi-protocol concurrency, capable of simultaneously processing real-time data streams from smart city IoT sensing devices and static basic geographic data from surveying and mapping departments. Static data sources mainly include urban building outline data, road network vector data, and underground pipeline topology data stored in PostgreSQL or PostGIS relational databases. These data typically have high-precision double-precision floating-point coordinate fields, with field types covering WKB binary geometric format and GeoJSON text format, and data volumes usually in the TB range. Streaming data sources originate from traffic flow monitoring cameras deployed on urban main roads, GNSS global navigation satellite system positioning terminals installed on buses and taxis, and meteorological sensors at environmental monitoring stations. These devices push status update data packets to the system at a frequency of tens of thousands of messages per second via MQTT message queues or Kafka distributed stream processing platforms. To cope with this multi-source heterogeneous input environment, the multimodal data cleaning engine incorporates a deeply customized transformation kernel based on the GDAL geospatial data abstraction library. This transformation kernel is configured to automatically identify the encoding format of the input data, for ESRI... Shapefile format files are automatically read by the multimodal data cleaning engine from .shp, .shx, and .shp files.The engine processes DBF files and parses their binary headers to extract geometric types and attribute field definitions. For GeoJSON data with complex nested structures, the multimodal data cleaning engine can enable a fast parser based on the SIMD single-instruction multiple-data-stream instruction set to flatten nested JSON objects into a columnar memory layout. In the coordinate system standardization stage, the heterogeneous data adaptive access and normalization subsystem incorporates an EPSG geodetic parameter database, automatically converting input CGCS2000 national geodetic coordinate system, WGS84 global satellite positioning coordinate system, or local independent coordinate system into Web coordinates. The Mercator projected coordinate system is EPSG:3857. The conversion process involves high-order polynomial fitting and datum transformation operations to ensure that the accuracy error of the converted coordinates is controlled within the centimeter range. The unified intermediate format conversion unit uses an extended GeoJSONSeq serialization format as its internal data exchange standard. This serialization format removes the curly brace closure structure of traditional GeoJSON and uses newlines to separate independent geographic feature objects, enabling parallel processing in a streaming, line-by-line manner. This completely avoids the risk of memory overflow caused by loading the entire large file into memory at once. Simultaneously, the streaming flattening mapping algorithm synchronously builds a type mapping table for attribute fields during processing. It maps variable-length string types in the original data to fixed-length UTF-8 encoded byte streams, uniformly maps dynamic precision numeric types to 64-bit double-precision floating-point numbers or 32-bit signed integers, and adds unit metadata descriptions to each field, such as labeling the velocity field as meters per second and the temperature field as degrees Celsius. This outputs topologically healthy and structurally uniform intermediate spatiotemporal data, laying a standardized data foundation for subsequent efficient processing.

[0020] In this embodiment, the spatiotemporal feature analysis and index construction middleware further includes a dynamic slicing rule adaptive generation module. This module performs feature density analysis based on a columnar analysis engine, specifically including: spatial sampling at the leaf node level of the quadtree index to calculate the number of nodes, line segment curvature, and polygon overlap in each grid region, thereby generating spatial feature density distribution entropy; establishing a density-parameter mapping model to invert the spatial feature density distribution entropy into tile cutting parameters; automatically increasing the simplification tolerance and enabling the on-demand discard strategy for dense regions with high entropy values; automatically increasing the detail precision for sparse regions with low entropy values; and generating partitioned heterogeneous slicing configuration files to drive the vector tile streaming production engine to perform differentiated processing.

[0021] Specifically, upon receiving the normalized intermediate data, the spatiotemporal feature analysis and index construction middleware immediately initiates a memory-based computation process. This process first initializes a virtual quadtree structure with a maximum depth of 22 levels in random access memory, corresponding to the highest zoom level in the map service. The dynamic tile rule adaptive generation module utilizes the high-speed scanning capability of the columnar analysis engine to traverse the spatial bounding boxes of all geographic features, mapping them to the leaf nodes of the quadtree. In the specific feature density analysis calculation, the system does not simply count the number of features, but executes a complex entropy calculation algorithm. The entropy calculation algorithm first calculates the total number of geometric vertices within each grid as a basic density index. Then, it iterates through all linear features within the grid, extracting the curvature of each line segment using the discrete point curvature calculation formula, and marking feature points with curvature greater than a threshold as key topological points. Simultaneously, for planar features, the algorithm calculates the overlap area ratio between polygons to assess the severity of visual occlusion. The system performs weighted normalization on the data from the three dimensions of vertex density, average curvature, and overlap to generate the spatial feature density distribution entropy value for the grid region. This spatial feature density distribution entropy value is a floating-point number between 0 and 1. A higher entropy value indicates denser geographic information and greater visual complexity in the area. Based on this spatial feature density distribution entropy value, the system calls a preset density-parameter mapping model. This model is a multidimensional lookup table stored in a hash table, defining the optimal combination of tiling parameters for different entropy value ranges. For example, when the entropy value of a grid cell located in the city center business district exceeds 0.8, the density-parameter mapping model automatically matches a set of high compression ratio parameters, sets the distance tolerance threshold of the Douglas-Puk simplification algorithm to 2.5 pixels, and enables a small feature removal strategy, automatically discarding screen projection areas smaller than 4 square pixels. For small polygons, the density-parameter mapping model simplifies the tolerance to 0.5 pixels while preserving all geometric details. This partitioned heterogeneous slicing configuration file is ultimately generated as a series of JSON-formatted instruction sets, each including specific spatial range coordinates and corresponding processing parameters. These instruction sets are precisely distributed to different worker threads of the production engine, thereby achieving on-demand allocation of computing resources and intelligent balance of output data quality. This effectively solves the engineering problem of "excessively large core area data loading slowly and rudimentary edge area data wasting details" caused by traditional globally unified parameter slicing.

[0022] Understandably, the multimodal data cleaning engine further includes a topology self-healing and geometry repair unit, configured to: automatically detect the topological integrity of geometric features during the data access phase, identify self-intersection, dangling nodes, and non-closed polygon errors; and reconstruct or remove abnormal geometric objects using a pre-defined geometry repair algorithm library to ensure that the data input to the production engine meets the geometric specifications of vector tile encoding, preventing rendering artifacts or production process interruptions caused by topology errors.

[0023] Specifically, the topology self-healing and geometry repair unit operates at the entry point of the data access memory buffer. When the original vector data is read in, the unit first initiates a self-intersection detection program based on the scan line algorithm. This program sorts all line segment endpoints by Y-axis coordinate and uses a balanced binary tree to maintain the intersection status of the current scan line and the geometry, thus quickly locating the abnormal points of self-intersection of polygon boundaries within logarithmic time complexity. Once a self-intersection error is detected, the system immediately calls the buffer reconstruction function in the geometry repair algorithm library to perform a buffer expansion operation with zero distance to the polygon. Mathematically, this operation can decompose complex self-intersecting polygons into a set of several valid simple polygons, thereby eliminating topological singularities. For dangling node errors, i.e., isolated breaks formed when the endpoints of linear features are not connected to other line segments, the unit uses a nearest neighbor search algorithm based on a KD tree to find the nearest valid node within a preset capture radius (e.g., 0.001 meters). If found, it automatically executes... The node snapping operation merges dangling points with neighboring nodes through displacement. If no such node is found and the line segment length is below a preset noise threshold, the line segment is directly discarded as data noise. For non-closed polygon errors, i.e., polygonal features where the first and last coordinates do not coincide, the repair unit automatically calculates the Euclidean distance between the first and last points. If the distance is within the allowable error range, a closed point is forcibly added; if the distance is too large, it is downgraded to a linear feature for processing. In addition, the topology self-healing and geometry repair unit is also responsible for detecting and repairing non-numerical anomalies and infinity anomalies in coordinate values, ensuring that all geometric coordinates are valid floating-point numbers. After the above series of automated detection and repair steps, the system records the original ID, error type, and repair measures of each modified data in the log database, forming a traceable data quality report. Data marked as "topology healthy" is seamlessly pushed to subsequent production stages, thereby preventing serious engineering accidents caused by individual dirty data leading to the collapse of the entire slicing process and ensuring the stable operation of the system 24 / 7 in an unattended state.

[0024] Understandably, the heterogeneous data adaptive access and normalization subsystem also includes a streaming attribute structure flattening unit, configured to: perform recursive attribute scanning for complex geographic entity objects with multi-layered nested structures; dynamically flatten the nested key-value pair structure into a single-layer attribute table; and standardize the field names to eliminate attribute structure differences between different data sources, providing a consistent data view for subsequent columnar storage analysis.

[0025] Specifically, the streaming attribute structure flattening unit uses a depth-first search strategy to traverse the input JSON object tree or XML document tree. In actual urban planning, a building object often includes deeply nested attributes; for example, "Building Attributes" includes "Structural Information," which in turn includes "Floor List," and each floor includes "Unit Distribution," etc. Traditional GIS software often loses deep information or stores it as large blocks of text when reading such data, which is not conducive to querying and analysis. This streaming attribute structure flattening unit maintains a dynamic key name stack during runtime. When the traverser delves into the nesting level, it concatenates the key name of the parent node with the key name of the current node using an underscore connector to generate a flattened, globally unique field name such as "Building Attributes_Structural Information_Number of Floors Above Ground." Simultaneously, this streaming attribute structure flattening unit incorporates intelligent type inference logic; if the array is being traversed... When an object is found to contain only numeric elements, it is converted into a specific statistical field or serialized into a comma-separated string for front-end parsing. To eliminate semantic ambiguity between different data sources, this streaming attribute structure flattening unit also mounts a field mapping engine based on a semantic dictionary. This engine predefines industry-standard field sets, such as mapping fields with different names like "build_h", "height", and "zg" in the source data to the standard field "building_height", and converting the unit to meters. For extended attributes that cannot be automatically mapped, the system retains them in a separate "extra_props" JSONB column to balance data integrity and structural standardization. After flattening, the attribute table structure fully conforms to the normalization requirements of relational databases and can be efficiently stored and retrieved by columnar in-memory formats such as Apache Arrow, greatly accelerating the calculation speed of attribute filtering and style matching in subsequent slicing processes.

[0026] Understandably, the vector tile streaming production engine executes a geofence-based partitioning and parallel slicing mechanism: based on the tile configuration file output by the adaptive generation module according to the dynamic slicing rules, the full data is divided into three logical processing units: high-density, medium-density, and low-density areas; independent computing resources are allocated to execute the tile generation tasks of each logical processing unit in parallel; and finally, the tile data packets of different density partitions are aggregated into a single tile dataset through a tile merging tool, achieving a balance between data quality and volume across the entire domain.

[0027] Specifically, the vector tile streaming production engine is deployed on a computing cluster composed of multiple high-performance servers, employing a master-slave distributed architecture. Upon receiving the tile configuration file, including density partitions, generated by the dynamic tile rule adaptive generation module, the master node first starts the task scheduler, using a geofencing algorithm to divide the entire map into several non-overlapping spatial task packages. Specifically, for urban center areas marked as "high-density areas," the system allocates the most powerful and memory-rich nodes (e.g., servers with 128 CPU cores and 512GB of memory) for processing, and starts more concurrent threads, each responsible for a smaller sub-tile area, to cope with the computational pressure of simplifying and encoding massive amounts of elements. For suburban areas marked as "low-density areas," the system packages and assigns them to edge nodes with weaker computing power or utilizes idle computing power for processing. During parallel cutting, each worker node runs a high-performance tile core written in C++. This core uses the underlying algorithms of the Tippecanoe library to strictly process intermediate data according to the received partitioning parameters (e.g., specific simplification tolerances and discarding strategies) to generate Protocols. The vector tile files are in Buffers format (.pbf). To address potential geometric breaks across partition boundaries, the vector tile streaming engine automatically sets pixel-level buffers at the boundaries of each partition. During computation, it reads data from adjacent regions to participate in topology construction, ensuring perfect connection between line segments and polygons at cutting lines. All tile files generated by child nodes are ultimately transmitted to the aggregation node via a high-speed internal network. The aggregation node runs the SQLite-based MBTiles merging tool, which uses a transaction mechanism to batch insert scattered tile data records into the final tile dataset file and build a tile index. This partitioned parallel mechanism cleverly utilizes the spatial locality principle of geographic data, decomposing the original full-scale tiling task, which would take several hours, into parallelizable micro-tasks. In actual testing, it reduced the processing time for the same scale of data by more than 70%, and the size of the generated tile packet was reduced by 40% while maintaining visual quality.

[0028] Understandably, the standardized service publishing bus is used to address the scalability issues of published services, including: a viewport-aware clipping unit, configured to parse the protocol buffer data stream in the tile data packet in real time when the server receives a client request, remove visual noise features according to the client's viewport parameters, and repackage the lightweight data packet for return; and an incremental hot update notification mechanism, configured to trigger the server's memory index refresh through a semaphore mechanism when the production engine generates new tile data, and calculate the differential data packet for the changed features, supporting the client to perform local rendering updates only for the changed area.

[0029] Specifically, the standardized service publishing bus serves as a bridge connecting backend data production and frontend user experience. Its viewport-aware clipping unit and incremental hot update notification mechanism completely transform the passive mode of traditional static file services. In the viewport-aware clipping unit's operational logic, when the server receives an HTTP GET request from the client, in addition to the standard XYZ tile coordinate parameters, it also parses the viewport bounding box parameters and the current camera's tilt and rotation angles carried in the request header. The server does not directly return the original tile file on disk; instead, it first loads the tile's ProtocolBuffers binary stream into memory for rapid deserialization. Subsequently, based on the viewport bounding box, it calculates the precise geographical range visible to the user's screen and uses geometric intersection operations to eliminate elements that, although belonging to the tile, are completely outside the user's field of view. For example, in a tilted 3D map browsing scene, the background elements of distant tiles are invisible, and transmitting this data is a waste of bandwidth. This viewport-aware clipping unit can complete this instant clipping operation within milliseconds and reserialize the remaining valid elements into smaller P... BF data packets are returned to the client; simultaneously, the incremental hot update notification mechanism maintains a long-lived WebSocket connection channel for real-time signaling interaction between the server and online clients; when the backend production engine completes the data update for a certain area (e.g., a road that is currently congested), it sends a semaphore containing the update range and version number to the publishing bus; the standardized service publishing bus immediately finds all online client connections that the current viewport covers in that area and pushes a lightweight "dirty tile" notification message through the WebSocket channel; after receiving the message, the client does not re-download the entire map, but only initiates an incremental request for a specific version for the affected tiles, which greatly reduces network traffic consumption in real-time map applications and makes it possible to achieve second-level data updates in mobile network environments.

[0030] Understandably, the viewport-aware clipping unit has a built-in end-side capability adaptation filter configured to: parse the client device type identifier and device pixel ratio parameter in the HTTP request header; calculate the projected area of ​​geometric elements on the screen at the current zoom level based on the effective visual information content algorithm; and filter out polygonal elements with a projected area smaller than a preset pixel threshold or line segment elements with a length smaller than a preset pixel threshold in real time, thereby achieving real-time compression of the transmission load for mobile devices.

[0031] Specifically, the edge capability adaptation filter is an intelligent decision-making component within the viewport-aware clipping unit, used to address the issue of balancing display quality and performance of the same map data on devices with different performance levels. When a request arrives, this edge capability adaptation filter first checks the HTTP User-Agent field and custom device pixel ratio and device performance level header information. If it identifies a request from a low-end smartphone with limited performance, such as a device performance level marked as "low" and a pixel ratio of 2.0, the system automatically activates an aggressive filtering strategy. This filtering strategy employs an effective visual information content (AVIP) algorithm, which simulates the client's rendering projection process on the server side, converting the geographic coordinates of each vector feature within the tile into screen pixel coordinates. Next, the AVIP algorithm calculates the number of pixels each polygon feature covers on the screen. If this value is less than a preset threshold, for example, 16 physical pixels (4x4 squares) on a low-end device, then the vector feature is considered to have extremely low visual contribution to the user's visual perception and falls under the category of "visually limited." "Perceived noise" should be eliminated; similarly, for linear features, if their projection length on the screen is less than 2 pixels, they should be ignored. In this way, the system can provide high-end PC workstations with complete data including all details, while providing low-end mobile devices with significantly reduced data. Both use the same backend data source, and differentiated distribution is achieved entirely through real-time calculations on the server side. Experimental data shows that in emergency command and dispatch scenarios, frontline personnel often use handheld terminals to work in weak network environments. After enabling this edge capability adaptation filter, the average loading time of map tiles was reduced from 3 seconds to less than 0.5 seconds, which greatly improved the system's actual response speed and effectively avoided the problem of mobile GPU overheating and frequency reduction caused by loading too many small features.

[0032] Understandably, the incremental hot update notification mechanism further includes a differential tile calculation engine: configured to cache fingerprint features of the previous version of tile data; when a change in source data is detected that causes tile regeneration, it compares the binary encoding differences between the old and new tile data; extracts the changed geometric features and attribute information, generates independent differential vector tile data packets, and sends a response message including update instructions to the client, instructing the client to merge the local cache with the differential data.

[0033] Specifically, the differential tile calculation engine is used to implement real-time dynamic maps in low-bandwidth environments. This engine maintains a fingerprint database of recently accessed tiles in server memory, where each fingerprint is a SHA-256 hash value calculated based on the tile's binary content. When the upstream streaming engine regenerates a tile (e.g., version V2) due to receiving new traffic flow data, the differential tile calculation engine first calculates the hash value of the new tile and compares it with the hash value of the old version (V1) in the cache. If the hash values ​​differ, it indicates a substantial change in content. In this case, the differential tile calculation engine does not directly overwrite the old file but instead initiates a differential extraction program. This program decodes both V1 and V2 PBF files, comparing layers and features one by one. The system uses the unique identifier of each feature as an anchor point to quickly identify newly added features, deleted features, and features with changed attributes or geometry. For changed features, the system... Only the changed state is extracted, such as the color attribute of a road changing from "green" (smooth traffic) to "red" (congested), or the location coordinates of a vehicle shifting. This change information is encapsulated into a custom "differential tile format," which only includes operation instructions (such as add, delete, update) and necessary data payloads, and its volume is usually only 1% to 5% of the total number of tiles. After generating the differential packet, the system pushes it to the client via HTTP response or WebSocket message. The client's map rendering engine has built-in differential merging logic, which can apply these differential data to the locally cached tile data structure during the idle period of the GPU rendering cycle, thereby achieving a smooth transition update of the image. This mechanism has extremely high engineering value in scenarios such as real-time updates of satellite remote sensing images or large-scale vehicle location tracking, making it possible to maintain real-time visualization of millions of dynamic elements across the city under limited 4G / 5G bandwidth resources.

[0034] Understandably, the spatiotemporal feature analysis and index building middleware uses a memory-resident columnar analysis database as its core component, configured to: perform high-speed memory-level scanning and statistics on intermediate data that is streamed directly without persisting it to disk; and serve as a read / write buffer between the production engine and the data source, using a vectorized execution engine to accelerate aggregation queries of large-scale spatial data and reduce disk input / output overhead.

[0035] Specifically, the memory-resident columnar analytics database uses a heavily modified DuckDB as its kernel. This database is embedded in the system's Java or Go main process, sharing memory space with the application and completely eliminating the serialization and network overhead associated with traditional databases communicating via JDBC / ODBC protocols. At system startup, the database is configured in pure memory mode, with the operating system allocating it a large page memory pool of tens of gigabytes or even terabytes. When heterogeneous data adaptively accesses and the normalization subsystem outputs intermediate data streams, this data is no longer written to temporary files on the hard drive, but is directly transmitted via Apache. Arrow-formatted memory pointer transfer maps to tables in the memory-resident columnar analysis database. This columnar storage structure stores data of the same attribute column contiguously, greatly improving CPU cache hit rate and allowing vectorized operations using the AVX-512 instruction set of modern CPUs for single-instruction multiple-data streams. For example, when performing global data density analysis, the system needs to statistically analyze the spatial distribution of hundreds of millions of points. Traditional row-based databases require reading and decoding row by row, while the columnar engine of this system can directly perform batch interval judgment and aggregation counting on coordinate columns, improving calculation speed by two orders of magnitude. In addition, this memory-resident columnar analysis database also acts as an intelligent read / write buffer. When the production engine needs to read data for tile cutting, it can use predicate pushdown technology to extract only the subset of data that fits the current tile spatial range and pass it to the calculation thread, avoiding a full table scan. Although the data mainly resides in memory, the system also has an asynchronous checkpointing mechanism that periodically dumps memory snapshots to high-speed NVMe in a low-priority background thread. On SSD hard drives, data is protected from complete loss due to power outages. However, during normal streaming job cycles, disk I / O operations are reduced to near zero, thus eliminating the IOPS bottleneck, which is the most common constraint in large-scale GIS data processing.

[0036] Understandably, the vector tile cutting and publishing system also includes a global resource adaptive scheduling module, configured to: predict the computational load of tile production in different regions based on the density heatmap generated by the spatiotemporal quadtree index; dynamically adjust the process concurrency and memory quota of the vector tile streaming production engine, prioritize the supply of computational resources to high-density regions, and automatically trigger the load balancing strategy of the viewport-aware pruning unit when the server detects high-concurrency requests.

[0037] Specifically, the global resource adaptive scheduling module is a secondary development based on the Kubernetes container orchestration platform, integrating prediction algorithms specifically designed for geospatial computing tasks. This module continuously monitors the real-time density heatmap generated by the spatiotemporal feature analysis middleware. This heatmap not only reflects the static distribution of data but also overlays the current system's user access intensity. Using time series prediction models (such as ARIMA or LSTM neural networks), it predicts the computational pressure of each spatial partition over a future period based on the load trend over the past 15 minutes. When it predicts that a certain area (such as the city center where a sudden event is occurring) is about to experience a data update peak or a surge in user access, the scheduler will invoke Docker... The API dynamically expands container instances on the corresponding compute node groups in advance to increase the concurrent processes of the vector tile streaming production engine, and temporarily increases the CPU weight and memory hard limit of these containers to ensure that processing tasks in high-density areas can obtain exclusive computing resources and avoid queuing delays caused by resource contention. Similarly, for low-load nighttime periods or sparse areas, the scheduler automatically reclaims excess container instances to release resources and save cloud server costs. On the service publishing side, when the load balancer detects that the QPS (queries per second) of tile requests sent to a specific area exceeds the warning threshold, the global resource adaptive scheduling module sends a degradation instruction to the service instances in that area, automatically triggering the viewport-aware pruning unit to enable a more aggressive load balancing strategy, such as temporarily reducing the precision level of the returned data or enabling a powerful cache reuse mode, sacrificing a small amount of real-time performance for the overall availability of the system. This allows the system to cope with various load challenges from daily stable operation to extreme emergency scenarios, ensuring that the GIS service is uninterrupted, undamaged, and uninterrupted during critical moments.

[0038] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0039] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0040] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0041] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is only one example of a series of equivalent or similar features.

Claims

1. A vector tile cut publishing system, characterized by, Comprise: A heterogeneous data adaptive access and normalization subsystem for receiving multi-modal heterogeneous input from spatial databases, file-type data, and streaming data sources, with a multi-modal data cleaning engine and a uniform intermediate format conversion unit, dynamically mapping nested attributes to columnar storage structures through a streaming flattening mapping algorithm, and performing coordinate system standardization, outputting topologically healthy and structurally uniform intermediate state spatio-temporal data; A spatio-temporal feature analysis and index construction middleware connected to the heterogeneous data adaptive access and normalization subsystem for constructing a spatio-temporal quadtree index in memory for intermediate state data, performing spatial aggregation statistics on full data, and generating metadata indexes reflecting the spatial distribution characteristics of the data; A vector tile streaming production engine driven by the metadata indexes to generate tiles, using a parallel computing architecture to convert intermediate state data into vector tile data packets, and writing them to a tile storage warehouse; A standardized service publishing bus for real-time monitoring of changes in the tile storage warehouse and providing a vector tile network service interface conforming to standard protocols.

2. The vector tile cut publishing system of claim 1, wherein, The spatio-temporal feature analysis and index construction middleware further comprises a dynamic slicing rule adaptive generation module that performs feature density analysis based on a columnar analysis engine, specifically including: By spatial sampling at the leaf node level of the quadtree index, calculating the number of nodes, line curvature, and polygon overlap in each grid region, and generating a spatial feature density distribution entropy; Establishing a density-parameter mapping model to invert the spatial feature density distribution entropy into tile cutting parameters, automatically adjusting the simplification tolerance and enabling on-demand discard strategy for dense areas with high entropy values, and automatically adjusting the detail precision for sparse areas with low entropy values, generating a partitioned heterogeneous slicing configuration file to drive the vector tile streaming production engine to perform differential processing.

3. The vector tile cut publishing system of claim 2, wherein, The multi-modal data cleaning engine further comprises a topology self-healing and geometry repair unit configured to: Automatically detect the topological integrity of geometric features during data access, identify self-intersection, hanging node, and non-closed polygon errors; Use a pre-set geometry repair algorithm library to reconstruct or discard abnormal geometric objects, ensuring that the data input to the production engine meets the geometric specifications of vector tile encoding, preventing rendering artifacts or production process interruptions caused by topological errors.

4. The vector tile cut publishing system of claim 2, wherein, The heterogeneous data adaptive access and normalization subsystem further comprises a streaming attribute structure flattening unit configured to: Perform recursive attribute scanning for complex geographic entity objects with multi-layer nested structures; Dynamically flatten nested key-value pair structures into single-layer attribute tables and perform standardized mapping of field names, eliminating attribute structure differences between different data sources, providing a consistent data view for subsequent columnar storage analysis.

5. The vector tile cut publishing system of claim 2, wherein, The vector tile streaming production engine performs a geographic fence-based partitioned parallel cutting mechanism: According to the slicing configuration file output by the dynamic slicing rule adaptive generation module, the full data is divided into three logical processing units: high-density, medium-density, and low-density areas; The independent computing resources are allocated to perform the tile generation tasks of the logical processing units in parallel, and finally the tile data packets of different density partitions are aggregated into a single tile data set by a tile merging tool to balance the global data quality and volume.

6. The vector tile cut publishing system of claim 1, wherein, The standardized service publishing bus comprises: A viewport-aware clipping unit configured to, when receiving a client request on the server side, analyze the protocol buffer data stream in the tile data packet in real time, eliminate visual noise elements according to the viewport parameters of the client, and re-encapsulate a lightweight data packet for return; An incremental hot update notification mechanism configured to, when the production engine generates new tile data, trigger the memory index refresh of the server side through a semaphore mechanism, and calculate the differential data packet of the changed elements to support the client to perform local rendering update only for the changed area.

7. The vector tile cut publishing system of claim 6, wherein, The viewport-aware clipping unit is internally provided with an end-side capability adaptation filter configured to: Analyze the client device type identifier and device pixel ratio parameters in the HTTP request header; calculate the projection area of the geometric elements on the screen at the current zoom level according to an effective visual information amount algorithm; Real-time filter out polygon elements with a projection area less than a preset pixel threshold or line segment elements with a length less than a preset pixel threshold to realize instant compression of the transmission load for mobile terminal devices.

8. The vector tile cut publishing system of claim 6, wherein, The incremental hot update notification mechanism further comprises a differential tile calculation engine: Configured to cache the fingerprint features of the tile data of the previous version, and when the tile regeneration caused by the source data change is monitored, compare the binary encoding differences between the new and old tile data; Extract the changed geometric elements and attribute information, generate an independent differential vector tile data packet, and send a response message including an update instruction to the client to instruct the client to merge the local cache and the differential data.

9. The vector tile cut publishing system of claim 6, wherein, The space-time feature analysis and index construction middleware adopts an in-memory columnar analysis database as a core component and is configured to: Directly perform memory-level high-speed scanning and statistics on the stream access intermediate state data without being persisted to the disk; As a read-write buffer between the production engine and the data source, the vectorized execution engine is used to accelerate the aggregation query of large-scale spatial data and reduce the disk input / output overhead.

10. The vector tile cut publishing system of claim 6, wherein, Further comprising a global resource adaptive scheduling module configured to: Based on the density heat map generated by the space-time quadtree index, predict the computing load of tile production in different areas; Dynamically adjust the process concurrency and memory quota of the vector tile stream production engine, preferentially guarantee the supply of computing resources in high-density areas, and automatically trigger the load balancing strategy of the viewport-aware clipping unit when high-concurrency requests are detected on the server side.

Images