A geospatial data parallel integration method based on spatial grid-aware routing

Abstract

This application discloses a method for parallel integration of geospatial data based on spatial grid-aware routing. The method includes: acquiring source information of geospatial data files and identifying their format and metadata; logically segmenting the data files into record blocks; calculating the bounding rectangle of the geometric objects of geospatial elements in each record block and determining the covering grid set according to spatial grid partitioning rules, then determining a unique master spatial grid identifier according to master attribution rules; constructing grid clusters based on the master spatial grid identifier, merging adjacent grids into scheduling partitions and routing them to corresponding parallel processing tasks; and having each task stream-read and process elements before batch writing them to the target storage. This invention overcomes the bottleneck of single-file parallel processing, achieves spatial locality-aware routing and dynamic load balancing, improves the spatial aggregation and query performance of target data, reduces the risk of memory overflow, and possesses high-throughput writing and precise breakpoint resumption capabilities.

Description

Technical Field

[0001] This application belongs to the field of data integration technology, and specifically relates to a method, apparatus, computer-readable storage medium, and electronic device for parallel integration of geospatial data based on spatial grid-aware routing. Background Technology

[0002] Geospatial data, as a crucial digital production factor, has been widely applied in smart cities, environmental monitoring, resource management, and autonomous driving. With the rapid development of geographic information acquisition technologies, the volume of geospatial data is growing exponentially, and its formats are becoming increasingly diverse, encompassing various types such as Shapefile, GeoJSON, GPKG, and GeoParquet. How to efficiently and reliably integrate large-scale, multi-format geospatial data into a unified target data storage (such as a spatial database, data warehouse, or search engine) has become a key technological challenge in the field of geospatial data management.

[0003] Existing geospatial data integration methods mainly employ the following strategies:

[0004] 1. Serial processing

[0005] This method reads and processes data sequentially, making it suitable for scenarios with small amounts of data. However, when faced with massive amounts of data at the TB or even PB level, the serial processing method is inefficient and time-consuming, making it difficult to meet the needs of real-time or near-real-time data integration.

[0006] 2. File-level parallel processing

[0007] Some improved methods split the integration task according to the number of files, allowing multiple concurrent processes or threads to process different files separately. This strategy has certain advantages when processing a large number of small files, but it still has the following significant limitations:

[0008] Single file bottleneck: When there is a single ultra-large-scale geospatial data file (such as a GeoJSON file containing hundreds of millions of elements of national administrative divisions or a large remote sensing image data package), file-level splitting cannot achieve parallel processing within a single file, and the integration efficiency is limited by the processing speed of the single file.

[0009] Uneven task load: If the size of the files to be integrated varies significantly, file-level splitting can easily lead to uneven task allocation, with some processing nodes being idle while others are overloaded, resulting in low utilization of computing resources and limited overall integration efficiency.

[0010] Memory overflow risk: Even if a block-based reading strategy is used for large files to alleviate memory pressure, the risk of memory overflow still exists if a large number of geographic features still need to be fully parsed, processed and buffered in a single task, and it is difficult to effectively utilize distributed computing resources for horizontal scaling.

[0011] Insufficient write efficiency on the target side: Existing methods often use a row-by-row insertion (such as executing INSERT statements one by one) when writing data to the target storage. This method has poor performance in high-throughput write scenarios, which can easily cause database I / O bottlenecks and reduce the overall integrated throughput.

[0012] Lack of spatial locality awareness: File-level partitioning typically fails to consider the spatial distribution characteristics of geographic features, leading to spatially adjacent features potentially being assigned to different processing tasks or written to different target storage regions. This can not only increase write conflicts and uneven node load in distributed spatial databases, but also hinder subsequent query optimization based on spatial locality, potentially resulting in decreased spatial query performance.

[0013] In summary, existing geospatial data integration methods still suffer from performance bottlenecks, low resource utilization, complex memory management, and a lack of spatial awareness when faced with single ultra-large files, uneven file sizes, and high-throughput write requirements. These issues make it difficult to meet the growing demand for large-scale geospatial data integration. Summary of the Invention

[0014] To overcome the aforementioned deficiencies in existing technologies, this application provides a novel parallel integration method and apparatus for geospatial data based on spatial grid-aware routing. More specifically, this application proposes a geospatial data integration method for efficient and parallel integration of large-scale, multi-format geospatial data, aiming to address the performance bottlenecks, low resource utilization, complex memory management, and lack of spatial awareness capabilities inherent in existing geospatial data integration methods when processing large-scale, heterogeneous geospatial data.

[0015] This invention achieves efficient, parallel, and intelligent integration of geospatial data through a collaborative scheme of refined record block segmentation, spatial grid encoding of geographic elements, spatial locality-aware routing, streaming batch writing, and fault-tolerant mechanisms.

[0016] Specifically, this application provides the following technical solutions:

[0017] The first aspect of this application provides a method for parallel integration of geospatial data based on spatial grid-aware routing, such as... Figure 1 As shown, the method includes the following steps:

[0018] S1. Obtain the source information of the geospatial data files to be integrated, and identify their file format type and metadata;

[0019] S2. Logically divide the geospatial data file into record blocks to generate one or more record blocks to be processed. The record block is the smallest schedulable processing unit containing a preset number of geospatial feature records or a data fragment not exceeding a preset byte threshold, and has a block number, source file identifier, and offset information.

[0020] S3. For each geospatial element in the record block, calculate the bounding rectangle of its geometric object, and determine its covering grid set according to the preset spatial grid division rules; then, according to the preset primary attribution determination rules, determine a primary spatial grid identifier from the covering grid set for task routing.

[0021] S4. Send the record block and its contained geospatial elements and their corresponding main spatial grid identifiers to the task scheduler for task allocation;

[0022] S5. The task scheduler constructs a grid cluster based on the main spatial grid identifier, merges adjacent grids within a preset adjacency order range into the same scheduling partition, and routes geospatial elements belonging to the scheduling partition to the corresponding parallel processing tasks according to the scheduling partition, thereby realizing spatial locality routing and dynamic load balancing.

[0023] S6. Each parallel processing task streams the assigned geospatial features, performs data format conversion, target field mapping, and geometric object conversion.

[0024] S7. Each parallel processing task writes the processed geospatial features into the target data storage in batches, and records the writing progress and status in real time. At the same time, it uses the uniqueness constraint or idempotent operation of the target data storage to prevent duplicate writing.

[0025] Furthermore, in the method of this application, the file format types mentioned in step S1 include, but are not limited to, Shapefile, GeoJSON, GPKG, and GeoParquet.

[0026] Furthermore, in the method of this application, the logical segmentation in step S2 employs different segmentation strategies based on the file format type:

[0027] When the file format type is GeoJSON, it is split according to the Feature stream;

[0028] When the file format type is Shapefile, split it according to the Record cursor;

[0029] When the file format is GPKG or FileGDB, split the file by cursor pagination.

[0030] When the file format is GeoParquet, split it by Row Group.

[0031] Furthermore, in the method of this application, the spatial grid division rules in step S3 include at least one of QuadTree, GeoHash, S2 Cell, and H3; the main attribution determination rules include at least one of the following: the grid where the geometric centroid is located is the main grid, the grid where the MBR center point is located is the main grid, and the grid with the largest coverage area is the main grid.

[0032] Furthermore, in the method of this application, the grid cluster construction and scheduling partition definition in step S5 are based on a preset adjacency order and grid level to achieve four-adjacency, eight-adjacency or k-ring adjacency of the grid; the dynamic load balancing includes dynamically adjusting the allocation strategy of the scheduling partition according to the real-time load of the parallel processing task and the expected processing complexity of the geospatial elements in the scheduling partition.

[0033] Furthermore, in the method of this application, the parallel processing tasks in step S6 that stream-read the allocated geospatial features also include the following processing:

[0034] Based on the acquired metadata and the target data source type, infer the target field and its data type, and assemble the table creation statement;

[0035] The visual interface allows operators to adjust target fields, including field type, field length, and field precision.

[0036] The geometric information of source geospatial features is converted into a geometric object representation supported by the target data storage.

[0037] Furthermore, in the method of this application, the batch writing in step S7, which uses the batch import interface or batch submission protocol supported by the target end according to the data storage type of the target end, includes:

[0038] When the target is PostgreSQL / PostGIS, use the COPY command or WKB-based batch loading;

[0039] When the target is MySQL, use the LOAD DATA INFILE command or a batch INSERT statement;

[0040] When the target is ClickHouse, use the batch insert API;

[0041] When the target is Elasticsearch, the Bulk API is used.

[0042] Furthermore, the method of this application also includes:

[0043] S8. During the batch write operation, a breakpoint resume point is generated, and the identifiers of successfully processed and written record blocks and their internal processing checkpoints are recorded, or the information of failed record blocks is recorded.

[0044] S9. When an error or interruption occurs, the task is retried or resumed based on the breakpoint or failure record block information.

[0045] A second aspect of this application provides a geospatial data parallel integration apparatus based on spatial grid-aware routing. When the apparatus operates, it implements the steps of the aforementioned geospatial data parallel integration method based on spatial grid-aware routing, such as... Figure 2 As shown, the device includes:

[0046] The data source access module is used to obtain the source information of the geospatial data files to be integrated and to identify their file format type and metadata.

[0047] The data segmentation module is used to logically segment the geospatial data file into record blocks, calculate the bounding rectangle of geospatial elements in each record block, determine the set of covering grids, and determine the main spatial grid identifier according to the main attribution determination rule.

[0048] The task scheduling module is used to construct a grid cluster based on the main spatial grid identifier, merge adjacent grids within a preset adjacency order range into the same scheduling partition, and route geospatial elements belonging to the scheduling partition to the corresponding parallel processing tasks according to the scheduling partition.

[0049] The parallel processing module is used to stream the assigned geospatial features and perform data format conversion, target field mapping, and geometric object conversion.

[0050] The batch write module is used to write the processed geospatial features into the target data storage in batches, and record the writing progress and status in real time. At the same time, it uses the uniqueness constraint or idempotent operation of the target data storage to prevent duplicate writing.

[0051] Furthermore, the device of this application also includes:

[0052] The fault tolerance and recovery module is used to generate breakpoint resume points or record block information for failed records during batch write operations, and to perform task retry or breakpoint resume when errors or interruptions occur.

[0053] A third aspect of this application provides an electronic device, including: a memory and a processor;

[0054] Memory: Used to store computer programs;

[0055] Processor: Used to execute the computer program to implement the steps of the aforementioned geospatial data parallel integration method based on spatial grid-aware routing.

[0056] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned method for parallel integration of geospatial data based on spatial grid-aware routing.

[0057] In summary, compared with the prior art, the solution of this application has the following advantages:

[0058] (1) The organic synergy between breaking through the single-file parallel bottleneck and optimizing load balancing: By dividing geospatial data files into logical record blocks, the problem of single ultra-large-scale files being unable to be processed in parallel is solved; on this basis, combined with a spatial grid-aware routing mechanism, spatially adjacent elements are routed to the same processing task or grid cluster, effectively avoiding the problem of uneven task distribution in traditional solutions, significantly improving the load balancing of parallel processing, and making full use of distributed computing resources. This "segmentation-awareness-routing" synergistic mechanism is the core innovation of this invention in terms of efficiency improvement.

[0059] (2) Resolving ambiguities and achieving efficient routing in cross-grid feature processing: This invention introduces a primary attribution determination rule, explicitly assigning a "primary spatial grid identifier" for each feature to be used for task routing. This avoids the problems of duplicate distribution, duplicate calculation, and data inconsistency that may occur when a feature spans multiple grids in traditional methods. At the same time, other auxiliary grid identifiers can be stored along with the feature, providing multi-dimensional assistance for subsequent spatial queries and indexes.

[0060] (3) Improve the spatial clustering and query performance of target data: Through spatial grid-aware routing, it is ensured that spatially adjacent data are processed by the same task during the processing and are eventually written to the same or adjacent areas of the target storage, forming a data distribution with high spatial locality. This provides a solid foundation for building efficient spatial indexes and performing spatial range queries in the target storage, and significantly improves the performance of spatial query and analysis operations.

[0061] (4) Synergy between reducing memory overflow risk and high-throughput writing: The adoption of streaming reading and record block splitting strategies avoids loading the entire large file or a large number of elements into memory at once, significantly reducing the risk of memory overflow. At the same time, after each parallel processing task buffers data in memory, it adopts the batch writing mechanism natively supported by the target database, which greatly improves the throughput of data writing to the target end and reduces database I / O overhead, thus jointly ensuring the stability and efficiency of the integration process.

[0062] (5) High robustness and accurate recoverability: A breakpoint resume mechanism is introduced to accurately record the identifiers of successfully processed record blocks and their internal processing checkpoints (such as Feature ID, offset, or Row Group index), as well as the retry function for failed record blocks. This enables data integration tasks to accurately recover from the breakpoint after encountering temporary failures or interruptions, avoiding the reprocessing of successfully processed data and improving the system's fault tolerance and data consistency.

[0063] (6) Simplified operation and intelligentization: Automated and parallel processing flow, unified format parsing and intelligent field mapping inference (and provide visual adjustment) reduce the technical background requirements and learning costs of operators and shorten the data integration cycle. Attached Figure Description

[0064] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this invention will be briefly introduced below. It should be noted that the drawings only show some embodiments of the invention. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.

[0065] Figure 1 This is a flowchart illustrating the operational steps of the geospatial data parallel integration method based on spatial grid-aware routing in this application.

[0066] Figure 2 This is a structural diagram of the geospatial data parallel integration device based on spatial grid-aware routing, as described in this application.

[0067] Figure 3 This is a schematic diagram illustrating the overall implementation process of the method in the embodiments of this application.

[0068] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0070] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.

[0071] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."

[0072] This invention provides the following technical solution:

[0073] In a first aspect, the present invention provides a method for parallel integration of geospatial data based on spatial grid-aware routing, the method comprising:

[0074] a. Obtain the source information of the geospatial data files to be integrated, and identify their file format type and metadata;

[0075] b. Logically divide the geospatial data file into record blocks to generate one or more record blocks to be processed, wherein the record block is the smallest schedulable processing unit containing a preset number of geospatial feature records or a data fragment not exceeding a preset byte threshold, and carries a block number, source file identifier and offset information;

[0076] c. For each geospatial feature in a record block, calculate the bounding rectangle (MBR) of its geometric object, and determine its covering grid set according to the preset spatial grid division rules; then, according to the preset primary attribution determination rules, determine a primary spatial grid identifier from the covering grid set for task routing.

[0077] d. Send the record block and its contained geospatial features and their corresponding master spatial grid identifiers to the task scheduler for task allocation;

[0078] e. The task scheduler constructs a grid cluster based on the main spatial grid identifier, merges adjacent grids within a preset adjacency order range into the same scheduling partition, and routes geospatial elements belonging to the scheduling partition to the corresponding parallel processing tasks according to the scheduling partition, thereby realizing spatial locality routing and dynamic load balancing.

[0079] f. Each parallel processing task streams the assigned geospatial features and performs data format conversion, target field mapping, geometric object conversion, and other processing.

[0080] g. Each parallel processing task writes the processed geospatial features into the target data storage in batches, and records the writing progress and status in real time. At the same time, it uses the uniqueness constraint or idempotent operation of the target data storage to prevent duplicate writing.

[0081] Optionally, the file format type identified in step a includes: Shapefile, GeoJSON, GPKG (GeoPackage), and GeoParquet.

[0082] Optionally, in step b, logical segmentation is performed by record blocks, and different segmentation strategies are adopted according to the file format type:

[0083] When the file format type is GeoJSON, it is split according to the Feature stream;

[0084] When the file format type is Shapefile, split it according to the Record cursor;

[0085] When the file format is GPKG or FileGDB, split the file by cursor pagination.

[0086] When the file format is GeoParquet, split it by Row Group.

[0087] Optionally, the preset spatial grid division rules in step c include, but are not limited to: QuadTree, GeoHash, S2 Cell, H3; the main attribution determination rules include, but are not limited to: using the grid where the geometric centroid is located as the main grid, using the grid where the MBR center point is located as the main grid, or using the grid with the largest coverage area as the main grid.

[0088] Optionally, the grid cluster construction and scheduling partition definition in step e are based on a preset adjacency order and grid level to achieve four-adjacency, eight-adjacency, or k-ring adjacency of the grid; the dynamic load balancing includes dynamically adjusting the allocation strategy of the scheduling partition according to the real-time load of the parallel processing tasks and the expected processing complexity of the geospatial elements within the scheduling partition.

[0089] Optionally, when each parallel processing task in step f processes the assigned geospatial features, it includes:

[0090] Based on the acquired metadata and the target data source type, infer the target field and its data type, and assemble the table creation statement;

[0091] The visual interface allows operators to adjust target fields, including field type, field length, and field precision.

[0092] The geometric information of source geospatial features is converted into a geometric object representation supported by the target data storage, such as WKB (Well-Known Binary) or WKT (Well-Known Text).

[0093] Optionally, the batch writing method in step g for the target data storage can employ different batch writing mechanisms depending on the target data storage type:

[0094] When the target is PostgreSQL / PostGIS, use the COPY command or WKB-based batch loading;

[0095] When the target is MySQL, use the LOAD DATA INFILE command or a batch INSERT statement;

[0096] When the target is ClickHouse, use the batch insert API;

[0097] When the target is Elasticsearch, the Bulk API is used.

[0098] Optionally, the method further includes:

[0099] h. During the batch write operation, generate breakpoint resume points, record the identifiers of successfully processed and written record blocks and their internal processing checkpoints (e.g., Feature ID, offset, or Row Group index), or record information of failed record blocks.

[0100] i. When an error or interruption occurs, the task can be retried or resumed based on the breakpoint or failure record block information to avoid repeated processing.

[0101] In a second aspect, the present invention provides a geospatial data parallel integration device based on spatial grid-aware routing, the device comprising:

[0102] Data source access module: used to obtain the source information of the geospatial data files to be integrated, and to identify their file format type and metadata;

[0103] Data segmentation module: used to logically segment the geospatial data file into record blocks, calculate the bounding rectangle (MBR) of geospatial elements in each record block, determine the set of overlay grids, and determine the main spatial grid identifier according to the main attribution determination rule;

[0104] Task scheduling module: used to construct grid clusters based on the main spatial grid identifier, merge adjacent grids within a preset adjacency order range into the same scheduling partition, and route geospatial elements belonging to the scheduling partition to the corresponding parallel processing tasks according to the scheduling partition.

[0105] Parallel processing module: used for streaming reading of assigned geospatial features, performing data format conversion, target field mapping, geometric object conversion, and other processing;

[0106] Batch write module: Used to write processed geospatial features into the target data storage in batches, and record the writing progress and status in real time. At the same time, it uses the uniqueness constraint or idempotent operation of the target data storage to prevent duplicate writing.

[0107] The device may further include:

[0108] Fault tolerance and recovery module: used to generate breakpoint resume points or record block information for failed records during batch write operations, and to retry tasks or resume breakpoints when errors or interruptions occur.

[0109] To more clearly illustrate the technical solution of this application, the following will provide further explanation through specific scenario embodiments.

[0110] Figure 3 This is a schematic diagram illustrating the overall implementation process of the method in this application embodiment. The detailed implementation process of this solution is as follows:

[0111] I. Data Source Configuration and File Discovery

[0112] 1. Source configuration: The user or system selects the file system (such as FTP, SFTP, HDFS, Minio, S3, or other object storage or local file system) to store geospatial data files.

[0113] 2. File format type selection: Specify the file format type of the geospatial data to be integrated. This solution supports multiple common formats such as Shapefile, GeoJSON, GPKG, and GeoParquet.

[0114] 3. File Discovery and Preliminary Metadata Acquisition: The system scans the specified file system path and discovers a list of files that meet the type requirements. For each discovered file, preliminary metadata parsing is performed to obtain its basic information, such as filename, file size, file path, and file type. This step prepares for subsequent detailed parsing and segmentation.

[0115] II. Metadata Parsing and Fragmentation Identification

[0116] 1. Detailed Metadata Parsing: For each geospatial data file to be integrated, deep parsing is performed using professional parsing tools (such as GeoTools, GDAL / OGR libraries, etc.). Detailed metadata information is obtained, including but not limited to:

[0117] Geospatial feature structure: the name of each attribute field, data type (such as string, integer, floating point, date) and the type of geometric field (such as point, line, polygon, multi-point, multi-line, multi-polygon, etc.).

[0118] Coordinate Reference System (CRS): Projection information of the data.

[0119] File characteristics: Whether the file supports record block-based streaming.

[0120] 2. Determine the segmentation strategy: Based on the parsed file format type and its characteristics, determine the appropriate record block segmentation strategy for the file. This is a crucial prerequisite for achieving parallel processing of a single large file.

[0121] III. Record Block Segmentation and Spatial Grid-Aware Routing

[0122] 1. Logical segmentation of record blocks: The core of this step is to logically divide a large file that may be physically contiguous into multiple independent and processable small units.

[0123] Definition: A "Record Block" is defined as the smallest schedulable processing unit continuously extracted from a source file. It contains a preset number of geospatial feature records (e.g., 1000 features) or a data fragment not exceeding a preset byte threshold (e.g., 1MB). Each Record Block carries a unique block number, a source file identifier, and offset information within the source file to support accurate tracking and recovery.

[0124] Segmentation strategy:

[0125] GeoJSON files: use a streaming parser to logically divide them into Feature streams based on Feature boundaries (usually JSON object delimiters), with each N Features or up to M bytes constituting a record block.

[0126] Shapefile files: can be segmented into individual geometric records by means of the Shapefile index file (.shx) or by direct parsing, and multiple records can be combined into a single record block.

[0127] GPKG / FileGDB files: utilize their internal database structure to split files by cursor pagination or Feature ID range.

[0128] GeoParquet files: Utilizing Parquet's Row Group feature, each Row Group naturally constitutes an independently readable record block.

[0129] 2. Extraction of spatial information of geospatial elements and determination of main spatial grid identifiers:

[0130] For each geospatial feature parsed from the record block, its geometric information is extracted.

[0131] For each geospatial feature's geometric object, calculate its minimum bounding rectangle (MBR).

[0132] Based on preset spatial grid division rules (e.g., S2 Cell, GeoHash, QuadTree, H3, etc.), the MBR is mapped to one or more spatial grid identifiers (Cell IDs) to form its overlay grid set. These identifiers represent the approximate spatial location of features on the Earth's surface.

[0133] Primary Attribution Determination Rule: To address the situation where a feature may cover multiple grids, and to avoid redundant processing and writing ambiguity, this invention introduces a primary attribution determination rule. This rule determines a unique primary spatial grid identifier from the set of covering grids, which is used for task routing of that feature. For example:

[0134] The grid containing the geometric centroid is used as the main grid.

[0135] The grid where the center point of the MBR is located is used as the main grid.

[0136] The grid with the largest coverage area is used as the main grid.

[0137] The grid with the highest level or the preset priority in the coverage grid set is used as the main grid.

[0138] The remaining grid identifiers serve as auxiliary spatial index information and can be stored along with the features or cached during processing, providing multi-dimensional support for subsequent query optimization.

[0139] 3. Task scheduling and spatial grid-aware routing:

[0140] The information, which includes record block data, geospatial features, and their corresponding master spatial grid identifiers, is submitted to the task scheduler of the master node (Manager).

[0141] Grid Cluster Construction: Instead of scheduling tasks directly on a single grid, the task scheduler constructs grid clusters based on the main space grid identifier. A grid cluster merges adjacent grids within a preset adjacency order range (e.g., four-adjacency, eight-adjacency, or k-ring adjacency) into a single scheduling partition. The merging threshold can be dynamically adjusted based on grid density, expected data volume, etc. This strategy effectively aggregates spatially closely related data, reducing data dependencies across task boundaries.

[0142] Spatial locality routing: Based on the scheduling partitions, the scheduler routes all geospatial feature sets (which may come from different record blocks or even different files) whose main spatial grid identifiers belong to the same scheduling partition to specific parallel processing worker nodes. For example, all features falling within the "scheduling partition A" area will be routed to Worker A for processing.

[0143] Dynamic load balancing: The scheduler monitors the CPU utilization, memory usage, processing queue length and other load information of the Workers in real time, and combines this with the expected processing complexity of geospatial elements within the scheduling partition (such as geometric complexity and number of elements) to dynamically adjust the allocation strategy of the scheduling partition or split high-load partitions to other idle Workers, so as to avoid the emergence of hot grids and ensure load balancing of all Workers.

[0144] IV. Parallel Processing and Streaming Batch Writing

[0145] 1. Data Transformation and Mapping: After receiving the assigned geospatial features, each Worker performs the following processing:

[0146] Field mapping and type conversion: Based on pre-acquired metadata and the target data source type, the system automatically infers the target field names and data types. For example, a "String" type attribute in the source GeoJSON can be mapped to a VARCHAR type in the target PostGIS database. The system provides a visual interface for operators to review and adjust the inference results (such as modifying field type, length, precision, and whether it is nullable), ensuring the flexibility and accuracy of the mapping.

[0147] Geometric object conversion: Convert geometric information in the source format into a geometric object representation supported by the target data storage, such as converting GeoJSON JSON geometric objects into PostGIS WKB (Well-Known Binary) or WKT (Well-Known Text) formats, and ensuring the consistency of coordinate reference systems.

[0148] 2. Streaming data buffer: The Worker buffers the processed geospatial features in memory, forming batches to be written.

[0149] 3. Batch write to the target: When the amount of buffered data reaches a preset threshold (such as N elements or M bytes) or a preset time interval, the Worker will write the entire batch of data at once through the batch write mechanism natively supported by the target data storage.

[0150] PostgreSQL / PostGIS: Utilizes the COPY FROM STDIN command for efficient batch writing via JDBC or other client libraries. For geometric data, it directly uses the WKB format for binary loading, ensuring that SRIDs are set correctly.

[0151] MySQL: Use the LOAD DATA INFILE command or the bulk INSERT INTO ... VALUES (...),(...), ... statement. For MySQL Spatial, ensure that the geometric data is converted to a recognizable format and that spatial indexes are created.

[0152] ClickHouse: Invokes its dedicated bulk insert API.

[0153] Elasticsearch: Use the Bulk API for bulk indexing.

[0154] Batch writing greatly reduces database transaction overhead and network round trips, significantly improving write throughput.

[0155] 4. Preventing Duplicate Writes: During batch writes, the system incorporates unique constraints on the target data storage, such as primary keys or unique indexes, to achieve idempotent writes. For existing records, options include skipping, updating, or reporting conflicts. For example, in PostgreSQL / PostGIS, the `INSERT... ON CONFLICT DO NOTHING / UPDATE (UPSERT)` statement can be used; in Elasticsearch, the Bulk API supports ID-based updates or inserts.

[0156] V. Error Handling and Resuming Breakpoints

[0157] 1. Progress and status recording: The master node and worker record the processing progress and write status of each record block in real time.

[0158] 2. Breakpoint Resume Point Generation: Periodically or after critical operations, the system generates breakpoint resume points, accurately recording the identifiers of successfully processed and written record blocks and their internal processing checkpoints, for example:

[0159] For GeoJSON streams, record the Feature ID or file offset that has been successfully processed.

[0160] For Shapefile, record the cursor position of the Record or the range of Feature IDs that have been successfully processed.

[0161] For GeoParquet, record the Row Group indexes that have been successfully processed.

[0162] 3. Failure Handling and Precise Retry: When an error occurs while a Worker is processing a record block or performing a batch write, the system captures the error information. The information of the failed record block (including its precise internal checkpoint) is recorded and can be reassigned by the task scheduler to another healthy Worker for precise retries, resuming processing only from the failed checkpoint. After the entire task is interrupted, the user can restart the task from the latest breakpoint. The system will only process incomplete or failed record blocks, avoiding duplicate processing of successfully processed data and ensuring data consistency.

[0163] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of apparatus, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.

[0164] like Figure 4As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the aforementioned parallel integration method for geospatial data based on spatial grid-aware routing.

[0165] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).

[0166] The operations described in this application can be implemented by writing computer program code using one or more programming languages ​​or a combination thereof. The programming languages ​​include, but are not limited to, the following types:

[0167] Object-oriented programming languages, such as Java, Smalltalk, C++, etc.

[0168] Conventional procedural programming languages, such as "C" or similar programming languages.

[0169] The execution methods of program code include, but are not limited to:

[0170] It runs entirely on the user's computer;

[0171] Part of it executes on the user's computer, and part of it executes on a remote computer;

[0172] Execute as a standalone software package;

[0173] It is executed entirely on a remote computer or server.

[0174] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.

[0175] Furthermore, this application also discloses a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the various steps of the geospatial data parallel integration method based on spatial grid-aware routing disclosed in this application.

[0176] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following:

[0177] (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.

[0178] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.

[0179] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.

[0180] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.

[0181] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.

[0182] (6) Fiber optic storage devices: storage media based on fiber optic technology.

[0183] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.

[0184] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.

[0185] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.

[0186] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.

[0187] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.

[0188] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program contains program code for executing the geospatial data parallel integration method based on spatial grid-aware routing disclosed in this application. When this computer program is executed by a processing device, it can achieve the functions defined in the embodiments of this application.

[0189] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.

[0190] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A parallel integration method for geospatial data based on spatial grid-aware routing, characterized in that, Includes the following steps: S1. Obtain the source information of the geospatial data files to be integrated, and identify their file format type and metadata; S2. Logically divide the geospatial data file into record blocks to generate one or more record blocks to be processed. The record block is the smallest schedulable processing unit containing a preset number of geospatial feature records or a data fragment not exceeding a preset byte threshold, and has a block number, source file identifier, and offset information. S3. For each geospatial element in the record block, calculate the bounding rectangle of its geometric object, and determine its covering grid set according to the preset spatial grid division rules; then, according to the preset primary attribution determination rules, determine a primary spatial grid identifier from the covering grid set for task routing. S4. Send the record block and its contained geospatial elements and their corresponding main spatial grid identifiers to the task scheduler for task allocation; S5. The task scheduler constructs a grid cluster based on the main spatial grid identifier, merges adjacent grids within a preset adjacency order range into the same scheduling partition, and routes geospatial elements belonging to the scheduling partition to the corresponding parallel processing tasks according to the scheduling partition, thereby realizing spatial locality routing and dynamic load balancing. S6. Each parallel processing task streams the assigned geospatial features, performs data format conversion, target field mapping, and geometric object conversion. S7. Each parallel processing task writes the processed geospatial features into the target data storage in batches, and records the writing progress and status in real time. At the same time, it uses the uniqueness constraint or idempotent operation of the target data storage to prevent duplicate writing.

2. The method according to claim 1, characterized in that, The file format types mentioned in step S1 include Shapefile, GeoJSON, GPKG, and GeoParquet.

3. The method according to claim 1, characterized in that, The logical segmentation described in step S2 employs different segmentation strategies based on the file format type: When the file format type is GeoJSON, it is split according to the Feature stream; When the file format type is Shapefile, split it according to the Record cursor; When the file format is GPKG or FileGDB, split the file by cursor pagination. When the file format is GeoParquet, split it by Row Group.

4. The method according to claim 1, characterized in that, The spatial grid division rules in step S3 include at least one of QuadTree, GeoHash, S2 Cell, and H3; the main grid assignment rules include at least one of the following: the grid where the geometric centroid is located is the main grid, the grid where the MBR center point is located is the main grid, and the grid with the largest coverage area is the main grid.

5. The method according to claim 1, characterized in that, The grid cluster construction and scheduling partition definition in step S5 are based on the preset adjacency order and grid level to achieve four-adjacency, eight-adjacency or k-ring adjacency of the grid; the dynamic load balancing includes dynamically adjusting the allocation strategy of the scheduling partition according to the real-time load of the parallel processing tasks and the expected processing complexity of the geospatial elements in the scheduling partition.

6. The method according to claim 1, characterized in that, Step S6, in which each parallel processing task streams the assigned geospatial features, also includes the following processing: Based on the acquired metadata and the target data source type, infer the target field and its data type, and assemble the table creation statement; The visual interface allows operators to adjust target fields, including field type, field length, and field precision. The geometric information of source geospatial features is converted into a geometric object representation supported by the target data storage.

7. The method according to claim 1, characterized in that, Step S7, which involves batch writing based on the target end's data storage type and using the batch import interface or batch submission protocol supported by the target end, includes: When the target is PostgreSQL / PostGIS, use the COPY command or WKB-based batch loading; When the target is MySQL, use the LOAD DATA INFILE command or a batch INSERT statement; When the target is ClickHouse, use the batch insert API; When the target is Elasticsearch, the Bulk API is used.

8. The method according to claim 1, characterized in that, The method further includes: S8. During the batch write operation, a breakpoint resume point is generated, and the identifiers of successfully processed and written record blocks and their internal processing checkpoints are recorded, or the information of failed record blocks is recorded. S9. When an error or interruption occurs, the task is retried or resumed based on the breakpoint or failure record block information.

9. A parallel integration device for geospatial data based on spatial grid-aware routing, characterized in that, The device, when in operation, implements the steps of the geospatial data parallel integration method based on spatial grid-aware routing as described in any one of claims 1-8, and the device comprises: The data source access module is used to obtain the source information of the geospatial data files to be integrated and to identify their file format type and metadata. The data segmentation module is used to logically segment the geospatial data file into record blocks, calculate the bounding rectangle of geospatial elements in each record block, determine the set of covering grids, and determine the main spatial grid identifier according to the main attribution determination rule. The task scheduling module is used to construct a grid cluster based on the main spatial grid identifier, merge adjacent grids within a preset adjacency order range into the same scheduling partition, and route geospatial elements belonging to the scheduling partition to the corresponding parallel processing tasks according to the scheduling partition. The parallel processing module is used to stream the assigned geospatial features and perform data format conversion, target field mapping, and geometric object conversion. The batch write module is used to write the processed geospatial features into the target data storage in batches, and record the writing progress and status in real time. At the same time, it uses the uniqueness constraint or idempotent operation of the target data storage to prevent duplicate writing.

10. The apparatus according to claim 9, characterized in that, The device further includes: The fault tolerance and recovery module is used to generate breakpoint resume points or record block information for failed records during batch write operations, and to perform task retry or breakpoint resume when errors or interruptions occur.

Images