Parallel processing method and system for complex spatial data

Abstract

The application discloses a parallel processing method and system for complex spatial data, and relates to the technical field of spatial data processing. The method comprises the following steps: receiving a spatial service task and abstracting the spatial service task into a spatial task object; clustering and grouping based on spatial adjacency relationship, and generating an ordered subtask sequence through space filling curve sorting and then enqueuing; actively pulling the subtask by a work node through a message queue pull mode to execute spatial calculation; managing the subtask life cycle through a multi-level state machine and performing a re-enqueue operation based on a hierarchical retry strategy; binding spatial adjacent subtasks to the same work node based on a cache-aware scheduling strategy, and dynamically adjusting global concurrency according to cache hit rate feedback; and performing aggregation processing on the subtask execution result through a multi-stage aggregation pipeline, and outputting a multi-modal processing result. The method enables the framework to have spatial data awareness, improves cache hit rate and load balancing capability, reduces task error rate, and supports multi-modal achievement output.

Description

Technical Field

[0001] This invention relates to the field of natural resource information technology, specifically to a parallel processing method and system for complex spatial data. Background Technology

[0002] The core production factor in the natural resources industry is vector spatial data composed of point, line, and surface elements. In typical business scenarios such as territorial spatial planning, land use surveys, and resource and environmental carrying capacity assessments, spatial data exhibits significant characteristics, including large data scale (the number of map features typically ranges from hundreds of thousands to millions, with single datasets reaching GB levels), high computational complexity (requiring complex operations such as spatial overlay analysis, buffer analysis, and topology checking), diverse output formats (generating Word documents, image atlases, PPT presentations, spatial datasets, etc.), and diverse data sources (involving comprehensive analysis of multiple types of data such as remote sensing imagery, land use status, and planning results). Spatial datasets in typical business scenarios usually contain 100,000 to 1 million map features, with single datasets reaching GB levels. They also require highly complex operations such as spatial overlay analysis, buffer analysis, and topology checking. The final output often needs to simultaneously cover multimodal results such as spatial datasets, thematic images, PPT presentation atlases, and Word analysis reports. Existing processing technologies struggle to simultaneously meet the business requirements of high efficiency, high reliability, and lightweight deployment, becoming a core bottleneck in the digital transformation of the natural resources industry.

[0003] Current parallel processing technologies for spatial data can be mainly divided into two categories:

[0004] One type is general-purpose distributed computing frameworks, typically represented by Celery asynchronous task queue, Airflow workflow orchestration engine, and Apache Spark distributed computing engine. These solutions provide general distributed task scheduling capabilities, but none of them have native spatial data awareness capabilities. They do not have built-in functions adapted to spatial data processing, such as spatial index building, coordinate transformation, and GIS service calls. If applied to natural resource business scenarios, a lot of secondary development is required, resulting in high development and adaptation costs and poor flexibility.

[0005] Another type is GIS-specific solutions, typically represented by ArcGIS GeoAnalytics Server vector big data analysis tools, GeoMesa distributed spatiotemporal data storage engine, and GeoTrellis raster data processing framework. These solutions have advantages in spatial data storage, querying, and service publishing, but they generally lack a business-level general task scheduling framework and have not formed a complete closed loop of "task splitting-distribution-execution-state management-result aggregation". Moreover, they are often tied to specific vendor ecosystems or rely on heavy basic components such as Hadoop and Spark, making their deployment and maintenance extremely difficult and unsuitable for medium-sized business scenarios common in the natural resources industry.

[0006] Both existing technical solutions share the following common, insurmountable drawbacks:

[0007] The lack of spatial awareness, the absence of spatial data adaptation logic in the general framework, and the failure of GIS-specific solutions to optimize task partitioning logic for the distribution characteristics of spatial data, often result in the use of a fixed number of linear splits, leading to the discrete distribution of spatial elements within the same subtask, frequent caching of GIS server cache, and limited processing efficiency.

[0008] Load balancing is not adaptive. Existing solutions mostly adopt a centralized scheduling mode, which relies on preset resource quantity estimates to allocate tasks. It cannot dynamically adjust according to the real-time processing capacity of the working nodes, which easily leads to an imbalance problem where high-performance nodes are idle and low-performance nodes are overloaded.

[0009] The fault tolerance mechanism is crude, mostly adopting fault tolerance logic of one-time failure marking at the task level or overall migration of the task at the node level. It does not set up a graded retry mechanism for occasional failures such as network fluctuations and momentary service unavailability, resulting in a high overall error rate of tasks.

[0010] The lack of multimodal output capability means that the output of existing solutions only supports raw data, map tiles or service interfaces, which cannot meet the needs of the land and space planning field for the automated generation of office documents such as PPT atlases and Word reports.

[0011] Custom task requirements are high; general frameworks require developers to master underlying logic such as message protocols and distributed locks; GIS-specific solutions often only provide pre-built analysis tools, making it difficult for business developers to quickly integrate new processing needs and resulting in extremely low efficiency in responding to business iterations. Deployment and maintenance are also cumbersome; existing GIS-specific solutions often rely on heavy components such as Hadoop, Spark, distributed databases, or ArcGIS Enterprise, leading to high deployment costs and complex maintenance, making them unsuitable for medium-sized business scenarios common in the natural resources industry.

[0012] It should be noted that while general-purpose frameworks such as Celery and Airflow can achieve basic distributed task scheduling, they lack spatial data awareness capabilities and do not support multimodal result aggregation output. Big data ecosystem solutions such as Spark, GeoMesa, and GeoTrellis, while possessing distributed spatial data processing capabilities, are heavily deployed and lack a business-level task scheduling framework layer. ArcGIS GeoAnalytics Server is strongly bound to the closed Esri ecosystem and lacks a tiered retry and fault tolerance mechanism. Direct HTTP communication as a substitute for message queues requires implementing load balancing and task locking independently, resulting in high complexity and poor reliability. Furthermore, existing distributed spatial data processing solutions are mostly designed for fixed tasks of specific data types. Their task splitting, distribution, and result aggregation logic are tightly coupled with specific business logic, requiring extensive refactoring of the scheduling layer code when business data types change, and lacking business-independent generic task abstraction capabilities.

[0013] In summary, existing technologies cannot simultaneously achieve native spatial data perception, adaptive load balancing, fine-grained fault tolerance, multimodal output, and lightweight deployment under a single architecture, making it difficult to meet the high-efficiency processing needs of complex spatial data in the natural resources industry. Therefore, there is an urgent need for a parallel processing framework and system for complex spatial data. Summary of the Invention

[0014] This invention provides a parallel processing method and system for complex spatial data, which solves the technical problem that existing parallel processing technologies for spatial data cannot simultaneously achieve native spatial data perception, adaptive load balancing, fine-grained fault tolerance, and multimodal output capabilities under a single architecture.

[0015] This invention is achieved through the following technical solution:

[0016] Firstly, this application provides a parallel processing method for complex spatial data, including:

[0017] Receive spatial business tasks, and abstract spatial business tasks as spatial task objects that include spatial extent, coordinate system and GIS service address;

[0018] Based on spatial adjacency, spatial task objects are clustered and grouped, and the subtasks after grouping are sorted according to the spatial filling curve. After generating an ordered sequence of subtasks, they are enqueued.

[0019] By using the Pull mode of the message queue, the worker node actively pulls the subtask and performs space calculations to obtain the subtask execution result;

[0020] The lifecycle states of subtasks are managed by a multi-level state machine, and subtasks in abnormal states are re-entered into the queue based on a hierarchical retry strategy.

[0021] Based on a cache-aware scheduling strategy, spatially adjacent subtasks are bound to the same working node, and the global concurrency is dynamically adjusted according to cache hit rate feedback.

[0022] The multi-stage aggregation pipeline performs multi-stage aggregation processing on the results of sub-tasks, outputting multi-modal processing results.

[0023] A further optimized solution involves using a message queue's Pull mode, where worker nodes actively pull subtasks and perform space calculations. Specifically, this includes:

[0024] When a worker node starts up, it performs dynamic registration with the main application and periodically sends a heartbeat keep-alive signal.

[0025] Work nodes exclusively pull matching subtasks from the message queue based on their real-time computing power;

[0026] The main application dynamically senses the online status and load of worker nodes based on their heartbeat keep-alive signals, achieving decentralized adaptive load balancing.

[0027] A further optimization scheme involves clustering spatial task objects based on spatial adjacency relationships and then sorting the grouped subtasks according to the spatial filling curve. Specifically, this includes:

[0028] Clustering and grouping are implemented based on R-tree spatial index construction and spatial adjacency relationships;

[0029] The complexity of subtasks is evaluated by combining the area of ​​the patches and the number of vertices, and the grouping results are adjusted for load balancing.

[0030] The adjusted subtasks are sequentially enqueued according to the spatial filling curve to maintain local continuity in geospatial space.

[0031] A further optimized solution involves managing the lifecycle states of subtasks using a multi-level state machine and performing re-queue operations on subtasks in abnormal states based on a tiered retry strategy. Specifically, this includes:

[0032] Define a multi-level state machine containing at least six state levels, wherein the state levels include at least a pending state, a running state, a success state, a first-level error state, a second-level error state, and a failure state;

[0033] When a subtask fails to execute, its status is set to a first-level error status and it is allowed to be re-entered into the queue for execution again.

[0034] When a subtask fails to execute again, its status is set to a Level 2 error status and a final retry is allowed.

[0035] If a subtask fails to execute after its last retry, its status is set to the failed state.

[0036] The results aggregation phase supports two aggregation strategies: strict fault tolerance mode and lenient fault tolerance mode.

[0037] A further optimization scheme is as follows: the cache-aware scheduling strategy binds spatially adjacent subtasks to the same working node and dynamically adjusts the global concurrency based on cache hit rate feedback, specifically including:

[0038] Based on the clustering results of spatially aware partitioning, spatially adjacent subtasks are sorted according to the spatial filling curve and then pushed into the task queue.

[0039] By using multi-queue grouping management, subtasks belonging to the same spatial cluster are bound to specific work nodes to adapt to the local caching characteristics of GIS services;

[0040] Real-time monitoring of GIS service cache hit rate; when the cache hit rate is lower than the preset hit rate threshold, reduce global concurrency to adapt to cache storage constraints.

[0041] When the cache hit rate is higher than the preset hit rate threshold, the global concurrency is increased to improve the overall throughput.

[0042] A further optimized solution is that the multi-stage aggregation process sequentially includes a result collection stage, a result verification stage, a format conversion stage, a result merging stage, and a multimodal output stage;

[0043] The result verification stage includes at least the verification of data integrity and logical consistency;

[0044] The format conversion stage includes at least coordinate system transformation and data format standardization;

[0045] The result merging stage includes at least spatial overlay analysis and attribute information linking;

[0046] The processing results output by the multimodal output stage include at least one or more of the following: spatial dataset, image set, PPT image set, and Word report.

[0047] A further optimized solution is that the abstract spatial business task is a spatial task object containing spatial extent, coordinate system, and GIS service address, specifically including:

[0048] Define the generic classes SubTask and SubTaskResult, and encapsulate the GisInfo data class which natively integrates spatial range, coordinate system, and GIS service address;

[0049] When integrating a business application, only the task splitting method, business processing method, and result aggregation method need to be implemented. The task scheduling layer is decoupled from the business layer through a generic interface.

[0050] Secondly, this application provides a parallel processing system for complex spatial data, comprising:

[0051] The task abstraction module is used to abstract received spatial business tasks into spatial task objects that include spatial extent, coordinate system, and GIS service address.

[0052] The spatial partitioning module is used to cluster spatial task objects based on spatial adjacency and sort the grouped subtasks according to the spatial filling curve to generate an ordered subtask sequence.

[0053] The message scheduling module is used to allow worker nodes to actively pull subtasks and perform space calculations through the Pull mode of the message queue.

[0054] The state management module is used to manage the lifecycle states of subtasks through a multi-level state machine and to perform re-queueing operations on subtasks in abnormal states based on a hierarchical retry strategy.

[0055] The cache-aware module is used to bind spatially adjacent subtasks to the same working node based on the cache-aware scheduling strategy, and dynamically adjust the global concurrency based on cache hit rate feedback.

[0056] The results aggregation module is used to perform multi-stage aggregation processing on the results of subtasks through a multi-stage aggregation pipeline, and output multi-modal processing results.

[0057] The further optimized solution is to adopt a master-slave architecture combined with a message queue design, which is divided into a business application layer, a task scheduling layer, a node execution layer and an infrastructure layer from top to bottom, and each layer is decoupled through generic interfaces;

[0058] The message scheduling module is also used to perform dynamic registration and heartbeat keep-alive when the worker node starts up, and to dynamically sense the online status and load of the worker node based on the heartbeat keep-alive signal of the worker node.

[0059] The multi-level state machine defined by the state management module contains at least six state levels, including at least the pending state, the running state, the success state, the first-level error state, the second-level error state, and the failure state.

[0060] A further optimized solution is that the multi-stage aggregation process performed by the result aggregation module includes, in sequence, a result collection stage, a result verification stage, a format conversion stage, a result merging stage, and a multimodal output stage;

[0061] The task abstraction module encapsulates the GisInfo data class, which natively integrates spatial range, coordinate system, and GIS service address to enable the system to have spatial awareness capabilities.

[0062] The processing results output by the multimodal output stage include at least one or more of the following: spatial dataset, image set, PPT image set, and Word report.

[0063] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0064] By abstracting spatial business tasks into spatial task objects that include spatial range, coordinate system, and GIS service address, the framework gains spatial data awareness capabilities. Combined with clustering and grouping based on spatial adjacency and spatial filling curve sorting, worker nodes continuously acquire spatially adjacent subtasks. Furthermore, by dynamically adjusting the global concurrency based on cache hit rate feedback to improve cache hit rate, the framework utilizes the worker nodes' proactive pull mode to achieve automatic load balancing. In addition, a multi-level state machine and hierarchical retry mechanism reduce the overall error rate caused by occasional failures. Finally, a multi-stage aggregation pipeline merges the subtask execution results into multiple modalities such as spatial datasets, image sets, PPT image sets, and Word reports. Moreover, the entire system does not rely on heavy basic components such as Hadoop and Spark; it can be deployed and run with only the main application, worker nodes, and message queues. Attached Figure Description

[0065] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0066] In the attached image:

[0067] Figure 1 A flowchart illustrating a parallel processing method for complex spatial data provided in an embodiment of this application;

[0068] Figure 2 A flowchart of the spatially-aware adaptive task partitioning method provided in the embodiments of this application;

[0069] Figure 3 Detailed flowchart of the task execution process provided in the embodiments of this application;

[0070] Figure 4 A state transition diagram of a six-level task state machine provided in the embodiments of this application;

[0071] Figure 5 This is an example of a thematic map of a village provided in an embodiment of this application;

[0072] Figure 6A schematic diagram of the master-slave architecture and core modules for parallel processing of complex spatial data provided in an embodiment of this application;

[0073] Figure 7 This is a schematic diagram of a layered architecture for a parallel processing system for complex spatial data, provided in an embodiment of this application. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0075] First, some of the technical terms used in this application will be explained to help those skilled in the art understand this application.

[0076] GIS: Geographic Information System;

[0077] HTTP: HyperText Transfer Protocol;

[0078] OGC: Open Geospatial Consortium;

[0079] ArcGIS: Environmental Systems Research Institute Geographic Information System.

[0080] GeoServer: Geospatial Server;

[0081] GisInfo: Geographic information data class.

[0082] Firstly, such as Figure 1 As shown, this application provides a parallel processing method for complex spatial data, including:

[0083] Step S1: Receive spatial business tasks. Abstract spatial business tasks into spatial task objects that include spatial extent, coordinate system, and GIS service address.

[0084] Step S2: Cluster the spatial task objects based on spatial adjacency, sort the subtasks after grouping according to the spatial filling curve, and enqueue them into an ordered subtask sequence.

[0085] Step S3: Using the Pull mode of the message queue, the worker node actively pulls the subtask and performs space calculations to obtain the subtask execution result;

[0086] Step S4: Perform multi-level state machine management on the lifecycle state of subtasks, and perform re-queueing operation on subtasks in abnormal states based on the hierarchical retry strategy.

[0087] Step S5: Based on the cache-aware scheduling strategy, bind spatially adjacent subtasks to the same working node, and dynamically adjust the global concurrency based on cache hit rate feedback;

[0088] Step S6: Perform multi-stage aggregation processing on the sub-task execution results through the multi-stage aggregation pipeline, and output multi-modal processing results.

[0089] This embodiment enables the framework to have spatial data awareness by abstracting spatial business tasks. Then, it clusters and groups tasks based on spatial adjacency relationships and sorts them into queues according to spatial filling curves. This allows adjacent spatial subtasks to be arranged consecutively to reuse GIS server cached data and improve cache hit rate. On this basis, the message queue pull mode allows worker nodes to actively pull tasks according to their own computing power to achieve adaptive load balancing. At the same time, a six-level state machine and a graded retry mechanism are used to reduce the error rate caused by occasional failures. Furthermore, the global concurrency is dynamically adjusted based on cache hit rate feedback. Finally, the subtask execution results are aggregated into multiple modal outputs through a multi-stage aggregation pipeline. The entire system does not rely on heavy basic components and can be deployed and run lightly with only the main application, worker nodes, and message queues.

[0090] In one embodiment, step S1: receiving a spatial service task, whereby the abstract spatial service task is a spatial task object containing a spatial extent, coordinate system, and GIS service address, specifically including the following steps:

[0091] Step S11: Obtain the spatial business task to be processed, extract the spatial range, coordinate system and target GIS service address information from the spatial business task, and generate a business data object to be encapsulated.

[0092] Specifically, the spatial operational tasks originate from natural resource surveys, land spatial planning, or ecological environment monitoring scenarios, and their tasks include at least spatial overlay analysis, buffer analysis, topology checking, or thematic map generation of spatial datasets. The spatial extent refers to the spatial coverage boundary of the spatial dataset, represented in geographic coordinates or projected coordinates.

[0093] Step S12: Write the extracted spatial range, coordinate system, and target GIS service address information into the GisInfo data class to obtain the encapsulated spatial information object.

[0094] Specifically, the GisInfo data class is a standardized information container for spatial data, containing at least a spatial extent field, a coordinate system field, and a GIS service address field. The spatial extent field records the smallest bounding rectangle or administrative boundary of the spatial dataset; the coordinate system field records the geographic coordinate system or projected coordinate system of the spatial data, ensuring the consistency of the spatial reference for subsequent spatial calculations; and the GIS service address field records the network access address of the target GIS service, pointing to ArcGIS, GeoServer, or OGC standard map services. The GisInfo data class, as a typical implementation of the generic parameter T, enables subtasks to possess spatial data awareness capabilities from the moment of creation.

[0095] Step S13: Based on the generic task abstraction model, construct a spatial task object using the GisInfo data class;

[0096] The space task object includes a task identifier, task type, business data, retry count, and lifecycle status.

[0097] The generic task abstract model includes the SubTask and SubTaskResult generic classes, which, together with WorkerInfo, form a three-layer abstract model of task-data-resource. The SubTask generic class is used to encapsulate subtask instances, and the SubTaskResult generic class is used to encapsulate subtask execution results. The two establish a one-to-one correspondence through task identifiers.

[0098] The business data fields of the SubTask generic class use a generic parameter T, with the GisInfo data class as the actual type parameter of the generic parameter T, so that the SubTask instance holds the encapsulated spatial information object.

[0099] The retry count of the SubTask generic class is used to record the number of times the subtask has been executed, and the lifecycle state is used to record the current execution stage state of the subtask.

[0100] The result data field of the SubTaskResult generic class uses a generic parameter R, which allows different result data types to be defined for different business scenarios.

[0101] Preferably, when integrating a business application, the framework is accessed by implementing the TaskHandler and TaskProcessor interfaces. The TaskHandler interface is defined on the main application side and includes methods for task splitting, result receiving, and result aggregation. The TaskProcessor interface is defined on the worker node side and includes business processing methods. Business developers do not need to understand the underlying details of distributed scheduling such as message queues, state machines, and load balancing. They only need to implement the above four core methods to access all the capabilities of the framework. The business layer and the task scheduling layer are decoupled through generic interfaces.

[0102] TaskHandler<T,R> and TaskProcessor<T,R> The method signatures for the two generic interfaces are designed as follows:

[0103] The TaskHandler interface is defined on the main application side and contains a List <SubTask <t>The `split(TaskConfig)` method is used to split the main task into a list of subtasks. `void receive(SubTaskResult)` <r>The `result()` method is used to receive and process the results of a single subtask. `void aggregate(List< ... <SubTaskResult <r>The `results` method is used to aggregate the sub-results into the final output after all subtasks have reached their final state; the `TaskProcessor` interface is defined on the worker node side and contains `SubTaskResult`. <r>process(SubTask <t>The `task` method is used to execute business logic at the execution node and return the result.

[0104] The method signatures for the two interfaces are designed as follows:

[0105] Table 1. Method signature design table for two interfaces

[0106]

[0107] The above interface method signatures define the specific communication contract between the business access layer and the framework layer; from an overall perspective, a summary of all core classes in the framework (including the two interfaces mentioned above) and their type parameters is shown in the following table:

[0108] Table 2 Core Classes of the Generic Task Abstraction Model

[0109]

[0110] This embodiment extracts and encapsulates the spatial extent, coordinate system, and GIS service address information from spatial business tasks into the GisInfo data class. Then, based on a generic task abstraction model, it constructs SubTask spatial task objects carrying spatial information. This enables the framework to natively possess spatial data awareness capabilities during the task creation phase, laying a data foundation for subsequent optimizations such as spatial partitioning and cache scheduling. Simultaneously, through the decoupled design of the TaskHandler and TaskProcessor interfaces, business developers only need to focus on the implementation of the three core business logics: split, process, and aggregate, without needing to delve into the underlying details of distributed scheduling such as message queues and state machines. This significantly lowers the barrier to entry for business integration and achieves complete decoupling between the task scheduling layer and the business layer.

[0111] In one embodiment, such as Figure 2 As shown, step S2: Clustering and grouping spatial task objects based on spatial adjacency relationships, sorting the grouped subtasks according to the spatial filling curve, and queuing the generated ordered subtask sequence, specifically includes the following steps:

[0112] Step S21: Obtain the set of SubTask spatial task objects constructed in step S1, construct an R-tree spatial index based on the spatial range information carried by each subtask, and determine the location range and adjacency relationship of each spatial element through the spatial adjacency of the R-tree.

[0113] The R-tree spatial index is used to organize spatial objects according to their minimum bounding rectangle, supporting spatial range queries and adjacency determination. When constructing the R-tree spatial index, the spatial range of each subtask in the spatial task object set is used as input. The minimum bounding rectangle of each spatial element is inserted into the R-tree node in sequence to form a hierarchical spatial index structure. The balance of the tree is maintained through the node splitting and adjustment mechanism of the R-tree.

[0114] Step S22: Based on the adjacency relationship in the R-tree spatial index, perform spatial clustering grouping on the map features, divide the spatially adjacent or continuous map features into the same partition, and output the spatial clustering grouping results.

[0115] The specific implementation of spatial clustering grouping of map features based on adjacency relationships in the R-tree spatial index is as follows: starting from the root node of the R-tree, the adjacency relationships between spatial elements are obtained, map features with common edges or common vertices are grouped into the same cluster group, and spatially adjacent or continuous map features are assigned to the same partition. The clustering threshold is adaptively adjusted according to the spatial data density to avoid excessive differences in group size.

[0116] Step S23: Balance the load of each partition in the spatial clustering grouping results and obtain the partition results after load balancing.

[0117] The load balancing adjustment method is as follows: the estimated processing complexity of each partition is evaluated based on the area and number of vertices of the patches in each partition. Partitions with estimated complexity exceeding the preset upper limit are split, and adjacent partitions with estimated complexity below the preset lower limit are merged, so that the estimated load of each partition tends to be balanced.

[0118] Step S24: Encapsulate each partition in the load-balanced partitioning result as a SubTask object, sort the subtasks according to the space filling curve, generate an ordered subtask sequence, and push it into the message queue.

[0119] The space-filling curve, either a Hilbert curve or a Z-order curve, is used to map the order of subtasks in a two-dimensional space to a one-dimensional sequence. When sorting subtasks according to the space-filling curve, the encoding value of each subtask's corresponding partition on the space-filling curve is calculated, and the subtasks are sorted in ascending order of encoding value, ensuring that spatially adjacent subtasks are in consecutive positions in the queue.

[0120] This embodiment obtains spatial feature adjacency relationships by constructing an R-tree spatial index, and performs spatial clustering and grouping of map features based on adjacency relationships, placing spatially adjacent map features into the same partition. After load balancing adjustment of the partitions, they are sorted and queued according to spatial fill curves, ensuring that spatially adjacent subtasks are arranged continuously in the queue. This spatially aware adaptive partitioning method provides a crucial data prerequisite for subsequent worker nodes to continuously acquire and process spatially adjacent tasks and reuse cached data from the GIS server, effectively solving the cache eviction problem caused by frequent switching of processing areas by worker nodes.

[0121] In one embodiment, such as Figure 3 As shown, step S3: Through the Pull mode of the message queue, the worker node actively pulls the subtask and performs space calculations to obtain the subtask execution result. This specifically includes the following steps:

[0122] Step S31: After the worker node starts up, it sends a registration request to the main application. After completing the registration, it periodically reports a heartbeat keep-alive signal to the main application so that the main application can obtain the online status and load information of the worker node.

[0123] Specifically, when a worker node starts up, it submits registration information to the main application via an HTTP interface. This registration information includes the node identifier, node address, number of available processor cores, and memory capacity. Upon receiving the registration information, the main application assigns a node identifier to the worker node and records its online status. After registration, each worker node sends a heartbeat keep-alive signal to the main application at preset time intervals. This heartbeat signal includes the node identifier, the number of tasks currently being processed, and CPU utilization. The main application updates the load status of the worker nodes based on the heartbeat keep-alive signal. If the main application does not receive a heartbeat keep-alive signal from a worker node within a preset time window, it marks that worker node as offline. New worker nodes only need to be started and registered to automatically join the cluster and participate in task processing, without requiring modifications to the main application configuration, thus supporting plug-and-play horizontal scaling.

[0124] Step S32: The worker node determines the pull frequency based on its current number of processing tasks and CPU utilization, and initiates a pull request to the task queue of the main application to trigger the main application to respond to the pull request by performing an exclusive retrieval operation from the task queue to obtain a sub-task to be executed and return it to the worker node.

[0125] Specifically, when the number of currently processed tasks falls below a preset threshold, a worker node initiates a fetch request to the main application. When the number of currently processed tasks reaches the preset threshold, the worker node pauses fetching until the number of currently processed tasks drops below the preset threshold. Upon receiving a fetch request, the main application retrieves a subtask from the head of the message queue. This retrieval operation is atomic; once retrieved, the subtask is immediately removed from the queue, ensuring that only one worker node retrieves the same subtask. When the message queue is empty, the main application returns an empty response, and the worker node waits for a preset time interval before re-initiating the fetch request.

[0126] Step S33: After the worker node obtains the subtask to be executed, it sets the subtask status to running, calls the business processing method to perform business logic processing on the subtask, and generates the subtask execution result.

[0127] Specifically, the worker node extracts business data from the subtask and calls the `process` method defined in the `TaskProcessor` interface to execute business logic processing. The input parameter of the `process` method is a `SubTask` object, and the output is a `SubTaskResult` object. Internally, the `process` method executes specific spatial calculation tasks, including GIS screenshotting, spatial overlay analysis, buffer analysis, or topology checking. When the business logic processing completes normally, the `process` method returns a `SubTaskResult` object and sets the success flag to true; when the business logic processing encounters an error, the `process` method returns a `SubTaskResult` object and sets the success flag to false, and the error information is recorded in the error field of the `SubTaskResult` object.

[0128] Step S34: The worker node submits the subtask execution result to the main application, so that the main application updates the lifecycle state of the subtask based on the execution result and notifies the worker node to continue executing the next subtask.

[0129] In this process, worker nodes submit SubTaskResult objects to the main application via an HTTP interface. The SubTaskResult object contains a task identifier, a success flag, result data, and error information. Upon receiving the execution result, the main application updates the subtask status to either success or error based on the success flag and stores the execution result in a result buffer. After submitting the execution result, the worker node returns to step S32 to continue sending pull requests to the main application for the next subtask, repeating this process until the message queue is empty.

[0130] Preferably, the main application periodically detects the heartbeat keep-alive signal of the worker nodes. When the heartbeat of a worker node times out, the worker node is marked as offline, and its currently executing subtask is re-enqueued, waiting for other worker nodes to pull and execute it.

[0131] This embodiment achieves decentralized adaptive load balancing by having worker nodes proactively pull tasks based on their processing capabilities. Nodes with high processing power automatically consume more tasks, while nodes with low processing power automatically consume fewer tasks, eliminating the need for the main application to implement complex centralized scheduling algorithms. Simultaneously, the dynamic registration and heartbeat keep-alive mechanism of worker nodes enables the cluster to support plug-and-play horizontal scaling, flexibly adjusting computing resources according to business load.

[0132] The following table compares the Pull mode of this invention with the traditional direct HTTP communication Push mode:

[0133] Table 3 Performance Comparison of Pull Mode and Push Mode

[0134]

[0135] In one embodiment, step S4: performing multi-level state machine management on the lifecycle states of subtasks, and performing re-queue operations on subtasks in abnormal states based on a hierarchical retry strategy, specifically including the following steps:

[0136] Step S41: After the worker node submits the subtask execution result, the main application updates the subtask status according to the success flag in the execution result and outputs the subtask record with the updated status.

[0137] The status update rules are as follows:

[0138]

[0139] A single subtask can be executed a maximum of 3 times (1 initial execution + 2 retries). When the cumulative number of failures reaches 3, it enters a failure state to avoid excessive consumption of computing resources.

[0140] Specifically, the subtask status includes pending, running, successful, first-level error, second-level error, and failed states. The pending state indicates the subtask has been queued and is waiting to be consumed; the running state indicates the subtask has been acquired by a worker node and is being executed; the successful state indicates the subtask has executed successfully and the result has been received; the first-level error state indicates the subtask failed on its first execution but can be retried; the second-level error state indicates the subtask failed a second time but can be retried; and the failed state indicates the subtask ultimately fails after reaching the maximum number of retries. The subtask initially has a pending state, and its status changes to running after a worker node acquires the subtask.

[0141] The state transition rules are as follows:

[0142] pending → running (Worker node acquires task);

[0143] running→success (Execution successful);

[0144] running→error1 (first failure);

[0145] running→error2 (second failure);

[0146] running→fail (Third failure, no further attempts);

[0147] error1 / error2 → pending (re-queueing and retrying).

[0148] Each subtask can be executed a maximum of 3 times (1 initial execution + 2 retries) to avoid excessive consumption of computing resources.

[0149] After receiving the execution result submitted by the worker node, the main application performs state transitions according to the success flag of the execution result and the cumulative number of failures of the subtasks, following these rules:

[0150] When the success flag is true, the status changes from the running state to the success state;

[0151] When the success flag is false and the cumulative number of failures equals 1, the status changes from running to Level 1 error status; when the success flag is false and the cumulative number of failures equals 2, the status changes from running to Level 2 error status.

[0152] When the success flag is false and the cumulative number of failures equals 3, the status changes from running to failed.

[0153] Subtasks in a Level 1 or Level 2 error state will be scanned and re-queued by the main application's scheduled task, and their status will be rolled back to pending state for retry; subtasks in a failed state will not be retried.

[0154] Step S42: The main application periodically scans for subtasks in the first or second level error state, performs a re-entry operation on retryable subtasks, rolls their state back to the pending state, and pushes them back into the message queue.

[0155] Specifically, for a subtask in a first-level error state, the main application increments its retry count by 1 and rolls back its status to a pending state, then pushes the subtask back into the message queue to wait for execution again; for a subtask in a second-level error state, the main application increments its retry count by 1 and rolls back its status to a pending state, then pushes the subtask back into the message queue to wait for one last execution opportunity.

[0156] Step S43: The main application obtains the execution results of each subtask and summarizes the results. It performs fault tolerance processing on the failed subtasks according to the fault tolerance mode configuration and outputs the final summarized result.

[0157] The aggregation phase supports two fault tolerance modes: strict fault tolerance mode requires all subtasks to succeed before aggregation can proceed, and aggregation will be blocked and await manual intervention if any subtask fails; lenient fault tolerance mode skips subtasks that fail, only aggregates the results of successful subtasks, and marks the missing parts in the final output.

[0158] Preferably, the fault tolerance mode is controlled by configuration parameters. When the service is accessed, the appropriate fault tolerance mode is selected according to the task scenario. For scenarios with high requirements for result integrity, a strict fault tolerance mode is selected, and for scenarios with low requirements for result integrity, a lenient fault tolerance mode is selected.

[0159] Step S44: The main application records the information of the failed subtasks to the log, displays the error information of the failed subtasks to the user, and waits for manual intervention.

[0160] This embodiment defines six state levels—pending, running, success, error1, error2, and fail—and employs a tiered retry mechanism with up to three attempts to achieve refined fault tolerance. For intermittent failures such as network fluctuations and momentary service unavailability, automatic retries can restore the system, preventing a single failure from causing the entire task to fail. For persistent failures, differentiated handling is achieved through strict or lenient fault tolerance modes, significantly improving the overall robustness of the system in unreliable network environments.

[0161] like Figure 4 As shown, the multi-level state machine defined in this application includes six state levels: pending, running, success, first error, second error, and failure. The entry conditions and descriptions for each state are shown in the table below:

[0162] Table 4. State Description Table for Level 6 Task State Machine

[0163]

[0164] In one embodiment, step S5: scheduling adjacent subtasks based on a cache-aware scheduling strategy and dynamically adjusting the global concurrency based on cache hit rate feedback, specifically includes the following steps:

[0165] Step S51: The main application obtains the spatial clustering grouping results generated in step S2, creates an independent sub-queue for each spatial clustering group, and places the sub-tasks of the same spatial cluster in the same sub-queue for continuous arrangement, so that the worker nodes can preferentially obtain tasks from the same sub-queue during the retrieval process, and outputs the grouped queue scheduling structure.

[0166] The main application allocates an independent sub-queue for each spatial cluster group based on the spatial clustering results. Subtasks belonging to the same spatial cluster are sorted according to the spatial filling curve and then continuously written to the sub-queue. When a worker node retrieves a subtask via the pull interface, it prioritizes pulling tasks continuously from the allocated sub-queue. When the sub-queue is empty, it falls back to the global queue to retrieve tasks from other regions. This design ensures that the same worker node processes tasks from the same spatial region centrally within a certain period, maximizing the GIS server's cache reuse effect.

[0167] Step S52: The main application obtains the GIS service cache hit rate statistics reported by each working node during the execution of subtasks, adjusts the global concurrency based on the comparison result of the cache hit rate and the preset threshold, and outputs the adjusted concurrency control parameters.

[0168] In this process, worker nodes record the cache hit rate for each GIS service call while performing spatial computation tasks, and periodically report the cache hit rate statistics to the main application. After receiving the statistics, the main application makes a comprehensive judgment based on the average response time of the GIS server, the current global cache hit rate, and a preset hit rate threshold. If the current cache hit rate is lower than the preset threshold or the average response time is too long, the global concurrency is reduced to decrease the number of worker nodes accessing the GIS service simultaneously, thus alleviating cache contention. If the current cache hit rate is higher than the preset threshold and the average response time is shorter, the global concurrency is increased to increase the number of worker nodes accessing the GIS service simultaneously, thereby improving overall throughput efficiency.

[0169] Step S53: The main application obtains the cache hit status information reported by each worker node, adjusts the multi-queue grouping strategy according to the feedback information, and sends it to each worker node so that the worker nodes can perform task retrieval according to the adjusted grouping strategy.

[0170] Specifically, each worker node generates cache hit status information after each GIS service call. The cache hit status information includes at least whether the cache was hit in that call and the identifier of the cache region that was hit. The worker nodes report the cache hit status information to the main application along with the subtask execution results. After receiving the cache hit status information from all worker nodes, the main application calculates the global cache hit rate for each spatial region. For spatial regions with a cache hit rate lower than a preset regional threshold, the scheduling weight of the corresponding subqueue for that region is reduced; for spatial regions with a cache hit rate higher than the preset regional threshold, the scheduling weight of the corresponding subqueue for that region is increased.

[0171] Preferably, the main application updates the scheduling weights of each spatial region sub-queue according to the following formula:

[0172] ;

[0173] in, Let be the scheduling weight of the i-th spatial region sub-queue at the current moment. Let be the cache hit rate of the i-th spatial region at the current time. The average cache hit rate for all spatial regions. The learning rate is used. Worker nodes pull tasks from different sub-queues based on the adjusted scheduling weights. This formula ensures that regions with above-average cache hit rates receive higher scheduling weights, and vice versa, forming a closed-loop control.

[0174] This embodiment uses multi-queue grouping management to bind subtasks of the same spatial cluster to specific work nodes, and dynamically adjusts the global concurrency based on the average response time and cache hit rate feedback of the GIS server, achieving a dynamic balance between improving cache hit rate and ensuring system throughput. This cache-aware scheduling strategy maximizes the use of the GIS server's local caching characteristics and reduces redundant calculations, making it one of the core elements for improving overall processing efficiency.

[0175] In one embodiment, step S6: Performing multi-stage aggregation processing on the subtask execution results through a multi-stage aggregation pipeline to output multi-modal processing results, specifically including the following steps:

[0176] Step S61: The main application obtains the execution results of the subtasks submitted by each worker node, stores the subtask execution results into the result buffer according to the task identifier, and outputs a complete list of subtask results after all subtasks reach the final state.

[0177] The main application receives SubTaskResult objects submitted by each worker node via HTTP and stores them in the result buffer according to the task identifier in the SubTaskResult. The main application continuously monitors the lifecycle status of all subtasks. When all subtasks reach the success or failure state, it determines that the aggregation condition is met, retrieves all subtask results from the result buffer, and outputs a complete list of subtask results.

[0178] Step S62: The main application obtains the complete list of subtask results, performs integrity verification on each subtask result, filters out results that fail verification, and obtains the list of valid results that pass verification.

[0179] Specifically, the main application iterates through the complete list of subtask results, performing three checks on each subtask result: result file existence check, verifying that the file paths referenced in the subtask result are accessible and the file content is complete; data format check, verifying that the data structure of the subtask result conforms to the expected format definition; and content consistency check, verifying that key fields such as spatial range and coordinate system in the subtask result are consistent with the corresponding information in the original spatial task object. Subtask results that fail the checks are marked and filtered, while subtask results that pass the checks are added to the list of valid results.

[0180] Step S63: The main application obtains the list of valid results that have passed the verification, performs format conversion on the results of each subtask according to the configured output type, and outputs a list of intermediate result objects in a unified format.

[0181] The output type is preset through the task configuration file as follows.

[0182] When the output type is a PPT image gallery, the screenshots in the results of each subtask will be converted into PPT page objects.

[0183] When the output type is a Word report, convert the statistical data in the results of each subtask into Word table objects or the analysis results into Word chart objects;

[0184] When the output type is a spatial dataset, the spatial data in the results of each subtask is preserved as a spatial feature set object.

[0185] The results of each subtask after conversion are combined into a list of intermediate result objects in a unified format.

[0186] Step S64: The main application obtains a list of intermediate result objects in a unified format, merges the intermediate result objects into a final output file according to a preset merging strategy, and outputs the merged final output file.

[0187] Specifically, the preset merging strategy is determined based on the output type. The PPT merging strategy appends each intermediate result object to the target PPT file sequentially according to the spatial order of the subtasks, forming a complete atlas; the Word merging strategy appends each intermediate result object to the target Word document sequentially according to the spatial order of the subtasks, forming a complete report; the dataset merging strategy merges spatial elements in each intermediate result object according to spatial fields and performs geometric stitching, forming a complete spatial dataset. The spatial order of the results from each subtask is maintained during the merging process.

[0188] Step S65: The main application obtains the final output file after merging, writes the final output file to the shared file system and records the file path, updates the overall task status to complete, and outputs the task completion status and the final output file path.

[0189] Preferably, when some subtasks enter a failed state, the multi-stage aggregation pipeline performs corresponding processing according to the fault tolerance mode configuration. In the relaxed fault tolerance mode, the corresponding results of the failed subtasks are automatically skipped, and partial aggregation is performed on the results of the successful subtasks, marking the position of the missing part in the final output; in the strict fault tolerance mode, if any subtask enters a failed state, the merging process is blocked, waiting for manual intervention.

[0190] This embodiment supports the automated generation of multimodal results such as PPT atlases, Word reports, and spatial datasets through a five-stage aggregation process of collection, verification, transformation, merging, and output. It fills the gap in the output capabilities of existing parallel spatial data processing solutions for office documents and meets the urgent needs of the natural resources field for diverse forms of results.

[0191] The specific process of multi-stage aggregation is shown in the table below:

[0192] Table 5. Five-stage processing flow chart for multimodal polymerization pipelines

[0193]

[0194] The thematic map output after multi-stage aggregation pipeline processing is as follows: Figure 5 As shown.

[0195] Based on the synergy of the aforementioned generic abstraction, spatial partitioning, adaptive load balancing, hierarchical fault tolerance, cache-aware scheduling, and multimodal aggregation technologies, this invention forms a parallel processing framework that deeply integrates the characteristics of spatial data processing services.

[0196] The technical effects of the present invention are further illustrated below through complete implementation examples:

[0197] Taking the task of generating a planning atlas for 200 administrative villages in a county-level administrative region as an example, each village needs to generate 8-10 thematic maps, which are ultimately merged into a complete county-wide atlas PPT. After the user submits the task, the main application uses spatial awareness partitioning to divide the 200 administrative villages into 15-20 groups based on spatial clustering. Five working nodes execute the screenshot task in parallel, continuously acquiring tasks within the same cluster group, resulting in a GIS service cache hit rate of approximately 70%. After all sub-tasks are completed, the aggregation pipeline merges the sub-PPTs into a complete atlas PPT. Compared to the traditional manual processing cycle of 3-5 days, this solution can complete the task in only 1-2 hours, improving processing efficiency by 30-60 times; the error rate is reduced from 5-10% in the traditional method to below 0.1%, and the result consistency reaches 100%, fully verifying the practicality and advancement of this solution.

[0198] Secondly, this application provides a parallel processing system for complex spatial data, including:

[0199] The task abstraction module is used to abstract received spatial business tasks into spatial task objects that include spatial extent, coordinate system, and GIS service address.

[0200] The spatial partitioning module is used to cluster and group spatial task objects based on spatial adjacency, and sort the grouped subtasks according to the spatial filling curve to generate an ordered subtask sequence.

[0201] The message scheduling module is used to allow worker nodes to actively pull subtasks and perform space calculations through the Pull mode of the message queue.

[0202] The state management module is used to manage the lifecycle states of subtasks using a multi-level state machine, and to perform re-queueing operations on subtasks in abnormal states based on a hierarchical retry strategy.

[0203] The cache-aware module is used to bind spatially adjacent subtasks to the same working node based on the cache-aware scheduling strategy, and dynamically adjust the global concurrency based on cache hit rate feedback.

[0204] The results aggregation module is used to perform multi-stage aggregation processing on the results of subtasks through a multi-stage aggregation pipeline, and output multi-modal processing results.

[0205] like Figure 6 As shown, the modules work together through a master-slave architecture and message queues to form a complete task scheduling and execution closed loop.

[0206] The further optimized solution is to adopt a master-slave architecture combined with a message queue design, which is divided into a business application layer, a task scheduling layer, a node execution layer and an infrastructure layer from top to bottom, and each layer is decoupled through generic interfaces;

[0207] The core components and main responsibilities of each architectural layer are shown in the table below:

[0208] Table 6. Responsibilities of Core Components in the Layered System Architecture

[0209]

[0210] like Figure 7 As shown, this system adopts a master-slave architecture combined with message queue design, and is divided into four layers from top to bottom: business application layer, task scheduling layer, node execution layer and infrastructure layer. The layers are decoupled through generic interfaces.

[0211] The data flow between the core components of the architecture forms a complete closed loop: users submit task requests through the business application layer, the task requests are passed to the task scheduling layer, the task splitting module splits them into subtasks and serializes them into the message queue; the worker nodes in the node execution layer actively pull subtasks from the message queue for execution, and after execution, they send the results back to the status management module of the task scheduling layer via HTTP interface; the status management module updates the status of the subtasks and triggers the result aggregation module to perform multimodal aggregation; the final result is persisted through file storage and displayed to the user through the business application layer. The entire process follows the logical link of "submission-splitting-pulling-execution-return-aggregation-output".

[0212] The message scheduling module is also used to perform dynamic registration and heartbeat keep-alive when the worker node starts up, and to dynamically sense the online status and load of the worker node based on the heartbeat keep-alive signal of the worker node.

[0213] The multi-level state machine defined by the state management module contains at least six state levels, including at least the pending state, the running state, the success state, the first-level error state, the second-level error state, and the failure state.

[0214] A further optimized solution is that the multi-stage aggregation process performed by the result aggregation module includes, in sequence, a result collection stage, a result verification stage, a format conversion stage, a result merging stage, and a multimodal output stage;

[0215] The task abstraction module encapsulates the GisInfo data class, which natively integrates spatial range, coordinate system, and GIS service address to enable the system to have spatial awareness capabilities.

[0216] The processing results output by the multimodal output stage include at least one or more of the following: spatial dataset, image set, PPT image set, and Word report.

[0217] The functions of each module in the parallel processing device for complex spatial data described above correspond to the steps in the embodiment of the parallel processing method for complex spatial data described above, and their functions and implementation processes will not be described in detail here.

[0218] Thirdly, embodiments of this application provide a parallel processing device for complex spatial data. The parallel processing device for complex spatial data can be a personal computer (PC), a laptop computer, a server, or other devices with data processing capabilities.

[0219] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0220] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within a parallel processing device for complex spatial data, as well as interfaces used for interconnecting the parallel processing device for complex spatial data with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0221] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0222] The processor can be a general-purpose processor, which can call a parallel processing program for complex spatial data stored in memory and execute the parallel processing method for complex spatial data provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the parallel processing program for complex spatial data is called can be referred to the various embodiments of the parallel processing method for complex spatial data in this application, and will not be repeated here.

[0223] Fourthly, embodiments of this application also provide a readable storage medium.

[0224] The present application has a readable storage medium storing a parallel processing program for complex spatial data, wherein when the parallel processing program for complex spatial data is executed by a processor, it implements the steps of the parallel processing method for complex spatial data as described above.

[0225] The method implemented when the parallel processing program for complex spatial data is executed can be referred to in the various embodiments of the parallel processing method for complex spatial data in this application, and will not be repeated here.

[0226] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0227] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / t> < / r> < / r> < / r> < / t>

Claims

1. A parallel processing method for complex spatial data, characterized in that, include: Receive spatial business tasks, and abstract spatial business tasks as spatial task objects that include spatial extent, coordinate system and GIS service address; Based on spatial adjacency, spatial task objects are clustered and grouped, and the subtasks after grouping are sorted according to the spatial filling curve. After generating an ordered sequence of subtasks, they are enqueued. By using the Pull mode of the message queue, the worker node actively pulls the subtask and performs space calculations to obtain the subtask execution result; The lifecycle states of subtasks are managed by a multi-level state machine, and subtasks in abnormal states are re-entered into the queue based on a hierarchical retry strategy. Based on a cache-aware scheduling strategy, spatially adjacent subtasks are bound to the same working node, and the global concurrency is dynamically adjusted according to cache hit rate feedback. The multi-stage aggregation pipeline performs multi-stage aggregation processing on the results of sub-tasks, outputting multi-modal processing results.

2. The parallel processing method for complex spatial data according to claim 1, characterized in that, The Pull mode via message queue, where worker nodes actively pull subtasks and perform space calculations, specifically includes: When a worker node starts up, it performs dynamic registration with the main application and periodically sends a heartbeat keep-alive signal. Work nodes exclusively pull matching subtasks from the message queue based on their real-time computing power; The main application dynamically senses the online status and load of worker nodes based on their heartbeat keep-alive signals, achieving decentralized adaptive load balancing.

3. The parallel processing method for complex spatial data according to claim 1, characterized in that, The process of clustering spatial task objects based on spatial adjacency and sorting the grouped subtasks according to spatial filling curves specifically includes: Clustering and grouping are implemented based on R-tree spatial index construction and spatial adjacency relationships; The complexity of subtasks is evaluated by combining the area of ​​the patches and the number of vertices, and the grouping results are adjusted for load balancing. The adjusted subtasks are sequentially enqueued according to the spatial filling curve to maintain local continuity in geospatial space.

4. The parallel processing method for complex spatial data according to claim 1, characterized in that, The process of managing the lifecycle states of subtasks using a multi-level state machine and re-queuing subtasks in abnormal states based on a tiered retry strategy specifically includes: Define a multi-level state machine containing at least six state levels, wherein the state levels include at least a pending state, a running state, a success state, a first-level error state, a second-level error state, and a failure state; When a subtask fails to execute, its status is set to a first-level error status and it is allowed to be re-entered into the queue for execution again. When a subtask fails to execute again, its status is set to a Level 2 error status and a final retry is allowed. If a subtask fails to execute after its last retry, its status is set to the failed state. The results aggregation phase supports two aggregation strategies: strict fault tolerance mode and lenient fault tolerance mode.

5. The parallel processing method for complex spatial data according to claim 1, characterized in that, The cache-aware scheduling strategy binds spatially adjacent subtasks to the same working node and dynamically adjusts the global concurrency based on cache hit rate feedback, specifically including: Based on the clustering results of spatially aware partitioning, spatially adjacent subtasks are sorted according to the spatial filling curve and then pushed into the task queue. By using multi-queue grouping management, subtasks belonging to the same spatial cluster are bound to specific work nodes to adapt to the local caching characteristics of GIS services; Real-time monitoring of GIS service cache hit rate; when the cache hit rate is lower than the preset hit rate threshold, reduce global concurrency to adapt to cache storage constraints. When the cache hit rate is higher than the preset hit rate threshold, the global concurrency is increased to improve the overall throughput.

6. The parallel processing method for complex spatial data according to claim 1, characterized in that, The multi-stage aggregation process includes, in sequence, the result collection stage, the result verification stage, the format conversion stage, the result merging stage, and the multimodal output stage; The result verification stage includes at least the verification of data integrity and logical consistency; The format conversion stage includes at least coordinate system transformation and data format standardization; The result merging stage includes at least spatial overlay analysis and attribute information linking; The processing results output by the multimodal output stage include at least one or more of the following: spatial dataset, image set, PPT image set, and Word report.

7. The parallel processing method for complex spatial data according to claim 1, characterized in that, The abstract spatial business task is a spatial task object that includes spatial extent, coordinate system, and GIS service address, specifically including: Define the generic classes SubTask and SubTaskResult, and encapsulate the GisInfo data class which natively integrates spatial range, coordinate system, and GIS service address; When integrating a business application, only the task splitting method, business processing method, and result aggregation method need to be implemented. The task scheduling layer is decoupled from the business layer through a generic interface.

8. A parallel processing system for complex spatial data, characterized in that, include: The task abstraction module is used to abstract received spatial business tasks into spatial task objects that include spatial extent, coordinate system, and GIS service address. The spatial partitioning module is used to cluster spatial task objects based on spatial adjacency and sort the grouped subtasks according to the spatial filling curve to generate an ordered subtask sequence. The message scheduling module is used to allow worker nodes to actively pull subtasks and perform space calculations through the Pull mode of the message queue. The state management module is used to manage the lifecycle states of subtasks through a multi-level state machine and to perform re-queueing operations on subtasks in abnormal states based on a hierarchical retry strategy. The cache-aware module is used to bind spatially adjacent subtasks to the same working node based on the cache-aware scheduling strategy, and dynamically adjust the global concurrency based on cache hit rate feedback. The results aggregation module is used to perform multi-stage aggregation processing on the results of subtasks through a multi-stage aggregation pipeline, and output multi-modal processing results.

9. The parallel processing system for complex spatial data according to claim 8, characterized in that, The architecture adopts a master-slave architecture combined with a message queue design, and is divided into a business application layer, a task scheduling layer, a node execution layer and an infrastructure layer from top to bottom. Each layer is decoupled through generic interfaces. The message scheduling module is also used to perform dynamic registration and heartbeat keep-alive when the worker node starts up, and to dynamically sense the online status and load of the worker node based on the heartbeat keep-alive signal of the worker node. The multi-level state machine defined by the state management module contains at least six state levels, including at least the pending state, the running state, the success state, the first-level error state, the second-level error state, and the failure state.

10. The parallel processing system for complex spatial data according to claim 8, characterized in that, The multi-stage aggregation process performed by the result aggregation module includes, in sequence, the result collection stage, the result verification stage, the format conversion stage, the result merging stage, and the multimodal output stage; The task abstraction module encapsulates the GisInfo data class, which natively integrates spatial range, coordinate system, and GIS service address to enable the system to have spatial awareness capabilities. The processing results output by the multimodal output stage include at least one or more of the following: spatial dataset, image set, PPT image set, and Word report.

Images