A visualization, orchestration and parallel execution method based on directed acyclic graph

Abstract

The application provides a visualization arrangement and parallel execution method based on a directed acyclic graph, relates to the technical field of geographic information systems and spatial data processing, and comprises the following steps: S1, modeling of a visualization canvas and an algorithm node; S2, design of a flow model data structure; S3, a CGraph DAG parallel execution engine; and S4, XML persistence and cross-session recovery. A directed acyclic graph (DAG) is used as a flow model, an adapter node is used to connect a geographic information processing algorithm to a parallel scheduling framework, a virtual layer placeholder mechanism is used to realize automatic memory-level transmission of parameters between algorithms, and a unique connection identifier is used to realize accurate restoration of a cross-session parameter source, so that the problems of a GIS multi-step processing flow being fragmented, being unable to be executed in parallel, lacking reusability, being difficult to track data flow, and a cross-session parameter source being unable to be accurately restored in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the fields of geographic information systems and spatial data processing technology, specifically to a visualization orchestration and parallel execution method based on directed acyclic graphs. Background Technology

[0002] In the field of Geographic Information System (GIS) data production, spatial data processing typically involves the sequential execution of multiple algorithmic steps, such as coordinate transformation → clipping → merging → exporting. Traditional workflows require operators to manually execute each algorithm tool sequentially, using the output file as input for the next step after each step. This approach presents the following significant problems:

[0003] (1) Fragmented process and low efficiency: Operators need to switch between different algorithm tool interfaces repeatedly, manually specify the input and output paths each time, and temporary files occupy a lot of storage space. They also cannot intuitively understand the structural relationship of the complete processing flow.

[0004] (2) Inability to express parallel relationships: For mutually independent processing branches (such as performing the same operation on multiple regions at the same time), the traditional serial execution method cannot utilize the parallel computing capabilities of multi-core processors, resulting in excessively long processing time;

[0005] (3) Lack of reusability: All algorithm parameters need to be configured from scratch for each processing task. The same or similar processing flow cannot be saved and reused, making it difficult to improve production efficiency;

[0006] (4) Difficulty in data flow tracing: In multi-step processing, it is difficult for operators to track the flow of data between each step, and the debugging and verification process is cumbersome.

[0007] (5) Cross-session parameter source cannot be restored: After the process model is saved and reloaded in the next session, the parameter configuration of the downstream node cannot identify which upstream connection it comes from. This is particularly prominent in scenarios where nodes have the same name or multiple connections point to the same parameter, resulting in the loss or mismatch of historical process configuration.

[0008] Patent CN201810292764.0, "Parallel Scheduling Method and Device for Remote Sensing Algorithm Flow" (Beijing Aerospace Hongtu), proposes a method for identifying node dependencies based on DAG topology and using a thread pool for parallel scheduling of dependency-free nodes. The differences between this invention and the present invention are: ① This patent only targets remote sensing image processing, focusing on specific scenarios such as forest monitoring, and does not support all types of GIS algorithms, including vector and raster algorithms; ② Its data transfer between algorithms depends on file paths, resulting in significant disk I / O overhead, and it lacks a direct transfer mechanism for memory-level layer objects (GsLayerPtr); ③ It lacks a two-way binding mechanism between virtual layer placeholder entries and connectionId, making it impossible to accurately restore the parameter source after saving / loading the flow model across sessions; ④ It does not support visual canvas drag-and-drop arrangement and lacks distinction between dual semantic connections (data stream connections / temporal dependency connections).

[0009] Patent CN202210256586.2, "A Distributed Scheduling Method for Remote Sensing Data Processing Plugins Based on a Workflow Engine" (54th Research Institute of China Electronics Technology Group Corporation), adopts a two-level distributed scheduling mode of "logic control service + master-slave nodes," supporting visual orchestration and batch scheduling of remote sensing data processing plugins. The differences between this invention and the present invention are: ① Plugin calls are completed using file-based parameter passing, without direct flow of memory-level layer objects; ② Multi-node cluster collaboration is achieved based on a microservice architecture, unlike the present invention's technical approach of single-machine localized C++ multi-threaded true parallelism; ③ It lacks a cross-session restoration mechanism for virtual layer bidirectional binding using connectionId as the anchor point; ④ Plugin scheduling is based on a standard workflow engine and cannot access the self-developed GIS algorithm base class (QgsProcessingAlgorithm).

[0010] Patent CN202010729398.8, "Task Flow Orchestration Method and Device, Electronic Equipment" (Inspur Cloud Information Technology Co., Ltd.): Provides a general-purpose timed task visual orchestration based on the Quartz framework, supporting drag-and-drop configuration and multi-flow linkage scheduling. The differences between this invention and the present invention are: ① It is geared towards general business task scheduling and does not support GIS spatial algorithms or self-developed algorithm base classes; ② It only achieves basic parallelism through shardable nodes and lacks C++-level multi-threaded true parallel scheduling capabilities; ③ It lacks a dedicated spatial layer memory transfer mechanism; ④ It does not have a connectionId virtual layer two-way binding cross-session restoration mechanism.

[0011] Patent CN110377287A, "A Workflow-Based GIS Data Processing Method," proposes a GIS processing scheme based on linear workflows. This scheme only supports sequential execution, fails to express the semantics of data flow between nodes, and does not support parallel branch execution. Furthermore, it lacks a mechanism to achieve precise two-way binding between virtual layer placeholder entries and upstream connections across sessions, using unique identifiers as anchor points.

[0012] Commercial software ArcGIS ModelBuilder provides a visual model building tool that supports chaining geoprocessing tools into a model and using the output of upstream tools as input for downstream tools. However, its data elements are passed in the form of datasets / file paths. Each tool needs to write its results to disk or a geodatabase before the downstream tool can read them, making it impossible to directly pass layer objects at the memory level. Furthermore, the model is executed sequentially and does not support asynchronous parallel scheduling of independent branches. It also lacks a mechanism for precisely binding virtual layer placeholders to upstream connections across sessions using unique connection identifiers as anchor points.

[0013] Commercial software FME Workbench (Safe Software): Visually orchestrates the construction of spatial data ETL processes using Transformers, allowing features to be passed between Transformers in the processing pipeline as in-memory streams.

[0014] QGIS Graphical Modeler (Processing Modeler), an open-source GIS modeling tool, supports visually connecting processing algorithms and can automatically use the output of upstream algorithms as the input of downstream algorithms.

[0015] The aforementioned solutions (including patents such as CN201810292764.0, CN202210256586.2, and CN202010729398.8, as well as various commercial / open-source software) are either workflow frameworks for general scenarios (not supporting GIS spatial algorithms and the platform's self-developed algorithm base class) or visualization orchestration tools in the GIS field (but have limitations such as serial execution, single connection semantics, dependency on file path transmission, and lack of cross-session virtual layer binding). None of them simultaneously achieve the complete technical solution of "adapting the self-developed geographic information processing algorithm base class (QgsProcessingAlgorithm) into a DAG node that can be scheduled in parallel by C++ multi-threading using the CGraph framework, expressing the relationship between algorithms with dual semantics of data flow connections and temporal dependency connections, automatically injecting upstream algorithm output into downstream algorithm parameters at the memory level through the virtual layer mechanism, and realizing cross-session bidirectional binding and restoration of virtual layers and connections using connectionId". Summary of the Invention

[0016] To address the shortcomings of existing technologies, the present invention aims to provide a visualization orchestration and parallel execution method based on a directed acyclic graph (DAG) to solve the problems mentioned in the background. This method uses a DAG as the process model, adapter nodes to connect geographic information processing algorithms to a parallel scheduling framework, a virtual layer placeholder mechanism to achieve automatic memory-level parameter transfer between algorithms, and unique identifiers for connections to accurately reconstruct the source of parameters across sessions. This solves the problems in existing technologies such as fragmented multi-step GIS processing flows, inability to execute in parallel, lack of reusability, difficulty in data flow tracing, and inability to accurately reconstruct the source of parameters across sessions.

[0017] To achieve the above objectives, the present invention provides a method for visual orchestration and parallel execution based on directed acyclic graphs, wherein the execution method includes the following steps:

[0018] S1. Visual Canvas and Algorithm Node Modeling

[0019] S1.1 Algorithm Node Creation Process:

[0020] ① Users select an algorithm from the algorithm tree panel and drag and drop it onto the canvas;

[0021] ② The system creates a QgsProcessNodeItem based on the algorithm ID, assigns a unique identifier (UUID) as the unique identifier of the node, and records the initial position;

[0022] ③ The user double-clicks the node to open the parameter configuration panel (QgsProcessingWidget) and fills in the input and output parameters;

[0023] S1.2, Drawing process of connecting lines:

[0024] ① The user clicks and drags the mouse from the output port of the source node to the input port of the target node;

[0025] ② The system determines the connection type based on the port semantics: if the connection is to a data source type parameter (VectorLayer / RasterLayer / FileDestination, etc.), a data flow connection is created (solid line, connectionType="datasource"); if the connection is to a control flow port, a time-dependent connection is created (dashed line, connectionType="prerequisite").

[0026] ③ After the connection is established, the system creates a virtual layer placeholder entry (InputLayerEntry, type=VirtualLayer) at the corresponding parameter position of the target node as a placeholder for the upstream result;

[0027] S1.3 Linkage between connection identifier and virtual layer placeholder: When establishing a data flow connection on the visualization canvas, a globally unique identifier (connectionId) is automatically assigned to the connection, and at the same time, a virtual layer placeholder entry with the connectionId as the index key is automatically created in the parameter interface of the downstream algorithm node as a reserved placeholder for the execution result of the upstream algorithm.

[0028] S2, Process Model Data Structure Design

[0029] The process model is managed by the ProcessModel class, which contains three core data elements:

[0030] ① Node Data (ProcessModelNode): Records node ID (UUID), algorithm identifier (algorithmId), canvas position (position), parameter mapping table (serializedParameters), virtual layer information (virtualLayerInfos, including connectionId two-way binding key and injection order order), field structure file path (fcsLayerPaths, used to restore field selection state across sessions). VirtualLayerInfo includes connectionId (two-way binding with connection UUID), displayName (UI display name), sourceNodeName (upstream node name), sourceLayerName (exactly matches the output layer during execution), and order (injection order when there are multiple connections), ensuring that the layer list with multiple input parameters is consistent with the connection establishment order;

[0031] ② Connection data (ProcessModelConnection): Records the source node ID (fromNodeId), target node ID (toNodeId), parameter names involved in the connection (fromOutput / toInput), connection type (connectionType: datasource / prerequisite), unique connection identifier (connectionId, UUID), folder to file transfer strategy (folderToFileMode: first / all / filter), and suffix filter list (folderToFileSuffixes).

[0032] ③ Topological Order: The system uses a standard topological ordering algorithm to sort the nodes, ensuring that upstream nodes execute before downstream nodes. If a loop is detected (i.e., the process has a circular dependency), the system refuses to execute and reports the specific loop path to the user to prevent execution deadlock or erroneous results caused by circular dependencies.

[0033] S3, CGraph DAG Parallel Execution Engine

[0034] S3.1 Upgrade the execution of the process model from the traditional serial ProcessModelExecutor to a parallel executor (CGraphExecutor) based on the CGraph framework. The execution process is divided into four stages:

[0035] ① Pipeline Construction: The system traverses all nodes in the ProcessModel, creating a CGraphAlgorithmNode instance (an adapter from QgsProcessingAlgorithm to CGraphGNode) for each node; encapsulating the node ID, algorithm ID, node parameters, and executor ID into node configuration parameters (CGraphNodeConfigParam) and passing them to the node; based on the connection relationships in the ProcessModel, calling the directed graph interface (GGraph): node registration (addGNode) adds the node to the CGraph directed graph, and calling the directed graph interface (GGraph): dependency edge creation (addGEdge) establishes the execution dependencies between nodes; the CGraph framework internally maintains a DAG structure and automatically identifies the set of nodes that can be executed in parallel;

[0036] ② Data Flow Establishment: The system iterates through all datasource connections, establishing a data flow association between the source node (upstream CGraphAlgorithmNode) and the target node (downstream CGraphAlgorithmNode). This involves calling the target node's upstream node registration (addUpstreamNode) to record upstream node references, enabling the reading of layer data from the upstream node's output parameters during execution. For folderToFile type connections, the connection configuration is stored in the target node's folder-to-file connection mapping table (folderToFileConnections). During execution, the folder is scanned, and file paths that meet the conditions are injected into the target parameters.

[0037] ③ Parallel execution of nodes (CGraphAlgorithmNode:run()): Each CGraphAlgorithmNode executes the run() method during CGraph scheduling: It calls the parameter preparation method (prepareParameters) to complete parameter merging and upstream injection: Internally, it sequentially executes serializedParameters deserialization and filters out invalid layer pointers, completes missing parameters with algorithm default values, and calls injectUpstreamData() to complete upstream data injection; injectUpstreamData() (an internal step of prepareParameters) sorts the parameters in virtualLayerInfos according to their connectionId and order, reads the GsLayerPtr list from all upstream nodes m_outputParam, and directly inserts the layer memory objects (GsVectorLayerPtr / GsRasterLayerPtr). The current node's execution parameter table is injected, allowing downstream algorithms to directly access the loaded layer objects in memory. A processing context (QgsProcessingContext) and processing feedback (QgsProcessingFeedback) are created, and algorithm->run() is called to execute the algorithm. After the algorithm is executed, the output layer parsing (parseAlgorithmOutputLayers) is called to parse the output results, load the output file path into a GsLayerPtr layer object, encapsulate it into a layer parameter object (CGraphLayerParam) and store it in the node output parameter (m_outputParam) for downstream nodes to read. Through the node log emitter (NodeLogEmitter) singleton, the execution logs and node events (Started / Completed / Failed) are routed to the corresponding UI log components with the executor identifier (executorId) as the key, supporting precise log routing when multiple executors are running concurrently.

[0038] Execution exception handling mechanism: Multi-level exception detection is implemented at each key node in the four-stage parallel execution to ensure that the execution failure of any node can be accurately captured and safely reported without affecting the overall system stability;

[0039] S3.2 Automatic Injection of Memory-Level Layer Objects: During the parallel execution preparation phase, downstream algorithm nodes read spatial layer memory objects from the output parameter container of upstream nodes based on the connectionId and injection order recorded in their virtual layer placeholder entries, and directly inject them into the execution parameter table of their own nodes, thereby realizing memory-level transfer of layer data between algorithms and eliminating intermediate disk file read and write operations.

[0040] S4, XML Persistence and Cross-Session Recovery

[0041] S4.1 The system supports persisting the complete process model as an XML file (.model) for reuse across sessions:

[0042] ① Save process (saveToXml): Iterate through all ProcessModelNodes, serialize the node parameters into XML strings using the parameter serialization method defined in the parameter definition, and store them in the serializedParameters mapping table; serialize the virtual layer information (connectionId, displayName, sourceNodeName, order) into child nodes; write fcsLayerPaths (field structure file path) into node attributes; serialize all ProcessModelConnections (including connectionType, fromOutput, toInput, folderToFileMode, and suffix filter list (folderToFileSuffixes)) into child elements under the connections node; and write the entire process to a file in standard XML format.

[0043] ② Loading Process (loadFromXml): Parse the XML file and rebuild the ProcessModelNode and ProcessModelConnection lists; rebuild the parameter objects based on the XML strings in serializedParameters; rebuild virtual layer placeholder entries in the node parameter panel based on virtualLayerInfos; load the field structure file based on fcsLayerPaths and restore the field selection state; restore the selected virtual layers for single-layer parameters (such as INPUT / JOIN for spatial connections) based on the selected virtual layer display names (selectedVirtualLayerDisplayNames); restore the position of each node and the connection shape in QGraphicsScene based on the node's position attribute.

[0044] S4.2, ConnectionId Two-Way Binding and Cross-Session Restoration: During the serialization and persistence of the process model, the connectionId and the virtual layer placeholder information are written together to the persistent file to establish a two-way binding relationship between the connection and the parameter placeholder. When the process model is reloaded across sessions, the connectionId is used as the anchor point to accurately restore the correspondence between which connection the downstream parameter comes from and which upstream node it originates from, solving the engineering problem of mismatch when nodes have the same name or multiple connections point to the same parameter.

[0045] Furthermore, in step S1, the system for creating the algorithm node provides an interactive canvas (QgsProcessManagementWidget) based on the Qt graphics view framework (QGraphicsView). The algorithm node (QgsProcessNodeItem) and the connecting arrow (QgsProcessArrowItem) are both implemented as subclasses of Qt graphics primitives (QGraphicsItem), supporting drag-and-drop, selection, movement, and scaling interactions.

[0046] Furthermore, in step S1.2, the triggering method of the time-dependent connection is as follows: when the user drags a connection from the source node to the target node, but there is no compatible data source parameter pair between the two nodes (i.e., the upstream output type and the downstream input type cannot match), the system automatically creates this connection as a time-dependent connection (dashed line, connectionType="prerequisite"), which represents a pure time constraint that "the upstream node must complete execution before the downstream node can start execution". The time-dependent connection does not trigger the creation of virtual layer placeholder entries, nor does it participate in the data injection logic of injectUpstreamData. During the Pipeline construction phase, its corresponding addGEdge call only establishes CGraph scheduling dependencies and does not establish data flow associations in the m_upstreamNodes list.

[0047] Furthermore, in step S1.2, the connection validity verification and linkage cleanup mechanism are implemented.

[0048] Type compatibility verification during connection establishment: The system automatically verifies based on the output-input type compatibility matrix: Vector output ↔ Vector-type input, Raster output ↔ Raster-type input, Folder output ↔ Folder / Raster / File input (supports 3D batch processing scenes), File output ↔ File input; Path string parameters (parameter names containing keywords such as dir / path / input) can be extended to receive folder outputs. A prompt will appear if the types are incompatible and connection creation will be refused, without producing any data model side effects.

[0049] Duplicate connection prevention mechanism: Before creating a connection, the system traverses the NodeConnection record set of the current canvas; if a connection with the same source node to the target node already exists, it will be silently refused to create, to prevent duplicate connections from causing downstream nodes to receive redundant upstream data injection.

[0050] Virtual layer cleanup mechanism for deleting nodes / connections: When deleting a node or connection, the system simultaneously cleans up all associated virtual layers: iterates through the downstream node parameter records by connectionId and deletes the corresponding entries; clears the layer pointers and XML serialization strings in the parameter cache; if there are temporary raster files, deletes the disk files simultaneously; finally, it updates the connection data model and refreshes the parameter panel to ensure that the data model and UI state are completely consistent.

[0051] Furthermore, in step S3.1, the parallel execution mechanism is illustrated by a process with three nodes (A→B, A→C, B+C→D):

[0052] ① CGraph analyzes the DAG structure and identifies A as the root node (in-degree 0), so A is executed first;

[0053] ②After A completes execution, the in-degree of both B and C drops to 0, and CGraph submits B and C to the thread pool for concurrent execution;

[0054] ③ After both B and C have been executed, the in-degree of D drops to 0. CGraph then executes D. In the run() method, D simultaneously reads the output layers of B and C as input through upstream data injection (injectUpstreamData).

[0055] ④ Once the entire process is complete, the parallel executor (CGraphExecutor) emits an execution completion signal (executionCompleted=true).

[0056] Furthermore, in step S3.1, the anomaly detection is divided into three layers:

[0057] ① During the initialization phase, the algorithm instance is searched by algorithm ID. If it is not registered, it is refused entry into the schedule and the reason is reported.

[0058] ② During the parameter validation phase, a pre-check of complete parameters is performed before run() is executed (e.g., missing required fields, non-existent paths, null layer pointers).

[0059] ③ During the execution phase, check the return flag of algorithm->run(). Failure at any level triggers a node failure event, which is highlighted in the log routing component, and the UI displays a resume execution button.

[0060] CGraphExecutor maintains a five-state machine (idle / running / paused / completed / errored), supporting both synchronous and asynchronous execution interfaces as well as pause, resume, and cancel controls.

[0061] Furthermore, in step S1, the virtual layer placeholder mechanism is as follows: when the data stream connection is established, a virtual layer placeholder entry is automatically created in the parameter interface of the downstream algorithm node as a reserved placeholder for the execution result of the upstream algorithm; during the execution phase, the placeholder entry is automatically replaced by the layer memory object actually output by the upstream algorithm, eliminating the operation step of manually configuring intermediate parameters by the operator.

[0062] Furthermore, in step S4, the bidirectional binding cross-session restoration mechanism for unique identifiers of connections involves: assigning a globally unique identifier to each data flow connection; during the serialization and persistence of the process model, writing the identifier and the virtual layer placeholder entry together into the persistence file to establish a bidirectional binding relationship between the connection and the parameter placeholder entry; and when loading the process model across sessions, using the identifier as the anchor point to accurately restore the correspondence between which connection the parameter comes from and which upstream node it originates from, thus solving the engineering problem of mismatch in complex scenarios such as node name conflicts or multiple connections pointing to the same parameter.

[0063] Furthermore, an adaptation method is provided for integrating geographic information processing algorithms into the directed acyclic graph parallel scheduling framework:

[0064] Design an adapter node class to encapsulate the geographic information processing algorithm base class into the graph node interface required by the parallel scheduling framework;

[0065] When the adapter node is executed in the parallel framework scheduling, it completes the complete orchestration of parameter merging, upstream data injection, algorithm instance execution, and output result writing back internally, and establishes a complete mapping between the geographic information processing algorithm execution contract and the parallel scheduling framework node interface contract;

[0066] By leveraging the happens-before semantic guarantee provided by the node in-degree zeroing mechanism in the parallel scheduling framework—that is, the current node is only scheduled by the thread pool after all the execution methods of the predecessor nodes have returned—the operation of upstream nodes writing layer memory objects to the output parameter container occurs before the operation of downstream nodes reading from the container. Thus, thread-safe read and write of memory layer objects in a multi-threaded concurrent scheduling environment can be achieved without applying an additional mutex lock to the output parameter container. This allows geographic information processing algorithms that were originally unable to be scheduled by the parallel framework to participate in lock-free memory flow multi-threaded parallel scheduling as directed acyclic graph nodes.

[0067] Furthermore, in the GIS algorithm visualization orchestration, two connection semantics are defined: data flow connection (datasource, solid line) and temporal dependency connection (temporal dependency prerequisite, dashed line). After the data flow connection is established, the system creates virtual layer placeholder entries in the downstream node. During execution, the upstream output is automatically injected into the corresponding parameters of the downstream algorithm based on the connection relationship. In the DAG connection, a folder-to-file connection type (Folder→File) is defined, supporting three file selection strategies: first (first file), all (all files), and filter (filter by suffix). During the execution phase, the system scans folders and dynamically injects the file paths that meet the conditions into the downstream algorithm parameters. Logs and node events are routed by executorId through a global singleton of the node log emitter (NodeLogEmitter). This supports multiple parallel executor (CGraphExecutor) instances running concurrently without log interference and with precise routing to their respective UI components.

[0068] The beneficial effects of this invention are:

[0069] 1. This invention improves processing efficiency: Traditional serial execution requires waiting for each step to complete before starting the next step. This invention, through CGraph parallel scheduling, allows nodes without dependencies to execute concurrently in a multi-threaded environment. According to Amdahl's Law, the actual speedup significantly increases with the number of independent branches—with 2 independent branches, the parallel execution time is approximately 1 / 1.5 to 1 / 2 of the serial execution time; with 4 branches, it's approximately 1 / 2.5 to 1 / 3.5; and with 8 or more branches, it's approximately 1 / 3.5 to 1 / 5. The specific value is affected by the I / O ratio of the single-branch algorithm and the DAG scheduling overhead. Compared to existing serial execution schemes, this invention can significantly shorten the total end-to-end execution time in multi-branch GIS processing scenarios, thus improving production efficiency.

[0070] 2. This invention eliminates the burden of manual path configuration: In traditional methods, operators need to manually fill in the output path of the previous step as the input parameter of the next step in each step. This invention automatically completes the parameter transfer between algorithms through a virtual layer mechanism, completely eliminating the operation step of manually specifying intermediate paths, while avoiding data transmission errors caused by incorrect path filling, and significantly improving production stability.

[0071] 3. This invention improves process reusability: The XML persistence mechanism enables the complete process model to be saved and loaded across sessions, and the standardized processing flow can be shared and reused among teams, reducing the workload of repetitive configuration and improving the degree of production standardization.

[0072] 4. This invention reduces the error rate of operation: By visualizing the DAG, the data flow between algorithms is clearly presented, and operators can intuitively check the process logic, reducing data errors caused by manual transmission of incorrect paths, and significantly improving data consistency.

[0073] 5. This invention supports multi-task concurrent management: The multi-executor log routing mechanism supports the concurrent execution of multiple process instances, and the logs are accurately routed to their respective UI components, meeting the needs of multi-task parallel production scenarios and significantly improving production throughput. Attached Figure Description

[0074] Figure 1 This is a flowchart of a visualization orchestration and parallel execution method based on a directed acyclic graph, as proposed in one embodiment of this disclosure.

[0075] Figure 2 This is a schematic diagram of the visual canvas nodes and connections of a visual orchestration and parallel execution method based on a directed acyclic graph proposed in one embodiment of this disclosure;

[0076] Figure 3 This is a flowchart of a visualization orchestration and parallel execution method based on a directed acyclic graph, as proposed in one embodiment of this disclosure, showing the data structure relationship diagram.

[0077] Figure 4 This is a flowchart of the CGraph parallel execution engine workflow, which proposes a visualization orchestration and parallel execution method based on a directed acyclic graph, according to an embodiment of this disclosure.

[0078] Figure 5 This is a schematic diagram of the XML process model file structure of a visualization orchestration and parallel execution method based on a directed acyclic graph proposed in one embodiment of this disclosure. Detailed Implementation

[0079] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0080] like Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 As shown, this disclosure proposes a visualization orchestration and parallel execution method based on directed acyclic graphs, the execution method including the following steps:

[0081] like Figure 2 As shown, S1, the visualization canvas, and the algorithm node modeling

[0082] The system for creating algorithm nodes provides an interactive canvas (QgsProcessManagementWidget) based on the Qt graphics view framework (QGraphicsView). The algorithm nodes (QgsProcessNodeItem) and the connecting arrows (QgsProcessArrowItem) are both implemented as subclasses of Qt graphics primitives (QGraphicsItem), supporting drag-and-drop, selection, movement, and scaling interactions.

[0083] S1.1 Algorithm Node Creation Process:

[0084] ① Users select an algorithm from the algorithm tree panel and drag and drop it onto the canvas;

[0085] ② The system creates a QgsProcessNodeItem based on the algorithm ID, assigns a unique identifier (UUID) as the unique identifier of the node, and records the initial position;

[0086] ③ The user double-clicks the node to open the parameter configuration panel (QgsProcessingWidget) and fills in the input and output parameters;

[0087] S1.2, Drawing process of connecting lines:

[0088] ① The user clicks and drags the mouse from the output port of the source node to the input port of the target node;

[0089] ② The system determines the connection type based on the port semantics: if the connection is to a data source type parameter (VectorLayer / RasterLayer / FileDestination, etc.), a data flow connection is created (solid line, connectionType="datasource"); if the connection is to a control flow port, a time-dependent connection is created (dashed line, connectionType="prerequisite").

[0090] ③ After the connection is established, the system creates a virtual layer placeholder entry (InputLayerEntry, type=VirtualLayer) at the corresponding parameter position of the target node as a placeholder for the upstream result;

[0091] The triggering method of the time-dependent connection:

[0092] When a user drags a connection from a source node to a target node, but there are no compatible data source parameter pairs between the two nodes (i.e., the upstream output type and the downstream input type cannot match), the system automatically creates this connection as a time-dependent connection (dashed line, connectionType="prerequisite"), which represents a pure time-series constraint that "the upstream node must complete execution before the downstream node can start execution". Time-dependent connections do not trigger the creation of virtual layer placeholder entries, nor do they participate in the data injection logic of injectUpstreamData.

[0093] During the Pipeline construction phase, the corresponding addGEdge call only establishes CGraph scheduling dependencies and does not establish data stream associations in the m_upstreamNodes list.

[0094] The connection validity verification and linkage cleanup mechanism

[0095] Type compatibility verification during connection establishment: The system automatically verifies based on the output-input type compatibility matrix: Vector output ↔ Vector-type input, Raster output ↔ Raster-type input, Folder output ↔ Folder / Raster / File input (supports 3D batch processing scenes), File output ↔ File input; Path string parameters (parameter names containing keywords such as dir / path / input) can be extended to receive folder outputs. A prompt will appear if the types are incompatible and connection creation will be refused, without producing any data model side effects.

[0096] Duplicate connection prevention mechanism: Before creating a connection, the system traverses the NodeConnection record set of the current canvas; if a connection with the same source node to the target node already exists, it will be silently refused to create, to prevent duplicate connections from causing downstream nodes to receive redundant upstream data injection.

[0097] Virtual layer cleanup mechanism for deleting nodes / connections: When deleting a node or connection, the system simultaneously cleans up all associated virtual layers: iterates through the downstream node parameter records by connectionId and deletes the corresponding entries; clears the layer pointers and XML serialization strings in the parameter cache; if there are temporary raster files, deletes the disk files simultaneously; finally, it updates the connection data model and refreshes the parameter panel to ensure that the data model and UI state are completely consistent.

[0098] like Figure 3 As shown, S2, process model data structure design

[0099] The process model is managed by the ProcessModel class, which contains three core data elements:

[0100] ① Node Data (ProcessModelNode): Records node ID (UUID), algorithm identifier (algorithmId), canvas position (position), parameter mapping table (serializedParameters), virtual layer information (virtualLayerInfos, including connectionId two-way binding key and injection order order), field structure file path (fcsLayerPaths, used to restore field selection state across sessions). VirtualLayerInfo includes connectionId (two-way binding with connection UUID), displayName (UI display name), sourceNodeName (upstream node name), sourceLayerName (exactly matches the output layer during execution), and order (injection order when there are multiple connections), ensuring that the layer list with multiple input parameters is consistent with the connection establishment order;

[0101] ② Connection data (ProcessModelConnection): Records the source node ID (fromNodeId), target node ID (toNodeId), parameter names involved in the connection (fromOutput / toInput), connection type (connectionType: datasource / prerequisite), unique connection identifier (connectionId, UUID), folder to file transfer strategy (folderToFileMode: first / all / filter), and suffix filter list (folderToFileSuffixes).

[0102] ③ Topological Order: The system uses a standard topological ordering algorithm to sort the nodes, ensuring that upstream nodes execute before downstream nodes. If a loop is detected (i.e., the process has a circular dependency), the system refuses to execute and reports the specific loop path to the user to prevent execution deadlock or erroneous results caused by circular dependencies.

[0103] like Figure 4 As shown, the S3 and CGraph DAG parallel execution engines

[0104] S3.1 Upgrade the execution of the process model from the traditional serial ProcessModelExecutor to a parallel executor (CGraphExecutor) based on the CGraph framework. The execution process is divided into four stages:

[0105] ① Pipeline Construction: The system traverses all nodes in the ProcessModel, creating a CGraphAlgorithmNode instance (an adapter from QgsProcessingAlgorithm to CGraphGNode) for each node; encapsulating the node ID, algorithm ID, node parameters, and executor ID into node configuration parameters (CGraphNodeConfigParam) and passing them to the node; based on the connection relationships in the ProcessModel, calling the directed graph interface (GGraph): node registration (addGNode) adds the node to the CGraph directed graph, and calling the directed graph interface (GGraph): dependency edge creation (addGEdge) establishes the execution dependencies between nodes; the CGraph framework internally maintains a DAG structure and automatically identifies the set of nodes that can be executed in parallel;

[0106] ② Data Flow Establishment: The system iterates through all datasource connections, establishing a data flow association between the source node (upstream CGraphAlgorithmNode) and the target node (downstream CGraphAlgorithmNode). This involves calling the target node's upstream node registration (addUpstreamNode) to record upstream node references, enabling the reading of layer data from the upstream node's output parameters during execution. For folderToFile type connections, the connection configuration is stored in the target node's folder-to-file connection mapping table (folderToFileConnections). During execution, the folder is scanned, and file paths that meet the conditions are injected into the target parameters.

[0107] ③ Parallel execution of nodes (CGraphAlgorithmNode:run()): Each CGraphAlgorithmNode executes the run() method during CGraph scheduling: It calls the parameter preparation method (prepareParameters) to complete parameter merging and upstream injection: Internally, it sequentially executes serializedParameters deserialization and filters out invalid layer pointers, completes missing parameters with algorithm default values, and calls injectUpstreamData() to complete upstream data injection; injectUpstreamData() (an internal step of prepareParameters) sorts the parameters in virtualLayerInfos according to their connectionId and order, reads the GsLayerPtr list from all upstream nodes m_outputParam, and directly inserts the layer memory objects (GsVectorLayerPtr / GsRasterLayerPtr). The current node's execution parameter table is injected, allowing downstream algorithms to directly access the loaded layer objects in memory. A processing context (QgsProcessingContext) and processing feedback (QgsProcessingFeedback) are created, and algorithm->run() is called to execute the algorithm. After the algorithm is executed, the output layer parsing (parseAlgorithmOutputLayers) is called to parse the output results, load the output file path into a GsLayerPtr layer object, encapsulate it into a layer parameter object (CGraphLayerParam) and store it in the node output parameter (m_outputParam) for downstream nodes to read. Through the node log emitter (NodeLogEmitter) singleton, the execution logs and node events (Started / Completed / Failed) are routed to the corresponding UI log components with the executor identifier (executorId) as the key, supporting precise log routing when multiple executors are running concurrently.

[0108] The parallel execution mechanism is illustrated by a process with three nodes (A→B, A→C, B+C→D):

[0109] ① CGraph analyzes the DAG structure and identifies A as the root node (in-degree 0), so A is executed first;

[0110] ②After A completes execution, the in-degree of both B and C drops to 0, and CGraph submits B and C to the thread pool for concurrent execution;

[0111] ③ After both B and C have been executed, the in-degree of D drops to 0. CGraph then executes D. In the run() method, D simultaneously reads the output layers of B and C as input through upstream data injection (injectUpstreamData).

[0112] ④ Once the entire process is complete, the parallel executor (CGraphExecutor) emits an execution completion signal (executionCompleted=true).

[0113] Execution exception handling mechanism: Multi-level exception detection is implemented at each key node in the four-stage parallel execution to ensure that the execution failure of any node can be accurately captured and safely reported without affecting the overall system stability;

[0114] The anomaly detection is divided into three layers:

[0115] ① During the initialization phase, the algorithm instance is searched by algorithm ID. If it is not registered, it is refused entry into the schedule and the reason is reported.

[0116] ② During the parameter validation phase, a pre-check of complete parameters is performed before run() is executed (e.g., missing required fields, non-existent paths, null layer pointers).

[0117] ③ During the execution phase, check the return flag of algorithm->run(). Failure at any level triggers a node failure event, which is highlighted in the log routing component, and the UI displays a resume execution button.

[0118] CGraphExecutor maintains a five-state machine (idle / running / paused / completed / errored), supporting both synchronous and asynchronous execution interfaces as well as pause, resume, and cancel controls.

[0119] like Figure 5 As shown, S4, XML persistence, and cross-session recovery

[0120] S4.1 The system supports persisting the complete process model as an XML file (.model) for reuse across sessions:

[0121] ① Save process (saveToXml): Iterate through all ProcessModelNodes, serialize the node parameters into XML strings using the parameter serialization method defined in the parameter definition, and store them in the serializedParameters mapping table; serialize the virtual layer information (connectionId, displayName, sourceNodeName, order) into child nodes; write fcsLayerPaths (field structure file path) into node attributes; serialize all ProcessModelConnections (including connectionType, fromOutput, toInput, folderToFileMode, and suffix filter list (folderToFileSuffixes)) into child elements under the connections node; and write the entire process to a file in standard XML format.

[0122] ② Loading Process (loadFromXml): Parse the XML file and rebuild the ProcessModelNode and ProcessModelConnection lists; rebuild the parameter objects based on the XML strings in serializedParameters; rebuild the virtual layer placeholder entries in the node parameter panel based on virtualLayerInfos; load the field structure file based on fcsLayerPaths and restore the field selection state; restore the selected virtual layers for single-layer parameters (such as INPUT / JOIN for spatial connections) based on the selected virtual layer display names (selectedVirtualLayerDisplayNames); restore the position and connection shape of each node in QGraphicsScene based on the node's position attribute.

[0123] This invention is not a simple overlay of known workflow orchestration techniques in the GIS field. Integrating a self-developed geographic information processing algorithm into a DAG parallel framework and achieving cross-session reusability of the workflow requires overcoming the following three technical challenges, and these techniques must work together to produce a synergistic effect:

[0124] The output of GIS algorithms is not a simple scalar, but a heavyweight layer object (GsLayerPtr) containing geometric data, attribute tables, field structures (FCS), coordinate reference systems, etc. Under CGraph's multi-threaded concurrent scheduling, the same upstream output may be read concurrently by multiple downstream nodes. Directly sharing mutable object pointers across threads will introduce data races and object lifecycle management risks, while reverting to file-to-disk transfer will generate a large amount of disk I / O overhead again.

[0125] This invention enables upstream nodes to write their output into the node output parameter container (m_outputParam) after execution. During the upstream data injection phase, downstream nodes read the layer memory object (GsLayerPtr) according to the connection relationship and directly inject it into their own node parameter table. This avoids downstream nodes repeatedly reading the same intermediate result from the disk and avoids the concurrency risk of cross-thread direct sharing of variable layer objects.

[0126] Concurrency safety guarantee mechanism: The above assertion of "avoiding concurrency risks" relies on the CGraph framework's internal guarantee of the partial order relationship of DAG execution. When CGraph establishes dependent edges using addGEdge, it implements happen-before semantics through node in-degree counting and condition variable mechanisms: a node will only be scheduled for execution by the thread pool when all of its predecessor nodes' run() have returned (the node's in-degree is zero). This happen-before guarantee ensures that the upstream node's writing to the end of GsLayerPtrrun() in m_outputParam occurs before the downstream node's reading of injectUpstreamData() from m_outputParam, without needing to apply an additional mutex lock to m_outputParam. This guarantees thread safety and avoids the additional overhead caused by lock contention.

[0127] The execution model of the self-developed algorithm base class QgsProcessingAlgorithm is a single synchronous call of "preparing parameters → executing the algorithm → producing results", with parameters passed in a key-value mapping manner; while CGraph's graph nodes (GNode) are driven by the run() return status code, and there is no channel for passing business parameters between nodes. The two are not aligned in four dimensions: parameter passing, object lifecycle, error handling and logging, and simple wrapping will not work.

[0128] The adapter (CGraphAlgorithmNode) of this invention completes the entire orchestration of "parameter merging → upstream data injection → algorithm instance construction and execution → output write-back" inside run(), establishing a complete mapping between the two sets of execution contracts, enabling GIS processing algorithms that could not be scheduled by the parallel framework to participate in multi-threaded parallel scheduling as DAG nodes.

[0129] After the process model is serialized to XML and reloaded in another session, the virtual layer placeholder entries in the downstream node parameter panel need to redetermine their corresponding upstream connections and upstream nodes. If matching is based solely on node names or parameter names, mismatches will occur in cases where nodes have the same name or multiple connections point to the same parameter, causing the flowchart to be unable to identify the source of the parameters.

[0130] This invention uses connectionId as a two-way binding key between virtual layer placeholder entries and connection relationships. Both saving and loading use it as the anchor point, thereby accurately restoring the correspondence of "which connection the parameters come from and which upstream node they originate from" in any complex connection scenario.

[0131] (a) Linkage between connection identifier and virtual layer placeholder - When establishing a data flow connection on the visualization canvas, a globally unique identifier (connectionId) is automatically assigned to the connection, and at the same time, a virtual layer placeholder entry with the connectionId as the index key is automatically created in the parameter interface of the downstream algorithm node as a reserved placeholder for the execution result of the upstream algorithm.

[0132] (b) Automatic injection of memory-level layer objects - During the parallel execution preparation phase, the downstream algorithm node reads the spatial layer memory object from the output parameter container of the upstream node according to the connectionId and injection order recorded in its virtual layer placeholder entry, and directly injects it into the execution parameter table of this node, realizing the memory-level transfer of layer data between algorithms and eliminating intermediate disk file read and write.

[0133] (c) ConnectionId bidirectional binding and cross-session restoration: When the process model is serialized and persisted, the connectionId and the virtual layer placeholder information are written together to the persistent file to establish a bidirectional binding relationship between the connection and the parameter placeholder; when the process model is reloaded across sessions, the connectionId is used as the anchor point to accurately restore the correspondence between which connection the downstream parameter comes from and which upstream node it originates from, solving the engineering problem of mismatch when nodes have the same name or multiple connections point to the same parameter;

[0134] The three technical means (a), (b), and (c) mentioned above work together and are indispensable: (a) establish a persistent and unique identifier foundation; (b) complete memory-level data transfer during parallel execution; and (c) ensure that the process model can be accurately reused across sessions. The three together achieve the overall technical effect of "memory-level layer transfer + true parallelism of C++ multithreading + accurate reuse across sessions". This overall effect cannot be achieved by any single means or a simple combination of existing technologies.

[0135] Virtual layer placeholder mechanism: When the data stream connection is established, a virtual layer placeholder entry is automatically created in the parameter interface of the downstream algorithm node as a reserved placeholder for the execution result of the upstream algorithm; during the execution phase, the system automatically replaces this placeholder entry with the actual layer memory object output by the upstream algorithm, eliminating the operation steps of manually configuring intermediate parameters by the operator;

[0136] The unique identifier for each data flow connection is used for bidirectional binding and cross-session restoration: a globally unique identifier is assigned to each data flow connection; during the serialization and persistence of the process model, this identifier and the virtual layer placeholder entries are written together to the persistence file to establish a bidirectional binding relationship between the connection and the parameter placeholder entries; when loading the process model across sessions, this identifier is used as the anchor point to accurately restore the correspondence between which connection the parameter comes from and which upstream node it originates from, solving the engineering problem of mismatch in complex scenarios such as node name conflicts or multiple connections pointing to the same parameter;

[0137] The method for adapting GIS algorithms to DAG nodes is as follows: An adapter node class is designed to encapsulate the base class of the geographic information processing algorithm into the graph node interface required by the parallel scheduling framework. During parallel framework scheduling, the adapter node internally completes the complete orchestration of parameter merging, upstream data injection, algorithm instance execution, and output result write-back, establishing a complete mapping between the geographic information processing algorithm execution contract and the parallel scheduling framework node interface contract. Simultaneously, leveraging the happens-before semantic guarantee provided by the parallel scheduling framework's node in-degree zeroing mechanism—that is, the current node is only scheduled by the thread pool after all predecessor node execution methods have returned—the operation of upstream nodes writing layer memory objects to the output parameter container occurs before the operation of downstream nodes reading from the container. This achieves thread-safe read / write of memory layer objects in a multi-threaded concurrent scheduling environment without imposing additional mutex locks on the output parameter container, enabling geographic information processing algorithms that were originally unschedulable by the parallel framework to participate as directed acyclic graph nodes in lock-free memory flow multi-threaded parallel scheduling.

[0138] Dual semantic connection: In GIS algorithm visualization orchestration, two connection semantics are defined: data flow connection (datasource, solid line) and temporal dependency connection (temporal dependency prerequisite, dashed line). After the data flow connection is established, the system creates virtual layer placeholder entries in the downstream node. During execution, the upstream output is automatically injected into the corresponding parameters of the downstream algorithm according to the connection relationship.

[0139] Folder to file connection: Define the folder to file connection type (Folder→File) in the DAG connection, and support three file selection strategies: first (first file), all (all files), and filter (filter by suffix). During the execution phase, the folder is scanned and the file paths that meet the conditions are dynamically injected into the downstream algorithm parameters.

[0140] Multi-executor log routing: This feature uses a global singleton instance of the NodeLogEmitter to route logs and node events by executorId. It supports multiple parallel executor instances running concurrently without log interference and allows for precise routing to their respective UI components.

[0141] First embodiment:

[0142] The following alternatives exist, but all have fundamental technical limitations: (1) Integrating the platform's GIS algorithm based on a Python scheduling framework (such as a custom Operator in Apache Airflow): Due to the limitations of the Python GIL, true C++ multi-threaded parallelism cannot be achieved; and the custom Operator is incompatible with the platform's QgsProcessingAlgorithm interface, requiring an additional encapsulation layer, and the automatic parameter injection mechanism cannot be directly adapted to the platform's interface. (2) Parallel execution based on process pools: The transfer of algorithm output results between processes requires a cross-process communication mechanism, which is highly coupled with the platform's QgsProcessingAlgorithm interface, making it difficult to directly implement the automatic parameter injection mechanism, and the complexity of inter-process synchronization is high. (3) Modifying based on the open-source QGIS framework: The QGIS graphical modeler (Processing Modeler) provides similar visual orchestration, but its layer transfer mechanism has interface differences with the platform's GsLayerPtr system, and does not support C++-level parallel scheduling based on CGraph. In summary, none of the above alternative solutions can fully realize the overall technical goals of this invention, which include the three collaborative technical means of connectionId-linked virtual layer placeholder, automatic injection of memory-level layer objects, and accurate restoration of cross-session parameter sources, as well as the seamless integration of CGraph C++ multi-threaded parallel scheduling and self-developed platform algorithm base class.

[0143] The foregoing has shown and described the basic principles and main features of the present invention and its advantages. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention.

[0144] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for visual orchestration and parallel execution based on directed acyclic graphs, characterized in that, The execution method includes the following steps: S1. Visual Canvas and Algorithm Node Modeling S1.1 Algorithm Node Creation Process: ① Users select an algorithm from the algorithm tree panel and drag and drop it onto the canvas; ② The system creates a QgsProcessNodeItem based on the algorithm ID, assigns a unique identifier (UUID) as the unique identifier of the node, and records the initial position; ③ The user double-clicks the node to open the parameter configuration panel (QgsProcessingWidget) and fills in the input and output parameters; S1.2, Drawing process of connecting lines: ① The user clicks and drags the mouse from the output port of the source node to the input port of the target node; ② The system determines the connection type based on the port semantics: if the connection is to a data source type parameter (VectorLayer / RasterLayer / FileDestination, etc.), a data flow connection is created (solid line, connectionType="datasource"); if the connection is to a control flow port, a time-dependent connection is created (dashed line, connectionType="prerequisite"). ③ After the connection is established, the system creates a virtual layer placeholder entry (InputLayerEntry, type=VirtualLayer) at the corresponding parameter position of the target node as a placeholder for the upstream result; S1.3 Linkage between connection identifier and virtual layer placeholder: When establishing a data flow connection on the visualization canvas, a globally unique identifier (connectionId) is automatically assigned to the connection, and at the same time, a virtual layer placeholder entry with the connectionId as the index key is automatically created in the parameter interface of the downstream algorithm node as a reserved placeholder for the execution result of the upstream algorithm. S2, Process Model Data Structure Design The process model is managed by the ProcessModel class, which contains three core data elements: ① Node Data (ProcessModelNode): Records node ID (UUID), algorithm identifier (algorithmId), canvas position (position), parameter mapping table (serializedParameters), virtual layer information (virtualLayerInfos, including connectionId two-way binding key and injection order order), field structure file path (fcsLayerPaths, used to restore field selection state across sessions). VirtualLayerInfo includes connectionId (two-way binding with connection UUID), displayName (UI display name), sourceNodeName (upstream node name), sourceLayerName (exactly matches the output layer during execution), and order (injection order when there are multiple connections), ensuring that the layer list with multiple input parameters is consistent with the connection establishment order; ② Connection data (ProcessModelConnection): Records the source node ID (fromNodeId), target node ID (toNodeId), parameter names involved in the connection (fromOutput / toInput), connection type (connectionType: datasource / prerequisite), unique connection identifier (connectionId, UUID), folder to file transfer strategy (folderToFileMode: first / all / filter), and suffix filter list (folderToFileSuffixes). ③ Topological Order: The system uses a standard topological ordering algorithm to sort the nodes, ensuring that upstream nodes execute before downstream nodes. If a loop is detected (i.e., the process has a circular dependency), the system refuses to execute and reports the specific loop path to the user to prevent execution deadlock or erroneous results caused by circular dependencies. S3, CGraph DAG Parallel Execution Engine S3.1 Upgrade the execution of the process model from the traditional serial ProcessModelExecutor to a parallel executor (CGraphExecutor) based on the CGraph framework. The execution process is divided into four stages: ① Pipeline Construction: The system traverses all nodes in the ProcessModel, creating a CGraphAlgorithmNode instance (an adapter from QgsProcessingAlgorithm to CGraph GNode) for each node; encapsulating the node ID, algorithm ID, node parameters, and executor ID into node configuration parameters (CGraphNodeConfigParam) and passing them to the node; based on the connection relationships in the ProcessModel, calling the directed graph interface (GGraph): node registration (addGNode) adds the node to the CGraph directed graph, and calling the directed graph interface (GGraph): dependency edge creation (addGEdge) establishes the execution dependencies between nodes; the CGraph framework internally maintains a DAG structure and automatically identifies the set of nodes that can be executed in parallel; ② Data Flow Establishment: The system iterates through all data source connections, establishing a data flow association between the source node (upstream CGraphAlgorithmNode) and the target node (downstream CGraphAlgorithmNode). This involves calling the target node's upstream node registration (addUpstreamNode) to record upstream node references, enabling the reading of layer data from the upstream node's output parameters during execution. For folderToFile type connections, the connection configuration is stored in the target node's folder-to-file connection mapping table (folderToFileConnections). During execution, the folder is scanned, and file paths that meet the conditions are injected into the target parameters. ③ Parallel execution of nodes (CGraphAlgorithmNode:run()): Each CGraphAlgorithmNode executes the run() method during CGraph scheduling: It calls the parameter preparation method (prepareParameters) to complete parameter merging and upstream injection: Internally, it sequentially executes serializedParameters deserialization and filters out invalid layer pointers, completes missing parameters with algorithm default values, and calls injectUpstreamData() to complete upstream data injection; injectUpstreamData() (an internal step of prepareParameters) sorts the parameters in virtualLayerInfos according to their connectionId and order, reads the GsLayerPtr list from all upstream nodes m_outputParam, and directly inserts the layer memory objects (GsVectorLayerPtr / GsRasterLayerPtr). The current node's execution parameter table is injected, allowing downstream algorithms to directly access the loaded layer objects in memory. A processing context (QgsProcessingContext) and processing feedback (QgsProcessingFeedback) are created, and algorithm->run() is called to execute the algorithm. After the algorithm is executed, the output layer parsing (parseAlgorithmOutputLayers) is called to parse the output results, load the output file path into a GsLayerPtr layer object, encapsulate it into a layer parameter object (CGraphLayerParam) and store it in the node output parameter (m_outputParam) for downstream nodes to read. Through the node log emitter (NodeLogEmitter) singleton, the execution logs and node events (Started / Completed / Failed) are routed to the corresponding UI log components with the executor identifier (executorId) as the key, supporting precise log routing when multiple executors are running concurrently. Execution exception handling mechanism: Multi-level exception detection is implemented at each key node in the four-stage parallel execution to ensure that the execution failure of any node can be accurately captured and safely reported without affecting the overall system stability; S3.2 Automatic injection of memory-level layer objects: During the parallel execution preparation phase, the downstream algorithm node reads the spatial layer memory object from the output parameter container of the upstream node according to the connectionId and injection order recorded in its virtual layer placeholder entry, and directly injects it into the execution parameter table of this node, realizing the memory-level transfer of layer data between algorithms and eliminating intermediate disk file read and write. S4, XML Persistence and Cross-Session Recovery S4.1 The system supports persisting the complete process model as an XML file (.model) for reuse across sessions: ① Save process (saveToXml): Iterate through all ProcessModelNodes, serialize the node parameters into XML strings using the parameter serialization method defined in the parameter definition, and store them in the serializedParameters mapping table; serialize the virtual layer information (connectionId, displayName, sourceNodeName, order) into child nodes; write fcsLayerPaths (field structure file path) into node attributes; serialize all ProcessModelConnections (including connectionType, fromOutput, toInput, folderToFileMode, and suffix filter list (folderToFileSuffixes)) into child elements under the connections node; and write the entire process to a file in standard XML format. ② Loading Process (loadFromXml): Parse the XML file and rebuild the ProcessModelNode and ProcessModelConnection lists; rebuild the parameter objects based on the XML strings in serializedParameters; rebuild virtual layer placeholder entries in the node parameter panel based on virtualLayerInfos; load the field structure file based on fcsLayerPaths and restore the field selection state; restore the selected virtual layers for single-layer parameters (such as INPUT / JOIN for spatial connections) based on the selected virtual layer display names (selectedVirtualLayerDisplayNames); restore the position of each node and the connection shape in QGraphicsScene based on the node's position attribute. S4.2, ConnectionId Two-Way Binding and Cross-Session Restoration: During the serialization and persistence of the process model, the connectionId and the virtual layer placeholder information are written together to the persistent file to establish a two-way binding relationship between the connection and the parameter placeholder. When the process model is reloaded across sessions, the connectionId is used as the anchor point to accurately restore the correspondence between which connection the downstream parameter comes from and which upstream node it originates from, solving the engineering problem of mismatch when nodes have the same name or multiple connections point to the same parameter.

2. The visualization orchestration and parallel execution method based on directed acyclic graphs according to claim 1, characterized in that, In step S1, the system for creating the algorithm node provides an interactive canvas (QgsProcessManagementWidget) based on the Qt graphics view framework (QGraphicsView). The algorithm node (QgsProcessNodeItem) and the connecting arrow (QgsProcessArrowItem) are both implemented as subclasses of Qt graphics primitives (QGraphicsItem), supporting drag-and-drop, selection, movement, and scaling interactions.

3. The visualization orchestration and parallel execution method based on directed acyclic graphs according to claim 2, characterized in that, In step S1.2, the triggering method of the time-dependent connection is as follows: When a user drags a connection from a source node to a target node, but there are no compatible data source parameter pairs between the two nodes (i.e., the upstream output type and the downstream input type cannot match), the system automatically creates this connection as a time-dependent connection (dashed line, connectionType="prerequisite"), which represents a pure time-series constraint that "the upstream node must complete execution before the downstream node can start execution". Time-dependent connections do not trigger the creation of virtual layer placeholder entries, nor do they participate in the data injection logic of injectUpstreamData. During the Pipeline construction phase, the corresponding addGEdge call only establishes CGraph scheduling dependencies and does not establish data stream associations in the m_upstreamNodes list.

4. The visualization orchestration and parallel execution method based on directed acyclic graphs according to claim 3, characterized in that, In step S1.2, the connection validity verification and linkage cleanup mechanism are implemented. Type compatibility check during connection establishment: The system automatically checks the type compatibility matrix based on the output-input type compatibility matrix: Vector output ↔ Vector input, Raster output ↔ Raster input, Folder output ↔ Folder / Raster / File input (supports 3D batch processing scenes), File output ↔ File input; The path string parameter (parameter name containing keywords such as dir / path / input) can be expanded to receive folder output. A prompt will appear indicating type incompatibility and the connection creation will be refused, without producing any side effects on the data model. Duplicate connection prevention mechanism: Before creating a connection, the system traverses the NodeConnection record set of the current canvas; if a connection with the same source node to the target node already exists, it will be silently refused to create, to prevent duplicate connections from causing downstream nodes to receive redundant upstream data injection. Virtual layer cleanup mechanism for deleting nodes / connections: When deleting a node or connection, the system simultaneously cleans up all associated virtual layers: iterates through the downstream node parameter records by connectionId and deletes the corresponding entries; clears the layer pointers and XML serialization strings in the parameter cache; if there are temporary raster files, deletes the disk files simultaneously; finally, it updates the connection data model and refreshes the parameter panel to ensure that the data model and UI state are completely consistent.

5. The visualization orchestration and parallel execution method based on directed acyclic graphs according to claim 1, characterized in that, In step S3.1, the parallel execution mechanism is illustrated by a process with three nodes (A→B, A→C, B+C→D): ① CGraph analyzes the DAG structure and identifies A as the root node (in-degree 0), so A is executed first; ②After A completes execution, the in-degree of both B and C drops to 0, and CGraph submits B and C to the thread pool for concurrent execution; ③ After both B and C have been executed, the in-degree of D drops to 0. CGraph then executes D. In the run() method, D simultaneously reads the output layers of B and C as input through upstream data injection (injectUpstreamData). ④ Once the entire process is complete, the parallel executor (CGraphExecutor) emits an execution completion signal (executionCompleted=true).

6. The visualization orchestration and parallel execution method based on directed acyclic graphs according to claim 5, characterized in that, In step S3.1, the anomaly detection is divided into three layers: ① During the initialization phase, the algorithm instance is searched by algorithm ID. If it is not registered, it is refused entry into the schedule and the reason is reported. ② During the parameter validation phase, a pre-check of complete parameters is performed before run() is executed (e.g., missing required fields, non-existent paths, null layer pointers). ③ During the execution phase, check the return flag of algorithm->run(). Failure at any level triggers a node failure event, which is highlighted via the log routing component, and the UI displays a resume execution button. CGraphExecutor maintains a five-state machine (idle / running / paused / completed / errored), supporting both synchronous and asynchronous execution interfaces as well as pause, resume, and cancel controls.

7. The visualization orchestration and parallel execution method based on directed acyclic graphs according to claim 1, characterized in that, In step S1, the virtual layer placeholder mechanism is as follows: when the data stream connection is established, a virtual layer placeholder entry is automatically created in the parameter interface of the downstream algorithm node as a reserved placeholder for the execution result of the upstream algorithm; during the execution phase, the placeholder entry is automatically replaced by the layer memory object actually output by the upstream algorithm, eliminating the operation step of manually configuring intermediate parameters by the operator.

8. The visualization orchestration and parallel execution method based on directed acyclic graphs according to claim 7, characterized in that, In step S4, the bidirectional binding cross-session restoration mechanism for connection unique identifiers assigns a globally unique identifier to each data stream connection. During the serialization and persistence of the process model, the identifier and the virtual layer placeholder entries are written together into the persistence file to establish a two-way binding relationship between the connection and the parameter placeholder entries. When loading the process model across sessions, this identifier is used as an anchor point to accurately restore the correspondence between which connection the parameter comes from and which upstream node it originates from, solving the engineering problem of mismatch in complex scenarios such as node name conflicts or multiple connections pointing to the same parameter.

9. The visualization orchestration and parallel execution method based on directed acyclic graphs according to claim 1, characterized in that, An adaptation method for integrating geographic information processing algorithms into the directed acyclic graph parallel scheduling framework: Design an adapter node class to encapsulate the geographic information processing algorithm base class into the graph node interface required by the parallel scheduling framework; When the adapter node is executed in the parallel framework scheduling, it completes the complete orchestration of parameter merging, upstream data injection, algorithm instance execution, and output result writing back internally, and establishes a complete mapping between the geographic information processing algorithm execution contract and the parallel scheduling framework node interface contract; By leveraging the happens-before semantic guarantee provided by the node in-degree zeroing mechanism in the parallel scheduling framework—that is, the current node is only scheduled by the thread pool after all the execution methods of the predecessor nodes have returned—the operation of upstream nodes writing layer memory objects to the output parameter container occurs before the operation of downstream nodes reading from the container. This enables thread-safe reading and writing of memory layer objects in a multi-threaded concurrent scheduling environment without applying an additional mutex lock to the output parameter container. This allows geographic information processing algorithms that were originally unable to be scheduled by the parallel framework to participate in lock-free memory flow multi-threaded parallel scheduling as directed acyclic graph nodes.

10. The visualization orchestration and parallel execution method based on directed acyclic graphs according to claim 9, characterized in that: In the GIS algorithm visualization orchestration, two connection semantics are defined: data flow connection (datasource, solid line) and temporal dependency connection (temporal dependency prerequisite, dashed line). After the data flow connection is established, the system creates virtual layer placeholder entries in the downstream node. During execution, the upstream output is automatically injected into the corresponding parameters of the downstream algorithm according to the connection relationship. The DAG connection defines a folder-to-file connection type (Folder→File), supporting three file selection strategies: first (first file), all (all files), and filter (filter by suffix). During the execution phase, the folder is scanned and the file paths that meet the conditions are dynamically injected into the downstream algorithm parameters. The NodeLogEmitter global singleton routes logs and node events by executorId, supporting multiple concurrent CGraphExecutor instances running concurrently without log interference and ensuring precise routing to their respective UI components.

Images