A Hybrid Orchestration Method for Batch and Stream Processing Tasks Based on a Process Engine

Abstract

This invention discloses a method for hybrid orchestration of batch and stream processing tasks based on a process engine. The process model includes a start event node, batch processing task nodes, stream processing task nodes, a gateway node, and an end event node; it breaks the limitations of the traditional static FGA model, supports hybrid orchestration of batch and stream processing in the same process model, and complex process control based on parallelism and inclusive gateways, thereby improving the flexibility of task modeling.

Description

[0001] This invention is a divisional application of Chinese invention patent application No. 2025112781128, filed on September 9, 2025, entitled "A Method and System for Arranging Multi-Source Heterogeneous Data Tasks Based on a Process Engine". Technical Field

[0002] This invention relates to the field of data integration and development technology, specifically to a method for hybrid orchestration of batch and stream processing tasks based on a process engine. Background Technology

[0003] In the field of civil aviation information technology (such as security check systems and baggage systems), there is a large amount of heterogeneous and complex data that requires batch processing, stream processing, cleaning, association, storage, and service-oriented encapsulation. However, existing data integration and development platforms (such as Apache NiFi and SeaTunnel) typically use static directed acyclic graphs (DAGs) as task scheduling models, which suffers from insufficient flexibility in task orchestration. Specifically, static DAG models struggle to support complex branching, parallelism, conditional judgments, and hybrid orchestration. Summary of the Invention

[0004] The purpose of this invention is to provide a hybrid orchestration method for batch processing and stream processing tasks based on a process engine, which can simultaneously support batch processing and stream processing tasks and has flexible task orchestration capabilities.

[0005] To achieve the above objectives, the solution of the present invention is: a method for hybrid orchestration of batch processing and stream processing tasks based on a process engine, comprising the following steps: Process modeling and task orchestration: The process is built based on a process engine. The process model includes a start event node, batch processing task nodes, stream processing task nodes, a gateway node, and an end event node, where: The start event node is configured as a timer event or a stream event to trigger subsequent batch processing tasks or stream processing tasks, respectively. The gateway node is used to logically control the execution results of batch processing events and stream processing events, determining the direction of subsequent processes. Specifically, it includes: Parallel branch gateway: receives a task and outputs it to multiple parallel batch processing tasks and / or stream processing tasks; The parallel merging gateway connects multiple parallel batch processing tasks and / or stream processing tasks, aggregates the results of all tasks, and then outputs them to the next task. The inclusive branch gateway receives a task, splits the output of the task into multiple branches, selects a branch based on conditions, and does not execute all of them; The inclusive merging gateway connects to multiple tasks, aggregates the results of all executed tasks in the inclusive branches, and then outputs them to the next task.

[0006] Furthermore, the timer event needs to be configured with a CRON expression to be triggered; the stream event does not need to be configured with a CRON expression, and the stream event can only be followed by a stream processing task node.

[0007] Furthermore, the batch processing task node supports JDBC data source type, MQ-PULL data source type, and batch processing plugin type: JDBC data source type: Configure JDBC data source and SQL statements; MQ data source type: Configure MQ data source, topic, queue, and batch size parameters that support PULL mode; Batch processing plugin type: Users can develop and upload Java plugins based on the SDK provided by the system to configure batch processing plugin task nodes.

[0008] Furthermore, if the output target is a JDBC data source, JDBC data source information needs to be configured, and insert statements need to be written, supporting the use of the $O{json path} expression to populate JSON attribute values; If the output target is an MQ data source, configure the MQ data source information and the topic or queue name.

[0009] Furthermore, the stream processing task node supports MQ-PUSH data source type and stream processing plugin type: MQ-PUSH data source type: Configure MQ data source, topic, and queue parameters; Stream processing plugin type: Users develop their own Java plugins based on the SDK provided by the system. Furthermore, when the inclusive branch gateway selects a branch based on conditions, it supports using the SELECT + HAVING syntax to analyze and judge the output dataset of the preceding task node.

[0010] Furthermore, in the batch processing task node and / or stream processing task node, the data output is configured as follows: directly outputting the batch data to the next node or to the target data source configured by the user; or, first using a Groovy script to process the data according to the agreed interface, and then outputting it to the next node or the target data source.

[0011] After adopting the above solution, the beneficial effects of the present invention are as follows: Achieve true dynamic orchestration of batch and stream tasks: Breaking the limitations of isolated design and static scheduling of batch and stream processing tasks in traditional data platforms, it supports unified modeling and visual orchestration of two types of heterogeneous tasks within the same BPMN process model, and realizes parallel triggering, conditional routing and result aggregation between tasks through parallel gateways and inclusive gateways, which greatly improves the flexibility of complex data process modeling. Attached Figure Description

[0012] Figure 1 This is a diagram illustrating the gateway nodes according to an embodiment of the present invention; Figure 2 This is an example diagram of a parallel branch gateway according to an embodiment of the present invention; Figure 3 This is an example diagram of a parallel merging gateway according to an embodiment of the present invention; Figure 4 This is an example diagram of an embodiment of the present invention, showing an inclusive branch gateway. Figure 5 This is an example diagram of an inclusive merging gateway according to an embodiment of the present invention; Figure 6 This is an example diagram of a hybrid orchestration of batch processing and stream processing tasks based on a process engine according to an embodiment of the present invention; Figure 7 This is a structural diagram of a multi-source heterogeneous data task orchestration system based on a process engine according to an embodiment of the present invention; Figure 8 This is a flowchart illustrating an application embodiment of the present invention; Figure 9 This is a flowchart illustrating the second application embodiment of the present invention. Detailed Implementation

[0013] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] This invention provides a method for hybrid orchestration of batch and stream processing tasks based on a workflow engine, comprising the following processes: <Process Modeling and Dynamic Task Orchestration> This invention provides users with visual process design capabilities by introducing a graphical modeling method based on BPMN (Business Process Model and Label), in order to support the dynamic and hybrid orchestration of complex data tasks.

[0015] Visual low-code modeling: The system provides an intuitive visual designer interface, allowing users to build data processing workflows by dragging and dropping, eliminating the need to write underlying scheduling code. This significantly lowers the barrier to workflow design and enables a low-code development experience.

[0016] Process model components: The process model consists of multiple nodes, and the icons for each node are as follows: Figure 1 As shown. These nodes are the foundation for implementing hybrid orchestration and complex process control. The specific node types and operating mechanisms are as follows: 1) Start Event Node (StartEvent): Configure the start event in the designer interface according to the actual business needs.

[0017] The start event supports timer events and stream events.

[0018] If it is a timer event, you need to configure the timer's CRON (an expression for a time rule) to trigger the timer event. The timer event can only be followed by -2) batch event nodes; If it is a streaming event, there is no need to configure CRON. The streaming event can only be connected to the following nodes: 3) Stream processing event nodes.

[0019] Subsequent tasks can be any batch or stream processing event node; that is, only the types of task nodes that can be followed by the starting event node are restricted.

[0020] 2) Batch processing event nodes (batch processing Task nodes): Supports JDBC data source types, MQ-PULL data source types, and batch processing plugin types: ① JDBC Data Source Type: Configures the JDBC data source and SQL statements (select, insert, select into) (query / insert / query and write). You can specify that the SQL execution result is directly output to the next node (applicable to select) or output to a user-configured target data source (applicable to insert / select into). Before outputting the SQL execution result, you can configure it to be processed by a Groovy script according to a predefined interface before being output.

[0021] ②MQ data source type: Configure parameters such as MQ data source that supports PULL mode, topic / queue, and batch size. This can be understood as: selecting an MQ in PULL mode (such as Kafka consumer, RabbitMQBasic.Get), specifying the consumption target through topic / queue, and balancing throughput and real-time performance through batchSize.

[0022] Data can be directly output to the next node or to a user-configured target data source; similarly, data can be processed using Groovy scripts according to a predefined interface before being output.

[0023] ③ Batch processing plugin type: Users can develop and upload Java plugins based on the SDK provided by the system to configure batch processing plugin task nodes and extend batch processing capabilities; it also supports output to the next node or output to the target data source configured by the user.

[0024] If the output target is a JDBC data source, you need to configure the JDBC data source information and write an INSERT statement that supports the $O{json path} expression to populate JSON attribute values; if the output target is an MQ data source, configure the MQ data source information and the topic or queue name.

[0025] Except for the first task, any processing task node in the process will use the output of the preceding task as the input data source if there is a preceding task.

[0026] 3) Stream processing event nodes (stream processing Task nodes): Supports MQ-PUSH data source type, and allows configuration of MQ data source and topic / queue parameters; It can be configured to process streaming data using Groovy scripts according to a predefined interface before outputting it, or it can be directly output to the next node or a user-defined target data source.

[0027] Stream processing plugin type: Users can develop their own Java plugins based on the SDK provided by the system to extend stream processing capabilities.

[0028] Dynamic orchestration capability: The workflow engine dynamically parses and executes the model at runtime. Based on the logical definitions of the gateway nodes (such as parallelism and conditional judgments), it schedules and coordinates the execution order and path of batch processing and stream processing tasks in real time, thereby addressing the complex and ever-changing data processing needs under different business scenarios and achieving true dynamic task orchestration.

[0029] This invention addresses the problems of poor scalability and maintainability of existing platforms by taking the following measures: Low-code / scripting: Task nodes support custom logic processing using Groovy scripts, eliminating the need for compilation and deployment, and offering high flexibility.

[0030] Pluggable architecture: An SDK is provided that allows users to develop custom batch processing and stream processing plugins to address extremely complex customization needs. The system achieves powerful extensibility through a mechanism that dynamically runs plugins.

[0031] Visual orchestration: Drag-and-drop process design through a visual interface reduces the cost of business logic changes.

[0032] 4) Gateway node: The main function of a gateway node is to logically control the execution results of batch processing events and stream processing events, determining the direction of subsequent processes. The gateway acts as a decision point or synchronization point in the process, responsible for dynamically adjusting process branches based on task execution results or data content.

[0033] Gateway nodes include parallel gateways and inclusive gateways, as detailed below: The Parallel Fork gateway supports parallel task distribution: it receives a task and outputs it to multiple parallel tasks. (See reference...) Figure 2 .

[0034] Parallel Join Gateway: Integrates multiple parallel tasks, aggregates the results of all tasks, and then outputs them to the next task. (See reference...) Figure 3 In addition, the gateway can merge data streams based on traceId and supports timeout policies to prevent processes from getting stuck indefinitely.

[0035] Inclusive Fork Gateway: Takes a task, splits it into multiple branches, and selects a branch based on conditions, not necessarily executing all of them. (See reference) Figure 4 It supports SELECT + HAVING syntax for judgment, such as Select Max(a)From #dataset[0] Having Max(a)>300; Inclusive Join Gateway: Integrates multiple tasks, aggregates the results of all executed tasks across all inclusive branches, and then outputs them to the next task. (Reference) Figure 5 .

[0036] 5) End Event Node: Once the process is complete, the system automatically sends the task completion status to the built-in message queue.

[0037] The following is through Figure 6 Here are some examples to illustrate the relationships between the nodes: Start event: Timer triggered, entering "Batch Task 1"; The parallel gateway connects to "Batch Processing Task 1" and outputs 3 parallel tasks (Stream Processing Task 2, Batch Processing Task 3 and Batch Processing Task 4). After "Stream Processing Task 2" is completed, it will proceed to "Stream Processing Plugin Task 5"; The inclusive gateway receives two tasks (batch task 3 and batch task 4) and outputs "batch task 6"; The parallel gateway connects to "Stream Processing Plugin Task 5" and "Batch Processing Task 6", outputs "Batch Processing Task 7", and the entire process ends when "Batch Processing Task 7" is completed.

[0038] This invention, "A Hybrid Orchestration Method for Batch and Stream Processing Tasks Based on a Process Engine," is combined with data source access and adaptation processes, traceId end-to-end tracing processes, data flow and execution processes, and result output and service-oriented processes to provide a multi-source heterogeneous data task orchestration method based on a process engine. The task orchestration system used in this method includes the following modules (see reference). Figure 7 ): Data access layer Data access module: responsible for adapting and abstracting multiple heterogeneous data sources (JDBC database, message queue MQ, etc.), uniformly converting them into an internal data model, and shielding the underlying differences.

[0039] Plugin / Extension Interface: Provides SDK interfaces, allowing users to extend custom data sources or processing logic.

[0040] Process orchestration and control layer Workflow Engine Module: The core module, based on the BPMN / process modeling concept, implements the orchestration of task nodes, branch gateways, condition controls, etc. The scheduling and execution control submodule supports batch processing tasks (CRON timed), stream processing tasks (real-time triggered), concurrent execution, and retry fault tolerance.

[0041] Task processing and computation layer Task Executor Module: Contains two types of task executors: batch processing task executors (JDBC / MQ-PULL / plugin tasks); stream processing task executors (MQ-PUSH / plugin tasks); Dynamic script and expression module: Supports expressions such as ${parameter} and $O{json path} with Groovy script parsing for flexible data processing rules.

[0042] Data flow and monitoring layer Message middleware module: responsible for data transmission between nodes, result caching, and transparent transmission of traceId; Monitoring and governance module: Provides full-link tracing with traceId, operation log collection, alarm detection, and data lineage analysis.

[0043] Result Service Layer Service-oriented configuration module: Supports outputting task processing results to databases and MQ message queues, and automatically publishing them as REST API services through configuration; Reports and Notifications Submodule: Automatically generates runtime reports and task completion notifications.

[0044] A multi-source heterogeneous data task orchestration method based on a process engine includes the following processes: <Data source access and adaptation> Multi-source access: Provide pluggable data source adaptation components. The system currently supports various common data sources such as JDBC databases (e.g., Oracle, SqlServer, MySQL, OpenGauss), MQ message queues (e.g., RabbitMQ, RocketMQ, Pulsar), etc., and can continue to develop and adapt new data sources according to project requirements.

[0045] Data standardization: Standardize and abstract the original heterogeneous data sources to shield underlying differences, that is, uniformly convert the data formats of different accessed data sources into an internal data model.

[0046] <Hybrid orchestration method for batch processing and stream processing tasks based on a process engine> <traceId full-link tracing> When each process instance starts to execute, that is, when the start event node (StartEvent) is triggered, the system will automatically generate a globally unique traceId, which is used to uniquely identify the specific process execution instance of this time.

[0047] When the start event is a timer event, a traceId is generated each time the timer is triggered, identifying a batch processing job instance. When the start event is a stream event, the arrival of each (or each batch) of messages will trigger the process and generate a traceId, identifying the processing instance of the message (or message batch).

[0048] The generated traceId will be transparently transmitted with the data among all task nodes and gateway nodes throughout the entire life cycle of the process instance to achieve full-link tracing. Regardless of whether the process logic involves parallel branches, conditional routing, or synchronous aggregation, all task nodes belonging to the same process instance share the same traceId.

[0049] This traceId, as a context identifier throughout the process, provides core support for the observability of the system and supports: Operation log association: All logs printed by nodes carry this traceId, and all logs of a complete process execution can be aggregated and queried through it, greatly improving the efficiency of fault troubleshooting.

[0050] Alarm event location: Any alarm information generated by the system will carry the traceId at that time, and operation and maintenance personnel can quickly locate the specific process instance and its context that triggered the alarm based on this.

[0051] Data lineage analysis: By recording traceId, data operations (read / write), and data objects, the system can accurately draw a complete flow and transformation map of data in the process, supporting upstream tracing and downstream impact analysis.

[0052] The traceId mechanism integrates a series of dispersed, potentially concurrently executed task processing processes into a traceable whole with a unified identifier, achieving transparency and observability across the entire data processing chain.

[0053] <Data Flow and Execution> 1) Data flow mechanism: Data transmission between task nodes is handled by the system's built-in message middleware. This middleware provides a unified communication channel for nodes. Sending nodes publish their processing results (encapsulated as an internal data model) to the middleware, and downstream consumer nodes subscribe to and retrieve the data. This mechanism shields the complexity of physical deployment and communication protocols between nodes, achieving decoupling, asynchronicity, and buffering of data transmission, thus remaining transparent to the user.

[0054] 2) Task scheduling and execution mechanism: The process engine has the following responsibilities: Example of a driving process: Based on the definition of the process model (such as a BPMN process), the execution order of task nodes is scheduled according to the path logic of the start event, task nodes, and gateway.

[0055] Coordinated Message Consumption: Works in conjunction with the built-in message middleware. When the engine schedules a task node for execution, the node pulls upstream data from the middleware; after execution, it pushes the results back to the middleware for downstream nodes to consume. The engine indirectly controls the data flow by coordinating the message consumption of nodes.

[0056] Supports concurrent execution: When the process reaches the parallel gateway, the engine will simultaneously schedule task nodes on multiple branches to execute, making full use of system resources and improving processing efficiency.

[0057] Fault tolerance and retries are provided: The engine monitors the execution status of each task node. If a node fails, the engine can automatically initiate a retry according to a preset strategy (such as fixed intervals or exponential backoff). For tasks that fail repeatedly, the engine can mark the process instance as failed and issue an alarm to ensure system reliability.

[0058] <Results Output and Service-Oriented Development> 1) After the process is completed, the system will output the processing results to the JDBC data source, MQ data source, etc., and then publish them as a REST API service through configuration.

[0059] 2) Once the process is complete, the system will automatically generate a running report and corresponding status events for users to query.

[0060] Application Example 1: Fusion Processing of Security Inspection Logs and Alarm Data In airport security check scenarios, there are two types of key data: passenger channel log data from the security check system database (including the number of passengers passing through the channel and the time information) and real-time alarm data from the passenger channel alarm message queue.

[0061] To detect operational anomalies at security checkpoints promptly and accurately, this embodiment utilizes the task orchestration method based on a workflow engine provided by this invention to fuse the two types of data, generating fused alarm data, which is then stored as an alarm report. The process is as follows (see flowchart illustration). Figure 8 : 1) StartEvent: Type: CRON timer event (e.g., triggered every 10 minutes).

[0062] The system generates a unique traceId to identify a single instance of process execution.

[0063] 2) Batch Task (JDBC): Configure the data source: Security inspection system database (Oracle).

[0064] Execute SQL: SELECT FROM passenger_log WHERE ts>last_time; Output: Passenger channel log dataset[0].

[0065] 3) Batch processing Task (MQ-PULL): Configure MQ: Alarm Message Queue (RocketMQ).

[0066] To fetch data: Configure topic = "SECURITY_ALARM"; batchSize = 10.

[0067] The data is processed using Groovy scripts, and the dataset is output[1].

[0068] 4) Gateway: Includes and merges two data streams; Input: Two data streams from dataset[0] and dataset[1].

[0069] Aggregation conditions: Aggregate the results by traceId to generate a fused dataset[2].

[0070] 5) Stream processing events (anomaly detection and new alarm generation): Real-time calculation and anomaly detection are performed on the dataset[2], for example, to determine whether the number of passengers in a channel exceeds the threshold in a certain period of time; If an anomaly is detected, a new alarm event is generated; The output is the merged alarm dataset[3], which is written to the alarm_report table in the database.

[0071] 6) EndEvent: Marks the end of the process.

[0072] Application Example 2: Airport Passenger Flow Forecasting and Dynamic Load Allocation at Security Checkpoints Using a 5-minute prediction interval, the system integrates historical channel logs, recent flight schedules, external features (weather / holidays), and real-time passenger flow to predict the throughput and estimated queuing time of each security checkpoint for the next 15–30 minutes. It automatically generates alarms (such as opening / closing channels and adjusting staffing ratios) when the SLA (Service Level Agreement, e.g., the maximum acceptable queuing time for passengers to pass through security does not exceed 10 minutes) or channel occupancy thresholds are exceeded, and outputs the results via MQ service.

[0073] Main data source JDBC: passenger_log (Historical channel logs for the past 60 days, including time granularity, number of users who passed, etc.) JDBC: flight_schedule (flight schedule for the next 2 hours, including departure times, gates, load factors, etc.) JDBC: ext_features (external features such as weather, holidays, and major events) MQ: PASSENGER_TURNSTILE (Real-time turnstile / counter passenger flow event) The execution process is as follows, please refer to the flowchart. Figure 9 : 1) StartEvent: Type: CRON Timer (triggered every 5 minutes); A unique traceId is generated when the process is triggered to identify the current process instance.

[0074] 2) Batch processing Task (JDBC: Historical Log): Query the passenger_log for the past 60 days, aggregate it to a 5-minute granularity, and form basic statistical characteristics (time period / weekday, holiday, flight peak window, etc.). Output dataset[hist].

[0075] 3) Batch processing Task (JDBC: Flight Schedule): Extract the flight_schedule for the next 2 hours and map the passenger flow contribution of the flights by time window (t, t+15, t+30) (which can be estimated by passenger capacity or aircraft type). Output dataset[flt].

[0076] 4) Batch processing Task (JDBC: External Feature): Acquire information on external influencing factors such as weather (rainfall / visibility) and holidays / major events; Output dataset[ext].

[0077] 5) Parallel Join: Parallel merging into 6) for feature construction; 6) Batch processing plugin (feature fusion + prediction inference): Perform feature engineering and fusion on the dataset [hist] / [flt] / [ext]; Load the pre-trained offline model and make predictions for the next 15 / 30 minutes (throughput, waiting time). Output dataset[pred].

[0078] 7) Batch processing plugin (real-time passenger flow + forecast correction): Pull the passenger flow count dataset[pred] (e.g., the last 5 minutes) from MQ: PASSENGER_TURNSTILE to perform nowcasting correction on the prediction results. Align dataset[pred] with dataset[rt] in terms of time window and channel dimension, and fuse them to obtain the corrected prediction.

[0079] Output dataset[fused].

[0080] 8) Inclusive branching (conditional routing): If the expected waiting time exceeds the target SLA or the channel occupancy rate exceeds 85%, an alarm task will be initiated.

[0081] 9) Batch processing tasks (alarm tasks): Production alerts are sent to the message queue. 10) Batch processing tasks (persistent result sets) The dataset[fused] is persisted to the security_lane_forecast table for use by other systems such as Kanban / BI.

[0082] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments and to explain the operating principles of the embodiments in conjunction with the relevant descriptions in the specification. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention. Components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the design of this case. All equivalent changes made based on the key design features of this case shall fall within the protection scope of this case.

Claims

1. A method for mixed orchestration of batch and stream processing tasks based on a process engine, characterized in that, Includes the following steps: Process modeling and task orchestration: The process is built based on a process engine. The process model includes a start event node, batch processing task nodes, stream processing task nodes, a gateway node, and an end event node, where: The start event node is configured as a timer event or a stream event to trigger subsequent batch processing tasks or stream processing tasks, respectively. The gateway node is used to logically control the execution results of batch processing events and stream processing events, determining the direction of subsequent processes. Specifically, it includes: Parallel branch gateway: receives a task and outputs it to multiple parallel batch processing tasks and / or stream processing tasks; The parallel merging gateway connects multiple parallel batch processing tasks and / or stream processing tasks, aggregates the results of all tasks, and then outputs them to the next task. The inclusive branch gateway receives a task, splits the output of the task into multiple branches, selects a branch based on conditions, and does not execute all of them; The inclusive merging gateway connects to multiple tasks, aggregates the results of all executed tasks in the inclusive branches, and then outputs them to the next task.

2. The batch processing and stream processing task hybrid orchestration method based on a workflow engine as described in claim 1, characterized in that: The timer event needs to be configured with a CRON expression to be triggered; the stream event does not need to be configured with a CRON expression, and the stream event can only be followed by a stream processing task node.

3. The batch processing and stream processing task hybrid orchestration method based on a workflow engine as described in claim 1, characterized in that: The batch processing task node supports JDBC data source type, MQ-PULL data source type, and batch processing plugin type: JDBC data source type: Configure JDBC data source and SQL statements; MQ data source type: Configure MQ data source, topic, queue, and batch size parameters that support PULL mode; Batch processing plugin type: Users can develop and upload Java plugins based on the SDK provided by the system to configure batch processing plugin task nodes.

4. The batch processing and stream processing task hybrid orchestration method based on a workflow engine as described in claim 3, characterized in that: If the output target is a JDBC data source, you need to configure the JDBC data source information and write an INSERT statement, which supports using the $O{json path} expression to populate JSON attribute values; If the output target is an MQ data source, configure the MQ data source information and the topic or queue name.

5. The batch processing and stream processing task hybrid orchestration method based on a workflow engine as described in claim 1, characterized in that: The stream processing task node supports MQ-PUSH data source type and stream processing plugin type: MQ-PUSH data source type: Configure MQ data source, topic, and queue parameters; Stream processing plugin type: Users develop their own Java plugins based on the SDK provided by the system.

6. The method for hybrid orchestration of multi-source heterogeneous data tasks based on a process engine as described in claim 1, characterized in that: When the inclusive branch gateway selects a branch based on conditions, it supports using the SELECT + HAVING syntax to analyze and judge the output dataset of the preceding task node.

7. The method for hybrid orchestration of multi-source heterogeneous data tasks based on a process engine as described in claim 1, characterized in that: In the batch processing task node and / or stream processing task node, the data output is configured as follows: directly output the batch data to the next node or to the target data source configured by the user; or, first use Groovy script to process the data according to the agreed interface, and then output it to the next node or the target data source.

Images