A data task orchestration method supporting end-to-end tracing and service-oriented architecture

Abstract

This invention discloses a data task orchestration method that supports end-to-end tracing and service-oriented architecture. It accesses multi-source heterogeneous data through pluggable data source adapter components and transforms the data into a unified internal data model. The process model includes start event nodes, batch processing task nodes, stream processing task nodes, gateway nodes, and end event nodes. It breaks the limitations of traditional static FGA models, supporting mixed orchestration of batch and stream processing within the same process model, as well as complex process control based on parallel and inclusive gateways. The traceId mechanism enables end-to-end task tracing, facilitating fault location and operational monitoring.

Description

[0001] This invention is a divisional application of Chinese invention patent application No. 2025112781128, filed on September 9, 2025, entitled "A Method and System for Arranging Multi-Source Heterogeneous Data Tasks Based on a Process Engine". Technical Field

[0002] This invention relates to the field of data integration and development technology, specifically to a data task orchestration method that supports end-to-end tracing and service-oriented architecture. Background Technology

[0003] In the field of civil aviation information technology (such as security check systems and baggage systems), there is a large amount of heterogeneous and complex data that requires batch processing, stream processing, cleaning, association, storage, and service-oriented encapsulation. However, existing data integration and development platforms (such as Apache NiFi and SeaTunnel) typically use static directed acyclic graphs (DAGs) as task scheduling models, which have the following shortcomings: Accessing multi-source heterogeneous data is complex: different types of data sources vary greatly, often requiring developers to manually write a lot of adaptation code.

[0004] Insufficient task orchestration flexibility: Static DAG models struggle to support complex branching, parallelism, conditional judgments, and hybrid orchestration.

[0005] Limited service-oriented architecture and governance: The data processing process lacks a full-link tracing mechanism, and the interface encapsulation and operational monitoring capabilities are insufficient. Summary of the Invention

[0006] The purpose of this invention is to provide a data task orchestration method that supports end-to-end tracing and service-oriented architecture, possessing flexible task orchestration capabilities, end-to-end tracing capabilities, and the ability to output results as services.

[0007] To achieve the above objectives, the solution of the present invention is: a data task orchestration method supporting end-to-end tracing and service-oriented architecture, comprising the following steps: Data source access and adaptation: Access multi-source heterogeneous data through pluggable data source adaptation components and transform the data into a unified internal data model; Process modeling and task orchestration: The process is built based on a process engine. The process model includes a start event node, batch processing task nodes, stream processing task nodes, a gateway node, and an end event node, where: The start event node is configured as a timer event or a stream event to trigger subsequent batch processing tasks or stream processing tasks, respectively. The gateway node is used to logically control the execution results of batch processing events and stream processing events, determining the direction of subsequent processes. Specifically, it includes: Parallel branch gateway: receives a task and outputs it to multiple parallel batch processing tasks and / or stream processing tasks; The parallel merging gateway connects multiple parallel batch processing tasks and / or stream processing tasks, aggregates the results of all tasks, and then outputs them to the next task. The inclusive branch gateway receives a task, splits the output of the task into multiple branches, selects a branch based on conditions, and does not execute all of them; The inclusive merging gateway connects to multiple tasks, aggregates the results of all executed tasks in the inclusive branches, and then outputs them to the next task.

[0008] End-to-end tracing: When a data processing workflow instance is triggered for execution, a globally unique trace identifier (traceId) is generated and passed through during the execution of processing task nodes and the data flow process; among which... When the start event is a timer event, a traceId is generated each time the timer is triggered, identifying a batch job instance; When the start event is a stream event, the arrival of each message or batch of messages will trigger the process and generate a traceId, which identifies the processing instance of that message or batch of messages. Based on the transmitted traceId, the execution process of this process instance is monitored across the entire chain and logs are correlated; this step includes at least the following functions: Run log association: When a task node is executed, log information carrying the same traceId will be aggregated and stored, and the complete process log can be queried by traceId. Alarm event localization: When a process execution error occurs or an alarm is triggered, the alarm information is associated with the traceId of the process instance that triggered the alarm; Data lineage analysis: Records data operations, data objects and transformation processes associated with traceId, and draws cross-task data flow and lineage maps based on traceId records of multiple process instances; Task execution and flow: The process engine schedules task nodes and gateway nodes for execution according to the process model, and transmits data between nodes through the system's internal message mechanism; Results output and service-oriented architecture: After the process is completed, the task processing results are output to the specified target data source, and then published as a service interface through configuration.

[0009] Furthermore, the timer event needs to be configured with a CRON expression to be triggered; the stream event does not need to be configured with a CRON expression, and the stream event can only be followed by a stream processing task node.

[0010] Furthermore, the batch processing task node supports JDBC data source type, MQ-PULL data source type, and batch processing plugin type: JDBC data source type: Configure JDBC data source and SQL statements; MQ data source type: Configure MQ data source, topic, queue, and batch size parameters that support PULL mode; Batch processing plugin type: Users can develop and upload Java plugins based on the SDK provided by the system to configure batch processing plugin task nodes.

[0011] Furthermore, if the output target is a JDBC data source, JDBC data source information needs to be configured, and insert statements need to be written, supporting the use of the $O{json path} expression to populate JSON attribute values; If the output target is an MQ data source, configure the MQ data source information and the topic or queue name.

[0012] Furthermore, the stream processing task node supports MQ-PUSH data source type and stream processing plugin type: MQ-PUSH data source type: Configure MQ data source, topic, and queue parameters; Stream processing plugin type: Users develop their own Java plugins based on the SDK provided by the system.

[0013] Furthermore, when the inclusive branch gateway selects a branch based on conditions, it supports using the SELECT + HAVING syntax to analyze and judge the output dataset of the preceding task node.

[0014] Furthermore, in the batch processing task node and / or stream processing task node, the data output is configured as follows: directly outputting the batch data to the next node or to the target data source configured by the user; or, first using a Groovy script to process the data according to the agreed interface, and then outputting it to the next node or the target data source.

[0015] Furthermore, in end-to-end tracing, the data lineage analysis is used to support upstream data tracing and downstream impact analysis.

[0016] Furthermore, the service interface is a REST API.

[0017] Furthermore, the method also includes a report generation step: after the process is completed, an operation report and status events are automatically generated for users to query.

[0018] After adopting the above solution, the beneficial effects of the present invention are as follows: 1. Hybrid orchestration and dynamic control: Breaking the limitations of traditional static DAG models, it supports hybrid orchestration of batch processing and stream processing in the same process model, as well as complex process control based on parallel and inclusive gateways, improving the flexibility of task modeling.

[0019] 2. End-to-End Observability: The traceId mechanism enables end-to-end task tracing, facilitating fault location and operational monitoring, and improving observability and reliability. Specifically, the traceId is generated when the initial event node is triggered and is deeply bound to different event types (batch events and streaming events), enabling precise management, location, and analysis of complex hybrid orchestration processes.

[0020] 3. Low-code extension: Supports flexible customization of processing logic through SDK plugins and Groovy scripts, eliminating the need for developers to write large amounts of code.

[0021] 4. Enhanced service-oriented output: The results of any task process can be quickly encapsulated as a REST API, improving the reusability of data services.

[0022] 5. Governance and lineage analysis: As a unified data identifier that is seamlessly transmitted between batch processing and stream processing nodes, traceId enables cross-task data lineage tracking, facilitating data quality management and compliance auditing. Attached Figure Description

[0023] Figure 1 This is a structural diagram of a multi-source heterogeneous data task orchestration system based on a process engine according to an embodiment of the present invention; Figure 2 This is a diagram illustrating the gateway nodes according to an embodiment of the present invention; Figure 3 This is an example diagram of a parallel branch gateway according to an embodiment of the present invention; Figure 4 This is an example diagram of a parallel merging gateway according to an embodiment of the present invention; Figure 5 This is an example diagram of an embodiment of the present invention, showing an inclusive branch gateway. Figure 6 This is an example diagram of an inclusive merging gateway according to an embodiment of the present invention; Figure 7 This is an example diagram of a multi-source heterogeneous data task orchestration process based on a process engine according to an embodiment of the present invention; Figure 8 This is a flowchart illustrating an application embodiment of the present invention; Figure 9 This is a flowchart illustrating the second application embodiment of the present invention. Detailed Implementation

[0024] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] This invention provides a data task orchestration method that supports end-to-end tracing and service-oriented architecture. The orchestration system used includes the following modules (see reference). Figure 1 ): Data access layer Data access module: responsible for adapting and abstracting multiple heterogeneous data sources (JDBC database, message queue MQ, etc.), uniformly converting them into an internal data model, and shielding the underlying differences.

[0026] Plugin / Extension Interface: Provides SDK interfaces, allowing users to extend custom data sources or processing logic.

[0027] Process orchestration and control layer Workflow Engine Module: The core module, based on the BPMN / process modeling concept, implements the orchestration of task nodes, branch gateways, condition controls, etc. The scheduling and execution control submodule supports batch processing tasks (CRON timed), stream processing tasks (real-time triggered), concurrent execution, and retry fault tolerance.

[0028] Task processing and computation layer Task Executor Module: Contains two types of task executors: batch processing task executors (JDBC / MQ-PULL / plugin tasks); stream processing task executors (MQ-PUSH / plugin tasks); Dynamic script and expression module: Supports expressions such as ${parameter} and $O{json path} with Groovy script parsing for flexible data processing rules.

[0029] Data flow and monitoring layer Message middleware module: responsible for data transmission between nodes, result caching, and transparent transmission of traceId; Monitoring and governance module: Provides full-link tracing with traceId, operation log collection, alarm detection, and data lineage analysis.

[0030] Result Service Layer Service-oriented configuration module: Supports outputting task processing results to databases and MQ message queues, and automatically publishing them as REST API services through configuration; Reports and Notifications Submodule: Automatically generates runtime reports and task completion notifications.

[0031] This invention provides a data task orchestration method that supports end-to-end tracing and service-oriented architecture, comprising the following steps: <Data Source Access and Adaptation> Multi-source access: Provides pluggable data source adaptation components. The system currently supports a variety of common data sources such as JDBC databases (e.g., Oracle, SQL Server, MySQL, OpenGauss) and MQ message queues (e.g., RabbitMQ, RocketMQ, Pulsar), and can continue to develop and adapt new data sources according to project needs.

[0032] Data standardization: Standardize and abstract the original heterogeneous data sources, shielding the underlying differences, that is, uniformly convert the data formats of different data sources into the internal data model.

[0033] <Process Modeling and Dynamic Task Orchestration> This invention provides users with visual process design capabilities by introducing a graphical modeling method based on BPMN (Business Process Model and Label), in order to support the dynamic and hybrid orchestration of complex data tasks.

[0034] Visual low-code modeling: The system provides an intuitive visual designer interface, allowing users to build data processing workflows by dragging and dropping, eliminating the need to write underlying scheduling code. This significantly lowers the barrier to workflow design and enables a low-code development experience.

[0035] Process model components: The process model consists of multiple nodes, and the icons for each node are as follows: Figure 2 As shown. These nodes are the foundation for implementing hybrid orchestration and complex process control. The specific node types and operating mechanisms are as follows: 1) Start Event Node (StartEvent): Configure the start event in the designer interface according to the actual business needs.

[0036] The start event supports timer events and stream events.

[0037] If it is a timer event, you need to configure the timer's CRON (an expression for a time rule) to trigger the timer event. The timer event can only be followed by -2) batch event nodes; If it is a streaming event, there is no need to configure CRON. The streaming event can only be connected to the following nodes: 3) Stream processing event nodes.

[0038] Subsequent tasks can be any batch or stream processing event node; that is, only the types of task nodes that can be followed by the starting event node are restricted.

[0039] 2) Batch processing event nodes (batch processing Task nodes): Supports JDBC data source types, MQ-PULL data source types, and batch processing plugin types: ① JDBC Data Source Type: Configures the JDBC data source and SQL statements (select, insert, select into) (query / insert / query and write). You can specify that the SQL execution result is directly output to the next node (applicable to select) or output to a user-configured target data source (applicable to insert / select into). Before outputting the SQL execution result, you can configure it to be processed by a Groovy script according to a predefined interface before being output.

[0040] ②MQ data source type: Configure parameters such as MQ data source that supports PULL mode, topic / queue, and batch size. This can be understood as: selecting an MQ in PULL mode (such as Kafka consumer, RabbitMQBasic.Get), specifying the consumption target through topic / queue, and balancing throughput and real-time performance through batchSize.

[0041] Data can be directly output to the next node or to a user-configured target data source; similarly, data can be processed using Groovy scripts according to a predefined interface before being output.

[0042] ③ Batch processing plugin type: Users can develop and upload Java plugins based on the SDK provided by the system to configure batch processing plugin task nodes and extend batch processing capabilities; it also supports output to the next node or output to the target data source configured by the user.

[0043] If the output target is a JDBC data source, you need to configure the JDBC data source information and write an INSERT statement that supports the $O{json path} expression to populate JSON attribute values; if the output target is an MQ data source, configure the MQ data source information and the topic or queue name.

[0044] Except for the first task, any processing task node in the process will use the output of the preceding task as the input data source if there is a preceding task.

[0045] 3) Stream processing event nodes (stream processing Task nodes): Supports MQ-PUSH data source type, and allows configuration of MQ data source and topic / queue parameters; It can be configured to process streaming data using Groovy scripts according to a predefined interface before outputting it, or it can be directly output to the next node or a user-defined target data source.

[0046] Stream processing plugin type: Users can develop their own Java plugins based on the SDK provided by the system to extend stream processing capabilities.

[0047] Dynamic orchestration capability: The workflow engine dynamically parses and executes the model at runtime. Based on the logical definitions of the gateway nodes (such as parallelism and conditional judgments), it schedules and coordinates the execution order and path of batch processing and stream processing tasks in real time, thereby addressing the complex and ever-changing data processing needs under different business scenarios and achieving true dynamic task orchestration.

[0048] This invention addresses the problems of poor scalability and maintainability of existing platforms by taking the following measures: Low-code / scripting: Task nodes support custom logic processing using Groovy scripts, eliminating the need for compilation and deployment, and offering high flexibility.

[0049] Pluggable architecture: An SDK is provided that allows users to develop custom batch processing and stream processing plugins to address extremely complex customization needs. The system achieves powerful extensibility through a mechanism that dynamically runs plugins.

[0050] Visual orchestration: Drag-and-drop process design through a visual interface reduces the cost of business logic changes.

[0051] 4) Gateway node: The main function of a gateway node is to logically control the execution results of batch processing events and stream processing events, determining the direction of subsequent processes. The gateway acts as a decision point or synchronization point in the process, responsible for dynamically adjusting process branches based on task execution results or data content.

[0052] Gateway nodes include parallel gateways and inclusive gateways, as detailed below: The Parallel Fork gateway supports parallel task distribution: it receives a task and outputs it to multiple parallel tasks. (See reference...) Figure 3 .

[0053] Parallel Join Gateway: Integrates multiple parallel tasks, aggregates the results of all tasks, and then outputs them to the next task. (See reference...) Figure 4 In addition, the gateway can merge data streams based on traceId and supports timeout policies to prevent processes from getting stuck indefinitely.

[0054] Inclusive Fork Gateway: Takes a task, splits it into multiple branches, and selects a branch based on conditions, not necessarily executing all of them. (See reference) Figure 5, support the SELECT + HAVING syntax judgment, for example, Select Max(a)From #dataset[0] Having Max(a)>300; Inclusive Join Gateway: Access multiple tasks, aggregate the actual executed task results of all inclusive branches, and then output to the next task. Refer to Figure 6 .

[0055] 5) End Event Node (EndEvent): Indicates the end of the process, and the system automatically sends the task completion status to the built-in message queue.

[0056] The following uses Figure 7 an example to explain the relationship between each node: Start Event: Triggered by a timer, enter "Batch Task 1"; The Parallel Gateway accesses "Batch Task 1" and outputs 3 parallel tasks (Stream Processing Task 2, Batch Task 3, and Batch Task 4); After "Stream Processing Task 2" is completed, it enters "Stream Processing Plugin Task 5"; The Inclusive Gateway accesses 2 tasks (Batch Task 3 and Batch Task 4) and outputs "Batch Task 6"; The Parallel Gateway accesses "Stream Processing Plugin Task 5" and "Batch Task 6" and outputs "Batch Task 7". After "Batch Task 7" is completed, the entire process ends.

[0057] <traceId Full Link Tracing> When each process instance starts to execute, that is, when the Start Event Node (StartEvent) is triggered, the system will automatically generate a globally unique traceId, which is used to uniquely identify the specific process execution instance of this time.

[0058] When the start event is a timer event, a traceId is generated each time the timer is triggered, identifying a batch processing job instance. When the start event is a stream event, the arrival of each (or each batch) of messages will trigger the process and generate a traceId, identifying the processing instance of the message (or message batch).

[0059] The generated traceId will be transparently transmitted along with the data between all task nodes and gateway nodes during the entire life cycle of the process instance to achieve full link tracing. Regardless of whether the process logic involves parallel branches, conditional routing, or synchronous aggregation, all task nodes belonging to the same process instance share the same traceId.

[0060] The traceId serves as a consistent context identifier, providing core support for system observability and enabling: Log association: All logs printed by all nodes carry this traceId, which can be used to aggregate and query all logs of a complete process execution, greatly improving the efficiency of troubleshooting.

[0061] Alarm event location: Any alarm information generated by the system will carry the traceId at that time, which can be used by operation and maintenance personnel to quickly locate the specific process instance that triggered the alarm and its context.

[0062] Data lineage analysis: By recording traceId, data operations (read / write), and data objects, the system can accurately draw a complete flow and transformation map of data in the process, supporting upstream tracing and downstream impact analysis.

[0063] The traceId mechanism integrates a series of dispersed, potentially concurrently executed task processing processes into a traceable whole with a unified identifier, achieving transparency and observability across the entire data processing chain.

[0064] <Data Flow and Execution> 1) Data flow mechanism: Data transmission between task nodes is handled by the system's built-in message middleware. This middleware provides a unified communication channel for nodes. Sending nodes publish their processing results (encapsulated as an internal data model) to the middleware, and downstream consumer nodes subscribe to and retrieve the data. This mechanism shields the complexity of physical deployment and communication protocols between nodes, achieving decoupling, asynchronicity, and buffering of data transmission, thus remaining transparent to the user.

[0065] 2) Task scheduling and execution mechanism: The process engine has the following responsibilities: Example of a driving process: Based on the definition of the process model (such as a BPMN process), the execution order of task nodes is scheduled according to the path logic of the start event, task nodes, and gateway.

[0066] Coordinated Message Consumption: Works in conjunction with the built-in message middleware. When the engine schedules a task node for execution, the node pulls upstream data from the middleware; after execution, it pushes the results back to the middleware for downstream nodes to consume. The engine indirectly controls the data flow by coordinating the message consumption of nodes.

[0067] Supports concurrent execution: When the process reaches the parallel gateway, the engine will simultaneously schedule task nodes on multiple branches to execute, making full use of system resources and improving processing efficiency.

[0068] Fault tolerance and retries are provided: The engine monitors the execution status of each task node. If a node fails, the engine can automatically initiate a retry according to a preset strategy (such as fixed intervals or exponential backoff). For tasks that fail repeatedly, the engine can mark the process instance as failed and issue an alarm to ensure system reliability.

[0069] <Results Output and Service-Oriented Development> 1) After the process is completed, the system will output the processing results to the JDBC data source, MQ data source, etc., and then publish them as a REST API service through configuration.

[0070] 2) Once the process is complete, the system will automatically generate a running report and corresponding status events for users to query.

[0071] Application Example 1: Fusion Processing of Security Inspection Logs and Alarm Data In airport security check scenarios, there are two types of key data: passenger channel log data from the security check system database (including the number of passengers passing through the channel and the time information) and real-time alarm data from the passenger channel alarm message queue.

[0072] To detect operational anomalies at security checkpoints promptly and accurately, this embodiment utilizes the task orchestration method based on a workflow engine provided by this invention to fuse the two types of data, generating fused alarm data, which is then stored as an alarm report. The process is as follows (see flowchart illustration). Figure 8 : 1) StartEvent: Type: CRON timer event (e.g., triggered every 10 minutes).

[0073] The system generates a unique traceId to identify a single instance of process execution.

[0074] 2) Batch Task (JDBC): Configure the data source: Security inspection system database (Oracle).

[0075] Execute SQL: SELECT FROM passenger_log WHERE ts>last_time; Output: Passenger channel log dataset[0].

[0076] 3) Batch processing Task (MQ-PULL): Configure MQ: Alarm Message Queue (RocketMQ).

[0077] To fetch data: Configure topic = "SECURITY_ALARM"; batchSize = 10.

[0078] The data is processed using Groovy scripts, and the dataset is output[1].

[0079] 4) Gateway: Includes and merges two data streams; Input: Two data streams from dataset[0] and dataset[1].

[0080] Aggregation conditions: Aggregate the results by traceId to generate a fused dataset[2].

[0081] 5) Stream processing events (anomaly detection and new alarm generation): Real-time calculation and anomaly detection are performed on the dataset[2], for example, to determine whether the number of passengers in a channel exceeds the threshold in a certain period of time; If an anomaly is detected, a new alarm event is generated; The output is the merged alarm dataset[3], which is written to the alarm_report table in the database.

[0082] 6) EndEvent: Marks the end of the process.

[0083] Application Example 2: Airport Passenger Flow Forecasting and Dynamic Load Allocation at Security Checkpoints Using a 5-minute prediction interval, the system integrates historical channel logs, recent flight schedules, external features (weather / holidays), and real-time passenger flow to predict the throughput and estimated queuing time of each security checkpoint for the next 15–30 minutes. It automatically generates alarms (such as opening / closing channels and adjusting staffing ratios) when the SLA (Service Level Agreement, e.g., the maximum acceptable queuing time for passengers to pass through security does not exceed 10 minutes) or channel occupancy thresholds are exceeded, and outputs the results via MQ service.

[0084] Main data source JDBC: passenger_log (Historical channel logs for the past 60 days, including time granularity, number of users who passed, etc.) JDBC: flight_schedule (flight schedule for the next 2 hours, including departure times, gates, load factors, etc.) JDBC: ext_features (external features such as weather, holidays, and major events) MQ: PASSENGER_TURNSTILE (Real-time turnstile / counter passenger flow event) The execution process is as follows, please refer to the flowchart. Figure 9 : 1) StartEvent: Type: CRON Timer (triggered every 5 minutes); A unique traceId is generated when the process is triggered to identify the current process instance.

[0085] 2) Batch processing Task (JDBC: Historical Log): Query the passenger_log for the past 60 days, aggregate it to a 5-minute granularity, and form basic statistical characteristics (time period / weekday, holiday, flight peak window, etc.). Output dataset[hist].

[0086] 3) Batch processing Task (JDBC: Flight Schedule): Extract the flight_schedule for the next 2 hours and map the passenger flow contribution of the flights by time window (t, t+15, t+30) (which can be estimated by passenger capacity or aircraft type). Output dataset[flt].

[0087] 4) Batch processing Task (JDBC: External Feature): Acquire information on external influencing factors such as weather (rainfall / visibility) and holidays / major events; Output dataset[ext].

[0088] 5) Parallel Join: Parallel merging into 6) for feature construction; 6) Batch processing plugin (feature fusion + prediction inference): Perform feature engineering and fusion on the dataset [hist] / [flt] / [ext]; Load the pre-trained offline model and make predictions for the next 15 / 30 minutes (throughput, waiting time). Output dataset[pred].

[0089] 7) Batch processing plugin (real-time passenger flow + forecast correction): Pull the passenger flow count dataset[pred] (e.g., the last 5 minutes) from MQ: PASSENGER_TURNSTILE to perform nowcasting correction on the prediction results. Align dataset[pred] with dataset[rt] in terms of time window and channel dimension, and fuse them to obtain the corrected prediction.

[0090] Output dataset[fused].

[0091] 8) Inclusive branching (conditional routing): If the expected waiting time exceeds the target SLA or the channel occupancy rate exceeds 85%, an alarm task will be initiated.

[0092] 9) Batch processing tasks (alarm tasks): Production alerts are sent to the message queue. 10) Batch processing tasks (persistent result sets) The dataset[fused] is persisted to the security_lane_forecast table for use by other systems such as Kanban / BI.

[0093] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments and to explain the operating principles of the embodiments in conjunction with the relevant descriptions in the specification. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention. Components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the design of this case. All equivalent changes made based on the key design features of this case shall fall within the protection scope of this case.

Claims

1. A data task orchestration method supporting end-to-end tracing and service-oriented architecture, characterized in that, The process includes the following steps: Data source access and adaptation: Access multi-source heterogeneous data through pluggable data source adaptation components and transform the data into a unified internal data model; Process modeling and task orchestration: The process is built based on a process engine. The process model includes a start event node, batch processing task nodes, stream processing task nodes, a gateway node, and an end event node, where: The start event node is configured as a timer event or a stream event to trigger subsequent batch processing tasks or stream processing tasks, respectively. The gateway node is used to logically control the execution results of batch processing events and stream processing events, determining the direction of subsequent processes. Specifically, it includes: Parallel branch gateway: It takes in a task and outputs it to multiple parallel batch processing tasks and / or stream processing tasks. The parallel merging gateway connects multiple parallel batch processing tasks and / or stream processing tasks, aggregates the results of all tasks, and then outputs them to the next task. The inclusive branch gateway receives a task, splits the output of the task into multiple branches, selects a branch based on conditions, and does not execute all of them; The inclusive merging gateway connects to multiple tasks, aggregates the results of all executed tasks in the inclusive branches, and then outputs them to the next task. End-to-end tracing: When a data processing workflow instance is triggered for execution, a globally unique trace identifier (traceId) is generated and passed through during the execution of processing task nodes and the data flow process; among which... When the start event is a timer event, a traceId is generated each time the timer is triggered, identifying a batch job instance; When the start event is a stream event, the arrival of each message or batch of messages will trigger the process and generate a traceId, which identifies the processing instance of that message or batch of messages. Based on the transmitted traceId, the execution process of this process instance is monitored across the entire chain and logs are correlated; this step includes at least the following functions: Run log association: When a task node is executed, log information carrying the same traceId will be aggregated and stored, and the complete process log can be queried by traceId; Alarm event localization: When a process execution error occurs or an alarm is triggered, the alarm information is associated with the traceId of the process instance that triggered the alarm; Data lineage analysis: Records data operations, data objects and transformation processes associated with traceId, and draws cross-task data flow and lineage maps based on traceId records of multiple process instances; Task execution and flow: The process engine schedules task nodes and gateway nodes for execution according to the process model, and transmits data between nodes through the system's internal message mechanism; Results output and service-oriented architecture: After the process is completed, the task processing results are output to the specified target data source, and then published as a service interface through configuration.

2. The data task orchestration method supporting end-to-end tracing and service-oriented architecture as described in claim 1, characterized in that: The timer event needs to be configured with a CRON expression to be triggered; the stream event does not need to be configured with a CRON expression, and the stream event can only be followed by a stream processing task node.

3. The data task orchestration method supporting end-to-end tracing and service-oriented architecture as described in claim 1, characterized in that: The batch processing task node supports JDBC data source type, MQ-PULL data source type, and batch processing plugin type: JDBC data source type: Configure JDBC data source and SQL statements; MQ data source type: Configure MQ data source, topic, queue, and batch size parameters that support PULL mode; Batch processing plugin type: Users can develop and upload Java plugins based on the SDK provided by the system to configure batch processing plugin task nodes.

4. The data task orchestration method supporting end-to-end tracing and service-oriented architecture as described in claim 3, characterized in that: If the output target is a JDBC data source, you need to configure the JDBC data source information and write an insert statement, which supports the use of the $O{json path} expression to populate JSON attribute values; If the output target is an MQ data source, configure the MQ data source information and the topic or queue name.

5. A data task orchestration method supporting end-to-end tracing and service-oriented architecture as described in claim 1, characterized in that: The stream processing task node supports MQ-PUSH data source type and stream processing plugin type: MQ-PUSH data source type: Configure MQ data source, topic, and queue parameters; Stream processing plugin type: Users develop their own Java plugins based on the SDK provided by the system.

6. The data task orchestration method supporting end-to-end tracing and service-oriented architecture as described in claim 1, characterized in that: When the inclusive branch gateway selects a branch based on conditions, it supports using the SELECT + HAVING syntax to analyze and judge the output dataset of the preceding task node.

7. A data task orchestration method supporting end-to-end tracing and service-oriented architecture as described in claim 1, characterized in that: In the batch processing task node and / or stream processing task node, the data output is configured as follows: directly output the batch data to the next node or to the target data source configured by the user; or, first use Groovy script to process the data according to the agreed interface, and then output it to the next node or the target data source.

8. The data task orchestration method supporting end-to-end tracing and service-oriented architecture as described in claim 1, characterized in that: In end-to-end tracing, the data lineage analysis is used to support upstream data tracing and downstream impact analysis.

9. The data task orchestration method supporting end-to-end tracing and service-oriented architecture as described in claim 1, characterized in that: The service interface is a REST API.

10. A data task orchestration method supporting end-to-end tracing and service-oriented architecture as described in claim 1, characterized in that: The method also includes a report generation step: after the process is completed, an operation report and status events are automatically generated for users to query.

Images