A multi-source heterogeneous data task arrangement method and system based on a process engine

Abstract

The application discloses a kind of multi-source heterogeneous data task arrangement method and system based on process engine, and unified internal data model is converted to by pluggable data source adaptation component access multi-source heterogeneous data;Process modeling and task arrangement: process model includes start event node, batch processing task node, stream processing task node, gateway node and end event node;Break the limitation of traditional static FGA model, support batch processing and stream processing in the same process model Hybrid arrangement, and complex process control based on parallel and inclusive gateway;Task whole-process tracking is realized by traceId mechanism, which is convenient for fault location and operation monitoring;Support through SDK plug-in and Groovy script support, realize the flexible customization of processing logic, and developers do not need to write code massively;Any task process result can be quickly packaged as REST API, improve data service reusability.

Description

Technical Field

[0001] This invention relates to the field of data integration and development technology, specifically to a method and system for orchestrating multi-source heterogeneous data tasks based on a process engine, applicable to the access, transformation, scheduling, tracking and service-oriented output of multi-source heterogeneous data. Background Technology

[0002] In the field of civil aviation information technology (such as security check systems and baggage systems), there is a large amount of heterogeneous and complex data that requires batch processing, stream processing, cleaning, association, storage, and service-oriented encapsulation. However, existing data integration and development platforms (such as Apache NiFi and SeaTunnel) typically use static directed acyclic graphs (DAGs) as task scheduling models, which have the following shortcomings:

[0003] Accessing multi-source heterogeneous data is complex: different types of data sources vary greatly, often requiring developers to manually write a lot of adaptation code.

[0004] Insufficient task orchestration flexibility: Static DAG models struggle to support complex branching, parallelism, conditional judgments, and hybrid orchestration.

[0005] Limited service-oriented architecture and governance: The data processing process lacks a full-link tracing mechanism, and the interface encapsulation and operational monitoring capabilities are insufficient.

[0006] Poor scalability and maintainability: It lacks low-code or plug-in extension mechanisms, making it difficult to adapt to rapid changes in complex business processes. Summary of the Invention

[0007] The purpose of this invention is to provide a multi-source heterogeneous data task orchestration method based on a process engine, which can simultaneously support batch processing and stream processing tasks, has flexible task orchestration and full-link tracing capabilities, and can realize result service output.

[0008] To achieve the above objectives, the solution of the present invention is: a multi-source heterogeneous data task orchestration method based on a process engine, comprising the following processes:

[0009] Data source access and adaptation: Access multi-source heterogeneous data through pluggable data source adaptation components and transform the data into a unified internal data model;

[0010] Process modeling and task orchestration:

[0011] The process model includes a start event node, a batch processing task node, a stream processing task node, a gateway node, and an end event node. The start event node is configured as a timer event or a stream event to trigger subsequent batch processing tasks or stream processing tasks, respectively.

[0012] In the processing task node, the data output configuration is as follows: directly output the batch data to the next node or to the target data source configured by the user; or, first use a Groovy script to process the batch data according to the agreed interface, and then output it to the next node or the target data source.

[0013] The gateway node includes a parallel gateway and an inclusive gateway, which are used to distribute, aggregate, or conditionally route task execution paths in parallel.

[0014] Link tracing: A unique trace identifier, traceId, is generated at the beginning of each process. The traceId is transmitted along the nodes throughout the process, enabling full-link tracing and monitoring.

[0015] Task execution and flow: The process engine schedules task nodes and gateway nodes for execution according to the process model, and transmits data between nodes through the system's internal message mechanism;

[0016] Results output and service-oriented architecture: After the process is completed, the task processing results are output to the specified target data source and then published as a REST API service through configuration.

[0017] Furthermore, the data source adaptation component supports JDBC databases and MQ message queues.

[0018] Furthermore, the batch processing task node supports JDBC data source type, MQ-PULL data source type, and batch processing plugin type:

[0019] JDBC data source type: Configure JDBC data source and SQL statements;

[0020] MQ data source type: Configure MQ data source, topic, queue, and batch size parameters that support PULL mode;

[0021] Batch processing plugin type: Users can develop and upload Java plugins based on the SDK provided by the system to configure batch processing plugin task nodes.

[0022] Furthermore, if the output target is a JDBC data source, the JDBC data source information needs to be configured, and insert statements need to be written. The expression `O{json path}` populates the JSON property value.

[0023] If the output target is an MQ data source, configure the MQ data source information and the topic or queue name.

[0024] Furthermore, the stream processing task node supports MQ-PUSH data source type and stream processing plugin type:

[0025] MQ-PUSH data source type: Configure MQ data source, topic, and queue parameters;

[0026] Stream processing plugin type: Users develop their own Java plugins based on the SDK provided by the system.

[0027] Furthermore, the gateway node specifically includes the following types:

[0028] Parallel Branch Gateway: Takes in one task and outputs to multiple parallel tasks;

[0029] Parallel Merging Gateway: Connects multiple parallel tasks, aggregates the results of all tasks, and then outputs them to the next task;

[0030] Inclusive Branch Gateway: When a task is received, it is split into multiple branches, and the branch is selected to execute the task based on the conditions;

[0031] Inclusive Merge Gateway: It connects to multiple tasks, aggregates and incorporates the actual executed task results from the inclusive branches, and then proceeds to subsequent tasks.

[0032] Furthermore, the traceId supports runtime logs, alarms, and lineage analysis.

[0033] Furthermore, once the process is complete, a runtime report and status events are automatically generated for users to query.

[0034] This invention also provides a multi-source heterogeneous data task orchestration system based on a process engine, comprising:

[0035] The data access layer is used to adapt to multiple heterogeneous data sources and uniformly convert the data into an internal data model.

[0036] The process orchestration and control layer is used to receive the process model configured by the user and to schedule and control the task nodes and gateway nodes in the process.

[0037] The task processing and computation layer is used to execute batch processing tasks and stream processing tasks scheduled by the process orchestration and control layer.

[0038] The data flow and monitoring layer is used to transfer data between task nodes, provide end-to-end tracking and monitoring functions, and generate operation reports;

[0039] The results service layer is used to output the task processing results and publish them as service interfaces.

[0040] Furthermore, the data access layer includes: a data access module responsible for adapting and abstracting multi-source heterogeneous data sources, uniformly converting them into an internal data model, and shielding underlying differences;

[0041] Plugin / Extension Interface: Provides SDK interfaces for users to extend custom data sources or processing logic;

[0042] The process orchestration and control layer includes:

[0043] Workflow Engine Module: Based on the BPMN workflow modeling concept, it realizes the orchestration of task nodes, branch gateways, and condition controls;

[0044] The scheduling and execution control submodule supports batch processing, stream processing, concurrent execution, and retry fault tolerance.

[0045] The task processing and computation layer includes:

[0046] Task Executor Module: Includes batch processing task executors and stream processing task executors, supporting the output of task processing results to a specified target data source;

[0047] Dynamic script and expression module: Supports expression and script parsing;

[0048] The data flow and monitoring layer includes:

[0049] Message middleware module: responsible for data transmission between nodes, result caching, and transparent transmission of traceId;

[0050] Monitoring and governance module: provides full-link tracing with traceId, operation log collection, alarm detection, and data lineage analysis;

[0051] The results service layer includes:

[0052] Service-oriented configuration module: Supports the automatic publication of data processed by tasks into a specified target data source as a REST API service through configuration;

[0053] Reports and Notifications Submodule: Automatically generates runtime reports and task completion notifications.

[0054] After adopting the above solution, the beneficial effects of the present invention are as follows:

[0055] 1. Hybrid orchestration and dynamic control: Breaking the limitations of traditional static DAG models, it supports hybrid orchestration of batch processing and stream processing in the same process model, as well as complex process control based on parallel and inclusive gateways, improving the flexibility of task modeling.

[0056] 2. End-to-end observability: The traceId mechanism enables full-process task tracing, facilitating fault location and operational monitoring, and improving observability and reliability.

[0057] 3. Low-code extension: Supports flexible customization of processing logic through SDK plugins and Groovy scripts, eliminating the need for developers to write large amounts of code.

[0058] 4. Enhanced service-oriented output: The results of any task process can be quickly encapsulated as a REST API, improving the reusability of data services.

[0059] 5. Governance and lineage analysis: Enables cross-task data lineage tracking, facilitating data quality management and compliance auditing. Attached Figure Description

[0060] Figure 1 This is a structural diagram of a multi-source heterogeneous data task orchestration system based on a process engine according to an embodiment of the present invention;

[0061] Figure 2 This is a diagram illustrating the gateway nodes according to an embodiment of the present invention;

[0062] Figure 3 This is an example diagram of a parallel branch gateway according to an embodiment of the present invention;

[0063] Figure 4 This is an example diagram of a parallel merging gateway according to an embodiment of the present invention;

[0064] Figure 5 This is an example diagram of an embodiment of the present invention, showing an inclusive branch gateway.

[0065] Figure 6 This is an example diagram of an inclusive merging gateway according to an embodiment of the present invention;

[0066] Figure 7 This is an example diagram of a multi-source heterogeneous data task orchestration process based on a process engine according to an embodiment of the present invention;

[0067] Figure 8 This is a flowchart illustrating an application embodiment of the present invention;

[0068] Figure 9 This is a flowchart illustrating the second application embodiment of the present invention. Detailed Implementation

[0069] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0070] This invention provides a multi-source heterogeneous data task orchestration system based on a workflow engine, comprising the following modules (see reference). Figure 1 ):

[0071] Data access layer

[0072] Data access module: responsible for adapting and abstracting multiple heterogeneous data sources (JDBC database, message queue MQ, etc.), uniformly converting them into an internal data model, and shielding the underlying differences.

[0073] Plugin / Extension Interface: Provides SDK interfaces, allowing users to extend custom data sources or processing logic.

[0074] Process orchestration and control layer

[0075] Workflow Engine Module: The core module, based on the BPMN / process modeling concept, implements the orchestration of task nodes, branch gateways, condition controls, etc.

[0076] The scheduling and execution control submodule supports batch processing tasks (CRON timed), stream processing tasks (real-time triggered), concurrent execution, and retry fault tolerance.

[0077] Task processing and computation layer

[0078] Task Executor Module: Contains two types of task executors: batch processing task executors (JDBC / MQ-PULL / plugin tasks); stream processing task executors (MQ-PUSH / plugin tasks);

[0079] Dynamic scripts and expression modules: Supported {parameter}, Expressions such as O{json path} are used with Groovy script parsing for flexible data processing rules.

[0080] Data flow and monitoring layer

[0081] Message middleware module: responsible for data transmission between nodes, result caching, and transparent transmission of traceId;

[0082] Monitoring and governance module: Provides full-link tracing with traceId, operation log collection, alarm detection, and data lineage analysis.

[0083] Result Service Layer

[0084] Service-oriented configuration module: Supports outputting task processing results to databases and MQ message queues, and automatically publishing them as REST API services through configuration;

[0085] Reports and Notifications Submodule: Automatically generates runtime reports and task completion notifications.

[0086] This invention provides a multi-source heterogeneous data task orchestration method based on a process engine, according to the aforementioned system, comprising the following processes:

[0087] <Data Source Access and Adaptation>

[0088] Multi-source access: Provides pluggable data source adaptation components. The system currently supports a variety of common data sources such as JDBC databases (e.g., Oracle, SQL Server, MySQL, OpenGauss) and MQ message queues (e.g., RabbitMQ, RocketMQ, Pulsar), and can continue to develop and adapt new data sources according to project needs.

[0089] Data standardization: Standardize and abstract the original heterogeneous data sources, shielding the underlying differences, that is, uniformly convert the data formats of different data sources into the internal data model.

[0090] <Process Modeling and Dynamic Task Orchestration>

[0091] This invention provides users with visual process design capabilities by introducing a graphical modeling method based on BPMN (Business Process Model and Label), in order to support the dynamic and hybrid orchestration of complex data tasks.

[0092] Visual low-code modeling:

[0093] The system provides an intuitive visual designer interface, allowing users to build data processing workflows by dragging and dropping, eliminating the need to write underlying scheduling code. This significantly lowers the barrier to workflow design and enables a low-code development experience.

[0094] Process model components:

[0095] The process model consists of multiple nodes, and the icons for each node are as follows: Figure 2 As shown. These nodes are the foundation for implementing hybrid orchestration and complex process control. The specific node types and operating mechanisms are as follows:

[0096] 1) Start Event Node (StartEvent):

[0097] Configure the start event in the designer interface according to the actual business needs.

[0098] The start event supports timer events and stream events.

[0099] If it is a timer event, you need to configure the timer's CRON (an expression for a time rule) to trigger the timer event. The timer event can only be followed by -2) batch event nodes;

[0100] If it is a streaming event, there is no need to configure CRON. The streaming event can only be connected to the following nodes: 3) Stream processing event nodes.

[0101] Subsequent tasks can be any batch or stream processing event node; that is, only the types of task nodes that can be followed by the starting event node are restricted.

[0102] 2) Batch processing event nodes (batch processing Task nodes):

[0103] Supports JDBC data source types, MQ-PULL data source types, and batch processing plugin types:

[0104] ① JDBC Data Source Type: Configures the JDBC data source and SQL statements (select, insert, select into) (query / insert / query and write). You can specify that the SQL execution result is directly output to the next node (applicable to select) or output to a user-configured target data source (applicable to insert / select into). Before outputting the SQL execution result, you can configure it to be processed by a Groovy script according to a predefined interface before being output.

[0105] ②MQ data source type: Configure parameters such as MQ data source that supports PULL mode, topic / queue, and batch size. This can be understood as: selecting an MQ in PULL mode (such as Kafka consumer, RabbitMQBasic.Get), specifying the consumption target through topic / queue, and balancing throughput and real-time performance through batchSize.

[0106] Data can be directly output to the next node or to a user-configured target data source; similarly, data can be processed using Groovy scripts according to a predefined interface before being output.

[0107] ③ Batch processing plugin type: Users can develop and upload Java plugins based on the SDK provided by the system to configure batch processing plugin task nodes and extend batch processing capabilities; it also supports output to the next node or output to the target data source configured by the user.

[0108] If the output target is a JDBC data source, then you need to configure the JDBC data source information and write the INSERT statement. The O{json path} expression populates the JSON attribute value; if the output target is an MQ data source, it configures the MQ data source information and the topic or queue name.

[0109] Except for the first task, any processing task node in the process will use the output of the preceding task as the input data source if there is a preceding task.

[0110] 3) Stream processing event nodes (stream processing Task nodes):

[0111] Supports MQ-PUSH data source type, and allows configuration of MQ data source and topic / queue parameters;

[0112] It can be configured to process streaming data using Groovy scripts according to a predefined interface before outputting it, or it can be directly output to the next node or a user-defined target data source.

[0113] Stream processing plugin type: Users can develop their own Java plugins based on the SDK provided by the system to extend stream processing capabilities.

[0114] Dynamic orchestration capability:

[0115] The workflow engine dynamically parses and executes the model at runtime. Based on the logical definitions of the gateway nodes (such as parallelism and conditional judgments), it schedules and coordinates the execution order and path of batch processing and stream processing tasks in real time, thereby addressing the complex and ever-changing data processing needs under different business scenarios and achieving true dynamic task orchestration.

[0116] This invention addresses the problems of poor scalability and maintainability of existing platforms by taking the following measures:

[0117] Low-code / scripting: Task nodes support custom logic processing using Groovy scripts, eliminating the need for compilation and deployment, and offering high flexibility.

[0118] Plug-in capability: An SDK is provided that allows users to develop custom batch processing and stream processing plugins to address extremely complex customization needs. The system achieves powerful extensibility through a mechanism that dynamically runs plugins.

[0119] Visual orchestration: Drag-and-drop process design through a visual interface reduces the cost of business logic changes.

[0120] 4) Gateway node:

[0121] The primary function of a gateway node is to logically control the execution results of batch and stream processing events, determining the direction of subsequent processes. The gateway acts as a decision point or synchronization point within the process, dynamically adjusting process branches based on task execution results or data content.

[0122] Gateway nodes include parallel gateways and inclusive gateways, as detailed below:

[0123] The Parallel Fork gateway supports parallel task distribution: it receives a task and outputs it to multiple parallel tasks. (See reference...) Figure 3 .

[0124] Parallel Join Gateway: Integrates multiple parallel tasks, aggregates the results of all tasks, and then outputs them to the next task. (See reference...) Figure 4 In addition, the gateway can merge data streams based on traceId and supports timeout policies to prevent processes from getting stuck indefinitely.

[0125] Inclusive Fork: One task is accessed and split into multiple branches. Branches are selected based on conditions and not all are executed. Refer to Figure 5 , which supports the SELECT + HAVING syntax for judgment. For example, Select Max(a)From #dataset[0] Having Max(a)>300;

[0126] Inclusive Join: Multiple tasks are accessed, and after aggregating the results of all tasks that have actually been executed in the inclusive branches, the output is sent to the next task. Refer to Figure 6 .

[0127] 5) End Event Node (EndEvent):

[0128] Indicates the end of the process, and the system automatically sends the task completion status to the built-in message queue.

[0129] The following uses the Figure 7 example to explain the relationship between each node: Start Event: Triggered by a timer, enter "Batch Task 1";

[0130] The Parallel Gateway accesses "Batch Task 1" and outputs 3 parallel tasks (Stream Processing Task 2, Batch Task 3, and Batch Task 4);

[0131] After "Stream Processing Task 2" is completed, it enters "Stream Processing Plugin Task 5";

[0132] The Inclusive Gateway accesses 2 tasks (Batch Task 3 and Batch Task 4) and outputs "Batch Task 6";

[0133] The Parallel Gateway accesses "Stream Processing Plugin Task 5" and "Batch Task 6" and outputs "Batch Task 7". After "Batch Task 7" is completed, the entire process ends.

[0134] <traceId Full Link Tracing>

[0135] When each process instance starts to execute, that is, when the Start Event Node (StartEvent) is triggered, the system will automatically generate a globally unique traceId, which is used to uniquely identify the specific process execution instance of this time.

[0136] When the start event is a timer event, a traceId is generated each time the timer is triggered, identifying a batch processing job instance. When the start event is a stream event, the arrival of each (or each batch) of messages will trigger the process and generate a traceId, identifying the processing instance of the message (or message batch).

[0137] The generated traceId will be transmitted along with the data to all task nodes and gateway nodes throughout the entire lifecycle of the process instance, enabling end-to-end tracing. Regardless of whether the process logic involves parallel branches, conditional routing, or synchronous aggregation, all task nodes belonging to the same process instance share the same traceId.

[0138] The traceId serves as a consistent context identifier, providing core support for system observability and enabling:

[0139] Log association: All logs printed by all nodes carry this traceId, which can be used to aggregate and query all logs of a complete process execution, greatly improving the efficiency of troubleshooting.

[0140] Alarm event location: Any alarm information generated by the system will carry the traceId at that time, which can be used by operation and maintenance personnel to quickly locate the specific process instance that triggered the alarm and its context.

[0141] Data lineage analysis: By recording traceId, data operations (read / write), and data objects, the system can accurately draw a complete flow and transformation map of data in the process, supporting upstream tracing and downstream impact analysis.

[0142] The traceId mechanism integrates a series of dispersed, potentially concurrently executed task processing processes into a traceable whole with a unified identifier, achieving transparency and observability across the entire data processing chain.

[0143] <Data Flow and Execution>

[0144] 1) Data flow mechanism:

[0145] Data transmission between task nodes is handled by the system's built-in message middleware. This middleware provides a unified communication channel for nodes. Sending nodes publish their processing results (encapsulated as an internal data model) to the middleware, and downstream consumer nodes subscribe to and retrieve the data. This mechanism shields the complexity of physical deployment and communication protocols between nodes, achieving decoupling, asynchronicity, and buffering of data transmission, thus remaining transparent to the user.

[0146] 2) Task scheduling and execution mechanism:

[0147] The process engine has the following responsibilities:

[0148] Example of a driving process: Based on the definition of the process model (such as a BPMN process), the execution order of task nodes is scheduled according to the path logic of the start event, task nodes, and gateway.

[0149] Coordinated Message Consumption: Works in conjunction with the built-in message middleware. When the engine schedules a task node for execution, the node pulls upstream data from the middleware; after execution, it pushes the results back to the middleware for downstream nodes to consume. The engine indirectly controls the data flow by coordinating the message consumption of nodes.

[0150] Supports concurrent execution: When the process reaches the parallel gateway, the engine will simultaneously schedule task nodes on multiple branches to execute, making full use of system resources and improving processing efficiency.

[0151] Fault tolerance and retries are provided: The engine monitors the execution status of each task node. If a node fails, the engine can automatically initiate a retry according to a preset strategy (such as fixed intervals or exponential backoff). For tasks that fail repeatedly, the engine can mark the process instance as failed and issue an alarm to ensure system reliability.

[0152] <Results Output and Service-Oriented Development>

[0153] 1) After the process is completed, the system will output the processing results to the JDBC data source, MQ data source, etc., and then publish them as a REST API service through configuration.

[0154] 2) Once the process is complete, the system will automatically generate a running report and corresponding status events for users to query.

[0155] Application Example 1: Fusion Processing of Security Inspection Logs and Alarm Data

[0156] In airport security check scenarios, there are two types of key data: passenger channel log data from the security check system database (including the number of passengers passing through the channel and the time information) and real-time alarm data from the passenger channel alarm message queue.

[0157] To detect operational anomalies at security checkpoints promptly and accurately, this embodiment utilizes the task orchestration method based on a workflow engine provided by this invention to fuse the two types of data, generating fused alarm data and storing it as an alarm report. The process is as follows (see flowchart illustration). Figure 8 :

[0158] 1) StartEvent:

[0159] Type: CRON timer event (e.g., triggered every 10 minutes).

[0160] The system generates a unique traceId to identify a single instance of process execution.

[0161] 2) Batch Task (JDBC):

[0162] Configure the data source: Security inspection system database (Oracle).

[0163] Execute SQL: SELECT FROM passenger_log WHERE ts>last_time;

[0164] Output: Passenger channel log dataset[0].

[0165] 3) Batch processing Task (MQ-PULL):

[0166] Configure MQ: Alarm Message Queue (RocketMQ).

[0167] To fetch data: Configure topic = "SECURITY_ALARM"; batchSize = 10.

[0168] The data is processed using Groovy scripts, and the dataset is output[1].

[0169] 4) Gateway: Includes and merges two data streams;

[0170] Input: Two data streams from dataset[0] and dataset[1].

[0171] Aggregation conditions: Aggregate the results by traceId to generate a fused dataset[2].

[0172] 5) Stream processing events (anomaly detection and new alarm generation):

[0173] Real-time calculation and anomaly detection are performed on the dataset[2], for example, to determine whether the number of passengers in a channel exceeds the threshold in a certain period of time;

[0174] If an anomaly is detected, a new alarm event is generated;

[0175] The output is the merged alarm dataset[3], which is written to the alarm_report table in the database.

[0176] 6) EndEvent: Marks the end of the process.

[0177] Application Example 2: Airport Passenger Flow Forecasting and Dynamic Load Allocation at Security Checkpoints

[0178] Using a 5-minute prediction interval, the system integrates historical channel logs, recent flight schedules, external features (weather / holidays), and real-time passenger flow to predict the throughput and estimated queuing time of each security checkpoint for the next 15–30 minutes. It automatically generates alarms (such as opening / closing channels and adjusting staffing ratios) when the SLA (Service Level Agreement, e.g., the maximum acceptable queuing time for passengers to pass through security is no more than 10 minutes) or the channel occupancy threshold is exceeded, and outputs the results via MQ service.

[0179] Main data source

[0180] JDBC: passenger_log (Historical channel logs for the past 60 days, including time granularity, number of users who passed, etc.)

[0181] JDBC: flight_schedule (flight schedule for the next 2 hours, including departure times, gates, load factors, etc.)

[0182] JDBC: ext_features (external features such as weather, holidays, and major events)

[0183] MQ: PASSENGER_TURNSTILE (Real-time turnstile / counter passenger flow event)

[0184] The execution process is as follows, please refer to the flowchart. Figure 9 :

[0185] 1) StartEvent:

[0186] Type: CRON Timer (triggered every 5 minutes);

[0187] A unique traceId is generated when the process is triggered to identify the current process instance.

[0188] 2) Batch processing Task (JDBC: Historical Log):

[0189] Query the passenger_log for the past 60 days, aggregate it to a 5-minute granularity, and form basic statistical characteristics (time period / weekday, holiday, flight peak window, etc.).

[0190] Output dataset[hist].

[0191] 3) Batch processing Task (JDBC: Flight Schedule):

[0192] Extract the flight_schedule for the next 2 hours and map the passenger flow contribution of the flights by time window (t, t+15, t+30) (which can be estimated by passenger capacity or aircraft type).

[0193] Output dataset[flt].

[0194] 4) Batch processing Task (JDBC: External Feature):

[0195] Acquire information on external influencing factors such as weather (rainfall / visibility) and holidays / major events;

[0196] Output dataset[ext].

[0197] 5) Parallel Join:

[0198] Parallel merging into 6) for feature construction;

[0199] 6) Batch processing plugin (feature fusion + prediction inference):

[0200] Perform feature engineering and fusion on the dataset [hist] / [flt] / [ext];

[0201] Load the pre-trained offline model and make predictions for the next 15 / 30 minutes (throughput, waiting time).

[0202] Output dataset[pred].

[0203] 7) Batch processing plugin (real-time passenger flow + forecast correction):

[0204] Pull the passenger flow count dataset[pred] (e.g., the last 5 minutes) from MQ: PASSENGER_TURNSTILE to perform nowcasting correction on the prediction results. Align dataset[pred] with dataset[rt] in terms of time window and channel dimension, and fuse them to obtain the corrected prediction.

[0205] Output dataset[fused].

[0206] 8) Inclusive branching (conditional routing):

[0207] If the expected waiting time exceeds the target SLA or the channel occupancy rate exceeds 85%, an alarm task will be initiated.

[0208] 9) Batch processing tasks (alarm tasks):

[0209] Production alerts are sent to the message queue.

[0210] 10) Batch processing tasks (persistent result sets)

[0211] The dataset [fused] is persisted to the security_lane_forecast table for use by other systems such as dashboards and BI.

[0212] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments and to explain the operating principles of the embodiments in conjunction with the relevant descriptions in the specification. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention. Components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0213] Furthermore, the directions such as front, back, left, and right mentioned in this embodiment are only for reference and do not represent the actual directions in use. In addition, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0214] The above description is only a preferred embodiment of the present invention and is not intended to limit the design of this case. All equivalent changes made based on the key design features of this case shall fall within the protection scope of this case.

Claims

1. A process engine based multi-source heterogeneous data task orchestration method, characterized in that, Comprise the following flows: Data source access and adaptation: access multi-source heterogeneous data through a pluggable data source adaptation component, and convert the data into a unified internal data model; Process modeling and task orchestration: build a process based on a BPMN model, the process model includes a start event node, a batch processing task node, a stream processing task node, a gateway node, and an end event node, wherein the start event node is configured as a timer event or a stream event to trigger subsequent batch processing tasks or stream processing tasks; in the processing task node, the output of the data is configured to: directly output the batch data to the next node or to a user-configured target data source; or first use a Groovy script to process the data according to the agreed interface, and then output to the next node or the target data source; The gateway node includes a parallel gateway and an inclusive gateway, which are used to distribute the task execution path in parallel, synchronize aggregation or conditional routing; The main role of the gateway node is to logically control the execution results of batch processing events and stream processing events, and determine the direction of the subsequent process, which specifically includes: Parallel branch gateway, accessing one task, outputting to multiple parallel batch processing tasks and / or stream processing tasks; Parallel merge gateway, accessing multiple parallel batch processing tasks and / or stream processing tasks, aggregating all task results and outputting to the next task; Inclusive branch gateway, accessing one task, splitting the output of the task into multiple branches, selecting branches according to conditions, and not all executing; Inclusive merge gateway, accessing multiple tasks, aggregating the results of all tasks that actually execute the inclusive branch, and then outputting to the next task; Link tracking: a unique traceId is generated at the beginning of each process, and the traceId is transmitted along the nodes in the entire process, realizing full-link tracking and monitoring; When the start event is a timer event, a traceId is generated every time the timer is triggered, identifying a batch processing job instance; When the start event is a stream event, the arrival of each message or batch of messages triggers the process and generates a traceId, identifying the processing instance for the message or message batch; The traceId supports data provenance analysis: the system can accurately draw the complete flow and transformation graph of data in the process by recording traceId, data operations and data objects, supporting upstream tracing and downstream impact analysis; Task execution and flow: the process engine schedules task nodes and gateway nodes for execution according to the process model, and transmits data between nodes through the system internal message mechanism; Result output and service: after the process ends, the task processing results output to the specified target data source can be published as a REST API service through configuration.

2. The process engine based multi-source heterogeneous data task orchestration method of claim 1, wherein: The data source adaptation component supports JDBC databases and MQ message queues. 3.The process engine based multi-source heterogeneous data task orchestration method of claim 1, wherein: The batch processing task node supports JDBC data source type, MQ-PULL data source type and batch processing plug-in type: JDBC data source type: configure JDBC data source and SQL statement; MQ data source type: configure MQ data source supporting PULL mode, topic, queue and batch size parameters; Batch plugin type: users develop and upload Java plugins based on the SDK provided by the system to configure batch plugin task nodes.

4. The multi-source heterogeneous data task orchestration method based on a process engine according to claim 3, characterized in that: If the output target is a JDBC data source, the JDBC data source information needs to be configured, and the insert statement needs to be written to support The O{json path} expression fills the JSON property value. If the output target is an MQ data source, configure the MQ data source information and the topic or queue name.

5. The process engine based multi-source heterogeneous data task orchestration method of claim 1, wherein: The stream processing task node supports MQ-PUSH data source type and stream processing plugin type: MQ-PUSH data source type: configure MQ data source, topic and queue parameters; Stream processing plugin type: users develop Java plugins based on the SDK provided by the system.

6. The multi-source heterogeneous data task orchestration method based on a process engine according to claim 1, characterized in that: The traceId supports running log and alarm analysis.

7. The multi-source heterogeneous data task orchestration method based on a process engine according to claim 1, characterized in that: After the process ends, automatically generate a running report and a state event for user query.

8. A process engine based multi-source heterogeneous data task orchestration system for performing the process engine based multi-source heterogeneous data task orchestration method according to any one of claims 1 to 7, characterized in that, Comprise: A data access layer for adapting multi-source heterogeneous data sources and uniformly converting data into an internal data model; A process orchestration and control layer for receiving a process model configured by a user and scheduling and controlling task nodes and gateway nodes in the process; A task processing and computing layer for executing batch tasks and stream tasks scheduled by the process orchestration and control layer; A data flow and monitoring layer for transmitting data between task nodes and providing full-link tracking and monitoring functions and generating a running report; A result service layer for publishing the output of task processing results as a service interface through configuration.

9. The multi-source heterogeneous data task orchestration system based on a process engine according to claim 8, characterized in that: The data access layer comprises: A data access module responsible for adapting and abstracting multi-source heterogeneous data sources, uniformly converting them into an internal data model, and shielding differences in the underlying layer; An plugin / extension interface providing an SDK interface for users to extend and customize data sources or processing logic; The process orchestration and control layer comprises: A process engine module based on the BPMN process modeling idea to realize the orchestration of task nodes, branch gateways and condition control; A scheduling and execution control submodule supporting batch processing, stream processing, concurrent execution and retry fault tolerance; The task processing and computing layer comprises: A task executor module including a batch task executor and a stream task executor; A dynamic script and expression module supporting expression and script parsing; The data flow and monitoring layer comprises: A message middleware module responsible for data transmission between nodes, result caching and transparent transmission of traceId; A monitoring and governance module providing traceId full-link tracking, running log collection, alarm detection and data bloodline analysis; The result service layer comprises: A service configuration module supporting the automatic publishing of data output to a target data source as a REST API service through configuration; A report and notification submodule for automatically generating a running report and a task completion notification.

Images