CHUNJUN data synchronization workflow scheduling method and system based on directed acyclic graph

Abstract

The invention discloses a CHUNJUN data synchronization workflow scheduling method and system based on a directed acyclic graph, and the method comprises the following steps: (1) packaging a CHUNJUN data synchronization task into standardized components, and exposing an input interface, an output interface and a configuration interface of each component; (2) receiving user operation through a visual interface to establish a component connection relationship, and executing connection normalization check; (3) constructing a directed acyclic graph (DAG) based on the connection relationship, and generating a task execution sequence; (4) triggering a CHUNJUN data synchronization task according to the execution sequence; according to the scheme, the construction time of a single task is compressed to be within 2 hours from 5-6 hours of a traditional command line mode, and the configuration error rate of a novice user is reduced by 60% through visual operation; and meanwhile, a DAG scheduling mechanism guarantees the reliability of multi-task sequential execution, the data synchronization failure rate is reduced by 60%, and the data intermediate station construction efficiency is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of electronic information, and in particular relates to a CHUNJUN data synchronization workflow scheduling method and system based on a directed acyclic graph. Background Art

[0002] Currently, the CHUNJUN tool is widely used in the distributed data processing field to synchronize heterogeneous data sources. However, its task configuration requires the use of command line scripts with specific syntax, which leads to three major technical bottlenecks for users: (1) The lack of visual operation capabilities makes it difficult for non-professionals to complete the configuration through the interface, and the error rate of novice scripts is as high as 40%; (2) Multi-task dependencies are difficult to define, and the order of task execution in complex data synchronization processes is out of control, affecting data integrity and accuracy; (3) Task status query and log debugging rely entirely on command line operations, and a single debugging session takes an average of 15-20 minutes. The above defects result in task construction and testing time as long as 5-6 hours, high professional and technical barriers, and low debugging efficiency, which seriously restricts the large-scale application of CHUNJUN technology in the construction of data middle platforms. Summary of the Invention

[0003] (1) Purpose of the invention

[0004] In order to overcome the above shortcomings, the purpose of the present invention is to provide a CHUNJUN data synchronization workflow scheduling method and system based on a directed acyclic graph, so as to solve the technical problems that when the CHUNJUN tool is used for heterogeneous data source synchronization, task configuration relies on command line scripting, and there are three major technical bottlenecks: lack of visual operation, difficulty in defining multi-task dependencies, and task status query and log debugging rely on command lines. These problems result in long build and test time, high technical barriers, and low debugging efficiency, which restrict its large-scale application.

[0005] (2) Technical solution

[0006] To achieve the above objectives, the present application provides the following technical solutions:

[0007] A directed acyclic graph-based CHUNJUN data synchronization workflow scheduling method includes the following steps:

[0008] (1) Encapsulate the CHUNJUN data synchronization task into standardized components, each of which exposes input interfaces, output interfaces, and configuration interfaces;

[0009] (2) Receive user operations through the visual interface to establish component connection relationships and perform connection compliance checks;

[0010] (3) Construct a directed acyclic graph (DAG) based on the connection relationship to generate the task execution sequence;

[0011] (4) Trigger the CHUNJUN data synchronization task according to the execution sequence;

[0012] This solution reduces the single-task construction time from 5-6 hours in traditional command line mode to less than 2 hours. The visual operation reduces the configuration error rate of novice users by 60%. At the same time, the DAG scheduling mechanism ensures the reliability of multi-task sequential execution, reduces the data synchronization failure rate by 60%, and significantly improves the efficiency of data center construction.

[0013] In some embodiments, step (3) includes the following sub-steps:

[0014] (3.1) Task node definition and configuration: Abstract the CHUNJUN data synchronization task into a DAG node; (3.2) Task dependency definition: Establish inter-task dependencies through visual drag and drop, generating directed edges;

[0015] (3.3) Topological sorting and execution sequence determination: A topological sorting algorithm is used to generate a loop-free execution sequence; (3.4) Dynamic sequence adjustment: When a task fails or resources change, the topological sequence is recalculated and completed nodes are skipped;

[0016] The dynamic sequence adjustment function maintains a 90% task completion rate even when 20% of nodes fail, avoiding the 50% redundant overhead caused by full-process rescheduling; actual measurements show that workflow execution efficiency is improved by 30% in resource fluctuation scenarios.

[0017] In some embodiments, step (3.2) includes a checking mechanism to prevent loop generation, blocking the connection when a circular dependency is detected;

[0018] The loop detection mechanism completely eliminates the risk of workflow deadlock, fundamentally ensuring the executability of complex synchronization processes. It is particularly suitable for cross-system data pipeline construction scenarios.

[0019] In some embodiments, step (3.3) uses the Kahn algorithm combined with task priority weights to perform topological sorting.

[0020] Priority-weighted scheduling reduces the waiting time of real-time tasks by 45%, ensuring that 99% of high-priority tasks are completed on time in scenarios such as bank statement synchronization.

[0021] In some embodiments, in step (2):

[0022] Connection compliance checks include input type matching, output type compatibility, input quantity thresholds, and output quantity limits.

[0023] Connection specification checking intercepts 90% of syntax errors during the configuration phase.

[0024] In some embodiments, the connection normalization check also includes a check whether the table name exists.

[0025] Table name verification eliminates data loss accidents caused by missing tables, shortens debugging time by 80%, and increases the configuration accuracy of novice users to over 95%.

[0026] In some embodiments, it also includes: (5) starting an intelligent retry mechanism for the task node and its sub-dependency chain that failed to execute.

[0027] The subgraph-level retry mechanism reduces retry resource consumption by 70%, and the grading strategy based on error type increases the retry success rate to 85%. In the data synchronization scenario of IoT devices in manufacturing enterprises, the time required for fault-tolerant processing is reduced by 50%.

[0028] In some embodiments, the intelligent retry mechanism identifies task failures through heartbeat detection and timeout thresholds.

[0029] In some embodiments, it also includes: (6) receiving the timing strategy, retry strategy and notification strategy set by the user, and automatically triggering the workflow execution according to the strategy.

[0030] Automated scheduling reduces manual intervention in nighttime synchronization tasks to zero, with a task completion rate of 99.5%; real-time notifications shorten fault response time from hours to minutes, meeting the stringent timeliness requirements of the financial industry.

[0031] Another aspect of the present invention provides a CHUNJUN data synchronization workflow scheduling system based on a directed acyclic graph, characterized by comprising:

[0032] A component packaging module for implementing the function of step (1) of claim 1;

[0033] A visual construction module for implementing the function of step (2) of claim 1;

[0034] A DAG generation module, configured to implement the function of step (3) of claim 1;

[0035] A scheduling execution engine is used to implement the function of step (4) of claim 1. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the CHUNJUN data synchronization workflow scheduling method based on directed acyclic graph of the present invention;

[0037] Figure 2 It is a flow chart of generating a directed acyclic workflow graph in the CHUNJUN data synchronization workflow scheduling method based on a directed acyclic graph of the present invention. DETAILED DESCRIPTION

[0038] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present invention.

[0039] The present invention provides a directed acyclic graph-based CHUNJUN data synchronization workflow scheduling method, comprising the following steps:

[0040] (1) Encapsulate the CHUNJUN data synchronization task into standardized components, each of which exposes input interfaces, output interfaces, and configuration interfaces;

[0041] (2) Receive user operations through the visual interface to establish component connection relationships and perform connection compliance checks;

[0042] (3) Construct a directed acyclic graph (DAG) based on the connection relationship to generate the task execution sequence;

[0043] (4) Trigger the CHUNJUN data synchronization task according to the execution sequence;

[0044] Specifically, this application encapsulates CHUNJUN data synchronization tasks (such as data migration from MySQL to Kafka) as standardized components. Each component exposes three types of interfaces through the API: the input interface uses the Socket protocol to receive upstream data, the output interface passes it to the downstream through RPC calls, and the configuration interface supports parameter settings in JSON format (such as field mapping rules, synchronization frequency). Furthermore, users drag and drop components through the WebSVG rendered visual interface to establish a connection, and the system performs connection compliance checks in real time. On this basis, a directed acyclic graph (DAG) is constructed based on the connection relationship, and a topological sorting algorithm is used to generate a task execution sequence. Finally, the scheduler starts from the node with an in-degree of zero (such as a data source acquisition node without a pre-dependency) and triggers the execution of CHUNJUN tasks in sequence.

[0045] Specifically, step (3) includes the following sub-steps:

[0046] First, the CHUNJUN tasks are abstracted into DAG nodes and bound to attributes (such as data source type and field mapping rules). Furthermore, users drag nodes to generate directed edges to define dependencies. Subsequently, a topological sorting algorithm is used to generate a loop-free execution sequence. It is worth noting that when a task fails or cluster resources change (such as adding a new compute node), the scheduler recalculates the topological sequence and automatically skips completed nodes. Preferably, the dynamic adjustment process only affects unexecuted nodes to avoid redundant scheduling.

[0047] Specifically, step (3.2) includes a validation mechanism to prevent loops. When visually dragging and dropping dependencies, the system triggers a loop detection mechanism in real time. This mechanism dynamically verifies paths based on a depth-first search (DFS) algorithm. If a circular dependency is detected (e.g., node A→B→A), the connection is immediately blocked and an error code E201 (e.g., "Dependency conflict: closed loop path") is returned via a pop-up window. This completely eliminates the risk of workflow deadlocks.

[0048] Specifically, in step (3.3), the topological sorting uses the Kahn algorithm and assigns priority weights to tasks: real-time tasks (such as transaction flow synchronization) are weighted 10, and offline tasks (such as log analysis) are weighted 5. When the in-degrees of multiple nodes simultaneously reach zero, the higher-weighted tasks are prioritized. Alternatively, a dynamic weighting strategy based on deadlines can be employed.

[0049] Specifically, in step (2), the real-time verifier is triggered during the connection operation to verify the following:

[0050] 1) Input type matching (e.g., MySQL output is only connected to Kafka input);

[0051] 2) Output compatibility (CSV format output cannot be connected to Parquet input);

[0052] 3) Input quantity ≤ 3;

[0053] 4) Output branches ≤ 5.

[0054] In particular, a table name verification module is added, and if the target table does not exist, an error code E101 (such as "target table not created") is returned. Preferably, a repair solution is provided that automatically prompts to execute DDL statements when a table name conflict occurs.

[0055] Preferably, the present application also adds an exception handling module: it identifies failed tasks through heartbeat detection (30-second polling) and a timeout threshold (300 seconds). When a task fails, the retry controller responds according to the error type: network timeout immediately retry, data verification failure delay retry for 5 minutes. Furthermore, only the failed node and its downstream nodes are restarted (for example, if a cleaning node fails, only the node and subsequent conversion nodes are restarted).

[0056] Preferably, this application adds a timing scheduling module to receive user configuration:

[0057] 1) Cron expressions (e.g. "02***" means execution every morning);

[0058] 2) Failure retry strategy (maximum 3 times);

[0059] 3) Enterprise WeChat notification API.

[0060] When a task fails, the notification adapter automatically pushes an alarm message containing the task ID and log link. Preferably, the retry engine creates an independent queue to execute subgraph retries.

[0061] On the other hand, the present application provides a CHUNJUN data synchronization workflow scheduling system based on a directed acyclic graph, which adopts a microservice architecture:

[0062] Component packaging module: This module runs in a Docker container (a single container is equipped with a 4-core CPU and 8GB of memory) and packages Chunjun tasks into standardized components.

[0063] Visual construction module: Renders the drag-and-drop interface based on Vue.js and receives user connection operations;

[0064] DAG generation module: extends the Apache Airflow engine to generate topology sequences;

[0065] Scheduling execution engine: deployed in a Kubernetes cluster, triggers tasks in sequence and supports automatic scaling.

[0066] Furthermore, each module communicates through the gRPC protocol, and the fault-tolerant processing module is independently deployed in a high-availability cluster.

[0067] It should be understood that the above-described specific embodiments of the present invention are merely illustrative or illustrative of the principles of the present invention and do not constitute limitations of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included within the scope of protection of the present invention. In addition, the appended claims are intended to cover all variations and modifications that fall within the scope and metes and bounds of the appended claims, or equivalents thereof.

Claims

1. A method for scheduling a CHUNJUN data synchronization workflow based on a directed acyclic graph, characterized in that: The following steps are involved: (1) Encapsulate the CHUNJUN data synchronization task into standardized components, each of which exposes input interfaces, output interfaces, and configuration interfaces; (2) Receive user operations through the visual interface to establish component connection relationships and perform connection compliance checks; (3) Construct a directed acyclic graph (DAG) based on the connection relationship to generate the task execution sequence; (4) Trigger the CHUNJUN data synchronization task according to the execution sequence.

2. The method according to claim 1, characterized in that Step (3) includes the following sub-steps: (3.1) Task node definition and configuration: abstract the CHUNJUN data synchronization task into a DAG node; (3.2) Task dependency definition: Dependencies between tasks are established by visually dragging and dropping, generating directed edges; (3.3) Topological sorting and execution sequence determination: A topological sorting algorithm is used to generate a loop-free execution sequence; (3.4) Dynamic sequence adjustment: When a task fails or resources change, the topology sequence is recalculated and completed nodes are skipped.

3. The method according to claim 2, wherein: Step (3.2) includes a checking mechanism to prevent loop generation, blocking the connection when a circular dependency is detected.

4. The method according to claim 2, wherein: In step (3.3), the Kahn algorithm is used in combination with the task priority weights for topological sorting.

5. The method according to claim 1, wherein In step (2): The connection specification check includes input type matching, output type compatibility, input quantity threshold and output quantity limit.

6. The method according to claim 5, characterized in that: In step (2): the connection standardization check also includes checking whether the table name exists.

7. The method according to claim 1, characterized in that Also includes: (5) Start an intelligent retry mechanism for the failed task node and its sub-dependency chain.

8. The method according to claim 7, wherein: The intelligent retry mechanism identifies task failures through heartbeat detection and timeout thresholds.

9. The method according to claim 1, characterized in that Also includes: (6) Receive the timing strategy, retry strategy and notification strategy set by the user, and automatically trigger the workflow execution according to the strategy.

10. A CHUNJUN data synchronization workflow scheduling system based on directed acyclic graph, characterized in that: include: A component packaging module for implementing the function of step (1) of claim 1; A visual construction module for implementing the function of step (2) of claim 1; A DAG generation module, configured to implement the function of step (3) of claim 1; A scheduling execution engine is used to implement the function of step (4) of claim 1.

Images