NiFi-based multi-task arrangement method and system

Abstract

The invention provides a multi-task arrangement method and system based on NiFi. According to the method, firstly, a data synchronization task is configured through a NiFi native interface, a NiFi system can generate a unique identifier for each task to serve as a mapping key of a scheduling center and the NiFi task, and the uniqueness of the tasks in cross-system interaction is ensured; a NiFi data interface is called through a communication protocol, asynchronous data synchronization is achieved, an available task list is pulled periodically, and data intercommunication between a dispatching center and a NiFi system is achieved. And dragging the task nodes from the available task list to the canvas, and defining a dependency relationship by drawing directed edges. And the scheduling center converts the visually arranged process into a task sequence executable by a computer by analyzing the task definition file, so as to ensure that the dependency relationship and the scheduling strategy fall to the ground. And the scheduling center sequentially calls the NiFi interfaces to trigger the tasks according to the task sequence in the execution plan, and converts the execution plan generated by the scheduling center into actual operation of the NiFi tasks, so that cross-system collaboration is realized, and the task scheduling efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multi-task orchestration method and system based on NiFi. Background Art

[0002] As a mature open source ETL tool in the big data field, NiFi is widely used in scenarios such as data governance and data synchronization. It configures data synchronization tasks through a visual interface and has the advantages of flexible process definition and rich data processing components. It can quickly realize the full process automation of data extraction, cleaning, conversion and loading, and has been widely used in the construction of data middle platforms in industries such as finance, e-commerce, and government affairs.

[0003] In the existing technology, the task scheduling mechanism of the NiFi native framework is based on an independent operation mode. Each task only supports independent triggering through a preset period and lacks the ability to configure dependencies between tasks. This requires developers to implement task concatenation through hard coding.

[0004] However, in complex data synchronization scenarios, there are often multi-task dependent scheduling requirements. Existing technologies implement task concatenation through hard coding, which increases the complexity of the system architecture and leads to low task scheduling efficiency. Summary of the Invention

[0005] An embodiment of the present invention provides a multi-task orchestration method based on NiFi, which can improve task scheduling efficiency. The method includes:

[0006] A1: Use the NiFi native interface to configure the data synchronization task, and use the NiFi system to create a unique identifier and a corresponding task definition file for each NiFi task. The unique identifier is the mapping key between the scheduling center and the NiFi system for each NiFi task, and the task definition file includes the task dependencies and logical rules.

[0007] A2: Based on a pre-agreed communication protocol, a connection is established with the NiFi system through the data interface module of the scheduling center, and a list of available tasks in the NiFi system is periodically collected, wherein the list of available tasks includes a task name, the unique identifier, and a task type;

[0008] A3: Based on the collected available task list, using the directed acyclic graph (DAG) interface of the scheduling center, choreographing the task execution process by dragging task nodes and drawing directed edges, wherein the task execution process is associated with the task definition file;

[0009] A4: When it is detected that the task scheduling trigger condition is met, the scheduling center analyzes the task dependencies and logic rules in the task definition file and generates a task execution plan;

[0010] A5: Based on the generated task execution plan, a control instruction is sent to the target NiFi task by calling the third-party interface of the NiFi system to control the execution of the target NiFi task.

[0011] Preferably,

[0012] The task dependencies include serial dependencies, parallel dependencies, and conditional branch dependencies between tasks;

[0013] The logic rules include data processing rules, data conversion logic and exception handling logic.

[0014] Preferably,

[0015] The task definition file further includes: a task priority attribute and a task scheduling time;

[0016] The A4 includes:

[0017] When it is detected that the task scheduling trigger condition is met, the scheduling center parses the task dependencies and logic rules in the task definition file;

[0018] Based on the task priority attributes preset in the task definition file, assign a corresponding priority to each NiFi task;

[0019] The task execution plan is generated based on the task definition file, the task priority attribute and the task scheduling time.

[0020] Preferably,

[0021] After A5, the method further includes:

[0022] Accessing NiFi logs through the scheduling center, wherein the NiFi logs include: real-time running status and failure records of the NiFi tasks;

[0023] When abnormal record data is detected in the NiFi log, an alarm message is generated.

[0024] Preferably,

[0025] After A5, the method further includes:

[0026] When the current NiFi node crashes, the scheduling center extracts the task information on the current NiFi node based on the pre-stored task configuration information;

[0027] Migrate unfinished tasks to the normally running target NiFi node and create a temporary task instance based on the task configuration information;

[0028] When the unfinished task is completed on the target NiFi node, the created temporary task instance is deleted through the scheduling center.

[0029] In a second aspect, an embodiment of the present invention provides a multi-task orchestration system based on NiFi, the system comprising:

[0030] Creation module: used to configure data synchronization tasks using the NiFi native interface, and create a unique identifier and a corresponding task definition file for each NiFi task through the NiFi system, wherein the unique identifier is the mapping key between the scheduling center and the NiFi system for each NiFi task, and the task definition file includes the task dependencies and logical rules;

[0031] Acquisition module: used to establish a connection with the NiFi system through the data interface module of the scheduling center based on a pre-agreed communication protocol, and periodically collect the available task list in the NiFi system, wherein the available task list includes the task name, the unique identifier created by the creation module, and the task type;

[0032] An orchestration module: based on the available task list collected by the collection module, using the directed acyclic graph (DAG) interface of the scheduling center, orchestrates the task execution process by dragging task nodes and drawing directed edges, wherein the task execution process is associated with the task definition file;

[0033] Processing module: When it is detected that the task scheduling trigger condition is met, the scheduling center parses the task dependencies and logical rules in the task definition file associated with the task execution process of the orchestration module to generate a task execution plan;

[0034] Execution module: Based on the task execution plan generated by the processing module, the execution module sends a control instruction to the target NiFi task by calling the third-party interface of the NiFi system to control the execution of the target NiFi task.

[0035] Preferably,

[0036] The processing module is further configured to execute:

[0037] When it is detected that the task scheduling trigger condition is met, the scheduling center parses the task dependencies and logic rules in the task definition file;

[0038] Based on the task priority attributes preset in the task definition file, assign a corresponding priority to each NiFi task;

[0039] The task execution plan is generated based on the task definition file, the task priority attribute and the task scheduling time.

[0040] Preferably,

[0041] After the execution module, it further includes: a log management module;

[0042] The log management module is configured to access the NiFi log through the dispatch center, wherein the NiFi log includes the real-time running status and failure records of the NiFi task, and generate an alarm message when abnormal record data is detected in the NiFi log.

[0043] Preferably,

[0044] After the execution module, it further includes: an exception management module;

[0045] The exception management module is used to perform:

[0046] When the current NiFi node crashes, the scheduling center extracts the task information on the current NiFi node based on the pre-stored task configuration information;

[0047] Migrate unfinished tasks to the normally running target NiFi node and create a temporary task instance based on the task configuration information;

[0048] When the unfinished task is completed on the target NiFi node, the created temporary task instance is deleted through the scheduling center.

[0049] In a third aspect, an embodiment of the present invention provides a multi-task orchestration system based on NiFi, comprising: at least one memory and at least one processor;

[0050] The at least one memory is configured to store a machine-readable program;

[0051] The at least one processor is configured to call the machine-readable program to execute any one of the methods described in the first aspect.

[0052] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the processor is caused to execute any one of the methods described in the first aspect.

[0053] Embodiments of the present invention provide a multi-task orchestration method and system based on NiFi. The method first configures data synchronization tasks, including data collection, cleaning, and conversion, through the native NiFi interface. Task logic is defined using NiFi's built-in processor. The NiFi system automatically generates a globally unique identifier for each task, which serves as a mapping key between the dispatch center and the NiFi task, ensuring the uniqueness of the task in cross-system interactions. A NiFi data interface is called via a communication protocol to achieve asynchronous data synchronization. A list of available tasks is periodically retrieved to enable data interoperability between the dispatch center and the NiFi system. Task nodes are then dragged from the available task list onto a canvas. Dependencies are defined by drawing directed edges. Each canvas node is bound to a NiFi task using a unique identifier, enabling visual definition of inter-task dependencies and execution logic, forming an executable process topology. The dispatch center parses the task definition file to convert the visually orchestrated process into a computer-executable task sequence, ensuring the implementation of dependencies and scheduling policies. The dispatch center sequentially calls NiFi interfaces to trigger tasks based on the task sequence in the execution plan, converting the execution plan generated by the dispatch center into the actual operation of NiFi tasks, achieving cross-system collaboration. Through the above steps, the scheduling center implements full lifecycle management from task definition, process orchestration, plan generation to execution control, filling the gap in NiFi's native lack of multi-task dependency scheduling capabilities. Compared with existing technologies, it can effectively improve task scheduling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 This is a flowchart of a multi-task orchestration method based on NiFi provided by one embodiment of the present invention;

[0056] Figure 2 This is a flowchart of another multi-task orchestration method based on NiFi provided by one embodiment of the present invention;

[0057] Figure 3 Schematic diagram of a multi-task orchestration system based on NiFi provided by one embodiment of the present invention;

[0058] Figure 4 Schematic diagram of another multi-task orchestration system based on NiFi provided by one embodiment of the present invention;

[0059] Figure 5This is a schematic diagram of another NiFi-based multi-task orchestration system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0061] like Figure 1 As shown, an embodiment of the present invention provides a multi-task orchestration method based on NiFi, which may include the following steps:

[0062] Step 101: Use the NiFi native interface to configure the data synchronization task, and create a unique identifier and a corresponding task definition file for each NiFi task through the NiFi system. The unique identifier is the mapping key between the scheduling center and the NiFi system for each NiFi task, and the task definition file includes the task dependencies and logical rules.

[0063] Step 102: Based on the pre-agreed communication protocol, a connection is established with the NiFi system through the data interface module of the scheduling center, and a list of available tasks in the NiFi system is periodically collected, wherein the list of available tasks includes a task name, a unique identifier, and a task type;

[0064] Step 103: Based on the collected available task list, the task execution process is arranged by dragging task nodes and drawing directed edges using the directed acyclic graph (DAG) interface of the scheduling center. The task execution process is associated with the task definition file.

[0065] Step 104: When it is detected that the task scheduling trigger condition is met, the scheduling center parses the task dependencies and logic rules in the task definition file and generates a task execution plan;

[0066] Step 105: Based on the generated task execution plan, a control instruction is sent to the target NiFi task by calling the third-party interface of the NiFi system to control the execution of the target NiFi task.

[0067] In an embodiment of the present invention, a multi-task orchestration method based on NiFi is provided. The method first configures data synchronization tasks through the NiFi native interface, including operations such as data collection, cleaning, and conversion. The task logic is defined using NiFi's built-in processor. The NiFi system can automatically generate a globally unique identifier for each task as a mapping key between the scheduling center and the NiFi task, ensuring the uniqueness of the task in cross-system interaction. The NiFi data interface is called through the communication protocol to achieve asynchronous data synchronization, and the available task list is periodically pulled to achieve data interoperability between the scheduling center and the NiFi system. Task nodes are then dragged from the available task list to the canvas, and dependencies are defined by drawing directed edges. Each canvas node is bound to the NiFi task through a unique identifier, enabling the visual definition of dependencies and execution logic between tasks, forming an executable process topology. The scheduling center converts the visually orchestrated process into a computer-executable task sequence by parsing the task definition file, ensuring the implementation of dependencies and scheduling strategies. The scheduling center calls the NiFi interface to trigger tasks in sequence according to the task sequence in the execution plan, converting the execution plan generated by the scheduling center into the actual operation of the NiFi task, achieving cross-system collaboration. Through the above steps, the scheduling center implements full lifecycle management from task definition, process orchestration, plan generation to execution control, filling the gap in NiFi's native lack of multi-task dependency scheduling capabilities. Compared with existing technologies, it can effectively improve task scheduling efficiency.

[0068] In order to solve the capability deficiency of native NiFi in multi-task dependency scenarios, in one embodiment of the present invention, the dependency relationship of the tasks in step 101 in the above embodiment includes serial dependency, parallel dependency and conditional branch dependency between tasks; the logical rules include data processing rules, data conversion logic and exception handling logic.

[0069] In this embodiment of the present invention, dependencies are used to clarify the order of task execution and branching logic, and logical rules are used to ensure the accuracy and robustness of data processing. The combination of the two upgrades NiFi from "single-task independent operation" to "complex process collaborative scheduling", meeting the core requirements of process flexibility and reliability in data synchronization scenarios. Through the visual orchestration and automated scheduling of the scheduling center, NiFi's task management is upgraded from "discrete mode" to "systematic process management", effectively solving the capacity deficiencies of native NiFi in multi-task dependency scenarios.

[0070] In order to improve the efficiency of task execution, in one embodiment of the present invention, the task definition file in step 101 in the above embodiment further includes: a task priority attribute and a task scheduling time;

[0071] The step 104 includes:

[0072] When it is detected that the task scheduling trigger condition is met, the scheduling center parses the task dependencies and logic rules in the task definition file;

[0073] Based on the task priority attributes preset in the task definition file, assign a corresponding priority to each NiFi task;

[0074] The task execution plan is generated based on the task definition file, the task priority attribute and the task scheduling time.

[0075] In this embodiment of the present invention, the task definition file contains serial, parallel, and conditional branch dependencies between tasks. Serial dependencies mean that tasks must be executed sequentially in a specific order. Parallel dependencies allow multiple tasks to be started simultaneously, improving processing efficiency. Conditional branch dependencies select subsequent task branches based on the execution results or data characteristics of previous tasks. The dispatch center analyzes these dependencies to construct a topological structure for task execution, clarifying the order and logical connections between tasks. Data processing rules are used to cleanse and filter data to ensure data quality. Data conversion logic converts data from one format to another to adapt to the needs of different systems. Exception handling logic develops response strategies for abnormal situations that may arise during task execution. The dispatch center analyzes these logical rules to determine the specific execution logic and handling methods for each task. Task definition files contain preset task priority attributes, which reflect the importance and urgency of each task within the overall task system. The dispatch center assigns a corresponding priority to each NiFi task based on these attributes. When resources are limited, high-priority tasks receive priority access to system resources, ensuring the normal operation of critical business operations. The dependencies and logical rules in the task definition file are the basis for generating execution plans. Based on this information, the scheduling center determines the execution order and specific operations of tasks to ensure that tasks can be executed according to the preset logic and process. When generating an execution plan, the scheduling center will prioritize high-priority tasks to ensure that these tasks are processed in a timely manner. When resources are limited, low-priority tasks may be delayed or suspended to ensure the smooth progress of high-priority tasks. The task scheduling time specifies the time point or time interval when the task begins execution. The scheduling center will combine the task scheduling time to reasonably arrange tasks on the timeline to achieve orderly execution of tasks. By integrating task definition files, task priority attributes and task scheduling time, the scheduling center can generate scientific and reasonable task execution plans, improve task execution efficiency, and ensure the stability and reliability of the entire data synchronization process.

[0076] In order to improve the reliability and stability of the entire data synchronization process, in one embodiment of the present invention, after step 105 in the above embodiment, the following steps are further included:

[0077] Accessing NiFi logs through the scheduling center, wherein the NiFi logs include: real-time running status and failure records of the NiFi tasks;

[0078] When abnormal record data is detected in the NiFi log, an alarm message is generated.

[0079] In an embodiment of the present invention, the NiFi log contains the real-time running status and failure records of NiFi tasks. This information is crucial for monitoring task execution and promptly identifying and resolving problems. By accessing the NiFi log, the dispatch center can centrally monitor all NiFi tasks. Once abnormal record data is detected, an alert message will be generated. Abnormal record data typically includes situations such as task execution failure, task execution time being too long, and data processing results not meeting expectations. The dispatch center will determine whether the data in the log is abnormal based on preset rules. When abnormal record data is detected, the dispatch center will immediately generate an alert message. The alert message will include detailed information about the abnormality, such as the time the abnormality occurred, the name of the task involved, and possible causes, so that operations and maintenance personnel can quickly understand the severity and general situation of the problem and take appropriate measures to address it. By accessing the NiFi log and monitoring and alerting abnormal record data, the dispatch center can promptly identify problems during NiFi task execution, reduce the impact of failures on business operations, and improve the reliability and stability of the entire data synchronization process. Furthermore, the detailed log records and alert information also provide a powerful basis for subsequent problem investigation and system optimization.

[0080] In order to improve system stability, in one embodiment of the present invention, after step 105 in the above embodiment, the following steps are further included:

[0081] When the current NiFi node crashes, the scheduling center extracts the task information on the current NiFi node based on the pre-stored task configuration information;

[0082] Migrate unfinished tasks to the normally running target NiFi node and create a temporary task instance based on the task configuration information;

[0083] When the unfinished task is completed on the target NiFi node, the created temporary task instance is deleted through the scheduling center.

[0084] In an embodiment of the present invention, in a NiFi-based data synchronization and task orchestration system, ensuring high system availability and task execution continuity is crucial. When the dispatch center detects that the current NiFi node has crashed, this means the node is unable to provide services normally, and tasks running on it may be interrupted. At this point, the dispatch center quickly extracts the task information for the current NiFi node based on pre-stored task configuration information. This pre-stored task configuration information is saved during system initialization or task creation and contains detailed information about each task. By extracting this information, the dispatch center can fully understand the tasks on the crashed node and their specific circumstances, preparing for subsequent task migration and resumption. The dispatch center selects a suitable target node from the normally functioning NiFi nodes. After determining the target node, the dispatch center migrates unfinished tasks to it. Based on the previously extracted task configuration information, a temporary task instance is created on the target node. This temporary task instance inherits all the configuration information of the original task, including the task's logical rules and dependencies. This process ensures that the task can continue to execute on the new node, and that the execution logic and results are consistent with the original task. When unfinished tasks are successfully executed on the target NiFi node, the dispatch center will perform cleanup operations and delete the previously created temporary task instances to release the node's system resources. Through the above steps, the system can quickly migrate and resume tasks in the face of abnormal situations such as NiFi node crashes, ensuring the continuity of data synchronization and processing processes, and improving system reliability and availability.

[0085] like Figure 2 As shown, in order to more clearly illustrate the technical solutions and advantages of the present invention, the following embodiment of the present invention provides a multi-task orchestration method based on NiFi for detailed description, which may specifically include the following steps:

[0086] Step 201: Use the NiFi native interface to configure the data synchronization task, and create a unique identifier and a corresponding task definition file for each NiFi task through the NiFi system, where the unique identifier is the mapping key between the scheduling center and the NiFi system for each NiFi task, and the task definition file includes the task's dependencies, logical rules, task priority attributes, and task scheduling time;

[0087] Specifically, task dependencies include serial dependencies, parallel dependencies, and conditional branch dependencies between tasks; logical rules include data processing rules, data conversion logic, and exception handling logic.

[0088] For example, the native interface of NiFi can be a Web UI, and the logical rules of the task can be defined by using NiFi built-in processors (such as FetchHTTP and UpdateDatabase). The unique identifier can be a UUID, the task definition file format can be JSON or XML, and the task scheduling time can be polling timing, corn timing manual, time polling, corn expression and other scheduling methods.

[0089] Step 202: Based on the pre-agreed communication protocol, a connection is established with the NiFi system through the data interface module of the scheduling center, and a list of available tasks in the NiFi system is periodically collected, wherein the list of available tasks includes a task name, a unique identifier, and a task type;

[0090] Step 203: Based on the collected available task list, the task execution process is arranged by dragging task nodes and drawing directed edges using the directed acyclic graph (DAG) interface of the scheduling center. The task execution process is associated with the task definition file.

[0091] Step 204: When it is detected that the task scheduling trigger condition is met, the scheduling center parses the task dependencies and logic rules in the task definition file;

[0092] Step 205: Assign a corresponding priority to each NiFi task based on the task priority attributes preset in the task definition file;

[0093] Step 206: Generate a task execution plan based on the task definition file, the task priority attribute, and the task scheduling time;

[0094] Step 207: Based on the generated task execution plan, a control instruction is sent to the target NiFi task by calling the third-party interface of the NiFi system to control the execution of the target NiFi task;

[0095] Step 208: Access the NiFi log through the scheduling center, where the NiFi log includes: the real-time running status and failure records of the NiFi task;

[0096] Step 209: When abnormal record data is detected in the NiFi log, an alarm message is generated;

[0097] Step 210: When the current NiFi node is detected to have crashed, the scheduling center extracts the task information on the current NiFi node based on the pre-stored task configuration information;

[0098] Step 211: Migrate the unfinished task to the normally running target NiFi node and create a temporary task instance based on the task configuration information;

[0099] Step 212: When the unfinished task is completed on the target NiFi node, the created temporary task instance is deleted through the scheduling center.

[0100] like Figure 3 As shown, an embodiment of the present invention provides a multi-task orchestration system based on NiFi, which includes:

[0101] Creation module 301: used to configure data synchronization tasks using the NiFi native interface, and create a unique identifier and a corresponding task definition file for each NiFi task through the NiFi system, wherein the unique identifier is a mapping key between the scheduling center and the NiFi system for each NiFi task, and the task definition file includes task dependencies and logical rules;

[0102] The acquisition module 302 is used to establish a connection with the NiFi system through the data interface module of the scheduling center based on a pre-agreed communication protocol, and periodically acquire a list of available tasks in the NiFi system, wherein the list of available tasks includes the task name, the unique identifier created by the creation module 301, and the task type;

[0103] The orchestration module 303 orchestrates the task execution process based on the available task list collected by the collection module 302 by using the directed acyclic graph (DAG) interface of the scheduling center by dragging task nodes and drawing directed edges, wherein the task execution process is associated with the task definition file;

[0104] Processing module 304: When it is detected that the task scheduling trigger condition is met, the scheduling center analyzes the task dependencies and logic rules in the task definition file associated with the task execution process of the orchestration module 303 and generates a task execution plan;

[0105] Execution module 305: Based on the task execution plan generated by the processing module 304, the execution module 305 sends a control instruction to the target NiFi task by calling the third-party interface of the NiFi system to control the execution of the target NiFi task.

[0106] like Figure 3 As shown, the processing module 304 is further configured to execute:

[0107] When it is detected that the task scheduling trigger condition is met, the scheduling center parses the task dependencies and logic rules in the task definition file;

[0108] Based on the task priority attributes preset in the task definition file, assign a corresponding priority to each NiFi task;

[0109] The task execution plan is generated based on the task definition file, the task priority attribute and the task scheduling time.

[0110] based on Figure 3 A multi-task orchestration system based on NiFi is shown in Figure 4 As shown, after the execution module 305, it further includes: a log management module 306;

[0111] The log management module 306 is configured to access the NiFi log through the dispatch center, wherein the NiFi log includes the real-time running status and failure records of the NiFi task, and generate an alarm message when abnormal record data is detected in the NiFi log.

[0112] based on Figure 4 A multi-task orchestration system based on NiFi is shown in Figure 5 As shown, after the log management module 306, it further includes: an exception management module 307;

[0113] The exception management module 307 is used to perform:

[0114] When the current NiFi node crashes, the scheduling center extracts the task information on the current NiFi node based on the pre-stored task configuration information;

[0115] Migrate unfinished tasks to the normally running target NiFi node and create a temporary task instance based on the task configuration information;

[0116] When the unfinished task is completed on the target NiFi node, the created temporary task instance is deleted through the scheduling center.

[0117] It is understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on a multi-task orchestration system based on NiFi. In other embodiments of the present invention, a multi-task orchestration system based on NiFi may include more or fewer components than shown in the figure, or combine or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0118] The information interaction, execution process, etc. between the units in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For specific contents, please refer to the description in the embodiment of the method of the present invention and will not be repeated here.

[0119] An embodiment of the present invention further provides a multi-task orchestration system based on NiFi, comprising: at least one memory and at least one processor;

[0120] at least one memory for storing a machine-readable program;

[0121] At least one processor is configured to call a machine-readable program to execute a multi-task orchestration method based on NiFi in any embodiment of the present invention.

[0122] An embodiment of the present invention further provides a computer-readable medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the processor executes a multi-task orchestration method based on NiFi in any embodiment of the present invention.

[0123] Specifically, a system or device equipped with a storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program codes stored in the storage medium.

[0124] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0125] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0126] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0127] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0128] Each embodiment of the present invention has at least the following beneficial effects:

[0129] 1. In an embodiment of the present invention, the method first configures the data synchronization task through the NiFi native interface, including operations such as data collection, cleaning, and conversion, and uses the NiFi built-in processor to define the task logic. The NiFi system can automatically generate a globally unique identifier for each task as a mapping key between the scheduling center and the NiFi task, ensuring the uniqueness of the task in cross-system interaction. The NiFi data interface is called through the communication protocol to achieve asynchronous data synchronization, and the available task list is periodically pulled to achieve data intercommunication between the scheduling center and the NiFi system. Then, the task node is dragged from the available task list to the canvas, and the dependency relationship is defined by drawing directed edges. Each canvas node is bound to the NiFi task through a unique identifier to achieve visual definition of the dependency relationship and execution logic between tasks, forming an executable process topology. The scheduling center converts the visually orchestrated process into a computer-executable task sequence by parsing the task definition file, ensuring that the dependency relationship and scheduling strategy are implemented. The scheduling center calls the NiFi interface to trigger the task in sequence according to the task order in the execution plan, and converts the execution plan generated by the scheduling center into the actual operation of the NiFi task, realizing cross-system collaboration. Through the above steps, the scheduling center realizes the full life cycle management from task definition, process arrangement, plan generation to execution control, filling the gap of NiFi's native lack of multi-task dependent scheduling capabilities. Compared with existing technologies, it can effectively improve task scheduling efficiency;

[0130] 2. In the embodiment of the present invention, the task execution order and branching logic are clarified through dependency relationships, and the accuracy and robustness of data processing are ensured through logical rules. The combination of the two upgrades NiFi from "single task independent operation" to "complex process collaborative scheduling", meeting the core requirements of process flexibility and reliability in data synchronization scenarios. Through the visual orchestration and automated scheduling of the scheduling center, NiFi's task management is upgraded from "discrete mode" to "systematic process management", effectively solving the capability deficiencies of native NiFi in multi-task dependency scenarios;

[0131] 3. In this embodiment of the present invention, the task definition file contains serial, parallel, and conditional branch dependencies between tasks. Serial dependencies mean that tasks must be executed sequentially in a specific order. Parallel dependencies allow multiple tasks to be started simultaneously, improving processing efficiency. Conditional branch dependencies select subsequent task branches based on the execution results or data characteristics of previous tasks. The dispatch center analyzes these dependencies to construct a topological structure for task execution, clarifying the order and logical connections between tasks. Data processing rules are used to cleanse and filter data to ensure data quality. Data conversion logic converts data from one format to another to adapt to the needs of different systems. Exception handling logic develops response strategies for abnormal situations that may arise during task execution. The dispatch center analyzes these logical rules to determine the specific execution logic and handling methods for each task. Task definition files contain preset task priority attributes, which reflect the importance and urgency of each task within the overall task system. The dispatch center assigns a corresponding priority to each NiFi task based on these attributes. When resources are limited, high-priority tasks receive priority access to system resources, ensuring the normal operation of critical business operations. The dependencies and logical rules in the task definition file form the basis for generating an execution plan. The scheduling center uses this information to determine the execution order and specific operations for tasks, ensuring that they are executed according to the pre-set logic and process. When generating an execution plan, the scheduling center prioritizes high-priority tasks to ensure they are processed promptly. When resources are limited, low-priority tasks may be delayed or suspended to ensure the smooth progress of high-priority tasks. The task scheduling time specifies the time point or time interval at which task execution begins. The scheduling center, based on the task scheduling time, rationally arranges tasks on the timeline to ensure orderly execution. By integrating the task definition file, task priority attributes, and task scheduling time, the scheduling center can generate a scientific and reasonable task execution plan, improve task execution efficiency, and ensure the stability and reliability of the entire data synchronization process.

[0132] It should be noted that not all steps and modules in the above processes and system structure diagrams are required, and certain steps or modules can be omitted according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or may be implemented by certain components in multiple independent devices.

[0133] In the above embodiments, the hardware unit can be realized by mechanical means or electrical means. For example, a hardware unit can include permanent dedicated circuits or logic (such as special processors, FPGA or ASIC) to complete the corresponding operations. The hardware unit can also include programmable logic or circuits (such as general-purpose processors or other programmable processors), which can be temporarily set up by software to complete the corresponding operations. Concrete implementation (mechanical means or dedicated permanent circuits or temporarily set circuits) can be determined based on the consideration on cost and time.

[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-task orchestration method based on NiFi, characterized in that: The method includes: A1: Use the NiFi native interface to configure the data synchronization task, and use the NiFi system to create a unique identifier and a corresponding task definition file for each NiFi task. The unique identifier is the mapping key between the scheduling center and the NiFi system for each NiFi task, and the task definition file includes the task dependencies and logical rules. A2: Based on a pre-agreed communication protocol, a connection is established with the NiFi system through the data interface module of the scheduling center, and a list of available tasks in the NiFi system is periodically collected, wherein the list of available tasks includes a task name, the unique identifier, and a task type; A3: Based on the collected available task list, using the directed acyclic graph (DAG) interface of the scheduling center, choreographing the task execution process by dragging task nodes and drawing directed edges, wherein the task execution process is associated with the task definition file; A4: When it is detected that the task scheduling trigger condition is met, the scheduling center analyzes the task dependencies and logic rules in the task definition file and generates a task execution plan; A5: Based on the generated task execution plan, a control instruction is sent to the target NiFi task by calling the third-party interface of the NiFi system to control the execution of the target NiFi task.

2. The method according to claim 1, characterized in that The task dependencies include serial dependencies, parallel dependencies, and conditional branch dependencies between tasks; The logic rules include data processing rules, data conversion logic and exception handling logic.

3. The method according to claim 1, characterized in that The task definition file further includes: a task priority attribute and a task scheduling time; The A4 includes: When it is detected that the task scheduling trigger condition is met, the scheduling center parses the task dependencies and logic rules in the task definition file; Based on the task priority attributes preset in the task definition file, assign a corresponding priority to each NiFi task; The task execution plan is generated based on the task definition file, the task priority attribute and the task scheduling time.

4. The method according to claims 1-3, characterized in that After A5, the method further includes: Accessing NiFi logs through the scheduling center, wherein the NiFi logs include: real-time running status and failure records of the NiFi tasks; When abnormal record data is detected in the NiFi log, an alarm message is generated; and / or, After A5, the method further includes: When the current NiFi node crashes, the scheduling center extracts the task information on the current NiFi node based on the pre-stored task configuration information; Migrate unfinished tasks to the normally running target NiFi node and create a temporary task instance based on the task configuration information; When the unfinished task is completed on the target NiFi node, the created temporary task instance is deleted through the scheduling center.

5. A multi-task orchestration system based on NiFi, characterized in that, The system includes: a creation module: used to configure data synchronization tasks using the NiFi native interface, and create a unique identifier and a corresponding task definition file for each NiFi task through the NiFi system, wherein the unique identifier is a mapping key between the scheduling center and the NiFi system for each NiFi task, and the task definition file includes task dependencies and logical rules; Acquisition module: used to establish a connection with the NiFi system through the data interface module of the scheduling center based on a pre-agreed communication protocol, and periodically collect the available task list in the NiFi system, wherein the available task list includes the task name, the unique identifier created by the creation module, and the task type; An orchestration module: based on the available task list collected by the collection module, using the directed acyclic graph (DAG) interface of the scheduling center, orchestrates the task execution process by dragging task nodes and drawing directed edges, wherein the task execution process is associated with the task definition file; Processing module: When it is detected that the task scheduling trigger condition is met, the scheduling center parses the task dependencies and logical rules in the task definition file associated with the task execution process of the orchestration module to generate a task execution plan; Execution module: Based on the task execution plan generated by the processing module, the execution module sends a control instruction to the target NiFi task by calling the third-party interface of the NiFi system to control the execution of the target NiFi task.

6. The system according to claim 5, characterized in that The processing module is further configured to execute: When it is detected that the task scheduling trigger condition is met, the scheduling center parses the task dependencies and logic rules in the task definition file; Based on the task priority attributes preset in the task definition file, assign a corresponding priority to each NiFi task; The task execution plan is generated based on the task definition file, the task priority attribute and the task scheduling time.

7. The system according to claim 5, characterized in that After the execution module, it further includes: a log management module; The log management module is configured to access the NiFi log through the dispatch center, wherein the NiFi log includes the real-time running status and failure records of the NiFi task, and generate an alarm message when abnormal record data is detected in the NiFi log.

8. The system according to claim 5, wherein: After the execution module, it further includes: an exception management module; The exception management module is used to perform: When the current NiFi node crashes, the scheduling center extracts the task information on the current NiFi node based on the pre-stored task configuration information; Migrate unfinished tasks to the normally running target NiFi node and create a temporary task instance based on the task configuration information; When the unfinished task is completed on the target NiFi node, the created temporary task instance is deleted through the scheduling center.

9. A multi-task orchestration system based on NiFi, characterized in that, include: at least one memory and at least one processor; The at least one memory is configured to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 4.

10. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 4.

Images