Multi-stage task dynamic allocation method and system based on DolphinScheduler

Abstract

The invention relates to a DolphinScheduler-based multi-stage task dynamic allocation method and system, and the method comprises the steps: constructing a state information table of each standard subtask, and dynamically determining the maximum number of standard subtasks processed by each service node, the state information comprising the starting and ending time of the standard subtasks and the ID of the service node; based on a directed acyclic graph of data processing, creating a first workflow for processing each standard subtask through DolphinScheduler; creating a second workflow, wherein the second workflow is used for obtaining the to-be-processed task and dynamically distributing the to-be-processed task to the adaptive node; and creating a third workflow which is used for distributing the task to a new service node when the task is overtime or failed.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud computing, and in particular relates to a multi-stage task dynamic allocation method and system based on DolphinScheduler. Background Art

[0002] With the rapid development of big data and cloud computing technologies, the amount of data that needs to be processed within enterprises is growing exponentially, and the types and complexity of data processing tasks are also increasing. This poses a severe challenge to traditional task scheduling methods, and there is an urgent need for a more efficient, flexible, and intelligent task scheduling system. Apache DolphinScheduler is an open source distributed workflow task scheduling system that assembles tasks into a scheduling workflow in the form of a DAG (directed acyclic graph) and provides a visual operation interface. It is convenient for users to schedule, monitor, and manage tasks.

[0003] There are two main ways to allocate tasks in DolphinScheduler:

[0004] 1. Static allocation: The tasks are pre-assigned to specific working nodes. This method is simple and easy, but it cannot be dynamically adjusted according to the real-time load of the node, which can easily cause resource waste or low task execution efficiency.

[0005] 2. Round-robin allocation: Tasks are assigned to each working node in turn. This method can balance the node load to a certain extent, but it cannot fully utilize the resource differences of the nodes. For example, some nodes may have stronger computing power or faster network bandwidth, and polling allocation cannot take advantage of these advantages.

[0006] The existing allocation method has the following problems:

[0007] 1. Low resource utilization: Both static allocation and round-robin allocation cannot fully utilize the resource differences of nodes, resulting in low resource utilization.

[0008] 2. Low task execution efficiency: When the node load is unbalanced, the task execution efficiency will be affected.

[0009] 3. Lack of flexibility: The existing allocation method is difficult to flexibly adjust according to actual conditions. For example, when a node fails, it is impossible to migrate the task to other nodes in time.

[0010] Unable to guarantee task dependencies: When executing a task that includes multiple steps, subsequent tasks require the completion of the predecessor tasks before they can be executed. This problem cannot be guaranteed at present. Summary of the invention

[0011] To solve the problem raised by the background technology, a first aspect of the present invention provides a multi-stage task dynamic allocation method based on DolphinScheduler, including: constructing a status information table for each standard subtask, and dynamically determining the maximum number of standard subtasks that each service node can process, wherein the status information includes the start and end time of the standard subtask and the ID of the service node; based on a directed acyclic graph of data processing, creating a first workflow for processing each standard subtask through DolphinScheduler; creating a second workflow, wherein the second workflow is used to obtain tasks to be processed and dynamically allocate the tasks to be processed to adapter nodes; creating a third workflow, wherein the third workflow is used to allocate tasks to new service nodes when a task times out or fails.

[0012] In some embodiments of the present invention, the acquisition of pending tasks and dynamic allocation of pending tasks to adapter nodes include: scanning pending tasks and splitting each pending task into multiple standard subtasks; acquiring the number of standard subtasks currently running on each service node according to the status information table of each standard subtask; and dynamically allocating the pending tasks to one or more service nodes with the least current tasks through DolphinScheduler based on the maximum number of standard subtasks processed by each service node.

[0013] Furthermore, it also includes: modifying the status information table of the service node assigned to the new standard subtask, and recording the start time and service node ID of each new standard subtask.

[0014] In some embodiments of the present invention, when a task times out or fails, allocating the task to a new service node includes: querying the start and end times of all currently running tasks, and calculating the execution duration of each task; comparing the execution duration of each task with the preset configuration time of the corresponding standard subtask, and based on the comparison result, stopping the execution of the corresponding standard subtask; and reallocating one or more marked subtasks that have stopped execution to other service nodes.

[0015] Furthermore, the execution time of each task is compared with the preset configuration time of the corresponding standard subtask, and based on the comparison result, the execution of the corresponding standard subtask is stopped, including: comparing the execution time of each task with the preset configuration time of the corresponding standard subtask: if the difference between the two exceeds a threshold, the corresponding standard subtask is stopped.

[0016] In the above embodiment, the directed acyclic graph based on data processing, creating a first workflow for processing each standard subtask through DolphinScheduler includes: creating a processing flow for processing each standard subtask through DolphinScheduler, and updating the status information of the corresponding node.

[0017] The second aspect of the present invention provides a multi-stage task dynamic allocation system based on DolphinScheduler, including: a construction module, used to construct a status information table for each standard subtask, and dynamically determine the maximum number of standard subtasks that each service node can process, the status information including the start and end time of the standard subtask and the ID of the service node; a first creation module, used to create a first workflow for processing each standard subtask through DolphinScheduler based on a directed acyclic graph of data processing; a second creation module, used to create a second workflow, the second workflow is used to obtain tasks to be processed and dynamically allocate tasks to be processed to adapter nodes; a third creation module, used to create a third workflow, the third workflow is used to allocate tasks to new service nodes when tasks time out or fail.

[0018] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the multi-stage task dynamic allocation method based on DolphinScheduler provided in the first aspect of the present invention.

[0019] A fourth aspect of the present invention provides a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the multi-stage task dynamic allocation method based on DolphinScheduler provided in the first aspect of the present invention is implemented.

[0020] The beneficial effects of the present invention are:

[0021] 1. Improve resource utilization: Dynamically allocate tasks according to the real-time load of the node, which can make full use of node resources and avoid resource waste.

[0022] 2. Improve task execution efficiency: Assigning tasks to the most suitable nodes for execution can shorten task execution time and improve task execution efficiency.

[0023] 3. Enhance the flexibility of the scheduling system: The allocation strategy can be flexibly adjusted according to actual conditions. For example, when a node fails, the task can be migrated to other nodes for execution, thereby improving the stability and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a basic flow chart of a multi-stage task dynamic allocation method based on DolphinScheduler in some embodiments of the present invention;

[0025] Figure 2Flowchart for configuring DolphinScheduler to scan for new tasks in some embodiments of the present invention

[0026] Figure 3 A flowchart of dynamically allocating tasks in some embodiments of the present invention;

[0027] Figure 4 is a flow chart of timeout task detection in some embodiments of the present invention;

[0028] Figure 5 It is a structural diagram of a multi-stage task dynamic allocation system based on DolphinScheduler in some embodiments of the present invention;

[0029] Figure 6 It is a schematic diagram of the structure of an electronic device in some embodiments of the present invention. DETAILED DESCRIPTION

[0030] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0031] refer to Figure 1 and Figure 2 In a first aspect of the present invention, a multi-stage task dynamic allocation method based on DolphinScheduler is provided, comprising:

[0032] S100. Construct a status information table for each standard subtask, and dynamically determine the maximum number of standard subtasks that each service node can process, wherein the status information includes the start and end time of the standard subtask and the ID of the service node; S200. Based on the directed acyclic graph of data processing, create a first workflow for processing each standard subtask through DolphinScheduler; S300. Create a second workflow, which is used to obtain pending tasks and dynamically allocate pending tasks to adapter nodes; S400. Create a third workflow, which is used to allocate tasks to new service nodes when tasks time out or fail.

[0033] It can be understood that the tasks mentioned in the present invention generally refer to tasks with a higher granularity than the standard subtasks; "tasks" can also be expressed as threads, processes, steps, processes, schedules or sequences; those skilled in the art can divide the above tasks and standard subtasks into two basic task levels according to actual conditions to represent a division of the data processing workflow. On this basis, the division according to the specific granularity belongs to the scope of this application.

[0034] In step S100 of some embodiments of the present invention, the directed acyclic graph based on data processing, creating a first workflow for processing each standard subtask through DolphinScheduler includes: creating a processing flow for processing each standard subtask through DolphinScheduler, and updating status information of corresponding nodes.

[0035] Specifically, the information of all service nodes is maintained in the database. A table is created for each step (standard subtask) task to record the node status of each step task and configure the maximum number of tasks processed by each service node. This table includes fields such as node ID, start / end time, task status, etc., and then a service node information table is created to record the service node ID, type, and other information.

[0036] In step S200 of some embodiments of the present invention, based on the directed acyclic graph of data processing, a first workflow for processing each standard subtask is created through DolphinScheduler;

[0037] Specifically, write a task assignment script, scan new tasks, split the tasks into multiple steps, store them in corresponding tables, and set the task status to be executed.

[0038] In step 300 of some embodiments of the present invention, obtaining the tasks to be processed and dynamically allocating the tasks to be processed to the adaptation nodes includes:

[0039] S301. Scan the pending tasks and split each pending task into multiple standard subtasks;

[0040] Specifically, create a scheduled task in DolphinScheduler and add pending tasks to the task table regularly. At the same time, change the Worker group of each node to "Default" and add an SQL node at the end of the task to update the task status; write a task assignment script, scan new tasks, split the tasks into multiple steps and store them in the corresponding tables, and set the task status to pending;

[0041] S302. According to the status information table of each standard subtask, obtain the number of standard subtasks currently running on each service node;

[0042] Specifically, query the pending, timed out, and failed tasks of each step (plus the condition: the previous step task must be completed) and the number of tasks currently running on each service node.

[0043] S303. Based on the maximum number of standard subtasks that each service node can process, the tasks to be processed are dynamically allocated to one or more service nodes with the least current tasks through DolphinScheduler.

[0044] Specifically, by comparing the configured task number threshold, the pending tasks are assigned to the service nodes with fewer tasks first, and the Worker group ID is passed in through the DolphinScheduler API to start the task. At the same time, the task status is changed to running, and the service node ID and start time are recorded in the table. Create a workflow for scheduled tasks, call the script regularly, query the pending tasks, assign the pending tasks to the relatively idle service nodes according to the current number of tasks for each service node, and start the task through the DolphinScheduler API. After the task is executed, the execution result (success, failure) is saved in the data table.

[0045] In step 400 of some embodiments of the present invention, when a task times out or fails, allocating the task to a new service node includes:

[0046] S401. Query the start and end time of all currently running tasks and calculate the execution time of each task; compare the execution time of each task with the preset configuration time of the corresponding standard subtask, and stop executing the corresponding standard subtask according to the comparison result;

[0047] Further, in step S401, the execution duration of each task is compared with the preset configuration time of the corresponding standard subtask, and based on the comparison result, the execution of the corresponding standard subtask is stopped, including: comparing the execution duration of each task with the preset configuration time of the corresponding standard subtask: if the difference between the two exceeds a threshold, stopping the corresponding standard subtask.

[0048] Specifically, write a timeout detection script to query all currently running tasks, calculate the task execution duration based on the current time and the task start time, and then compare the timeout time configured for each step task. If it exceeds the threshold, the task is updated to the timeout state and the API is called to stop the task.

[0049] S402. Reallocate one or more stopped labeling subtasks to other service nodes.

[0050] Specifically, a scheduled task is created in DolphinScheduler to scan timed-out or failed tasks and assign them to other service nodes for task re-running.

[0051] In a specific embodiment of the present invention, the following steps are included:

[0052] 1. Create a table for each step task in the database (node ​​ID, start / end time, task status and other fields), and then create a service node information table to record the service node ID, type and other information;

[0053] 2. Create a processing flow for each step task in DolphinScheduler, change the Worker group of each node to "Default", and add a SQL node at the end of the task to update the task status;

[0054] 3. Write a task assignment script, scan new tasks, split the tasks into multiple steps and store them in the corresponding tables, set the task status to pending, and then query the pending, timed out, and failed tasks of each step (plus the condition: the previous step task needs to be completed) and the number of tasks currently running on each service node. By comparing the configured task number threshold, the pending tasks are preferentially assigned to the service nodes with fewer tasks. The Worker group ID is passed in through the DolphinScheduler API to start the task, and the task status is changed to running, and the service node ID and start time are recorded in the table;

[0055] 4. Create a scheduled task in DolphinScheduler to schedule the script in step 3;

[0056] 5. Write a timeout detection script to query all currently running tasks, calculate the task execution duration based on the current time and the task start time, and then compare the timeout time configured for each step task. If it exceeds the threshold, update the task to the timeout state and call the API to stop the task;

[0057] 6. Create a scheduled task in DolphinScheduler to schedule the script in step 5;

[0058] 7. Run the scheduled tasks in steps 4 and 6 to complete the task execution. You only need to modify the maximum number of running nodes of each service node to complete the dynamic allocation of tasks.

[0059] Example 2

[0060] refer to Figure 5According to a second aspect of the present invention, a multi-stage task dynamic allocation system 1 based on DolphinScheduler is provided, comprising: a construction module 11, for constructing a status information table for each standard subtask, and dynamically determining the maximum number of standard subtasks that each service node can process, wherein the status information includes the start and end time of the standard subtask and the ID of the service node; a first creation module 12, for creating a first workflow for processing each standard subtask through DolphinScheduler based on a directed acyclic graph of data processing; a second creation module 13, for creating a second workflow, wherein the second workflow is used to obtain tasks to be processed and dynamically allocate tasks to be processed to adapter nodes; a third creation module, for creating a third workflow 14, wherein the third workflow is used to allocate tasks to new service nodes when a task times out or fails.

[0061] Furthermore, the second creation module 13 includes: a scanning unit, used to scan the tasks to be processed and split each task to be processed into multiple standard subtasks; an acquisition unit, used to obtain the number of standard subtasks currently running on each service node according to the status information table of each standard subtask; an allocation unit, used to dynamically allocate the tasks to be processed to one or more service nodes with the least current tasks through DolphinScheduler based on the maximum number of standard subtasks processed by each service node.

[0062] Example 3

[0063] refer to Figure 6 According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the multi-stage task dynamic allocation method based on DolphinScheduler in the first aspect of the present invention.

[0064] The electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0065] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0066] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above functions defined in the method of the embodiment of the present disclosure are executed. It should be noted that the computer-readable medium described in the embodiment of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In an embodiment of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wire, optical cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0067] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device. The computer-readable medium carries one or more computer programs. When the one or more programs are executed by the electronic device, the electronic device:

[0068] Computer program code for performing the operations of embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, Python, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0069] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0070] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-stage task dynamic allocation method based on DolphinScheduler, characterized in that: include: Construct a status information table for each standard subtask and dynamically determine the maximum number of standard subtasks that each service node can process. The status information includes the start and end time of the standard subtask and the ID of the service node. Based on the directed acyclic graph of data processing, the first workflow for processing each standard subtask is created through DolphinScheduler; Creating a second workflow, wherein the second workflow is used to obtain tasks to be processed and dynamically allocate the tasks to be processed to adapter nodes; A third workflow is created, where the third workflow is used to allocate the task to a new service node when the task times out or fails.

2. The multi-stage task dynamic allocation method based on DolphinScheduler according to claim 1 is characterized in that: The obtaining of pending tasks and the dynamic allocation of pending tasks to the adaptation nodes include: Scan pending tasks and split each pending task into multiple standard subtasks; According to the status information table of each standard subtask, obtain the number of standard subtasks currently running on each service node; Based on the maximum number of standard subtasks that each service node can process, the tasks to be processed are dynamically allocated to one or more service nodes with the least current tasks through DolphinScheduler.

3. The multi-stage task dynamic allocation method based on DolphinScheduler according to claim 2 is characterized in that: Also includes: Modify the status information table of the service node assigned to the new standard subtask, and record the start time and service node ID of each new standard subtask.

4. The multi-stage task dynamic allocation method based on DolphinScheduler according to claim 1 is characterized in that: When the task times out or fails, allocating the task to a new service node includes: Query the start and end time of all currently running tasks, and calculate the execution time of each task; Compare the execution time of each task with the preset configuration time of the corresponding standard subtask, and stop executing the corresponding standard subtask according to the comparison result; Reassign one or more stopped annotation subtasks to other service nodes.

5. The multi-stage task dynamic allocation method based on DolphinScheduler according to claim 4 is characterized in that: The step of comparing the execution time of each task with the preset configuration time of the corresponding standard subtask and stopping the execution of the corresponding standard subtask according to the comparison result includes: The execution time of each task is compared with the preset configuration time of the corresponding standard subtask: if the difference between the two exceeds the threshold, the corresponding standard subtask is stopped.

6. The multi-stage task dynamic allocation method based on DolphinScheduler according to claim 1 is characterized in that: The directed acyclic graph based on data processing, creating a first workflow for processing each standard subtask through DolphinScheduler includes: Create a processing flow for each standard subtask through DolphinScheduler and update the status information of the corresponding node.

7. A multi-stage task dynamic allocation system based on DolphinScheduler, characterized in that: include: A construction module is used to construct a status information table for each standard subtask and dynamically determine the maximum number of standard subtasks that each service node can process. The status information includes the start and end time of the standard subtask and the ID of the service node. A first creation module, for creating a first workflow for processing each standard subtask through DolphinScheduler based on a directed acyclic graph of data processing; A second creation module, used to create a second workflow, wherein the second workflow is used to obtain tasks to be processed and dynamically allocate the tasks to be processed to adapter nodes; The third creation module is used to create a third workflow, and the third workflow is used to allocate the task to a new service node when the task times out or fails.

8. The multi-stage task dynamic allocation system based on DolphinScheduler according to claim 7 is characterized in that: The second creation module includes: A scanning unit, used for scanning pending tasks and splitting each pending task into multiple standard subtasks; An acquisition unit, used to acquire the number of standard subtasks currently running on each service node according to the status information table of each standard subtask; The allocation unit is used to dynamically allocate the pending tasks to one or more service nodes with the least current tasks through DolphinScheduler based on the maximum number of standard subtasks that each service node can process.

9. An electronic device, comprising: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the multi-stage task dynamic allocation method based on DolphinScheduler as described in any one of claims 1 to 6.

10. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method for minimizing the road network data range of the electronic horizon is implemented as described in any one of claims 1 to 6.

Images