Quartz-based distributed task scheduling framework and use thereof

Abstract

Provided are a Quartz-based distributed task scheduling framework and a use thereof. A task flow execution engine framework is constructed by means of a view, task execution, a timed task, an alert, and a component, and pooled execution and persistence of a task flow are supported. The view comprises a task flow execution view, used for observing an execution situation of the task flow, to keep abreast of the status and progress of the task execution, so that quick troubleshooting is facilitated. The task execution comprises a self-developed task flow execution engine, used for being responsible for execution of the task flow. The timed task comprises a periodic timeout detection Quartz distributed task and a periodic task heartbeat update task. Tasks can be scheduled for a plurality of machines in a distributed manner, to ensure high availability and scalability of the tasks. The alert comprises an abnormal situation reporting alert, used for providing an alert regarding an abnormal situation, to ensure the reliability and stability of the task. The component comprises MySQL, Quartz, Prometheus, and Alertmanager, used for implementing the entire framework.

Description

A distributed task scheduling framework based on Quartz and its application

[0001] Related applications

[0002] This application claims priority to Chinese patent application No. 2023117227740, filed on December 15, 2023, entitled “A distributed task scheduling framework based on Quartz and its application,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of computer technology, and in particular to a Quartz-based distributed task scheduling framework and its application. Background Art

[0004] In the database field, users generally use the database to perform operations on the page console, such as instance management, account management, data management, disaster recovery management, etc. These operation tasks generally correspond to multiple steps, and each step may even have nested subtasks. However, the traditional task execution method usually uses task distribution and execution on a single machine. This method has the problem of single point failure and cannot guarantee the high availability of tasks. Other task execution engines in the industry are not flexible enough when processing task flows, that is, tasks consisting of multiple steps. They cannot support nested scheduling of task flows and cannot reuse existing task flows. The distributed task scheduling framework in traditional technology also has shortcomings when handling abnormal situations such as machine failures. It cannot implement functions such as stateless services and automatic restart. The execution of task flows lacks observability, and it is impossible to timely understand the status and progress of task execution, making it difficult to quickly troubleshoot problems. The scalability of traditional task execution is also relatively poor. It is difficult for new tasks to reuse existing tasks, and the execution of task flows cannot be scheduled to different machines based on machine availability, making it impossible to achieve distributed and highly available scheduling of task flows.

[0005] The traditional method includes: registering multiple scheduling applications in a distributed queue at a zookeeper registration center; calling a first scheduling application among the multiple registered scheduling applications to allocate a distributed lock to the first scheduling application so that only the first scheduling application can execute tasks; and when the first scheduling application obtains the allocated distributed lock, using the first scheduling application to execute the corresponding task.

[0006] The traditional method also includes: a master node receives task pull requests sent by multiple task execution nodes at a preset frequency; the master node assigns tasks to a target task execution node based on the received task pull requests, where the target task execution node is one of the multiple task execution nodes; the master node receives the task execution results sent by the target task execution node after executing the assigned task; and, upon detecting a failure in the target task execution node, the master node transfers the executing task queue of the target task execution node to another surviving task execution node. This avoids single points of failure and improves system stability and availability.

[0007] Traditional methods also include: users define, launch, and operate workflows by calling interface service APIs; the distributed workflow coordinator Coordinator integrates the distributed timing engine Quartz to schedule workflows and add them to the workflow dispatch distributed message queue MQ, receives workflows and processes their task dependencies, and adds the coordinated business tasks to be executed to the task dispatch distributed message queue MQ; the distributed task executor Worker receives each business task from the task dispatch distributed message queue MQ and executes it, and calls back the task execution results to the distributed workflow coordinator Coordinator through the task callback distributed message queue MQ; finally, the Coordinator persists the task execution results to the database for feedback to users; the Coordinator focuses on logical coordination processing to ensure that workflow coordination processing and task execution are fully decoupled, thereby improving the system's throughput, scalability, and scalability.

[0008] A traditional system includes: the system is integrated by a coordinator and multiple Quartz service engines, the kernel of each Quartz service engine includes multiple task management units, and the task management units are preset with a first load threshold; the coordinator is used to monitor the first load level of the task management units, and is used to dynamically expand a new task management unit when the first load level of any task management unit reaches the first load threshold; the provided Quartz-based distributed task scheduling method is applied to the above system, and by expanding the task management units of Quartz, the load of a single task management unit is reduced, and the system is no longer limited by the performance bottleneck problem of Quartz. In addition, task routing can be controlled on the control plane without increasing the workload on the application side.

[0009] The above traditional technologies have the problems raised by this background technology: the traditional task execution method has single point failure, poor observability, and poor scalability. To solve this problem, this application proposes a distributed task scheduling framework based on Quartz and its application. Summary of the Invention

[0010] The purpose of this section is to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the present application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions shall not be used to limit the scope of the present application.

[0011] In view of the problems existing in a Quartz-based distributed task scheduling framework and its application in the above-mentioned traditional technology, this application is proposed.

[0012] Therefore, the purpose of this application is to provide a distributed task scheduling framework based on Quartz and its application.

[0013] To solve the above technical problems, this application provides the following technical solutions: view, task execution, scheduled tasks, alarms and components;

[0014] The view includes a task flow execution view, which is used to observe the execution of the task flow;

[0015] The task execution includes a self-developed task flow execution engine, which is responsible for the execution of the task flow;

[0016] The scheduled tasks include regular timeout detection quartz distributed tasks and regular task heartbeat update tasks;

[0017] The alarm includes an abnormal situation reporting alarm, which is used to generate the alarm for the abnormal situation;

[0018] The components include MySQL, Quartz, Promethus and Alertmanager, which are used to implement the entire framework.

[0019] As a solution of a Quartz-based distributed task scheduling framework and its application described in this application, the task flow task_flow contains multiple steps, each step is a subtask flow, which can be nested. For example, the instance creation task flow contains multiple steps. The data items included in the task flow are as follows:

[0020] taskCode, which represents the task ID. Each type of task ID identifies a type of task flow, such as creating an instance;

[0021] taskID, which represents the id of the task flow and uniquely identifies the task flow;

[0022] parentTaskID, which indicates the parent task flow ID of the task flow. For a nested subtask flow, the parent task flow ID is identified by this field.

[0023] CurrentStepNo, indicates the step currently executed by the task flow;

[0024] status, indicating the status of the task flow;

[0025] version, which indicates the heartbeat timestamp of the task flow. This timestamp is updated periodically when the local machine schedules the execution of the task.

[0026] As a solution of a Quartz-based distributed task scheduling framework and its application described in this application, each task flow specifically executes each task step task_step, and the task step is the actual execution content. The data items included in the task step are as follows:

[0027] stepName, which indicates the class name of the task step implementation and is used to reflect the instantiation object;

[0028] orderNo: indicates the step number of the task step in the task flow. When executing the task flow, the tasks are executed in order.

[0029] childTaskID, which identifies a subtask flow of the task step;

[0030] status: indicates the status of the task step.

[0031] As a solution of a Quartz-based distributed task scheduling framework and its application described in this application, the process of submitting and executing the task flow is as follows:

[0032] When a user creates an instance, suspends an instance, migrates a task when it times out, retries a task, or performs other operations, a task flow task is created first. The task flow task is queued into different task flow queues according to the instance and supports priority scheduling. The task flow with a high priority is executed first. At this time, the execution of the task flow is triggered, and tasks are dequeued from the task flow queue to execute the task flow. Specifically, the thread pool of the task flow is used for scheduling and execution to improve the efficiency of execution. After the current task flow is executed, the situation in the task flow queue is judged. If the task flow queue is empty, the process of submitting the task flow for execution is terminated. If the task flow queue is not empty, the execution process of the task flow is repeated until all the task flow tasks are executed.

[0033] As a solution of a Quartz-based distributed task scheduling framework and its application described in this application, the task flow queue adopts ConcurrentLinkedDeque in the Java concurrent package. The task flow queue is a double-ended queue based on a linked list. When in use, it enters the queue from the end of the queue and exits the queue from the head of the queue. In addition, it adopts the form of two queues, one queue stores high-priority tasks, and the other stores tasks of ordinary task flows. When scheduling execution, tasks are extracted from the high-priority queue for execution first.

[0034] As a solution of a Quartz-based distributed task scheduling framework and its application described in this application, wherein: the periodic timeout detection quartz distributed task is used to periodically detect whether there are timed tasks in the task flow in the database, and submit the timed tasks to the task execution engine for execution;

[0035] The periodic task heartbeat update task is used to periodically update the heartbeat timestamp of the locally executed task flow;

[0036] The process of distributed task scheduling is as follows:

[0037] First, the task is scheduled, and then an available machine is selected for scheduling and execution. If the machine fails at this time, the distributed task is scheduled to another machine for execution. If the machine does not fail at this time, the process of distributed task scheduling is ended.

[0038] As a solution of a Quartz-based distributed task scheduling framework and its application described in this application, the process of distributed scheduling of task flows is as follows:

[0039] First, create the task flow task, and then select an available machine to run the task flow. When each task flow is executed on the local machine, the execution heartbeat timestamp of the task flow will be reported to MySQL. If a machine failure, program exception or other reasons occur during the operation, the distributed task will be subjected to heartbeat timeout detection. If the task times out, it will be scheduled to run on another machine. Periodic tasks will not be executed on the failed machine.

[0040] As a solution of a Quartz-based distributed task scheduling framework and its application described in this application, the alarm mechanism process is as follows:

[0041] When the task flow is executed, if the execution is successful, the alarm mechanism process is terminated;

[0042] If the operation fails, it will cause the instance to be abnormal, which can trigger the alarm. Quartz will use regular distributed scheduling tasks to report the instance abnormal status to PushGateway. Then Promethus will regularly pull data, calculate alarm rules, push alarm data to AlertManager, and send it to the alarm recipient to implement the alarm process.

[0043] In addition, during the execution of the task flow task, the success or failure of the task flow task will be persisted in real time, and the user can observe the running status of the task flow task on the page.

[0044] A computer device includes a memory for storing instructions and a processor for executing the instructions, so that the device implements a Quartz-based distributed task scheduling framework and its application.

[0045] A computer-readable storage medium stores a computer program, which, when executed, implements a Quartz-based distributed task scheduling framework and its application.

[0046] Beneficial effects of this application: This application builds a task flow execution engine framework through views, task execution, scheduled tasks, alarms and components, supports pooled execution and persistence of task flows, can be easily expanded, supports nested scheduling of task flows, has a mechanism for reusing existing task flows, and does not rely on other components. It can provide lib packages for easy reuse in other scenarios that require task flow scheduling; the view includes a task flow execution view, which is used to observe the execution of task flows, and displays the execution status of task flows in the form of a page view, so that the status and progress of task execution can be understood in a timely manner, which is convenient for quick problem solving; task execution includes a self-developed task flow execution engine, which is responsible for the execution of task flows and provides the execution process and status of task flows. The system is persistent; scheduled tasks include periodic timeout detection Quartz distributed tasks and periodic task heartbeat update tasks, which can distribute and schedule tasks to multiple machines, realize abnormal migration of task flows, task flow interruption restart, retry, etc., thereby ensuring high availability and scalability of tasks, and supporting cluster deployment to ensure that tasks are triggered and executed only once, and can support flexible periodic tasks, etc.; alarms include abnormal situation reporting alarms, which are used to warn of abnormal situations, support triggering alarms and retries of abnormal tasks, and ensure the reliability and stability of tasks; components include MySQL, Quartz, Promethus and Alertmanager, which are used to implement the entire framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0048] FIG1 is a diagram showing the overall system architecture of a Quartz-based distributed task scheduling framework and its application;

[0049] FIG2 is a flowchart of a task flow submission and execution described in a Quartz-based distributed task scheduling framework and its application in this application;

[0050] FIG3 is a diagram of a dual-ended concurrent queue of a task flow described in a Quartz-based distributed task scheduling framework and its application in this application;

[0051] FIG4 is a diagram of task running states described in a Quartz-based distributed task scheduling framework and its application in this application;

[0052] FIG5 is a diagram of the task step operation status of a distributed task scheduling framework based on Quartz and its application in this application;

[0053] FIG6 is a diagram of distributed task scheduling described in a Quartz-based distributed task scheduling framework and its application in this application;

[0054] FIG7 is a distributed task scheduling diagram of a Quartz-based distributed task scheduling framework and its application described in this application;

[0055] FIG8 is a flowchart of a task observation and failure alarm described in a Quartz-based distributed task scheduling framework and its application in this application;

[0056] FIG9 is a diagram showing the steps of creating a task for a master-slave instance in a distributed task scheduling framework based on Quartz and its application in this application;

[0057] [Corrected 23.01.2025 according to Rule 91] Figure 10 is a step diagram of the subtask flow described in a Quartz-based distributed task scheduling framework and its application in this application.

[0058] [Corrected 23.01.2025 in accordance with Article 91] DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0060] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can also be implemented in other ways different from those described herein. Those skilled in the art can make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0061] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present application. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0062] Furthermore, this application is described in detail with reference to schematic diagrams. For ease of illustration, when describing the embodiments of this application, cross-sectional views of device structures may be partially enlarged and not to scale. Furthermore, these schematic diagrams are merely illustrative and should not limit the scope of protection of this application. Furthermore, in actual production, the three-dimensional dimensions of length, width, and depth should be included.

[0063] Example 1

[0064] In this embodiment, a system overall architecture diagram of a Quartz-based distributed task scheduling framework and its application is provided, as shown in Figure 1. A Quartz-based distributed task scheduling framework and its application include views, task execution, scheduled tasks, alarms and components.

[0065] The view includes the task flow execution view, which is used to observe the execution of the task flow.

[0066] The task flow task_flow contains multiple steps. Each step is a sub-task flow, which can be nested. For example, the instance creation task flow contains multiple steps. The data items included in the task flow are as follows:

[0067] taskCode, which represents the task ID. Each type of task ID identifies a type of task flow, such as creating an instance;

[0068] taskID, which represents the ID of the task flow and uniquely identifies the task flow;

[0069] parentTaskID, which indicates the parent task flow ID of a task flow. For a nested subtask flow, the parent task flow ID is identified by this field.

[0070] CurrentStepNo, indicates the step currently executed by the task flow;

[0071] status, indicating the status of the task flow;

[0072] version, which indicates the heartbeat timestamp of the task flow. This timestamp is updated periodically when the local machine schedules the execution of the task.

[0073] The status of the task flow is shown in Table 1:

[0074] Table 1 Task flow status

[0075] Task execution includes a self-developed task flow execution engine, which is responsible for the execution of task flows.

[0076] Each task flow specifically executes each task step task_step. The task step is the actual execution content. The data items included in the task step are as follows:

[0077] stepName, which indicates the class name of the task step implementation and is used to reflect the instantiation object;

[0078] orderNo: indicates the step number of the task step in the task flow. When executing the task flow, the tasks are executed in sequence.

[0079] childTaskID, a subtask flow of the label task step;

[0080] status: indicates the status of the task step.

[0081] Among them, the status of the task steps is shown in Table 2:

[0082] Table 2 Status of task steps

[0083] The process of task flow submission and execution is shown in Figure 2, and the specific explanation is as follows:

[0084] When a user creates an instance, pauses an instance, migrates a task when it times out, retries a task, or performs other operations, a task flow task is created first. The task flow task is queued into different task flow queues according to the instance and supports priority scheduling. Task flows with high priorities are executed first. At this time, the execution of the task flow is triggered, and tasks are dequeued from the task flow queue to execute the task flow. Specifically, the task flow thread pool is used for scheduling and execution to improve execution efficiency. After the current task flow is executed, the situation in the task flow queue is judged. If the task flow queue is empty, the task flow submission execution process is terminated. If the task flow queue is not empty, the task flow execution process is repeated until all task flow tasks are executed.

[0085] The task flow queue uses ConcurrentLinkedDeque in the Java concurrency package. The task flow queue is a double-ended queue based on a linked list. It has the advantages of high efficiency, thread safety, double-ended queue characteristics, support for empty elements and strong scalability. It is more suitable for task scheduling scenarios. When used, it enters the queue from the end and exits the queue from the head, as shown in Figure 3. In addition, it adopts the form of two queues, one queue to store high-priority tasks, and the other to store tasks of ordinary task flows. When scheduling, tasks are extracted from the high-priority queue for execution.

[0086] After the task flow is dequeued from the queue, the specific task flow execution is triggered. The task flow involves multiple states. The entire process is that when the task flow is executed, multiple steps in the task flow will be executed, and they will be executed in sequence. Initialization is performed when it is put into operation. The work done by initialization is to initialize all the steps to be executed by the task flow and persist them into MySQL. If the initialization fails, it enters the EXEC_NEW state. After the initialization is completed, it enters the READY state, and then starts to execute. Each step of the task flow is executed in sequence. The current step to be executed is known through the currentStepNo attribute, the current step parameters are obtained from MySQL, and the object is dynamically created through the reflection mechanism to instantiate the task step object. Execution. If the step is a nested subtask flow, it enters the ABORT state and waits for the completion of the subtask flow. After each step is completed, it checks whether it is the last task of the task flow, that is, whether the execution is completed. Specifically, it judges the currentStepNo attribute of the task flow to see whether it is greater than or equal to the number of steps to be executed. Each time a step is completed, the attribute will increase, thereby judging whether a task flow has ended. During the execution process, if an abnormal error is found, it enters the TERMINATE state. If a retry is performed, it will enter the QUERY_RETRY state and queue for execution. After being put into operation, it re-enters the READY state for execution. The whole process is shown in Figure 4.

[0087] During the task flow execution process, the execution of each step also involves multiple state transitions. The core process is that the step enters the READY state, starts execution and enters the RUNNING state, enters the FAIL state if there is an exception, enters the ABORT state if it is a subtask flow step, and enters the SUCCESS state after execution is completed. In addition, if the step is in the SKIP state, it will be skipped. The whole process is shown in Figure 5.

[0088] Scheduled tasks include periodic timeout detection of Quartz distributed tasks and periodic task heartbeat update tasks.

[0089] Periodic timeout detection Quartz distributed tasks are used to periodically detect whether there are timed tasks in the task flow in the database. If there is a timeout, it will be submitted to the task execution engine for execution.

[0090] The periodic task heartbeat update task is used to periodically update the heartbeat timestamp of the locally executed task flow.

[0091] In order to solve the abnormal situations that may occur during task execution, such as machine failure that causes the task to be unable to continue execution, the distributed task scheduling framework Quartz is introduced. This framework is a powerful open source task scheduling framework that is flexible, reliable, and distributed. It also supports cluster deployment, periodic tasks, calendar tasks, etc. One of its features is distributed task scheduling, which will schedule tasks and only schedule them to one available machine. The process is shown in Figure 6. The process of distributed task scheduling is as follows:

[0092] First, the task is scheduled, and then an available machine is selected for scheduling and execution. If the machine fails at this time, the distributed task is scheduled to another machine for execution. If the machine does not fail at this time, the task distributed scheduling process ends.

[0093] In order to solve the problem of abnormal task execution due to machine failure or program problems during the execution of the task flow, the task flow execution engine is combined with Quartz. The distributed task scheduling feature of Quartz is used to deploy the service on multiple machines. When the above problem occurs, the task flow will be scheduled to another machine for execution. The specific implementation is that when each task flow is executed on the local machine, the execution heartbeat timestamp information of the task flow will be reported to MySQL, specifically saved in the version field. When a machine failure, program problem or other reasons cause the task flow to be unable to continue to execute, there will be a Quartz periodic task to regularly check whether the timestamp of the task flow has timed out. If it times out, the task flow will be put into local operation. The periodic task will not be executed on the faulty machine. The distributed scheduling function of Quartz is used to achieve high-availability distributed scheduling of task flows under abnormal conditions. The whole process is shown in Figure 7. The process of distributed scheduling of task flows is as follows:

[0094] First, create a task flow task, then select an available machine to run the task flow. When each task flow is executed on the local machine, the task flow execution heartbeat timestamp will be reported to MySQL. If a machine failure, program exception, or other reasons occur during the operation, the distributed task will be subjected to a heartbeat timeout check. If the task times out, it will be scheduled to run on another machine. Periodic tasks will not be executed on the failed machine.

[0095] Alarms include reporting alarms for abnormal situations.

[0096] Under normal circumstances, the combination of the task flow execution engine framework and the distributed scheduling Quartz can achieve distributed high-availability scheduling of task flows. However, during the execution of the task flow, execution may fail. In this case, an alarm mechanism is provided. The alarm mechanism process is shown in Figure 8. The specific process is as follows:

[0097] When the task flow runs successfully, the alarm mechanism process ends;

[0098] If the operation fails, it will cause the instance to be abnormal, which can trigger an alarm. Quartz will use regular distributed scheduling tasks to report the instance abnormal status to PushGateway. Then Promethus will regularly pull data, calculate alarm rules, push alarm data to AlertManager, and send it to the alarm recipient to implement the alarm process.

[0099] In addition, during the execution of the task flow task, the success and failure of the operation will be persisted in real time, and users can observe the operation status of the task flow task on the page.

[0100] Components include MySQL, Quartz, Promethus, and Alertmanager.

[0101] Example 2

[0102] The application of a Quartz-based distributed scheduling framework in the self-developed task flow execution engine includes the following:

[0103] [Corrected 23.01.2025 according to Rule 91] Application 1: Create a master-slave instance task, which aims to illustrate the process of creating a task flow with nested subtasks and executing the task flow, and also illustrates the mechanism of reusing existing task flows by subtask flows and the observability of task flows; first, the user issues a task to create an instance, which is persisted to MySQL. The core data items include task id: 1201-1668879808996605952, task code: 1201, task description: master-slave instance creation task, instance id, parameters including deployment machine information, heartbeat time, current execution step: currentStepNo, initialized to 0, and submit the task to the task flow execution engine. The task execution engine obtains the task flow implementation class corresponding to the task code through the task code, and maintains a code-to-class mapping table The execution object of the task flow is dynamically created through the reflection mechanism and queued, triggering scheduled execution. The first step of execution is to initialize the task flow and persist the steps required to execute the master / slave instance creation task to MySQL. The initialization steps are shown in Figure 9. You can see that the master / slave instance creation task includes 16 steps, of which the 15th step is a subtask. The task flow sequentially retrieves the corresponding step data based on the currentStepNo, obtains the corresponding implementation class through the step_name, and instantiates the step object through the reflection mechanism to execute the corresponding step. When executing step 15, a new subtask flow is created and submitted to the task execution engine. The subtask flow is a whitelist creation task and includes 6 steps, as shown in Figure 10. After execution, the final step is executed, and the instance creation is completed. This shows that both the task flow and the task steps are persisted to MySQL. In addition, during the task execution, execution exceptions are also persisted. This allows you to observe the execution status of the task flow. The execution status of the task flow can be viewed on the page.

[0104] Application 2: Cluster instance pause task, which aims to illustrate distributed scheduling in the event of a service failure. The task will be scheduled to another machine to achieve high availability of task flow execution without user perception. Multiple machines will be deployed when deploying the service. Here, two machines are deployed to execute user task operations. First, a 4-shard MEMDB cluster is created, with 2 nodes per shard, one master and one backup, for a total of 8 Redis nodes. The 4 shards have 4 proxy nodes, deployed on 4 machines. The page initiates a cluster instance pause task. You will see that the task includes multiple steps, such as pausing the Redis node and pausing the proxy node. After submitting the task, manually stop one of the services. You can see that the task is not interrupted but continues to execute. From the log of another service, you can see that the task is executed by the service, that is, it is distributed and scheduled to another machine for execution. Finally, the execution of the paused task is completed without user perception.

[0105] Application three: Data cleaning task, aims to illustrate the process of task flow failure alarm, through manual simulation of task failure, here use the docker stop command to manually pause the node of a single node instance, and then trigger the data cleaning task on the page. The task will be abnormal because the node is not running, resulting in task flow failure. The task execution engine will record the exception to the instance. In addition, there will be a Quartz scheduled task to push the instance status to the pushgateway. The instance status includes normal and abnormal status. By configuring the alarm rules in advance, the alarm will be pushed to the alertmanager by promethus and sent to the alarm recipient, such as the enterprise WeChat alarm. After receiving the alarm, the abnormal instance ID is known, and then the page observes the execution status of the task flow. It is found that the redis node is not running. The redis node is manually started, and then the task is retried to realize the execution of the cleaning task flow.

[0106] Example 3

[0107] In this embodiment, a computer device is provided, including a memory and a processor, the memory is used to store instructions, and the processor is used to execute the instructions, so that the computer device performs the steps of implementing the above-mentioned Quartz-based distributed task scheduling framework and its application.

[0108] Example 4

[0109] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed, the steps of implementing the above-mentioned Quartz-based distributed task scheduling framework and its application are implemented.

[0110] The computer-readable storage medium includes various media for storing program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0111] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application, and all of these should be included in the scope of the claims of the present application.

[0112] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0113] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A Quartz-based distributed task scheduling framework and its application, characterized by: include, Views, task execution, scheduled tasks, alarms, and components; The view includes a task flow execution view, which is used to observe the execution of the task flow; The task execution includes a self-developed task flow execution engine, which is responsible for the execution of the task flow; The scheduled tasks include regular timeout detection quartz distributed tasks and regular task heartbeat update tasks; The alarm includes an abnormal situation reporting alarm, which is used to generate the alarm for the abnormal situation; The components include MySQL, Quartz, Promethus and Alertmanager, which are used to implement the entire framework.

2. The Quartz-based distributed task scheduling framework and its application as claimed in claim 1, characterized in that: The task flow task_flow contains multiple steps, each step is a sub-task flow, which can be nested. For example, the instance creation task flow contains multiple steps. The data items contained in the task flow are as follows: taskCode, which represents the task ID. Each type of task ID identifies a type of task flow, such as creating an instance; taskID, which represents the id of the task flow and uniquely identifies the task flow; parentTaskID, which indicates the parent task flow ID of the task flow. For a nested subtask flow, the parent task flow ID is identified by this field. CurrentStepNo, indicates the step currently executed by the task flow; status, indicating the status of the task flow; version, which indicates the heartbeat timestamp of the task flow. This timestamp is updated periodically when the local machine schedules the execution of the task.

3. The Quartz-based distributed task scheduling framework and its application as claimed in claim 2, characterized in that: Each task flow specifically executes each task step task_step, which is the actual execution content. The data items included in the task step are as follows: stepName, which indicates the class name of the task step implementation and is used to reflect the instantiation object; orderNo: indicates the step number of the task step in the task flow. When executing the task flow, the tasks are executed in order. childTaskID, which identifies a subtask flow of the task step; status: indicates the status of the task step.

4. The Quartz-based distributed task scheduling framework and its application as claimed in claim 3, characterized in that: The process of submitting and executing the task flow is as follows: When a user creates an instance, suspends an instance, migrates a task when it times out, retries a task, or performs other operations, a task flow task is created first. The task flow task is queued into different task flow queues according to the instance and supports priority scheduling. The task flow with a high priority is executed first. At this time, the execution of the task flow is triggered, and tasks are dequeued from the task flow queue to execute the task flow. Specifically, the thread pool of the task flow is used for scheduling and execution to improve the efficiency of execution. After the current task flow is executed, the situation in the task flow queue is judged. If the task flow queue is empty, the process of submitting the task flow for execution is terminated. If the task flow queue is not empty, the execution process of the task flow is repeated until all the task flow tasks are executed.

5. The Quartz-based distributed task scheduling framework and its application as claimed in claim 4, characterized in that: The task flow queue adopts ConcurrentLinkedDeque in the Java concurrent package. The task flow queue is a double-ended queue based on a linked list. When in use, entries are entered from the end of the queue and entries are exited from the head of the queue. In addition, two queues are adopted, one queue stores high-priority tasks and the other stores tasks of ordinary task flows. When scheduling execution, tasks are extracted from the high-priority queue for execution first.

6. The Quartz-based distributed task scheduling framework and its application as claimed in claim 5, characterized in that: The periodic timeout detection quartz distributed task is used to regularly detect whether there are timed tasks in the task flow in the database, and submit the timed tasks to the task execution engine for execution; The periodic task heartbeat update task is used to periodically update the heartbeat timestamp of the locally executed task flow; The process of distributed task scheduling is as follows: First, the task is scheduled, and then an available machine is selected for scheduling and execution. If the machine fails at this time, the distributed task is scheduled to another machine for execution. If the machine does not fail at this time, the process of distributed task scheduling is ended.

7. The Quartz-based distributed task scheduling framework and its application as claimed in claim 6, characterized in that: The process of distributed scheduling of task flows is as follows: First, create the task flow task, and then select an available machine to run the task flow. When each task flow is executed on the local machine, the execution heartbeat timestamp of the task flow will be reported to MySQL. If a machine failure, program exception or other reasons occur during the operation, the distributed task will be subjected to heartbeat timeout detection. If the task times out, it will be scheduled to run on another machine. Periodic tasks will not be executed on the failed machine.

8. The Quartz-based distributed task scheduling framework and its application as claimed in claim 7, characterized in that: The alarm mechanism process is as follows: When the task flow is executed, if the execution is successful, the alarm mechanism process is terminated; If the operation fails, it will cause the instance to be abnormal, which can trigger the alarm. Quartz will use regular distributed scheduling tasks to report the instance abnormal status to PushGateway. Then Promethus will regularly pull data, calculate alarm rules, push alarm data to AlertManager, and send it to the alarm recipient to implement the alarm process. In addition, during the execution of the task flow task, the success or failure of the task flow task will be persisted in real time, and the user can observe the running status of the task flow task on the page.

9. A computer device, characterized in that: include, a memory for storing instructions; A processor is used to execute the instruction so that the device executes and implements a Quartz-based distributed task scheduling framework and its application as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, a Quartz-based distributed task scheduling framework and its application are implemented as described in any one of claims 1 to 8.

Images