Distributed multi-system timing operation task management system

Abstract

The invention relates to a distributed multi-system timing operation task management system which comprises a dispatching center and a plurality of actuators. The system comprises a scheduling center, a task management module, an actuator management module, a scheduler, a log management module and a monitoring report module. The task management module is used for editing task parameters of the timed operation task and determining whether the timed operation task takes effect or not; the actuator management module is used for registering an actuator; the scheduler is used for scheduling the effective timing operation task to each registered online actuator according to a preset strategy; the log management module is used for recording a scheduling result of the scheduler; each executor is used for performing timed execution according to the task parameters of the effective timed operation tasks according to the scheduling result of the scheduler, and feeding back the execution state to the scheduling center; and the monitoring report module is used for generating monitoring information according to the execution state of each actuator for the timing operation task.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed systems, and in particular to a distributed multi-system scheduled job task management system. Background Art

[0002] With the rapid development of information technology and the popularization of the Internet, distributed system architecture has become a key technology for processing massive data and improving system performance. A distributed system consists of multiple independent, intercommunicating computers. By distributing tasks to different computers for processing, it achieves fast computing speed, high fault tolerance, strong scalability, and easy management and maintenance. This architecture is particularly suitable for scenarios that require processing large amounts of data and complex business logic. Scheduled task management plays an important role in distributed systems. In a distributed multi-system environment, the background scheduled job service is responsible for performing various regular or periodic tasks, which may involve key business operations such as data backup, log cleaning, and statistical analysis.

[0003] Traditional task management performs single-point management on each system. The single-point scheduled task management method has problems such as relatively simple functions, poor interactivity, low task deployment efficiency, and high development and maintenance costs.

[0004] Therefore, how to manage and schedule the scheduled tasks of distributed multi-systems and reduce development and maintenance costs is an urgent problem to be solved. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a distributed multi-system scheduled job task management system, which can solve the problem that the distributed multi-system scheduled task scheduling in the prior art cannot perform centralized visual monitoring and authority management.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present application provides a distributed multi-system scheduled job task management system, comprising: a scheduling center and a plurality of executors;

[0008] The dispatch center includes a task management module, an executor management module, a scheduler, a log management module and a monitoring report module;

[0009] The task management module is used to edit the task parameters of the scheduled job task and to determine whether the scheduled job task is effective;

[0010] The executor management module is used to register the executor;

[0011] The scheduler is used to schedule the effective scheduled job tasks to each registered online executor according to the preset strategy;

[0012] The log management module is used to record the scheduling results of the scheduler;

[0013] Each of the executors is used to perform scheduled execution according to the scheduling result of the scheduler and the task parameters of the effective scheduled job tasks, and feed back the execution status to the scheduling center;

[0014] The monitoring report module is used to generate monitoring information according to the execution status of each executor for the scheduled job task.

[0015] In one implementation of the present application, the task management module provides a web page for users to edit task parameters of scheduled job tasks, including at least one of task name, task description, execution frequency, execution time, and execution code.

[0016] In an implementation of the present application, the task parameters also include the scheduled job task and its subtasks, as well as the sequence or dependency relationship between the subtasks.

[0017] In one implementation of the present application, the scheduler schedules the effective scheduled job tasks according to a preset scheduling cycle; when the scheduler detects that any executor is online or offline, the effective scheduled job tasks are reallocated according to the preset strategy in the next scheduling cycle.

[0018] In one implementation of the present application, the prediction strategy includes a cluster deployment strategy of the scheduler, a cluster deployment strategy of the executor, a blocking processing strategy of the scheduler, a routing strategy of the executor, and a failover strategy of the executor.

[0019] In one implementation of the present application, when the executor is executing a scheduled job task, if a predefined task timeout period is exceeded, the task is interrupted and recorded as a task execution failure.

[0020] In one implementation of the present application, the executor retries the task according to a set number of task failure retries.

[0021] In one implementation of the present application, the monitoring report module sends an alarm message to a preset email address after the executor exceeds a set number of task failure retries.

[0022] In one implementation of the present application, the executor is used to send a rolling log to the scheduling center when executing a scheduled job task.

[0023] In one implementation of the present application, the executor is further configured to register scheduled tasks with the scheduling center at regular intervals through a registration thread.

[0024] The present invention adopts the above technical solution, which has the following advantages:

[0025] Solve the problem that scheduled tasks cannot monitor alarms. Use a custom data set query language to query the task status information stored in the database, configure this query statement on your own IT network management monitoring platform for monitoring, and notify relevant operation and maintenance personnel in real time in the event of failure.

[0026] Solve the problem that the scheduling strategy of scheduled tasks cannot be modified in real time. Persist the detailed information related to the task to the database, provide relevant visual interface functions for adding tasks, editing tasks, deleting tasks, starting tasks, and stopping tasks to edit the detailed information of the task strategy in real time, so as to achieve real-time dynamic changes in task scheduling strategies.

[0027] Solve the problems of difficult operation and maintenance and high maintenance costs. By using an open source framework to separate the task executor and the scheduler, the executor is registered with the scheduler registration center, and the specific task is executed through the scheduler. If the task scheduler and the specific execution task are written into a service, and the program is tightly coupled to handle the task, then when the strategy needs to be changed, the task processing needs to be upgraded to solve it, and upgrading the service may be difficult. However, by separating the task from the executor, the purpose of updating the strategy in real time and making the strategy effective in real time can be achieved. Improve the efficiency of software operation and maintenance, reduce costs, and reduce operational risks.

[0028] Solve the problem of being unable to manage scheduled jobs in multiple systems by rights and domains. Differentiate system-level permissions by adding system codes to executors, manage permissions of individual systems by users, roles, and permissions. After logging into the system, each system displays the executor and task information related to its own system. It can also perform related operations on the data of its own system, achieving the purpose of online management of scheduled tasks by rights and roles. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a module function block diagram of the distributed multi-system scheduled job task management system of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0031] In view of the fact that traditional task management performs single-point management on each system, the single-point scheduled task management method has the problems of relatively simple functions, poor interactivity, low task deployment efficiency, high development and maintenance costs, etc., the present application provides a distributed multi-system scheduled job task management system, including: a scheduling center and several executors; the scheduling center includes a task management module, an executor management module, a scheduler, a log management module and a monitoring report module; the task management module is used to edit the task parameters of the scheduled job task and to determine whether the scheduled job task is effective; the executor management module is used to register the executor; the scheduler is used to schedule the effective scheduled job task to each registered online executor according to a preset strategy; the log management module is used to record the scheduling results of the scheduler; each of the executors is used to perform scheduled execution according to the task parameters of the effective scheduled job task according to the scheduling results of the scheduler, and feed back the execution status to the scheduling center; the monitoring report module is used to generate monitoring information according to the execution status of the scheduled job task by each executor.

[0032] The method provided in the present application is described in more detail below in conjunction with the accompanying drawings of the present application to illustrate its process and technical advantages.

[0033] exist Figure 1 , a functional block diagram of the system provided by the present invention is illustrated.

[0034] The present application provides a distributed multi-system scheduled job task management system, including: a scheduling center and several executors; the scheduling center includes a task management module, an executor management module, a scheduler, a log management module and a monitoring report module.

[0035] Task management module, which implements task definition and configuration.

[0036] In the task management system, clear description, execution frequency, execution time, execution command or script and other information are defined for each scheduled job task.

[0037] The scheduler module is used to implement task allocation management.

[0038] It uses appropriate scheduling algorithms, such as polling, priority, load balancing, etc., to determine the execution order and allocation method of tasks, and dynamically adjusts the execution order of tasks according to the usage of system resources and the priority of tasks.

[0039] The executor starts the execution of the task at the specified time or under the trigger conditions.

[0040] The dispatch center monitors the execution status of the task, including whether it is being executed, the execution progress, whether it is successful or failed, etc.

[0041] For failed tasks, retries or alarms are performed according to the configured strategy.

[0042] The dispatch center provides logging and alarm functions: record the execution log of each task, including execution time, execution results, error information, etc. Set up an alarm mechanism to notify relevant personnel via email, SMS, etc. when a task fails or reaches a specific alarm condition.

[0043] The dispatch center also has security and authority management functions: specifically ensuring the security of the task management system to prevent unauthorized access and operation. And implementing authority management, different users or roles can only operate or view the tasks they are authorized to.

[0044] exist Figure 1 In specific application scenarios, its characteristics are as follows:

[0045] 1. Support CRUD operations on tasks through Web pages, which is easy to operate;

[0046] 2. Dynamic: Supports dynamic modification of task status, starting / stopping tasks, and terminating running tasks, with immediate effect;

[0047] 3. Dispatching center HA (centralized): The dispatching adopts a centralized design. The "dispatching center" is implemented based on cluster Quartz and supports cluster deployment, which can ensure the HA of the dispatching center;

[0048] 4. Executor HA (distributed): tasks are executed in a distributed manner, and the task "executor" supports cluster deployment, which can ensure task execution HA;

[0049] 5. Registration Center: The executor will automatically register tasks periodically, and the scheduling center will automatically discover the registered tasks and trigger execution. At the same time, it also supports manual entry of the executor address;

[0050] 6. Elastic expansion and reduction: Once a new executor machine is online or offline, the task will be reallocated during the next scheduling;

[0051] 7. Routing strategy: A variety of routing strategies are provided when the executor cluster is deployed, including: first, last, polling, random, consistent HASH, least frequently used, least recently used, failover, busy transfer, etc.;

[0052] 8. Failover: When the task routing strategy is set to "failover", if a machine in the executor cluster fails, it will automatically failover to a normal executor to send the scheduling request.

[0053] 9. Blocking handling strategy: The handling strategy when the scheduling is too dense and the executor has no time to handle it. The strategies include: single machine serial (default), discarding subsequent scheduling, and overwriting previous scheduling;

[0054] 10. Task timeout control: Supports custom task timeout, and the task will be automatically interrupted when the task runs overtime;

[0055] 11. Task failure retry: Supports customizing the number of task failure retries. When a task fails, it will be actively retried according to the preset number of failure retries. Among them, sharding tasks support failure retries at the sharding granularity.

[0056] 12. Task failure alarm: By default, email failure alarm is provided, and an extension interface is reserved to facilitate the expansion of SMS, DingTalk and other alarm methods;

[0057] 13. Shard broadcast task: When the executor cluster is deployed, if the task routing strategy is selected as "shard broadcast", a task scheduling will broadcast and trigger all executors in the cluster to execute a task. Shard tasks can be developed according to the shard parameters;

[0058] 14. Dynamic Sharding: Sharding broadcast tasks are sharded based on the executor dimension, supporting dynamic expansion of executor clusters to dynamically increase the number of shards and collaboratively process business; it can significantly improve task processing capabilities and speed when performing large-scale business operations.

[0059] 15. Event triggering: In addition to the "Cron method" and "task dependency method" to trigger task execution, it supports event-based triggering of tasks. The scheduling center provides an API service for triggering a single execution of a task, which can be flexibly triggered according to business events.

[0060] 16.Task progress monitoring: support real-time monitoring of task progress;

[0061] 17.Rolling real-time log: supports online viewing of scheduling results, and supports real-time viewing of the complete execution log output by the executor in Rolling mode;

[0062] 18.Task dependency: supports configuring subtask dependencies. When the parent task is completed and executed successfully, it will actively trigger the execution of a subtask. Multiple subtasks are separated by commas.

[0063] 19. Consistency: The "scheduling center" ensures the consistency of cluster distributed scheduling through DB locks. One task scheduling will only trigger one execution;

[0064] 20. Customized task parameters: support online configuration of scheduling task parameters, effective immediately;

[0065] 21. Scheduling thread pool: The scheduling system triggers scheduling operation through multiple threads to ensure accurate execution of scheduling without being blocked;

[0066] 22. Data encryption: Data encryption is performed on the communication between the dispatch center and the executor to improve the security of dispatch information;

[0067] 23. Email alarm: support email alarm when task fails, support configuration of multiple email addresses to send alarm emails in batches;

[0068] 24. Push to Maven Central Warehouse: The latest stable version will be pushed to Maven Central Warehouse for easy access and use by users;

[0069] 25. Operation report: supports real-time viewing of operation data, such as the number of tasks, scheduling times, number of executors, etc.; and scheduling reports, such as scheduling date distribution chart, scheduling success distribution chart, etc.;

[0070] 26. Fully asynchronous: The task scheduling process is fully asynchronously designed and implemented, such as asynchronous scheduling, asynchronous operation, asynchronous callback, etc., which effectively reduces the traffic peak for intensive scheduling and theoretically supports the operation of tasks of any duration;

[0071] 27. Cross-platform: Natively provides a general HTTP task Handler (Bean task, "HttpJobHandler"); the business side only needs to provide an HTTP link, without restrictions on language or platform;

[0072] 28.Containerization: Provide official Docker images and update and push them to Dockerhub in real time, further making the product ready to use out of the box;

[0073] 29. Thread pool isolation: The scheduling thread pool is isolated and split, and slow tasks are automatically downgraded to the "Slow" thread pool to avoid exhausting the scheduling threads and improve system stability;

[0074] 30. User management: Support online management system users, with two roles: administrator and general operation and maintenance user;

[0075] 31. Permission control: Permission control is performed at the executor dimension. Administrators have full permissions, and ordinary users need to be assigned executor permissions before related operations are allowed to achieve the purpose of subsystem management.

[0076] In summary, the technical advantages of this solution include:

[0077] Solve the problem that scheduled tasks cannot monitor alarms. Use a custom data set query language to query the task status information stored in the database, configure this query statement on your own IT network management monitoring platform for monitoring, and notify relevant operation and maintenance personnel in real time in the event of failure.

[0078] Solve the problem that the scheduling strategy of scheduled tasks cannot be modified in real time. Persist the detailed information related to the task to the database, provide relevant visual interface functions for adding tasks, editing tasks, deleting tasks, starting tasks, and stopping tasks to edit the detailed information of the task strategy in real time, so as to achieve real-time dynamic changes in task scheduling strategies.

[0079] Solve the problems of difficult operation and maintenance and high maintenance costs. By using an open source framework to separate the task executor and the scheduler, the executor is registered with the scheduler registration center, and the specific task is executed through the scheduler. If the task scheduler and the specific execution task are written into a service, and the program is tightly coupled to handle the task, then when the strategy needs to be changed, the task processing needs to be upgraded to solve it, and upgrading the service may be difficult. However, by separating the task from the executor, the purpose of updating the strategy in real time and making the strategy effective in real time can be achieved. Improve the efficiency of software operation and maintenance, reduce costs, and reduce operational risks.

[0080] Solve the problem of being unable to manage scheduled jobs in multiple systems by rights and domains. Differentiate system-level permissions by adding system codes to executors, manage permissions of individual systems by users, roles, and permissions. After logging into the system, each system displays the executor and task information related to its own system. It can also perform related operations on the data of its own system, achieving the purpose of online management of scheduled tasks by rights and roles.

[0081] In the several embodiments provided by the present invention, it should be understood that the disclosed method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0082] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (Processor) to perform some steps of the above-mentioned method of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0083] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A distributed multi-system scheduled job task management system, characterized in that: include: A dispatch center and several actuators; The dispatch center includes a task management module, an executor management module, a scheduler, a log management module and a monitoring report module; The task management module is used to edit the task parameters of the scheduled job task and to determine whether the scheduled job task is effective; The executor management module is used to register the executor; The scheduler is used to schedule the effective scheduled job tasks to each registered online executor according to the preset strategy; The log management module is used to record the scheduling results of the scheduler; Each of the executors is used to perform scheduled execution according to the scheduling result of the scheduler and the task parameters of the effective scheduled job tasks, and feed back the execution status to the scheduling center; The monitoring report module is used to generate monitoring information according to the execution status of each executor for the scheduled job task.

2. The distributed multi-system scheduled job task management system according to claim 1, characterized in that: The task management module provides a web page for users to edit the task parameters of the scheduled job tasks, including at least one of the task name, task description, execution frequency, execution time, and execution code.

3. The distributed multi-system scheduled job task management system according to claim 2 is characterized in that: The task parameters also include the scheduled job task and its subtasks, as well as the sequence or dependency relationship between the subtasks.

4. The distributed multi-system scheduled job task management system according to claim 1, characterized in that: The scheduler schedules the effective scheduled job tasks according to a preset scheduling cycle; when the scheduler detects that any executor is online or offline, the effective scheduled job tasks are reallocated according to the preset strategy in the next scheduling cycle.

5. The distributed multi-system scheduled job task management system according to claim 4, characterized in that: The prediction strategy includes a cluster deployment strategy of the scheduler, a cluster deployment strategy of the executor, a blocking processing strategy of the scheduler, a routing strategy of the executor, and a failover strategy of the executor.

6. The distributed multi-system scheduled job task management system according to claim 1, characterized in that: When the executor is executing a scheduled job task, if the predefined task timeout period is exceeded, the task is interrupted and recorded as a task execution failure.

7. The distributed multi-system scheduled job task management system according to claim 6, characterized in that: The executor retries the task according to the set task failure retry times.

8. The distributed multi-system scheduled job task management system according to claim 7, characterized in that: The monitoring report module sends an alarm message to a preset email address after the executor exceeds the set number of task failure retries.

9. The distributed multi-system scheduled job task management system according to claim 1, characterized in that: The executor is used to send rolling logs to the scheduling center when executing scheduled job tasks.

10. The distributed multi-system scheduled job task management system according to claim 1, characterized in that: The executor is also used to register scheduled tasks with the scheduling center at regular intervals through a registration thread.

Images