Distributed task scheduling system based on rabbitmq, redis and mongodb

Abstract

A distributed task scheduling system based on RabbitMQ, Redis and MongoDB. The task scheduling system comprises a task system service, Redis, MongoDB, RabbitMQ and an SDK, wherein the task system service comprises a data and object storage management module, a message queue management module, a task management module, a timing task module and a workflow module. By means of the design of a task system service, Redis, MongoDB, RabbitMQ and an SDK, the functions of an asynchronous task, a timing task and a workflow in a distributed and micro-service environment are provided. By means of the technical solution of a message queue, trigger-based scheduling of a task is realized, and a reliable and unique task scheduling capability exists; moreover, a timing task trigger capability exists, and thus the present invention is suitable for a scenario in which the execution of a certain task is triggered in a timing or delayed manner; a workflow capability is provided, and thus serial and parallel execution of a plurality of tasks can be performed; and capabilities such as task rollback and interruption are provided, and thus a system which unifies the asynchronous task, the timing task and the workflow is achieved.

Description

A distributed task scheduling system based on RabbitMQ, Redis and MongoDB Technical Field

[0001] The present invention relates to the fields of IT and software development, and in particular to a distributed task scheduling system based on RabbitMQ, Redis and MongoDB. Background Art

[0002] In the era of distribution, microservices, and cloud native, there is a demand for task asynchronization, scheduling, and workflow orchestration in sub-sectors such as system control, task processing, and process management. Therefore, the emergence of various message middlewares has solved the problem in distributed environments and provided solutions for asynchronous tasks. Subsequently, scheduled task controllers such as Crontab, Spring Task, and Quartz appeared, but they did not provide workflow solutions. The emergence of XXL-JOB provides both scheduled task capabilities and task dependency functions, partially meeting workflow scenarios. However, this solution cannot provide capabilities such as process rollback. Therefore, there is a lack of a system that unifies asynchronous tasks, scheduled tasks, and workflows. Summary of the Invention

[0003] The purpose of the present invention is to provide a distributed task scheduling system based on RabbitMQ, Redis and MongoDB to solve the problems raised in the above background technology.

[0004] To achieve the above-mentioned objectives, the present invention provides the following technical solutions: a distributed task scheduling system based on RabbitMQ, Redis and MongoDB, wherein the task scheduling system includes a task system service, Redis, MongoDB, RabbitMQ and SDK, wherein the Redis is used for distributed timing, the MongoDB is used for storing data, the RabbitMQ is used for message transmission, and the SDK is used for accessing the task scheduling system. The task system service includes a data and object storage management module, a message queue management module, a task management module, a timed task module and a workflow module, and the Redis includes a distributed timer module.

[0005] Preferably, the distributed timer module provides timing capabilities for task timeout retry and timed tasks, a timer is provided inside the distributed timer module, and the distributed timer module starts two coroutines when the task system service process.

[0006] Preferably, the two coroutines include a timed record acquisition loop and a recycling queue recovery loop.

[0007] Preferably, the data and object storage management module is used to store object data in MongoDB, and the object data includes Task objects of management tasks, timed task objects TimerTask, management workflows WorkflowDefinition and Workflow objects.

[0008] Preferably, the message queue management module is used to manage RabbitMQ ， The RabbitMQ is used as a message middleware.

[0009] Preferably, the task management module is used for creating, querying, deleting, publishing, retrying and status management of tasks. The task management module is used to manage Task objects, and the Task object consists of two parts: metadata Object and Spec.

[0010] Preferably, the scheduled task module is used to create, query, delete scheduled tasks and create asynchronous tasks when scheduled tasks expire.

[0011] Preferably, the scheduled task includes metadata Object and details TimerSchedulerSpec, and the details TimerSchedulerSpec include multiple modes, timing configuration, user data, and asynchronous task configuration for scheduled creation. The multiple modes include Timer mode, Ticker mode, and Crontab mode. The timing configuration includes Ticker time interval Duration, Crontab configuration CronSpec, and time zone location option.

[0012] When the Timer mode is used to create a scheduled task, a distributed timer is used to set the expiration time to the current time + Duration, and an asynchronous task is created after the expiration time.

[0013] The Ticker mode is used to create an asynchronous task after the expiration date, and insert an additional timer item with the expiration time + Duration to achieve regular interval triggering;

[0014] The Crontab pattern is used to generate the next expiration time for the next scheduled item using CronSpec syntax.

[0015] Preferably, the workflow module is used to provide the capability of executing multiple asynchronous task strings in parallel and the rollback and undo functions of the workflow.

[0016] Preferably, the SDK is used to provide a task registration interface, an asynchronous task creation interface, a result callback interface, a task processing process and a heartbeat maintenance of the system. The interface call between the SDK and the system adopts the GRPC protocol. The task system service is internally provided with a RESTful interface, and the RESTful interface is used to provide creation, viewing, modification, pause, deletion, start, approval, rejection and withdrawal functions.

[0017] Technical effects and advantages of the present invention:

[0018] The present invention provides a one-stop service for asynchronous tasks, scheduled tasks and workflows in a distributed and microservice environment through the design of task system services, Redis, MongoDB, RabbitMQ and SDK. It realizes the triggering and scheduling of tasks by adopting the technical solution of message queues, and has reliable and unique task scheduling. It also has the ability to trigger scheduled tasks, which is suitable for scenarios where the execution of a task is triggered on a scheduled or delayed basis. It has the ability to provide workflows, can execute multiple tasks in series or in parallel, and provides the ability to roll back and interrupt tasks, thereby realizing a unified system of asynchronous tasks, scheduled tasks and workflows. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG1 is a schematic diagram of the task scheduling system architecture of the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] The present invention provides a distributed task scheduling system based on RabbitMQ, Redis and MongoDB as shown in Figure 1. The task scheduling system includes a task system service, Redis, MongoDB, RabbitMQ and SDK. Redis is used for distributed timing, MongoDB is used to store data, RabbitMQ is used for message transmission, and SDK is used to access the task scheduling system. The task system service includes a data and object storage management module, a message queue management module, a task management module, a timed task module and a workflow module. Redis includes a distributed timer module.

[0022] Specifically, the SDK is used to provide task registration interface, asynchronous task creation interface, result callback interface, task processing process and system heartbeat maintenance. The interface call between the SDK and the system adopts the GRPC protocol. The task system service is internally set up with a RESTful interface. The RESTful interface is used to provide creation, viewing, modification, pause, deletion, start, approval, rejection and withdrawal functions. The architecture of the task system service decentralizes the work nodes to the access service, decoupling scheduling and execution. At the same time, the system as a whole provides an external interface without restricting the access language and architecture. There is no single point in the architectural design, so it can be horizontally expanded and can meet the task scheduling requirements of hundreds of thousands or even millions of tasks per second, solving the problems of single function and performance bottlenecks of existing products in many industries. The RESTful interface is specifically used to provide task creation, viewing, modification, and deletion functions, creation, viewing, modification, pause, and deletion functions of scheduled tasks, and creation, editing, deletion, start, approval, rejection, and withdrawal functions of workflows.

[0023] Specifically, RabbitMQ is a reliable, open source message broker and queue system used to support data transmission between distributed applications. Redis is an open source in-memory data structure storage system that not only supports key-value pair storage but also provides operations on multiple data structures. MongoDB is an open source, high-performance, document-based distributed database that supports multiple data types.

[0024] Specifically, the distributed timer module provides timing capabilities for task timeout retry and scheduled tasks. A timer is set inside the distributed timer module. The distributed timer module starts two coroutines when the task system service process is in progress. The two coroutines include a timing record acquisition loop and a recycling queue recovery loop.

[0025] Furthermore, the timing capability of the distributed timer module is implemented based on the Redis sorted set data structure and the Redis Lua script. The way to create a timing record is to insert a value that is the expiration time timestamp and member that is the record ID into the sorted set with the key suffix timerkey (hereinafter referred to as timerkey). Each task system service process starts two coroutines. In the timing record acquisition loop, the Lua script is called to atomically execute the following process: obtain a record with a value less than or equal to the current time from timerkey, delete the record from timerkey, and add the record with the value as the current time plus 5s timestamp and member as the record ID to the sorted set with the key suffix recyclekey (hereinafter referred to as recyclekey), and return the record ID. After the server obtains the expired record, it executes the corresponding handler. If the execution is successful, the record corresponding to recyclekey is deleted. If the execution fails, the loop is executed directly. The record will be restored by the recycling queue and restored to the timing queue for re-execution. If there is no expired record at present, sleep After 1 second, the execution loop process is recorded and the recycling queue is restored in the loop. The Lua script is called to execute the following process: a record with a value less than or equal to the current time is obtained from the recyclekey, the record is deleted from the recyclekey, and the record is added back to the timerkey. The value is the current time stamp. Timekey, recyclekey, and the expired record acquisition coroutine and the recycling queue recovery coroutine constitute a distributed timer module. The module can set the timer name, which is recorded in the timerkey and recyclekey suffixes. Timer modules with different names can execute different expired record processing processes. The module with the same name can be executed in multiple or all service processes. The above is the distributed timer method and implementation method.

[0026] Specifically, the data and object storage management module is used to store object data in MongoDB. The object data includes the Task object for managing tasks, the TimerTask object for managing timed tasks, the WorkflowDefinition and Workflow objects for managing workflows. The message queue management module is used to manage RabbitMQ as a message middleware.

[0027] Furthermore, MongoDB adopts a one-master-two-slave Replica deployment method. The message queue management module encapsulates the Golang driver amqp. In addition to the basic Exchange and Queue creation, binding, and deletion functions and message publishing and subscription functions, it also provides functions such as automatic reconnection of soft and hard connections in case of errors.

[0028] Specifically, the task management module is used to create, query, delete, publish, retry and manage the status of tasks. The task management module is used to manage Task objects, which consist of two parts: metadata Object and Spec.

[0029] Furthermore, Spec includes: 1. taskType is used to distinguish whether the task is an asynchronous task, a scheduled task or a workflow task; 2. channel is used as the key for task publishing and subscription, and different tasks use different channels; 3. generator ID indicates the ID of the scheduled task object or workflow object that generates the task; 4. Status; 5. progress; 6. retry configuration, including the number of retries, retry strategy, backoff strategy, etc.; 7. data; 8. result information; when the task is created, the retry timer will be set according to the retry configuration of the task configuration. If the configuration does not retry, it will be skipped, and then an MQ message will be sent. The expiration time of the message is consistent with the retry timer time, and the task status is set to Invalid. The task is executed in the business service. After the business service consumes the MQ message, it obtains the task details from the task management module and sets the status to Running, executes the processing function, and after the processing function is successfully executed, ack The MQ message also calls the Done(true) interface to the task management module, deletes the retry timer, and sets the task status to Success. After the processing function is successfully executed, the MQ message also calls the Done(false) interface to the task management module to set the task status to Fail. After waiting for the retry timer to time out, the MQ message is resent to allow the service executing the task to consume the message again and retry. When the processing service crashes, the process of another service will re-consume the MQ message and retry directly. The idempotence problem of the automatic retry function needs to be handled by the processing function. The above is the asynchronous task abstraction method and implementation method. This method is to asynchronousize the high-performance calls of the microservice architecture, use message queues for decoupling, and abstract such asynchronous calls into tasks for management.

[0030] Specifically, the scheduled task module is used to create, query, delete scheduled tasks and create asynchronous tasks when scheduled tasks expire. Scheduled tasks include metadata Object and details TimerSchedulerSpec. The details TimerSchedulerSpec include multiple modes, timing configuration, user data and asynchronous task configuration for scheduled creation. Multiple modes include Timer mode, Ticker mode and Crontab mode. Timing configuration includes Ticker time interval Duration. Crontab configures CronSpec and time zone location options. When creating a scheduled task in Timer mode, a distributed timer is used to set the expiration time to the current time + Duration, and an asynchronous task is created after the expiration. The Ticker mode is used to create an asynchronous task after the expiration and additionally insert a scheduled item with the end of the expiration time and the current time + Duration to achieve regular interval triggering. The Crontab mode is used to generate the next expiration time for the next scheduled item using CronSpec syntax.

[0031] Furthermore, TimerSchedulerSpec includes: 1. mode mode, including Timer mode, Ticker mode and Crontab mode; 2. timing configuration, Ticker time interval Duration, crontab configuration CronSpec and time zone location options; 3. user data Data; 4. asynchronous task configuration TaskSpec created at regular intervals. In Timer mode, when creating a scheduled task, a distributed timer is used to set the expiration time to the current time + Duration, and an asynchronous task is created after the expiration; in Ticker mode, in addition to creating an asynchronous task after the expiration, an additional timing item with the end of the expiration time and the current time + Duration is inserted to achieve regular interval departure; in Crontab mode, the next timing item uses CronSpec syntax to generate the next expiration time; the scheduled task can be deleted, and the timing item will be deleted accordingly after deletion. The work of the scheduled task is based on the two methods of distributed timer and asynchronous task abstraction. The key points create scheduled tasks, and regularly create corresponding asynchronous tasks, execute consumption tasks, and execute corresponding tasks.

[0032] Specifically, the workflow module is used to provide the capability of executing multiple asynchronous task strings in parallel and the rollback and cancellation functions of the workflow.

[0033] Furthermore, the workflow module contains two classes. WorkflowDefinition is used to edit and save workflow templates. It defines the node NodeTemplate of a workflow, the execution sequence relationship between each node, the task information executed by each node, and the node is distinguished by nodeType whether it is automatically executed or requires consent to execute. It is determined by stepBackAfterReject to which step the rollback will fall back; Workflow is the workflow record that is actually created and executed. It is created according to the corresponding WorkflowDefinition definition and its status is changed during execution. It contains the following contents: 1. name workflow name; 2. currentIndex current execution node; 3. nodes all nodes of the workflow is a two-dimensional array to realize the serial and parallel execution of nodes. The outer layer is a serial node array, and the inner layer represents the nodes of the unified step parallel execution; 4. nodesRe cord is a record of executed nodes, recording the execution results; 5. data is the user data of the workflow; 6. stage is the stage of the workflow, including not started, started, normal end, rejected and abnormal end; in addition to the content of NodeTemplate, the definition of each node also includes the state node state, and the node state includes the following states: Invalid illegal state, Ready preparation state, WaitOption waiting operation state, Working state, Rejecting rejection state, Withdrawing rollback state, Accepted completed state, Rejected rejection state, Withdrawn rollback state and Fail failure state. The workflow line of sight method is based on the distributed timer and asynchronous task abstraction, and extends the workflow definition function to enable different asynchronous tasks to be executed in parallel order according to the string set by the dotted line, and provide consent, rejection and revocation functions.

[0034] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A distributed task scheduling system based on RabbitMQ, Redis and MongoDB, characterized in that: The task scheduling system includes task system services, Redis, MongoDB, RabbitMQ and SDK, the Redis is used for distributed timing, the MongoDB is used to store data, the RabbitMQ is used for message transmission, and the SDK is used to access the task scheduling system. The task system service includes a data and object storage management module, a message queue management module, a task management module, a timed task module and a workflow module, and the Redis includes a distributed timer module.

2. A distributed task scheduling system based on RabbitMQ, Redis and MongoDB according to claim 1, characterized in that: The distributed timer module provides timing capabilities for task timeout retry and timing tasks. A timer is provided inside the distributed timer module. The distributed timer module starts two coroutines when the task system service process is in progress.

3. A distributed task scheduling system based on RabbitMQ, Redis and MongoDB according to claim 2, characterized in that: The two coroutines include a timing record acquisition loop and a recycling queue recovery loop.

4. A distributed task scheduling system based on RabbitMQ, Redis and MongoDB according to claim 1, characterized in that: The data and object storage management module is used to store object data in MongoDB, and the object data includes Task objects of management tasks, timed task objects TimerTask, management workflows WorkflowDefinition and Workflow objects.

5. A distributed task scheduling system based on RabbitMQ, Redis and MongoDB according to claim 1, characterized in that: The message queue management module is used to manage RabbitMQ, and the RabbitMQ is used for message middleware.

6. A distributed task scheduling system based on RabbitMQ, Redis and MongoDB according to claim 1, characterized in that: The task management module is used for creating, querying, deleting, publishing, retrying and status management of tasks. The task management module is used to manage Task objects, which are composed of two parts: metadata Object and Spec.

7. A distributed task scheduling system based on RabbitMQ, Redis and MongoDB according to claim 1, characterized in that: The scheduled task module is used to create, query, delete scheduled tasks and create asynchronous tasks when scheduled tasks expire.

8. A distributed task scheduling system based on RabbitMQ, Redis and MongoDB according to claim 7, characterized in that: The scheduled task includes metadata Object and details TimerSchedulerSpec, the details TimerSchedulerSpec includes multiple modes, timing configuration, user data and asynchronous task configuration created at a scheduled time, the multiple modes include Timer mode, Ticker mode and Crontab mode, the timing configuration includes Ticker time interval Duration, Crontab configuration CronSpec and time zone location option; When the Timer mode is used to create a scheduled task, a distributed timer is used to set the expiration time to the current time + Duration, and an asynchronous task is created after the expiration time. The Ticker mode is used to create an asynchronous task after expiration, and insert an additional timer item of the expiration time + Duration to achieve regular interval departure; The Crontab pattern is used to generate the next expiration time for the next scheduled item using CronSpec syntax.

9. The distributed task scheduling system based on RabbitMQ, Redis and MongoDB according to claim 1, characterized in that: The workflow module is used to provide the parallel execution capability of multiple asynchronous task strings and the rollback and cancellation functions of the workflow.

10. The distributed task scheduling system based on RabbitMQ, Redis and MongoDB according to claim 1, characterized in that: The SDK is used to provide task registration interface, asynchronous task creation interface, result callback interface, task processing process and system heartbeat maintenance. The interface call between the SDK and the system adopts the GRPC protocol. The task system service is internally provided with a RESTful interface, and the RESTful interface is used to provide creation, viewing, modification, suspension, deletion, start, approval, rejection and withdrawal functions.

Images