Lightweight distributed mutual exclusion task scheduling system based on RabbitMQ and Redis

Abstract

The invention relates to the technical field of task scheduling, in particular to a lightweight distributed mutual exclusion task scheduling system based on RabbitMQ and Redis. Comprising a node management module used for managing nodes in a cluster, the nodes comprise a main node and task nodes, the main node is used for sending a scheduling instruction of a task, and the task nodes are used for performing polling competition on the main node and executing the scheduling instruction issued by the main node; the task receiving module is used for receiving a task submitted by a service system and sending the task to the node management module; the node management module comprises a task distribution module, and the task distribution module identifies task types submitted by a service system through task nodes, queries whether message queues with the same tasks exist in a message queue RabbitMQ or not, sends the tasks to the corresponding message queues if the message queues with the same tasks exist, and creates the message queues with the task types if the message queues with the same tasks do not exist. The lightweight mutual exclusion task scheduling system can be provided, and the requirements of application scenes can be met.

Description

Technical Field

[0001] The present invention relates to the technical field of task scheduling, and in particular to a lightweight distributed mutually exclusive task scheduling system based on RabbitMQ and Redis. Background Art

[0002] A distributed system is a software system built on a network. Because of the characteristics of software, a distributed system has a high degree of cohesion and transparency.

[0003] In Java projects, common task scheduling systems are XXL-Job and Power Job, which are scheduled task scheduling systems used to call Java methods or scripts built into the system, while common message queues such as RabbitMQ and Kafka are just simple distributed message queues.

[0004] There is an application scenario in a distributed system where tasks with the same attributes cannot be executed in parallel in the distributed system. The existing open source distributed task scheduling system is too large to meet the needs of this application scenario. A lightweight mutually exclusive task scheduling system is urgently needed to meet the needs of this application scenario. Summary of the invention

[0005] The technical problem solved by the present invention is to provide a lightweight distributed mutually exclusive task scheduling system based on RabbitMQ and Redis. A lightweight mutually exclusive task scheduling system can be provided to meet the needs of application scenarios.

[0006] The basic solution provided by the present invention is: a lightweight distributed mutually exclusive task scheduling system based on RabbitMQ and Redis, including a node management module, a task acceptance module, a task allocation module, a listener management module and a task management module;

[0007] A node management module is used to manage nodes in the cluster, wherein the nodes include a master node and a task node. The master node is used to send a scheduling instruction for a task, and the task node polls the competing master node to execute the scheduling instruction issued by the master node.

[0008] The task acceptance module is used to accept tasks submitted by the business system and send them to the node management module;

[0009] The node management module includes the task allocation module

[0010] The task allocation module identifies the task type submitted by the business system through the task node, and queries whether there is a message queue with the same task in the message queue RabbitMQ. If so, the task is sent to the corresponding message queue. If not, a message queue of the task type is created.

[0011] The listener management module determines whether there is a listener for the task message queue. If not, the cluster task queue listener is reallocated through the master node;

[0012] The task management module is used for the task nodes to start destroying the monitoring and adding new monitoring operations according to the destroy and add message queue monitoring instructions issued by the master node. After receiving the task, each task node receives the task in the message queue one by one. After receiving the task, each node scans the task type in the business system and executes the task.

[0013] The principle and advantage of the present invention is that a lightweight task scheduling system is implemented through RabbitMQ and Redis to serially execute tasks with the same attributes in a distributed system. In addition, the distributed task system can be automatically elastically scaled, unlike Zookeeper, where half of the cluster will be unavailable if it is offline.

[0014] Further, the node management module includes a node startup module;

[0015] The node startup module is used to start two task nodes, including node 1 and node 2. The task cluster is automatically formed through node 1 and node 2. Each node submits the node survival status to the Rdeis cache and continuously refreshes the node survival status.

[0016] Furthermore, the task allocation module is specifically used to refresh the submission status of type A task to the Redis cache when node 1 receives type A task submitted by the business system, and query whether there is a message queue A that executes type A task from the message queue RabbitMQ. If so, the task is sent to message queue A; if not, message queue A is created and the task is dropped to message queue A.

[0017] Further, the listener management module;

[0018] After receiving the message that needs to be based on the cluster task queue, it is used to recalculate the message queue that each node should listen to through the currently surviving nodes in the cluster, the message queues that the cluster should survive, and the message queues currently listened to by each node, to ensure that each message queue has only one listener and is evenly distributed to each node, and to notify each node through the message bus of the message queue listeners that need to be added or deleted.

[0019] Furthermore, it also includes an offline management module, which is used to make other task nodes compete for the master node when the master node is offline, and then recalculate and allocate cluster task queue monitoring.

[0020] Furthermore, the offline management module is also used for the master node to determine whether the cluster status has changed through cyclic timing tasks when the task node is online or offline. If it has changed, it will also trigger recalculation and allocation of cluster task queue monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a logical block diagram of an embodiment of a lightweight distributed mutually exclusive task scheduling system based on RabbitMQ and Redis of the present invention;

[0022] Figure 2 This is a logical block diagram of an embodiment of a lightweight distributed mutually exclusive task scheduling system based on RabbitMQ and Redis of the present invention. DETAILED DESCRIPTION

[0023] The following is further described in detail through specific implementation methods:

[0024] The symbols in the drawings of the specification include:

[0025] The embodiment is basically as shown in the attached Figure 1 and attached Figure 2 As shown:

[0026] A lightweight distributed mutually exclusive task scheduling system based on RabbitMQ and Redis, including a node management module, a task acceptance module, a task allocation module, a listener management module and a task management module;

[0027] A node management module is used to manage nodes in the cluster, wherein the nodes include a master node and a task node. The master node is used to send a scheduling instruction for a task, and the task node polls the competing master node to execute the scheduling instruction issued by the master node.

[0028] The task acceptance module is used to accept tasks submitted by the business system and send them to the node management module;

[0029] The node management module includes the task allocation module

[0030] The task allocation module identifies the task type submitted by the business system through the task node, and queries whether there is a message queue with the same task in the message queue RabbitMQ. If so, the task is sent to the corresponding message queue. If not, a message queue of the task type is created.

[0031] The listener management module determines whether there is a listener for the task message queue. If not, the cluster task queue listener is reallocated through the master node;

[0032] The task management module is used for the task nodes to start destroying the monitoring and adding new monitoring operations according to the destroy and add message queue monitoring instructions issued by the master node. After receiving the task, each task node receives the task in the message queue one by one. After receiving the task, each node scans the task type in the business system and executes the task.

[0033] The principle and advantage of the present invention is that a lightweight task scheduling system is implemented through RabbitMQ and Redis to serially execute tasks with the same attributes in a distributed system. In addition, the distributed task system can be automatically elastically scaled, unlike Zookeeper, where half of the cluster will be unavailable if it is offline.

[0034] The node management module includes a node startup module;

[0035] The node startup module is used to start two task nodes, including node 1 and node 2. The task cluster is automatically formed through node 1 and node 2. Each node submits the node survival status to the Rdeis cache and continuously refreshes the node survival status.

[0036] The task allocation module is specifically used to refresh the submission status of type A tasks to the Redis cache when node 1 receives type A tasks submitted by the business system, and to query from the message queue RabbitMQ whether there is a message queue A that executes type A tasks. If so, the task is sent to message queue A; if not, message queue A is created and the task is dropped to message queue A.

[0037] The listener management module;

[0038] After receiving the message that needs to be based on the cluster task queue, it is used to recalculate the message queue that each node should listen to through the currently surviving nodes in the cluster, the message queues that the cluster should survive, and the message queues currently listened to by each node, to ensure that each message queue has only one listener and is evenly distributed to each node, and to notify each node through the message bus of the message queue listeners that need to be added or deleted.

[0039] It also includes an offline management module, which is used to enable other task nodes to compete for the master node when the master node is offline, and then recalculate and allocate cluster task queue monitoring.

[0040] The offline management module is also used for the master node to determine whether the cluster status has changed through cyclic timing tasks when the task node is online or offline. If it has changed, it will also trigger recalculation and allocation of cluster task queue monitoring.

[0041] The above are only embodiments of the present invention. Common knowledge such as the known specific structures and characteristics in the scheme is not described in detail here. Ordinary technicians in the relevant field know all the common technical knowledge in the technical field to which the invention belongs before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply conventional experimental means before that date. Ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the scope of protection of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A lightweight distributed mutually exclusive task scheduling system based on RabbitMQ and Redis, characterized by: It includes node management module, task acceptance module, task allocation module, listener management module and task management; A node management module is used to manage nodes in the cluster, wherein the nodes include a master node and a task node. The master node is used to send a scheduling instruction for a task, and the task node polls the competing master node to execute the scheduling instruction issued by the master node. The task acceptance module is used to accept tasks submitted by the business system and send them to the node management module; The node management module includes the task allocation module The task allocation module identifies the task type submitted by the business system through the task node, and queries whether there is a message queue with the same task in the message queue RabbitMQ. If so, the task is sent to the corresponding message queue. If not, a message queue of the task type is created. The listener management module determines whether there is a listener for the task message queue. If not, the cluster task queue listener is reallocated through the master node; The task management module is used for the task nodes to start destroying the monitoring and adding new monitoring operations according to the destroy and add message queue monitoring instructions issued by the master node. After receiving the task, each task node receives the task in the message queue one by one. After receiving the task, each node scans the task type in the business system and executes the task.

2. According to a RabbitMQ and Redis-based lightweight distributed mutually exclusive task scheduling system according to claim 1, it is characterized in that: The node management module includes a node startup module; The node startup module is used to start two task nodes, including node 1 and node 2. The task cluster is automatically formed through node 1 and node 2. Each node submits the node survival status to the Rdeis cache and continuously refreshes the node survival status.

3. A lightweight distributed mutually exclusive task scheduling system based on RabbitMQ and Redis according to claim 2, characterized in that: The task allocation module is specifically used to refresh the submission status of type A tasks to the Redis cache when node 1 receives type A tasks submitted by the business system, and to query from the message queue RabbitMQ whether there is a message queue A that executes type A tasks. If so, the task is sent to message queue A; if not, message queue A is created and the task is dropped to message queue A.

4. A lightweight distributed mutually exclusive task scheduling system based on RabbitMQ and Redis according to claim 3, characterized in that: The listener management module; After receiving the message that needs to be based on the cluster task queue, it is used to recalculate the message queue that each node should listen to through the currently surviving nodes in the cluster, the message queues that the cluster should survive, and the message queues currently listened to by each node, to ensure that each message queue has only one listener and is evenly distributed to each node, and to notify each node through the message bus of the message queue listeners that need to be added or deleted.

5. A lightweight distributed mutually exclusive task scheduling system based on RabbitMQ and Redis according to claim 4, characterized in that: It also includes an offline management module, which is used to enable other task nodes to compete for the master node when the master node is offline, and then recalculate and allocate cluster task queue monitoring.

6. A lightweight distributed mutually exclusive task scheduling system based on RabbitMQ and Redis according to claim 5, characterized in that: The offline management module is also used for the master node to determine whether the cluster status has changed through cyclic timing tasks when the task node is online or offline. If it has changed, it will also trigger recalculation and allocation of cluster task queue monitoring.

Images