Distributed Timed Task Scheduling Method and System Based on Message Queue

Abstract

The present invention relates to a distributed timed task scheduling method based on a message queue, which comprises the following steps: S1. A timed task production module creates different timed tasks according to different scheduling message types and delivers messages; S2. A timed task processing module schedules the timed tasks delivered by the timed task production module; S3. A timed task configuration module sets up a dead letter queue and a group of delay queues with different time granularities; S4. The delay queue specifies a survival time for the incoming timed tasks; S5. A timed task consumption module listens to and consumes the dead letter queue. The advantages of the present invention are as follows: Without using a database for data persistence of timed scheduling tasks, the present invention enables the business system to create different types of timed task scheduling tasks according to the actual business scenario, ensuring the timeliness and integrity of the business in the timed scenario, and at the same time avoiding the problem of repeated scheduling of timed tasks in a distributed cluster environment.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and specifically to a distributed timed task scheduling method and system based on a message queue. Background Art

[0002] With the government's strong promotion of the construction of the electronic government procurement system, the scenario of using timed task scheduling in the process of electronic bidding and tendering activities is increasing. In the existing electronic bidding and tendering system during the bidding activity process, for scenarios such as reminding unbid suppliers to upload bidding documents 48 hours before the project bidding deadline, reminding suppliers who have successfully signed up but have not downloaded the tender documents within 5 minutes, reminding registered suppliers to enter the bidding hall 30 minutes before remote bidding, and timeout of the call number in the supplier evaluation waiting hall, etc., the timed function is required to perform task scheduling.

[0003] Currently, relatively popular distributed timed task frameworks include Quartz, xx-job, etc. They have their own advantages and disadvantages. For example, Quartz supports data storage-based timed tasks, and the task maintainability is relatively high. The disadvantage is that Quartz depends on the database cluster, has a library table dependency, and is a relatively old implementation method. XX-job supports elastic expansion and fault migration. The disadvantage is that the framework itself is relatively heavy, not pure enough, also has a library table dependency, and requires maintaining a relatively large number of information items.

[0004] If using a pure delay queue to implement timed tasks, due to the characteristic of the queue being first in first out (FIFO), assuming that all messages are in the same queue, only the first message at the top of the queue will be consumed preferentially. If the first message has not expired, the subsequent expired messages will have to wait and will be consumed only after the first message is consumed. Summary of the Invention

[0005] The purpose of the present invention is to provide a high-performance and highly available distributed timed task scheduling method and system based on a message queue.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A distributed timed task scheduling method based on a message queue, which includes the following steps:

[0008] S1. The timed task production module creates different timed tasks according to different scheduling message types and delivers messages;

[0009] S2. The timed task processing module schedules the timed tasks delivered by the timed task production module, executes the timed task scheduling when the execution time is reached, otherwise delivers them to the timed task configuration module;

[0010] S3. Set up a dead-letter queue and a group of delay queues with different time granularities in the timed task configuration module; when the timed task processing module schedules, calculate the time difference between the current time and the execution time of the timed task, and deliver the timed task to the next-level delay queue with the closest time difference interval to it.

[0011] S4. The delay queue specifies a survival time for the incoming timed task, and the survival time is consistent with the time granularity of the delay queue. The timed task enters the dead-letter queue after reaching the survival time.

[0012] S5. The timed task consumption module listens to and consumes the dead-letter queue, and determines whether to redeliver the timed task; when the timed task needs to be redelivered, it returns to the timed task processing module, and repeats steps S2 - S5 until the timed task reaches the execution time, triggering the execution of the timed task scheduling.

[0013] A distributed timed task scheduling system based on a message queue, which includes

[0014] A timed task production module, which creates different timed tasks according to different scheduling message types and delivers messages.

[0015] A timed task configuration module, which is used to configure a dead-letter queue and a group of delay queues with different time granularities; the delay queue specifies a survival time for the incoming timed task, and the survival time is consistent with the time granularity of the delay queue. The timed task enters the dead-letter queue after reaching the survival time.

[0016] A timed task processing module, which is used to schedule the timed tasks delivered by the timed task production module and the timed tasks returned by the dead-letter queue, execute the timed task scheduling after reaching the execution time, otherwise deliver them to the timed task configuration module; when delivering to the timed task configuration module, the timed task processing module calculates the time difference between the current time and the execution time of the timed task, and delivers the timed task to the next-level delay queue with the closest time difference interval to it.

[0017] A timed task consumption module, which is used to listen to and consume different message queues and determine whether to redeliver the messages.

[0018] Compared with the prior art, the advantages of the present invention are as follows: The present invention can, without using a database to persist data for timed scheduling tasks, enable the business system to create different types of timed task scheduling tasks according to the actual business (such as an electronic bidding system) scenario, ensure the timeliness and integrity of the business in the timed scenario, and at the same time avoid the problem of repeated scheduling of timed tasks in a distributed cluster environment.

[0019] 1. The scheduling message persistence does not rely on the database storage form. Relying on the data storage mechanism of RabbitMQ, the message data will be persisted to the disk, ensuring that the data can still be restored after the service crashes and restarts.

[0020] 2. The service framework is a lightweight structure. Utilizing the natural integration advantage of Spring Boot with RabbitMQ, it does not require excessive external dependencies. Only the relevant dependencies of RabbitMQ need to be introduced to use it. Brief Description of the Drawings

[0021] Figure 1 It is a schematic diagram of the configuration of the timed task.

[0022] Figure 2 It is a flowchart of the timed task scheduling.

[0023] Figure 3 It is a schematic diagram of the timed task scheduling. Detailed Implementation Manner

[0024] The content of the present invention will be described in detail below in conjunction with the drawings of the specification and the embodiments:

[0025] As Figures 1-3 shown is a schematic diagram of an embodiment of a distributed timed task scheduling method and system based on a message queue provided by the present invention.

[0026] A distributed timed task scheduling method based on a message queue, which includes the following steps:

[0027] Step S1. The timed task production module creates different timed tasks according to different scheduling message types and delivers messages.

[0028] Step S2. The timed task processing module schedules the timed tasks delivered by the timed task production module. When the execution time is reached, the timed task scheduling is executed; otherwise, it is delivered to the timed task configuration module.

[0029] Step S3. A dead letter queue and a group of delay queues with different time granularities are set in the timed task configuration module. When the timed task processing module schedules, it calculates the time difference between the current time and the execution time of the timed task, and delivers the timed task to the next-level delay queue with the closest time difference interval. For example, if the time difference is 5 hours and there are 6-hour and 2-hour delay queues set, the timed task will be delivered to the 2-hour delay queue. If the timed task is delivered to the upper-level delay queue with the closest time difference interval, it will cause the message to expire.

[0030] The number of delay queues and the configuration of time granularities in the timed task configuration module are adjusted according to the business situation.

[0031] A reasonable time granularity delay queue configuration can effectively avoid the risk of exceptions occurring during the transmission and consumption of messages in the queue. It is not the case that the more delay queue configurations, the better.

[0032] The timing task configuration module contains at least one delay queue with a time granularity of 1 second.

[0033] Step S4. The delay queue specifies a survival time for the incoming timing tasks. The survival time is consistent with the time granularity of the delay queue. When the timing task reaches the survival time, it enters the dead letter queue.

[0034] The timing tasks entering the same delay queue have the same specified survival time. The survival time is similar to a countdown. From the moment the timing task enters the delay queue, the survival time starts counting down. After the countdown is completed, the timing task is directly delivered to the dead letter queue. For timing tasks in the same delay queue at different times, the remaining time of their survival time is different.

[0035] The delay queue is not configured with consumers. When the timing task reaches the survival time, it directly enters the dead letter queue.

[0036] According to the dead letter exchange (DLX) feature of the message queue, when the message reaches the expiration time (TTL), through the dead letter related configuration of the message, the message will be delivered to the specified dead letter queue. The DLX and TTL are used to simulate the delay queue function, that is, after the message is sent to the delay queue, no consumers are configured for the delay queue. After waiting until the expiration time, the message will be delivered to the dead letter queue.

[0037] Step S5. The timing task consumption module listens to and consumes the dead letter queue, and determines whether to redeliver the timing task. When the timing task needs to be redelivered, it returns to the timing task processing module, and steps S2 - S5 are repeated until the timing task reaches the execution time, triggering the execution of the timing task scheduling.

[0038] The timing tasks include one-time timing tasks and periodic tasks.

[0039] The periodic task supports the Cron expression. The Cron expression is a string composed of seven sub - expressions, and the format rule is: second minute hour day of month month day of week year (optional), such as executing a scheduling task every 5 minutes (** / 5***?). In the expression, "*" specifies all values. For example, if "minute" is set to "*", it means every minute. " / " is a special unit, indicating "every". For example, "0 / 5" means executing once every 5 minutes, where "0" means starting from 0 minutes.

[0040] The one-time scheduled task specifies the execution time in the message body (such as 9:00 am one day later). The scheduling task supports address calls (which can be http, rpc, etc.), and also supports passing in parameters during task scheduling by setting custom callback parameters.

[0041] The structure of the scheduled task message body is as follows. The scheduled task type is specified in scheduleWayDto. For example, a periodic scheduling task is executed once every 10 seconds. The task scheduling parameters are specified in paramObj. scheduleUrl is the address of the scheduling task, and payload is the custom callback parameter, which will be carried during scheduling for invocation.

[0042]

[0043]

[0044] Running instance of the one-time scheduled task:

[0045] Suppose the scheduled task configuration module includes 7-day delay queues, 1-day delay queues, 5-hour delay queues, 1-hour delay queues, 30-minute delay queues, 5-minute delay queues, 1-minute delay queues, 30-second delay queues, 5-second delay queues, and 1-second delay queues.

[0046] If the scheduled task needs to be executed 3 hours later, the scheduled task message is delivered to the 1-hour delay queue. After reaching the survival time, the scheduled task enters the dead-letter queue. The scheduled task consumption module redelivers the scheduled task to the scheduled task processing module. At this moment, the execution time of the scheduled task becomes 2 hours later. The scheduled task message is then delivered to the 1-hour delay queue. Repeat the above judgment logic. When the scheduled task reaches the execution time, the execution of the scheduled task scheduling is triggered.

[0047] When the periodic task reaches the execution time and triggers the execution of the task scheduling, the periodic task is redelivered to the delay queue by the scheduled task processing module and waits for the next execution time.

[0048] Running instance of the periodic task:

[0049] If a periodic task is executed every 30 minutes starting from 0 minutes, the cron expression is "0 0 / 30 ***? *". It is specified that the delay queues are 10 minutes, 5 minutes, 1 minute, etc. When the system receives a periodic task message, it judges the difference between the current time and the execution time. If the difference from the execution time is 8 minutes, the timed task message will be delivered to the next-level delay queue, that is, the 5-minute queue. When the timed task message reaches the execution time and is executed, the system judges that the timed task message is a periodic task, and then delivers the timed task message to the delay queue again to wait for the next execution. This repeated execution and scheduling constitute a periodic task.

[0050] The timed task configuration module is connected to the server cluster through the Direct routing mode. The server cluster includes several nodes, and each node includes a timed task production module and a timed task processing module.

[0051] The persistence of message content depends on the message confirmation (ACK) mechanism of RabbitMQ itself. That is, from the consumer receiving the message to the system successfully performing timed scheduling and then feeding back successful consumption, RabbitMQ will delete the message from the queue only after receiving the feedback. In the case of a service cluster, the message confirmation (ACK) mechanism ensures that when a consumer encounters a server exception or other exceptional situations, RabbitMQ will push the message to other consumers, and RabbitMQ will delete the message data only after the consumer sends an ACK feedback.

[0052] RabbitMQ mainly supports three routing modes: Direct, Fanout, and Topic. The present invention uses the unicast (Direct) routing mode. In the Direct mode, the message consumption mode is a competitive one. When a queue is configured with multiple consumers, a message will be consumed by only one consumer, ensuring that there will be no situation where a message is repeatedly consumed when the system deploys a service cluster.

[0053] The present invention mainly improves on the basis of the RabbitMQ message mechanism and uses related mechanism features such as the RabbitMQ dead letter exchanger (DLX), message expiration (TTL), and message ACK to design a distributed timed task scheduling system based on a message queue, which includes:

[0054] A timed task production module, which creates different timed tasks according to different scheduling message types and delivers messages.

[0055] A timed task configuration module, which is used to configure a dead letter queue and a group of delay queues with different time granularities; the delay queue specifies a survival time for the incoming timed task, and the survival time is consistent with the time granularity of the delay queue. The timed task enters the dead letter queue after reaching the survival time.

[0056] A scheduled task processing module, which is used to schedule the scheduled tasks delivered by the scheduled task production module and the scheduled tasks returned by the dead letter queue, execute the scheduled task scheduling when the execution time is reached, otherwise deliver them to the scheduled task configuration module; when delivering to the scheduled task configuration module, the scheduled task processing module calculates the time difference between the current time and the scheduled task execution time, and delivers the scheduled task to the next-level delay queue with the closest time difference interval; the scheduled task processing module can perform various address call methods (such as http, rpc, etc.) according to the input parameters.

[0057] A scheduled task consumption module, which is used to listen and consume different message queues and judge whether to redeliver the messages.

[0058] A scheduled expression parsing module, which is used to parse the scheduling expressions in the task message body of the scheduled task processing module; periodic tasks follow the Cron expression rules, and one-time scheduled tasks specify the specific execution time.

[0059] The distributed scheduled task scheduling system based on the message queue of the present invention achieves the following purposes:

[0060] 1. Through the message persistence mechanism of RabbitMQ, solve the dependence of traditional scheduled task scheduling on the database.

[0061] 2. Through the message consumption mechanism of RabbitMQ, improve the support of the scheduled task scheduling system for the distributed cluster environment.

[0062] 3. Through the simplicity of the integration of the RabbitMQ framework by Spring boot, solve the problem of the overweight of traditional scheduled task frameworks.

[0063] 4. Through the agreement of the message structure, solve the scheduled tasks in various task scenarios such as periodic scheduling and one-time scheduling.

Claims

1. A distributed timed task scheduling method based on a message queue, characterized in that It includes the following steps: S1. The timed task production module creates different timed tasks according to different scheduling message types and delivers messages; S2. The timed task processing module schedules the timed tasks delivered by the timed task production module, executes the timed task scheduling when the execution time is reached, otherwise delivers them to the timed task configuration module; S3. The timed task configuration module sets up a dead letter queue and a group of delay queues with different time granularities; When the timed task processing module schedules, it calculates the time difference between the current time and the execution time of the timed task, and delivers the timed task to the next-level delay queue with the closest time difference interval; S4. The delay queue specifies a survival time for the incoming timed task, and the survival time is consistent with the time granularity of the delay queue. The timed task enters the dead letter queue after reaching the survival time; S5. The timed task consumption module listens to and consumes the dead letter queue, and determines whether to redeliver the timed task; when the timed task needs to be redelivered, it returns to the timed task processing module, and steps S2 - S5 are repeated until the timed task reaches the execution time and triggers the execution of the timed task scheduling.

2. The distributed timing task scheduling method based on a message queue according to claim 1, characterized in that: The timed tasks include one-time timed tasks and periodic tasks.

3. The distributed timed task scheduling method based on a message queue according to claim 2, characterized in that: When the periodic task reaches the execution time and triggers the execution of the task scheduling, the periodic task is redelivered to the delay queue by the timed task processing module and waits for the next execution time.

4. The distributed timed task scheduling method based on a message queue according to claim 1, characterized in that: The configuration of the number and time granularity of the delay queues in the timed task configuration module is adjusted according to the business situation.

5. The distributed timed task scheduling method based on a message queue according to claim 1, characterized in that: The timed task configuration module contains at least one delay queue with a time granularity of 1 second.

6. The distributed timed task scheduling method based on a message queue according to claim 1, characterized in that: The timed task configuration module includes a 7-day delay queue, a 1-day delay queue, a 5-hour delay queue, a 1-hour delay queue, a 30-minute delay queue, a 5-minute delay queue, a 1-minute delay queue, a 30-second delay queue, a 5-second delay queue, and a 1-second delay queue.

7. The distributed timed task scheduling method based on a message queue according to claim 1, characterized in that: The delay queue is not configured with consumers, and the timed task directly enters the dead letter queue after reaching the survival time.

8. The distributed timed task scheduling method based on a message queue according to any one of claims 1-7, characterized in that: The timed task configuration module connects to the server cluster through the Direct routing mode. The server cluster includes several nodes, and each node includes a timed task production module and a timed task processing module.

9. A distributed timed task scheduling system based on a message queue, characterized in that, It includes a timed task production module that creates different timed tasks according to different scheduling message types and delivers messages; A timed task configuration module for configuring a dead letter queue and a group of delay queues with different time granularities; The delay queue specifies a survival time for the incoming timed task, and the survival time is consistent with the time granularity of the delay queue. The timed task enters the dead letter queue after reaching the survival time; A timed task processing module for scheduling the timed tasks delivered by the timed task production module and the timed tasks returned by the dead letter queue, executing the timed task scheduling when the execution time is reached, otherwise delivering them to the timed task configuration module; when delivering to the timed task configuration module, the timed task processing module calculates the time difference between the current time and the execution time of the timed task, and delivers the timed task to the next-level delay queue with the closest time difference interval; The timed task consumption module is used to listen and consume different message queues and determine whether to redeliver messages.

10. The distributed timed task scheduling system based on a message queue according to claim 9, characterized in that: It also includes a timed expression parsing module, which is used to parse the scheduling expression in the task message body of the timed task processing module; periodic tasks follow the Cron expression rules, and one-time timed tasks specify the specific execution time.

Images