Distributed timing task scheduling method based on message queue and electronic device

Abstract

The application relates to a distributed timing task scheduling method and system based on a message queue, which comprises the following steps: setting a scheduling console, wherein the scheduling console comprises one main scheduler node and at least one standby scheduler node; establishing a plurality of message queues, each of which has a unique identifier; the main scheduler node receives a timing task creation instruction, generates a scheduling message, adds an identifier, and puts the scheduling message into the message queue corresponding to the identifier after the scheduling message reaches the execution time condition; setting a scheduling client, adding an identifier on the operation class of the scheduling client, and binding different operation classes to the message queues corresponding to the identifiers when the scheduling client starts; and the scheduling client listens to the message queue bound thereto and executes the scheduling message in the message queue. Compared with the prior art, the application designs a high-availability scheduling console and a high-availability and load-balancing scheduling client to accept scheduling instructions and execute scheduling tasks.

Description

Technical Field

[0001] This invention relates to a scheduling scheme for timed tasks, and in particular to a distributed timed task scheduling method and system based on message queues. Background Technology

[0002] As software applications become increasingly feature-rich, backend systems often encounter data that requires offline or scheduled processing. Examples include timeout status checks in order systems, scheduled updates of cached data, scheduled emails sent to users, and even periodically calculated reports. Consequently, scheduled task processing functionality inevitably becomes a default requirement for many systems.

[0003] There are many existing scheduling solutions, the most common being setting scheduled tasks through the operating system. For example, traditional Linux scheduled tasks are implemented based on Crontab, while Windows uses the Task Scheduler feature. In the Java ecosystem, the Quartz scheduled task framework has also emerged. These frameworks meet most current scheduling needs, but relatively speaking, these frameworks only work well in single-machine scenarios and perform poorly in multi-machine scenarios. On the one hand, both the scheduler and the scheduling client can become points of failure, and once the scheduler crashes, the entire scheduling system will cease to function. On the other hand, due to the limited performance of a single machine, if there are many scheduling tasks, the execution performance of a single machine is very likely to become the system bottleneck. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art by providing a distributed timed task scheduling method and electronic device based on message queues.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A distributed timed task scheduling method based on message queues includes the following steps:

[0007] A scheduling console is set up, which includes one master scheduler node and at least one backup scheduler node. The backup scheduler node switches to become the master scheduler node when the master scheduler node fails.

[0008] Establish multiple message queues, each with a unique identifier;

[0009] The master scheduler node receives a command to create a scheduled task, generates a scheduling message according to the command, adds an identifier to the scheduling message, and puts the scheduling message into the message queue corresponding to its identifier after the scheduling message meets the execution time condition.

[0010] Configure a scheduling client, add identifiers to the operation classes of the scheduling client, and when the scheduling client starts, different operation classes are bound to the message queues corresponding to their identifiers;

[0011] The scheduling client listens to the message queue it is bound to and executes the scheduling messages in the message queue.

[0012] Furthermore, the scheduling console includes a status monitoring module, a node selection module, and multiple scheduler nodes. The node selection module is used to select a master scheduler node from the multiple scheduler nodes, and the status monitoring module is used to monitor the status of the master scheduler node.

[0013] Furthermore, the node selection module selects the master scheduler node from multiple scheduler nodes as follows: multiple scheduler nodes elect a master node through ZooKeeper, and the master node is used as the master scheduler node.

[0014] Furthermore, the master scheduler node and each alternative scheduler node are equipped with different scheduling schemes, and the master scheduler node adds an identifier to the scheduling message according to the scheduling scheme it is equipped with.

[0015] Furthermore, the alternative scheduler node obtains the scheduling message generated and an identifier added by the master scheduler node in real time.

[0016] Furthermore, it also includes:

[0017] Establish a dead-letter queue to store unprocessed scheduled messages that have timed out from each message queue;

[0018] The master scheduler node listens to the dead-letter queue and retryes or terminates the scheduling messages in the dead-letter queue.

[0019] Furthermore, it also includes:

[0020] A message acknowledgment queue is established, and the scheduling client places a success message or a failure message into the message acknowledgment queue after executing the scheduling message;

[0021] The master scheduler node listens to the message acknowledgment queue and retryes or terminates the scheduling messages in the message acknowledgment queue.

[0022] Furthermore, the same message queue can be bound to multiple scheduling clients.

[0023] Furthermore, the master scheduler node is connected to the scheduling client and monitors the status of the scheduling client.

[0024] An electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; wherein:

[0025] Memory, used to store computer programs;

[0026] The processor is used to execute the above-described distributed timed task scheduling method steps by running the program stored in memory.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] (1) A highly available scheduling console was designed. The scheduling console is in a one-master-multiple-standby mode, including one master scheduler node and at least one standby scheduler node. When the current master scheduler node fails, a new master scheduler node can be selected from the standby scheduler nodes, thus avoiding the collapse of the entire scheduling system due to scheduler failure.

[0029] (2) The scheduling tasks are distributed through multiple message queues and identifiers. At the same time, the scheduling client supports horizontal expansion. Multiple scheduling clients can be bound to the same message queue, which realizes the balanced distribution of scheduling tasks and avoids single point of failure of the scheduling client.

[0030] (3) Design a dead letter queue. Once a task is not processed for a certain period of time, it will enter the dead letter queue to avoid task blocking. Design a message confirmation queue. Each time a scheduled task is successfully or unsuccessfully processed, the scheduling result must be fed back to the scheduling console for further processing. The dead letter queue and the message confirmation queue can provide feedback on the execution status of the scheduled task. Attached Figure Description

[0031] Figure 1 This is a schematic diagram illustrating the application of a distributed timed task scheduling method. Detailed Implementation

[0032] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, providing detailed implementation methods and specific operating procedures. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them, and the scope of protection of the present invention is not limited to the following embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] The term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the invention. In the description of the invention, it should be understood that the terms "first," "second," and "third," etc., in the specification, claims, and accompanying drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0034] This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or server products, the method can be executed in the order shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment), or the execution order of steps without timing constraints can be adjusted.

[0035] This application provides a distributed scheduled task method based on message queues, such as... Figure 1 As shown, it includes the following steps:

[0036] Set up a scheduling console, which includes one master scheduler node and at least one backup scheduler node. The backup scheduler node will switch to become the master scheduler node when the master scheduler node fails.

[0037] Establish multiple message queues, each with a unique identifier;

[0038] The master scheduler node receives the instruction to create a scheduled task, generates a scheduling message based on the instruction, adds an identifier to the scheduling message, and puts the scheduling message into the message queue corresponding to its identifier after the scheduling message meets the execution time condition.

[0039] Configure a scheduling client, add identifiers to the operation classes of the scheduling client, and when the scheduling client starts, different operation classes are bound to the message queues corresponding to their identifiers;

[0040] The scheduling client listens to the message queue it is bound to and executes the scheduling messages in the message queue.

[0041] In this application, the core of the scheduling console is high availability, which is a one-master-multiple-standby mode. The scheduling console includes a status monitoring module, a node selection module, and multiple scheduler nodes. The node selection module is used to select the master scheduler node from multiple scheduler nodes. The status monitor is used to monitor the status of the master scheduler node to determine whether the current master scheduler node has failed. When a failure occurs, the node selection module selects a new master scheduler node from the standby scheduler nodes.

[0042] In this application, all scheduler nodes request the master node lock from ZooKeeper. Upon successful request, only one machine becomes the master (i.e., the primary scheduler node), and the others become backup master nodes (i.e., backup scheduler nodes). If the existing master fails, another backup machine can take over as the new master. Only the master node is responsible for receiving and sending scheduling instructions; the backup scheduler nodes do not operate. These backup scheduler nodes can be configured with Cron expressions to trigger scheduling tasks at specified times and send the corresponding scheduling messages to the scheduling queue.

[0043] The scheduling client is the executor of scheduling operations. The scheduling client needs to develop corresponding operation classes that implement scheduling tasks, and annotate these operation classes with a unique key for the scheduling task. When the application starts, different annotations will bind the operation classes to different message queues. When a message arrives in the corresponding queue, it indicates that a scheduling message has arrived. The scheduling client retrieves the scheduling message and executes the corresponding operation class. It's understandable that a scheduling client can have multiple operation classes, therefore a single scheduling client will listen to multiple message queues.

[0044] Message queues inherently enable the distribution of scheduling messages, achieving balanced distribution. Simultaneously, scheduling clients support horizontal scaling; multiple scheduling clients can be bound to the same message queue. The master scheduler node connects to the scheduling clients and monitors their status, preventing single points of failure from affecting message execution. Even if a single scheduling client fails, the scheduling messages in the message queue can still be executed through other bound scheduling clients.

[0045] Multiple message queues are established based on the RabbitMQ server. Different scheduled messages are sent to the Exchange with corresponding Routing Keys. The RabbitMQ Exchange then distributes the message to the designated consumer queue.

[0046] At the same time, a dead-letter queue is established to store scheduling messages that have timed out and not been processed in each message queue. Once a task has not been processed for a certain period of time, it will enter the dead-letter queue to avoid task blocking. The main scheduler node listens to the dead-letter queue and retryes or terminates the scheduling messages in the dead-letter queue.

[0047] Simultaneously, a message acknowledgment queue is established. After the scheduling client executes the scheduling message, it puts the success message or failure message into the message acknowledgment queue. Each time a scheduling task is successfully or unsuccessfully processed, the scheduling result must be fed back to the scheduling control end for further processing. The main scheduler node listens to the message acknowledgment queue and retryes or terminates the scheduling messages in the message acknowledgment queue.

[0048] In this embodiment, as Figure 1 As shown, two scheduler nodes are started. After starting, one of them will be elected as the master through Zookeeper to provide services to the outside world, while the other is a standby node.

[0049] The system generates a scheduled task creation command and sends it to the scheduling console through manual triggering, third-party API triggering, or user-configured scheduling. The scheduled task is then configured on the scheduling console page. Key information involved includes:

[0050] Table 1 Key Information of Scheduled Tasks

[0051]

[0052]

[0053] Implement a specific task execution class in the scheduling client, such as OrderExpiredProcessing, and annotate the class with the corresponding routingKey as orderExpired. Start the scheduling client, which will automatically bind to the RabbitMQ consumer queue, which will listen for scheduling messages from the Exchange with routingKey = orderExpired.

[0054] The scheduling console triggers a scheduling message after the execution time condition is met. For example, triggering a "timeout order processing" message with routingKey = orderExpired. After this message is passed to RabbitMQ, the scheduling client will receive the corresponding scheduling message and execute the corresponding operation class to trigger the specific operation. The specific scheduling message structure is as follows:

[0055]

[0056] If a scheduling message is executed successfully, the scheduling client will send a success message back to the "message acknowledgment queue." The master node will listen to messages in this queue and set the corresponding status to success. If a scheduling message is executed unsuccessfully, the scheduling client will send a failure message back to the "message acknowledgment queue." The master node will listen to messages in this queue. If a retry setting is in place, the scheduling message will be resent to the queue, and the number of retries will be recorded. If the scheduling message no longer meets the retry requirements, the scheduling will be terminated.

[0057] The console allows users to save data to the database during each scheduling message sending and receiving operation, enabling effective tracking of scheduling message execution.

[0058] In this application, the scheduling console and scheduling client effectively implement the distribution and retry of scheduling messages based on reasonable queue settings. At the same time, both the scheduling console and scheduling client can be deployed at multiple levels, achieving high availability.

[0059] In a further design, the master scheduler node and each alternative scheduler node are equipped with different scheduling schemes. The master scheduler node adds an identifier to the scheduling message according to its own scheduling scheme, thereby avoiding the situation where a certain scheduling problem causes the master scheduler node and the new master scheduler node to have the same failure, thus improving the reliability of the scheduling console.

[0060] After receiving a scheduled task creation instruction, the master scheduler node generates a scheduling message based on the execution time conditions, required resources, and other personalized configuration information recorded in the instruction. It then adds an identifier to the scheduling message according to the scheduling scheme, associating the message with the identifier with the corresponding message queue. Upon meeting the execution time conditions, the message is placed into its associated message queue. The scheduler node's scheduling scheme can be a load-balanced scheme, considering the number of scheduling clients bound to each message queue, the available resources of the scheduling clients, the number of scheduling messages already associated with each message queue, their required resources, and the personalized configuration information of the scheduling messages, all while adding identifiers to the scheduling messages. Alternatively, it can be a scheme that minimizes the number of scheduling clients and maximizes utilization, considering the resource usage of each scheduling client at different time periods, the execution time conditions of the scheduling messages, and their required resources. Other scheduling schemes that consider different factors are also possible.

[0061] To ensure a smooth switchover to the backup scheduler node after a primary scheduler node failure, the backup scheduler node receives scheduling messages generated and identified by the primary scheduler node in real time. This ensures that the scheduling of already generated scheduling messages is not affected after the switchover. Specifically, a separate set of backup delay queues with different time granularities can be established, independent of each scheduler node. Scheduling messages generated and identified by the primary scheduler node are placed into the corresponding delay queue in real time based on their execution time. The backup scheduler node can access this set of delay queues. Thus, even if the primary scheduler node fails, its generated and identified scheduling messages can still be found in this set of delay queues, facilitating scheduling control by the new primary scheduler node.

[0062] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0063] Furthermore, a portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. The program instructions invoking the methods of this application may be stored in a fixed or removable recording medium, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in the working memory of a computer device operating according to the program instructions. Here, one embodiment of this application includes an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the apparatus is triggered to operate the methods and / or technical solutions based on the foregoing embodiments of this application.

[0064] This application also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; wherein:

[0065] Memory, used to store computer programs;

[0066] The processor is used to execute the above-described distributed timed task scheduling method steps by running the program stored in memory.

[0067] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A distributed timed task scheduling method based on message queues, characterized in that, Includes the following steps: A scheduling console is set up, which includes one master scheduler node and at least one backup scheduler node. The backup scheduler node switches to become the master scheduler node when the master scheduler node fails. Establish multiple message queues, each with a unique identifier; The master scheduler node receives a command to create a scheduled task, generates a scheduling message according to the command, adds an identifier to the scheduling message, and puts the scheduling message into the message queue corresponding to its identifier after the scheduling message meets the execution time condition. Configure a scheduling client, add identifiers to the operation classes of the scheduling client, and when the scheduling client starts, different operation classes are bound to the message queues corresponding to their identifiers; The same message queue can be bound to multiple scheduling clients, and the master scheduler node is connected to the scheduling clients and monitors the status of the scheduling clients; The scheduling client listens to the message queue it is bound to and executes the scheduling messages in the message queue.

2. The distributed timed task scheduling method based on message queues according to claim 1, characterized in that, The scheduling console includes a status monitoring module, a node selection module, and multiple scheduler nodes. The node selection module is used to select a master scheduler node from the multiple scheduler nodes, and the status monitoring module is used to monitor the status of the master scheduler node.

3. The distributed timed task scheduling method based on message queues according to claim 2, characterized in that, The node selection module selects the master scheduler node from multiple scheduler nodes as follows: multiple scheduler nodes elect a master node through ZooKeeper, and the master node is used as the master scheduler node.

4. The distributed timed task scheduling method based on message queues according to claim 1, characterized in that, The master scheduler node and each of the alternative scheduler nodes are equipped with different scheduling schemes. The master scheduler node adds an identifier to the scheduling message according to the scheduling scheme it is equipped with.

5. A distributed timed task scheduling method based on a message queue according to claim 4, characterized in that, The alternative scheduler node receives scheduling messages generated and labeled with identifiers from the main scheduler node in real time.

6. The distributed timed task scheduling method based on message queues according to claim 1, characterized in that, Also includes: Establish a dead-letter queue to store unprocessed scheduled messages that have timed out from each message queue; The master scheduler node listens to the dead-letter queue and retryes or terminates the scheduling messages in the dead-letter queue.

7. A distributed timed task scheduling method based on a message queue according to claim 1, characterized in that, Also includes: A message acknowledgment queue is established, and the scheduling client places a success message or a failure message into the message acknowledgment queue after executing the scheduling message; The master scheduler node listens to the message acknowledgment queue and retryes or terminates the scheduling messages in the message acknowledgment queue.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other through the communication bus; wherein: Memory, used to store computer programs; A processor for executing the method steps of any one of claims 1 to 7 by running a program stored in memory.

Images