Message queue-based task scheduling method and computer readable storage medium

Abstract

The application discloses a task scheduling method based on a message queue and a computer readable storage medium, and the method comprises the following steps: according to a task request initiated by a business party, a message body in a preset format is constructed and sent to a message queue, the message body comprises a task index, a task list and a task execution mode, the task execution mode comprises sequential execution, parallel execution, a rollback operation and re-execution; the message queue is periodically scanned, a newly enqueued message body is obtained as a current message body; according to the task index, the task list and the task execution mode in the current message body, a current task is determined, and according to the task execution mode in the current message body, the current task is executed; according to the execution result of the current task, the task index and / or the task execution mode in the current message body are modified, a new message body is obtained and sent to the message queue, or the current message body is removed from the message queue. The application can improve the performance of message processing and the flexibility of task scheduling.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of task scheduling, in particular to a task scheduling method based on a message queue and a computer readable storage medium. BACKGROUND

[0002] From the previous monolithic architecture to the current microservice architecture, hundreds of services call and depend on each other. From the initial Internet with 100 online users on a server to the current WeChat with 1 billion daily active users. A solution is needed to decouple the relationship between services, control the reasonable and timely use of resources, and buffer traffic peaks, etc. For this purpose, a message queue is introduced, which is commonly used to achieve: asynchronous processing, service decoupling, and traffic control. The message queue mainly schedules tasks through asynchronous processing to achieve asynchronous processing, service decoupling, and traffic control.

[0003] In actual applications, sometimes some tasks are required to be asynchronous, and a group of tasks is synchronous, and the group of tasks and another group of tasks are asynchronous. At this time, synchronous task scheduling of the message queue needs to be implemented. The existing technical solution is to set the priority and dependency relationship in the message queue to ensure the order and processing of the message. After receiving a message, it is first determined whether another message with a higher priority exists. If it exists, it is not consumed. The defect of this solution is that polling and waiting may occur. For example, a group of tasks consists of 10 sub-tasks. Therefore, 10 messages are required. When the message with the highest priority is being consumed, the other 9 messages will be sent to the consumer. The consumer will determine to return and wait until the message with the highest priority is consumed. Only then will the next message be consumed. In this scenario, the number of messages is large. When the message with the highest priority is being consumed, the other messages will be polling and waiting. The polling and waiting will continuously send and receive messages, affecting the traffic and throughput of the message queue.

[0004] In addition, rollback operation is required for synchronous task scheduling. For example, if a 10-step task fails at the 5th step, the previous 4 steps that have been successfully executed need to be rolled back. The existing message queue can only implement distributed transactions for a group of tasks to be successfully executed or failed at the same time. It cannot be executed successfully in sequence and rolled back in sequence. It also cannot execute all tasks in the entire task group from the beginning. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a task scheduling method based on a message queue and a computer readable storage medium, which can improve the performance of message processing and the flexibility of task scheduling.

[0006] To solve the above technical problems, the technical solution adopted by the present application is as follows: a task scheduling method based on a message queue, comprising:

[0007] According to the task request initiated by the business party, a message body in a preset format is constructed, and the message body is sent to a message queue, the message body including a task index, a task list and a task execution mode, the task execution mode including sequential execution, parallel execution, rollback operation and re-execution;

[0008] Periodically scanning the message queue, a newly enqueued message body is obtained as a current message body;

[0009] According to the task index, the task list and the task execution mode in the current message body, a current task is determined, and according to the task execution mode in the current message body, the current task is executed;

[0010] According to the execution result of the current task, the task index and / or the task execution mode in the current message body are modified to obtain a new message body, and the new message body is sent to the message queue, or the current message body is removed from the message queue.

[0011] The application further provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method.

[0012] The application has the advantages that: a message body is encapsulated according to a business requirement and sent to a message queue, a message body is obtained by listening to the message queue and parsed, a corresponding task is executed according to a task index and a task execution mode, the task index is modified after the task is successfully executed and the modified task index is re-delivered to the message queue, so that the next task is continuously executed, and the tasks are executed in this way until all the tasks are executed, and the message body is removed from the message queue. The application can include multiple tasks in one message body, thereby greatly reducing the amount of message data. The re-delivery of the modified message body can eliminate the need for polling, thereby greatly reducing the flow and query performance loss of the system and improving the use flexibility. Since the message queue is asynchronously executed, the tasks corresponding to the same message body can be synchronously executed, thereby improving the flexibility of task scheduling. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 The figure is a flowchart of a task scheduling method based on a message queue according to the application;

[0014] Figure 2 The figure is a flowchart of the method according to the first embodiment of the application;

[0015] Figure 3 The figure is an implementation schematic diagram of the second embodiment of the application. DETAILED DESCRIPTION

[0016] To explain the technical content of the present application, the purposes and effects achieved, the following embodiments are combined with the drawings.

[0017] Please refer to Figure 1 A task scheduling method based on a message queue, comprising:

[0018] According to the task request initiated by the business party, a message body of a preset format is constructed, and the message body is sent to the message queue, the message body including a task index, a task list and a task execution mode, the task execution mode including sequential execution, parallel execution, rollback operation and re-execution;

[0019] Periodically scan the message queue to obtain a newly enqueued message body as a current message body;

[0020] According to the task index, the task list and the task execution mode in the current message body, a current task is determined, and the current task is executed according to the task execution mode in the current message body;

[0021] According to the execution result of the current task, the task index and / or the task execution mode in the current message body are modified to obtain a new message body, and the new message body is sent to the message queue, or the current message body is removed from the message queue.

[0022] From the above description, the beneficial effects of the present application are that the performance of message processing can be improved, and the flexibility of task scheduling can be improved.

[0023] Further, the task execution mode includes sequential execution, parallel execution, rollback operation and re-execution.

[0024] Further, according to the task index, the task list and the task execution mode in the current message body, the current task is determined, and the current task is executed according to the task execution mode in the current message body, specifically:

[0025] If the task execution mode in the current message body is sequential execution, rollback operation or re-execution, the corresponding task ID is obtained from the task list of the current message body according to the task index in the current message body, the task corresponding to the task ID is taken as the current task, and the current task is executed;

[0026] If the task execution mode in the current message body is parallel execution, the tasks corresponding to each task ID in the task list in the current message body are taken as the current tasks, and the current tasks are executed simultaneously.

[0027] From the above description, the tasks can be executed sequentially, and multiple tasks can also be executed simultaneously.

[0028] Further, the task index and / or the task execution mode in the current message body are modified according to the execution result of the current task to obtain a new message body, and the new message body is sent to the message queue, or the current message body is removed from the message queue, specifically:

[0029] If the current task is executed successfully, it is determined whether the tasks corresponding to the current message body have all been executed completely.

[0030] If not, the task index in the current message body is modified according to the task execution mode in the current message body to obtain a new message body, and the new message body is sent to the message queue.

[0031] If yes, the current message body is removed from the message queue.

[0032] If the current task fails, the current message body is modified according to the task execution mode in the current message body to obtain a new message body, and the new message body is sent to the message queue, or the current message body is removed from the message queue and an exception record is made.

[0033] Further, the task index in the current message body is modified according to the task execution mode in the current message body, specifically:

[0034] If the task execution mode in the current message body is sequential execution or re-execution, the value of the task index in the current message body is incremented by one.

[0035] If the task execution mode in the current message body is rollback operation, the value of the task index in the current message body is decremented by one.

[0036] Further, the current message body is modified according to the task execution mode in the current message body to obtain a new message body, and the new message body is sent to the message queue, or the current message body is removed from the message queue and an exception record is made, specifically:

[0037] If the task execution mode in the current message body is sequential execution or parallel execution, the task execution mode in the current message body is modified to rollback operation, or the task execution mode in the current message body is modified to re-execution, and the task index in the current message body is reset to 1 to obtain a new message body, and the new message body is sent to the message queue.

[0038] If the task execution mode in the current message body is rollback operation or re-execution, the current message body is removed from the message queue, and an exception information is recorded, the exception information including the current message body, the task ID of the current task and the failure reason.

[0039] From the above description, by performing the exception record, the abnormal information in the system running process can be captured and recorded in time, which helps the subsequent developers to quickly locate and solve the problems, and also helps the debugging and optimization of the system.

[0040] Further, after recording the exception information, further comprising:

[0041] According to the exception information, an alarm is performed.

[0042] From the above description, by performing the alarm, the alarm information can be automatically sent to the relevant personnel, so as to take timely measures to solve the problem.

[0043] Further, further comprising:

[0044] If the current task execution is successful, log recording is performed.

[0045] From the above description, by performing the log recording, the observability of the system can be enhanced, and the problem tracking and debugging are facilitated.

[0046] Further, further comprising:

[0047] If the task execution mode in the current message body is a rollback operation and the tasks corresponding to the current message body have all been successfully rolled back, the task execution mode in the current message body is restored to sequential execution or parallel execution, a new message body is obtained, and the new message body is sent to the message queue.

[0048] The application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the method.

[0049] Embodiment one

[0050] Please refer to Figure 2 , the embodiment one of the application is: a task scheduling method based on a message queue, which can be applied to a micro-service architecture. As shown in Figure 2 , the method of the embodiment comprises the following steps:

[0051] S1: according to a task request initiated by a business party, a message body in a preset format is constructed, and the message body is sent to a message queue.

[0052] In this embodiment, the format of the message body is {index, taskIds, mode}, wherein index represents a task index, i.e., a task sequence number, that is, the first task in the task list taskIds to be executed currently, for example, index of 1 represents the first task in the task list to be executed currently; taskIds represents a task list, i.e., a list of task ID numbers to be executed; and mode represents a task execution mode, different values of which correspond to different execution modes, in this embodiment, 0 represents sequential execution (synchronous execution), 1 represents parallel execution, 2 represents a rollback operation, and 3 represents re-execution.

[0053] For example, assuming that the business party needs to sequentially execute three tasks A, B and C, the task IDs of which are 1, 2 and 3 respectively, the constructed message body is {index: 1, taskIds: [1, 2, 3], mode: 0}; assuming that the business party needs to execute three tasks A, B and C in parallel, the constructed message body is {index: 1, taskIds: [1, 2, 3], mode: 1}; and assuming that the business party needs to rollback a task, the constructed message body is {index: 3, taskIds: [1, 2, 3], mode: 2}.

[0054] S2: periodically scan the message queue to obtain a newly enqueued message body as a current message body.

[0055] S3: determine a current task according to the task index, the task list and the task execution mode in the current message body, and execute the current task according to the task execution mode in the current message body.

[0056] Specifically, if the task execution mode in the current message body is sequential execution, a rollback operation or re-execution, a task corresponding to a task ID in the task list in the current message body is obtained according to the task index in the current message body, the task corresponding to the task ID is taken as the current task, and the current task is executed; if the task execution mode in the current message body is parallel execution, all tasks corresponding to all task IDs in the task list in the current message body are taken as the current task, and the current task is executed simultaneously, i.e., the tasks are executed simultaneously through multiple threads.

[0057] For example, assuming that the current message body is {index: 1, taskIds: [3, 6, 7], mode: 0}, i.e., tasks 3, 6 and 7 are sequentially executed, since the task index index is 1 at this time, the current task is the first task in the task list taskIds, i.e., task 3.

[0058] Assuming the current message body is {index: 1, taskIds: [3, 6, 7], mode: 1}, that is, tasks 3, 6 and 7 are executed in parallel, since the task execution mode at this time is parallel execution, the current task, that is, all tasks in the task list taskIds, that is, tasks 3, 6 and 7, start 3 threads to execute tasks 3, 6 and 7 at the same time.

[0059] Assuming the current message body is {index: 3, taskIds: [3, 6, 7], mode: 2}, that is, tasks 3, 6 and 7 are rolled back, since the task index index at this time is 3, the current task is the third task in the task list taskIds, that is, task 7. Task 7 is rolled back (each task is configured with a rollback method).

[0060] S4: Determine whether the current task is executed successfully, if yes, execute step S5, if not, execute step S8.

[0061] S5: Determine whether the tasks corresponding to the current message body have all been executed, if yes, execute step S7, if not, execute step S6.

[0062] Specifically, if the task execution mode in the current message body is sequential execution or re-execution, when the value of the task index is the same as the length of the task list, it indicates that the tasks corresponding to the current message body have all been executed.

[0063] If the task execution mode in the current message body is parallel execution, the completion of each task in the task list is obtained respectively, if all tasks are executed successfully, it indicates that the tasks corresponding to the current message body have all been executed.

[0064] If the task execution mode in the current message body is a rollback operation, when the value of the task index is 1, it indicates that the tasks in the task list have all been rolled back.

[0065] S6: Modify the task index in the current message body according to the task execution mode in the current message body, obtain a new message body, and send the new message body to the message queue.

[0066] Specifically, if the task execution mode in the current message body is sequential execution or re-execution, the value of the task index in the current message body is incremented by one; if the task execution mode in the current message body is a rollback operation, the value of the task index in the current message body is decremented by one.

[0067] For example, assuming the current message body is {index: 1, taskIds: [3, 6, 7], mode: 0}, after task 3 is successfully executed, the value of task index index is increased by one, i.e., the value of task index index is modified to 2, to obtain a new message body {index: 2, taskIds: [3, 6, 7], mode: 0}, and then the new message body is sent to the message queue, so that subsequent tasks 6 are executed according to the new message body. In this way, the subsequent tasks are executed according to the new message body. When task 7 is also successfully executed, it indicates that the tasks corresponding to the current message body have all been successfully executed, and the current message body can be removed from the message queue.

[0068] For example, assuming the current message body is {index: 1, taskIds: [3, 6, 7], mode: 0}, after task 3 is successfully executed, the value of task index index is increased by one, i.e., the value of task index index is modified to 2, to obtain a new message body {index: 2, taskIds: [3, 6, 7], mode: 0}, and then the new message body is sent to the message queue, so that subsequent tasks 6 are executed according to the new message body. In this way, the subsequent tasks are executed according to the new message body. When task 7 is also successfully executed, it indicates that the tasks corresponding to the current message body have all been successfully executed, and the current message body can be removed from the message queue.

[0069] S7: Remove the current message body from the message queue. That is, the tasks corresponding to the current message body have all been successfully executed, and the current message body can be removed from the message queue.

[0070] For example, assuming the current message body is {index: 1, taskIds: [3, 6, 7], mode: 1}, when the three tasks are all successfully executed, i.e., the tasks corresponding to the current message body have all been executed, the current message body can be removed from the message queue.

[0071] Further, in another optional embodiment, if the current message body is modified from a previous message body, and the task execution mode in the current message body is a rollback operation, after the tasks corresponding to the current message body have all been successfully rolled back, the task execution mode in the current message body can be restored to the task execution mode in the previous message body, and the corresponding tasks are executed according to the previous message body.

[0072] For example, assuming that the message body initially constructed according to the business requirement is {index: 1, taskIds: [3, 6, 7], mode: 0}, that is, sequentially execute task 3, task 6 and task 7, when task 6 is executed, the execution fails, and it is required to roll back the task and the tasks before it, therefore the task execution mode is modified to a rollback operation, and a new message body {index: 2, taskIds: [3, 6, 7], mode: 2} is obtained. Then task 6 and task 3 are rolled back in turn, when task 3 is successfully rolled back, the task execution mode is restored, a new message body {index: 1, taskIds: [3, 6, 7], mode: 0} is obtained, and then task 3, task 6 and task 7 are sequentially executed.

[0073] That is, in the case where the tasks corresponding to the current message body are all rolled back, the current message body can be directly removed from the message queue, or can be restored to the original task execution mode to continue executing the tasks.

[0074] S8: Determine whether the task execution mode in the current message body is sequential execution or parallel execution, if yes, execute step S9, if not, that is, the task execution mode in the current message body is a rollback operation or re-execution, execute step S10.

[0075] S9: Modify the current message body to obtain a new message body, and send the new message body to the message queue.

[0076] That is, according to the business requirement, the task execution mode in the current message body is modified to a rollback operation, at this time the task index does not need to be modified; or the task execution mode in the current message body is modified to re-execution, at this time the task index needs to be reset to 1.

[0077] S10: Remove the current message body from the message queue, and record the exception.

[0078] In a specific application scenario, the task execution mode in the message body constructed according to the business requirement is generally sequential execution or parallel execution, if a task fails in the middle, it is switched to a rollback operation or re-execution. If the task rollback fails or the re-execution fails, it is considered that an exception occurs, the tasks corresponding to the current message body are no longer processed, it is directly removed from the message queue, and the exception information is recorded. In the embodiment, the exception information includes the current message body, the task ID of the current task and the failure reason, and can also include the input parameter and the output parameter of the current task.

[0079] Further, after the current task is executed successfully, logging is also performed. Through exception recording and logging, abnormal information and logs in the system running process can be captured and recorded in time, which helps developers quickly locate and solve problems, and also helps system debugging and optimization.

[0080] Further, after exception recording, an alarm is performed, that is, an alarm message is automatically sent to relevant personnel to take timely measures to solve the problem.

[0081] In the embodiment, by encapsulating the message body, different task execution modes are set, so that the tasks corresponding to the same message body can be executed synchronously, asynchronously, or concurrently. Through free switching configuration, tasks can be rolled back and redone in sequence, which has more advantages than traditional distributed transactions. Compared with the existing scheme in which multiple tasks require multiple messages and other messages need to be constantly polled except the message being consumed, the embodiment can greatly reduce the amount of message data by including multiple tasks in only one message body. Moreover, by modifying the message body for re-delivery, polling can not be needed, thereby greatly reducing the traffic and query performance loss of the system and improving the use flexibility. By recording all operation logs and abnormal information, the observability of the system can be enhanced, helping developers quickly locate and solve problems. By using a message queue as middleware, the design of the system can be simplified, and decoupling and coordination between different systems or components can be achieved, reducing the development and maintenance cost of the system. Since the message queue itself is executed asynchronously, that is, the message body in the message queue is executed asynchronously, while the tasks corresponding to the same message body can be executed synchronously, the flexibility of task scheduling can be improved.

[0082] Embodiment Two

[0083] The embodiment is a specific implementation scenario of the above-mentioned embodiment.

[0084] As shown in Figure 3 , the embodiment includes the following parts:

[0085] 1. Message encapsulator; the message encapsulator is used to construct a suitable message body according to the business demand.

[0086] 2. Message sender; the message sender is used to send the message body to the message queue. In the embodiment, the message queue can be an internal message queue service such as Kafka, RabbitMQ, or Amazon SQS, or a custom message queue service.

[0087] 3. Controller; the controller is the brain of the entire task scheduling center, responsible for managing and controlling tasks in the message queue. There can be multiple controllers listening to different types of message bodies, obtaining the message body and calling the task processor to execute the corresponding task, and performing corresponding processing according to the task execution result, such as modifying the message body and re-releasing it to the message queue, recording exceptions, sending alarm information, or performing result analysis.

[0088] 4. Task processor; the task processor is a consumer system or component for executing tasks. The controller passes the parsed message body to the consumer system or component of the work center, which executes the corresponding task. The consumer system or component can be multiple concurrent instances to improve the throughput and response performance of the system. The task processor executes tasks according to different task execution modes. After the task execution is completed, the execution result is fed back to the controller.

[0089] 5. Exception handling and log monitoring; in the entire task scheduling center, an exception handling and log recording mechanism needs to be designed and implemented to capture and record abnormal information and error logs in the system running process in a timely manner; these information can help developers quickly locate and solve problems, and also help system debugging and optimization. Further, when an abnormal situation occurs, the exception handling and log recording mechanism can automatically send alarm information to relevant personnel to take timely measures to solve the problem.

[0090] Example three

[0091] This embodiment is a computer readable storage medium corresponding to the above embodiment, which stores a computer program, the program is executed by a processor to realize the steps of the task scheduling method based on the message queue in the above embodiment, and the same technical effect can be achieved, which will not be repeated here.

[0092] In summary, the task scheduling method based on the message queue provided by the present application can include multiple tasks in only one message body, which can greatly reduce the amount of message data; and by modifying the message body and re-delivering it, polling can be avoided, which can greatly reduce the traffic and query performance loss of the system, and also improve the use flexibility; at the same time, since the message queue itself is executed asynchronously, i.e. the message body in the message queue is executed asynchronously, and the tasks corresponding to the same message body can be executed synchronously, which can improve the flexibility of task scheduling; by recording exceptions and logs, it is helpful for subsequent developers to quickly locate and solve problems, and also helps system debugging and optimization.

[0093] The above merely illustrates the embodiments of the present application, and is not intended to limit the patent scope of the present application. Any equivalent transformation or direct or indirect application in the related technical field based on the content of the present application specification and drawings is also included in the patent protection scope of the present application.

Claims

1. A message queue based task scheduling method, characterized in that, The application relates to a task execution method and device. According to a task request initiated by a business party, a message body in a preset format is constructed, and the message body is sent to a message queue, wherein the message body comprises a task index, a task list and a task execution mode, and the task execution mode comprises sequential execution, parallel execution, rollback operation and re-execution. The message queue is periodically scanned to obtain a newly-queued message body as a current message body. According to the task index, the task list and the task execution mode in the current message body, a current task is determined, and the current task is executed according to the task execution mode in the current message body. According to the execution result of the current task, the task index and / or the task execution mode in the current message body are modified to obtain a new message body, and the new message body is sent to the message queue or the current message body is removed from the message queue. If the current task is successfully executed, it is judged whether all tasks corresponding to the current message body have been executed. If not, the task index in the current message body is modified according to the task execution mode in the current message body to obtain a new message body, and the new message body is sent to the message queue. If yes, the current message body is removed from the message queue. If the current task is unsuccessfully executed, the current message body is modified according to the task execution mode in the current message body to obtain a new message body, and the new message body is sent to the message queue or the current message body is removed from the message queue, and abnormal information is recorded. If the task execution mode in the current message body is sequential execution, rollback operation or re-execution, a task ID corresponding to the task index in the current message body is obtained from the task list in the current message body, the task corresponding to the task ID is taken as the current task, and the current task is executed.

2. The message queue based task scheduling method according to claim 1, wherein, If the task execution mode in the current message body is parallel execution, all tasks corresponding to the task IDs in the task list in the current message body are taken as the current tasks, and the current tasks are simultaneously executed. If the task execution mode in the current message body is sequential execution or re-execution, the value of the task index in the current message body is increased by one. If the task execution mode in the current message body is rollback operation, the value of the task index in the current message body is decreased by one.

3. The message queue based task scheduling method of claim 1, wherein, If the current task is successfully executed, it is judged whether all tasks corresponding to the current message body have been executed. If not, the task index in the current message body is modified according to the task execution mode in the current message body to obtain a new message body, and the new message body is sent to the message queue. If yes, the current message body is removed from the message queue.

4. The message queue based task scheduling method of claim 1, wherein, If the current task is unsuccessfully executed, the current message body is modified according to the task execution mode in the current message body to obtain a new message body, and the new message body is sent to the message queue or the current message body is removed from the message queue, and abnormal information is recorded. If the task execution mode in the current message body is sequential execution or parallel execution, the task execution mode in the current message body is modified to a rollback operation, or the task execution mode in the current message body is modified to re-execution, while the task index in the current message body is reset to 1, a new message body is obtained, and the new message body is sent to the message queue; If the task execution mode in the current message body is a rollback operation or re-execution, the current message body is removed from the message queue, and exception information is recorded, the exception information including the current message body, the task ID of the current task, and the failure cause.

5. The message queue based task scheduling method of claim 1, wherein, After the exception information is recorded, the method further comprises: According to the exception information, an alarm is given.

6. The message queue based task scheduling method of claim 1, wherein, Further comprising: If the current task is executed successfully, log recording is performed.

7. The message queue based task scheduling method of claim 1, wherein, Further comprising: If the task execution mode in the current message body is a rollback operation and all tasks corresponding to the current message body have been successfully rolled back, the task execution mode in the current message body is restored to sequential execution or parallel execution, a new message body is obtained, and the new message body is sent to the message queue.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-7.

Images