Batch task remote scheduling method based on RabbitMQ

Abstract

The invention discloses a remote scheduling method for batch tasks based on RabbitMQ, which comprises the following steps: task definition: defining and packaging the batch tasks, and integrating the defined task related information to form an independent and manageable task object; the task objects defined in batches are classified and stored in corresponding sets according to the task types and the batch task number set by the user, and the task state is marked as a to-be-published state; task publishing is carried out, wherein the packaged task object set is integrally taken out and sequentially published into RabbitMQ; performing task routing and distribution; performing task execution; a task execution result message is received, the task information table is updated, the task state is processed, and the result information of the batch task is packaged and returned; and performing execution monitoring and exception handling: monitoring the execution condition of the task in real time through a monitoring component, and handling and recording the occurred exception in time.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to a batch task remote scheduling method based on RabbitMQ. Background Technology

[0002] With the rapid development of information technology, enterprises and organizations are facing an increasing demand for batch task processing. Batch tasks can be, for example, image processing websites where users upload a large number of images that need to be compressed, cropped, and processed; large websites generate massive amounts of logs daily that need to be stored, analyzed, and reported in batches; and security systems perform data synchronization and migration between internal and external networks, as well as data cleaning and other processing tasks.

[0003] Traditional batch task scheduling methods often face many challenges, such as uneven task allocation, low system resource utilization, and limited ability to process large-scale tasks. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention provides a batch task remote scheduling method based on RabbitMQ to solve the problems of low efficiency, poor reliability and insufficient scalability in existing batch task scheduling.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A batch task remote scheduling method based on RabbitMQ includes the following steps:

[0007] Define tasks: Define and encapsulate batch tasks, generating information including unique task ID, message content, task type, task status, and priority. Integrate the defined task-related information to form an independent and manageable task object.

[0008] Perform task encapsulation: Categorize the batch-defined task objects according to task type and the number of batch tasks set by the user and store them in the corresponding collections, and mark the task status as pending release.

[0009] To publish a task: retrieve the entire collection of encapsulated task objects and publish them sequentially to RabbitMQ, record the task information table, and modify the task status to published status.

[0010] Perform task routing and allocation: Distribute task messages to the corresponding queues through RabbitMQ's routing mechanism;

[0011] Task execution: After receiving a task, the execution node performs the task and sends the execution results to a custom message receiving queue after completion.

[0012] Provide result feedback: Receive task execution result messages, update the task information table, process the task status, and encapsulate and return the result information of this batch of tasks;

[0013] Execution monitoring and exception handling: The monitoring component monitors the execution of tasks in real time and handles and records any exceptions in a timely manner.

[0014] In one possible implementation, the unique task ID is used to uniquely identify and track each specific task throughout the system; the message content is specific instructions, data, or other necessary information related to task execution; the task type is information that distinguishes task categories, used to uniformly process batch tasks according to task type; the task status includes any one of pending release, released, completed, or failed, used to reflect the progress of the task at different stages; and the priority is used to determine the order in which tasks are executed in the execution queue.

[0015] In one possible implementation, the step of sending the execution results to a custom message receiving queue after execution is completed specifically includes: when the execution node successfully completes the task, the processing results of the task are summarized and organized, and sent to a specially set custom message receiving queue. The custom message receiving queue is used to centrally collect the execution results of each task so that they can be processed, analyzed or fed back in a unified manner later.

[0016] In one possible implementation, receiving the task execution result message includes obtaining detailed information about the task execution status sent by the execution node, including whether the task was successfully completed and the data generated during the execution process.

[0017] In one possible implementation, updating the task information table includes updating the relevant information to the task information table after receiving the execution result, including recording the actual completion time of the task, a detailed description of the execution result, and whether any errors occurred.

[0018] In one possible implementation, the processing task status includes changing the task status based on the execution result: if the task is successfully completed, the status is marked as completed; if an error occurs or the expected goal is not achieved, it is marked as "failed" or "requires reprocessing".

[0019] In one possible implementation, the encapsulation and return of the result information of this batch task includes organizing and packaging the overall results of this batch task and preparing them for return in a unified and standardized format.

[0020] In one possible implementation, the monitoring component used to monitor the execution of the task in real time is RabbitMQ, a tool or module for collecting and analyzing task execution-related data, used to continuously track the progress of the task, including information on the task's current status, execution progress, and resource usage.

[0021] In one possible implementation, the timely handling and recording of anomalies includes tasks failing, execution times exceeding expectations, and task stagnation due to resource shortages. Handling methods include automatically restarting the task, reallocating resources, adjusting the task's execution priority, or notifying the administrator for manual intervention. At the same time, each anomaly is recorded in detail, including the time of occurrence, type, information about related tasks, and the handling measures taken.

[0022] The present invention has the following beneficial effects:

[0023] (1) Improve task scheduling efficiency: By utilizing RabbitMQ's asynchronous processing and high concurrency features, tasks can be quickly allocated and executed.

[0024] (2) Enhance system reliability: Ensure that tasks are not lost through message confirmation mechanism and persistent storage.

[0025] (3) Good scalability: It supports the dynamic addition of execution nodes, easily copes with the ever-increasing workload. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the steps of a batch task remote scheduling method based on RabbitMQ, according to an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] See Figure 1 The diagram shows a flowchart of a batch task remote scheduling method based on RabbitMQ according to an embodiment of the present invention, including the following steps:

[0029] S10, Define tasks: Define and encapsulate batch tasks, generating information including unique task ID, message content, task type, task status, and priority. Integrate the defined task-related information to form an independent and manageable task object.

[0030] S20, Perform task encapsulation: Store the batch-defined task objects in the corresponding collections according to the task type and the number of batch tasks set by the user, and mark the task status as pending release.

[0031] S30, Perform task publishing: Take out the encapsulated task object collection as a whole and publish it to RabbitMQ in sequence, record the task information table and modify the task status to published status;

[0032] S40 performs task routing and allocation: using RabbitMQ's routing mechanism, task messages are distributed to the corresponding queues;

[0033] S50, perform task execution: After receiving the task, the execution node performs the specific execution of the task, and after the execution is completed, the execution result is sent to the custom message receiving queue.

[0034] S60, provide result feedback: receive task execution result messages, update the task information table, process the task status and encapsulate and return the result information of this batch of tasks;

[0035] S70 performs execution monitoring and exception handling: The monitoring component monitors the execution status of tasks in real time and handles and records any exceptions in a timely manner.

[0036] Specifically, the task definition in S10 describes the processing of batch tasks in the initial stage, that is, defining and encapsulating tasks to form task objects with specific attributes. Batch tasks mean that they are not single, isolated tasks, but a collection of related or similar tasks. Defining these tasks involves clarifying key elements such as the task's goal, scope, input, and output. Therefore, task definitions generally include information such as a unique task ID, message content, task status, and priority. The defined task-related information is integrated together to form an independent, manageable unit, which is the task object.

[0037] Furthermore, a unique task ID is used to uniquely identify and track each specific task throughout the system, facilitating subsequent querying, management, and monitoring. The message content contains specific instructions, data, or other necessary information related to task execution. The task type is key information distinguishing task categories; it can be object information, allowing for unified processing of batch tasks based on task type. Task status can have multiple states, such as pending release, released, completed, and failed, reflecting the task's progress at different stages. Priority determines the order in which tasks are executed in the queue; higher-priority tasks may be processed first.

[0038] In a specific application example, such as in a facial recognition processing system, batch tasks can process a batch of facial information to be retrieved. Each facial recognition task has a unique ID, contains information such as facial details, and its status may be "pending retrieval." The priority can be set according to the urgency of the facial recognition requested by the client.

[0039] Specifically, in S20, task encapsulation is a crucial step in the processing and preparation of tasks. The batch-defined task objects represent numerous task objects with clearly defined attributes and requirements. Task type means categorizing tasks according to their nature, function, and operational characteristics, such as data processing tasks, file transfer tasks, and computation tasks. The user-defined batch task quantity indicates that the user can specify the number of tasks to process each time based on actual needs or the system's processing capacity. Categorized storage in corresponding sets means that different types and quantities of tasks are stored in separate storage areas, allowing for centralized management of similar tasks and improving system organization and efficiency. Marking a task status as "pending release" clearly informs the system that these tasks are ready but have not yet entered the actual execution or release stage. Tasks in the "pending release" state await subsequent scheduling and processing.

[0040] In a specific application example, such as in an image processing system, batch tasks might involve operations like cropping or color correction on different images. Based on the task type (cropping or color correction) and the user-defined quantity of 50 tasks to process each time, these tasks are categorized and stored in different collections, and marked as pending publication.

[0041] Specifically, in S30, task publishing is a crucial step in the task processing flow, marking the transition of a task from the preparation phase to the actual execution phase. Retrieving the entire set of encapsulated task objects means retrieving the entire prepared task set at once, rather than processing individual task objects. This improves publishing efficiency and reduces system overhead from frequent operations. RabbitMQ is a message queue system; publishing tasks to it ensures ordered task allocation and processing. Sequential publishing ensures tasks enter the processing flow in a specific order, avoiding confusion and conflicts. A task information table is used to record detailed information about published tasks. This table may contain various task attributes, such as task ID, publishing time, estimated completion time, and processing node. Recording this information facilitates subsequent task tracking, monitoring, and management. Timely updating the task status when it is changed to published status is crucial for the overall system's state management. The published status indicates that the task has been pushed into the processing flow, awaiting further allocation and execution.

[0042] Specifically, in S50, task execution is the core of the entire task processing flow, determining whether the task can achieve its intended goal. An execution node can be understood as a work unit or computing resource responsible for actually processing the task. When an execution node receives a task assigned to it, it begins preparations for execution. The actual execution of the task means that the execution node will utilize its computing power, data resources, and relevant algorithms to complete the actual processing of the task according to the operations and requirements specified by the task. For example, if it is a data computation task, the execution node may start the corresponding computing program to process the data; if it is a file transfer task, the execution node will establish a connection and execute the file transfer operation. After successfully completing the task, the execution node will summarize and organize the task processing results and send them to a specially designated custom message receiving queue. This queue is used to centrally collect the execution results of various tasks for subsequent unified processing, analysis, or feedback.

[0043] In a specific application example, in a distributed image recognition system, each execution node receives an image recognition task and uses its own image processing capabilities and recognition algorithms to perform the recognition operation. After completing the recognition, the recognition results (such as the identified object category, confidence level, etc.) are uniformly sent to a custom result receiving queue.

[0044] Specifically, in S60, result feedback is the final crucial step in the task processing flow, ensuring that task completion status is properly handled and communicated. Receiving task execution result messages means the system receives detailed information about the task execution status sent by the execution nodes. These messages contain key information such as whether the task was successfully completed, data generated during execution, or problems encountered. Updating the task information table means that after receiving the execution results, the system updates the relevant information in the task information table. This may include recording the actual completion time of the task, a detailed description of the execution results, and whether any errors occurred, ensuring accurate and complete recording of information throughout the task's entire lifecycle. Processing the task status involves changing the task's status based on the execution results. If the task is successfully completed, the status may be marked as "Completed"; if an error occurs or the expected goal is not achieved, it may be marked as "Failed" or "Requires Reprocessing," etc., for subsequent follow-up and processing. Encapsulating and returning the result information of this batch of tasks means organizing and packaging the overall results of this batch of tasks into a unified and standardized format for return to the user or other relevant systems. This allows the recipient to easily obtain and understand the execution status of the entire batch of tasks.

[0045] Specifically, in S70, execution monitoring and exception handling are crucial mechanisms for ensuring smooth task execution and stable system operation. The monitoring component is a tool or module in RabbitMQ used to collect and analyze task execution-related data, continuously tracking task progress, including current status, execution progress, resource usage, and other information. For example, it can monitor whether a task is progressing according to the predetermined timeline and whether it is consuming excessive computing resources or memory. Timely handling and recording of exceptions are essential. These exceptions may include task failure, execution time exceeding expectations, resource shortages causing task stagnation, and different actions taken depending on the specific task information. Handling methods may include automatically restarting the task, reallocating resources, adjusting task execution priority, or notifying the administrator for manual intervention. Simultaneously, each exception is recorded in detail, including the time and type of the exception, information about related tasks, and the measures taken. These records are invaluable for subsequent problem analysis, system optimization, and troubleshooting.

[0046] The batch task remote scheduling method based on RabbitMQ described above, through specific processes and strategies, fully utilizes the features of RabbitMQ to achieve efficient, stable, and scalable batch task remote scheduling.

[0047] It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A batch task remote scheduling method based on RabbitMQ, characterized in that, Includes the following steps: Define tasks: Define and encapsulate batch tasks, generating information including unique task ID, message content, task type, task status, and priority. Integrate the defined task-related information to form an independent and manageable task object. Perform task encapsulation: Categorize the batch-defined task objects according to task type and the number of batch tasks set by the user and store them in the corresponding collections, and mark the task status as pending release. To publish a task: retrieve the entire collection of encapsulated task objects and publish them sequentially to RabbitMQ, record the task information table, and modify the task status to published status. Perform task routing and allocation: Distribute task messages to the corresponding queues through RabbitMQ's routing mechanism; Task execution: After receiving a task, the execution node performs the task and sends the execution results to a custom message receiving queue after completion. Provide result feedback: Receive task execution result messages, update the task information table, process the task status, and encapsulate and return the result information of this batch of tasks; Execution monitoring and exception handling: The monitoring component monitors the execution of tasks in real time and handles and records any exceptions in a timely manner.

2. The batch task remote scheduling method based on RabbitMQ as described in claim 1, characterized in that, The unique task ID is used to uniquely identify and track each specific task throughout the system; the message content is the specific instructions, data, or other necessary information related to task execution; the task type is information that distinguishes task categories, used to process batch tasks uniformly according to task type; the task status includes any one of pending release, released, completed, or failed, used to reflect the progress of the task at different stages; the priority is used to determine the order in which tasks are executed in the execution queue.

3. The batch task remote scheduling method based on RabbitMQ as described in claim 1, characterized in that, The step of sending the execution results to a custom message receiving queue after execution is completed specifically includes: when the execution node successfully completes the task, the processing results of the task are summarized and organized, and sent to a specially set custom message receiving queue. The custom message receiving queue is used to centrally collect the execution results of each task so that they can be processed, analyzed or fed back in a unified manner later.

4. The batch task remote scheduling method based on RabbitMQ as described in claim 1, characterized in that, The receipt of task execution result messages includes obtaining detailed information about the task execution status sent by the execution node, including whether the task was successfully completed and the data generated during the execution process.

5. The batch task remote scheduling method based on RabbitMQ as described in claim 1, characterized in that, The updated task information table includes updating the relevant information to the task information table after receiving the execution result, including recording the actual completion time of the task, a detailed description of the execution result, and whether any errors occurred.

6. The batch task remote scheduling method based on RabbitMQ as described in claim 1, characterized in that, The processing task status includes changing the task status according to the execution result. If the task is successfully completed, the status is marked as completed; if an error occurs or the expected goal is not achieved, it is marked as "failed" or "re-processing is required".

7. The batch task remote scheduling method based on RabbitMQ as described in claim 1, characterized in that, The encapsulation and return of the results of this batch task includes organizing and packaging the overall results of this batch task in a unified and standardized format for return.

8. The batch task remote scheduling method based on RabbitMQ as described in claim 1, characterized in that, The monitoring component mentioned in the real-time monitoring of task execution is RabbitMQ, a tool or module used to collect and analyze task execution-related data, and to continuously track task progress, including information on the task's current status, execution progress, and resource usage.

9. The batch task remote scheduling method based on RabbitMQ as described in claim 1, characterized in that, The timely handling and recording of anomalies includes task execution failure, execution time exceeding expectations, and task stagnation due to resource shortages. Handling methods include automatically restarting the task, reallocating resources, adjusting the task execution priority, or notifying the administrator for manual intervention. At the same time, each anomaly is recorded in detail, including the time of occurrence, type, information of related tasks, and the handling measures taken.

Images