Distributed processing method and system

Abstract

The invention discloses a distributed processing method and system. The method comprises a task center and at least one task processing node. According to task distribution and execution processes, tasks in different stages are divided into seven states: to-be-processed, downloading, processing, uploading, processing completion, to-be-retry and processing termination. The task center is responsible for receiving tasks, distributing the tasks and managing task processing nodes, and can record registration information of the task processing nodes and start and stop the task processing nodes according to node information and task load conditions; and the task processing node is responsible for receiving and processing tasks. A task processing node of the method adopts a task processing process based on a parallel assembly line, a task dynamic fault tolerance and grading retry mechanism is provided on this basis, and when a problem occurs in task processing or the task processing node quits legally or illegally, the task processing process is completed. And performing state-level recovery or task-level recovery on the task according to the task state and the error frequency in task processing, recovering the task to the state before the error, and continuing to process. According to the method, the resource utilization rate of task processing nodes can be effectively improved, negative effects caused by problems such as node faults and task processing errors are reduced, and the task processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a distributed processing method and system. Background Art

[0002] Distributed processing is a computing method for handling large-scale, complex tasks. It distributes tasks across multiple computers in a network, improving efficiency through collaborative work. Compared to single-machine processing, distributed systems can reduce the pressure on individual nodes, shorten processing time, and enhance reliability, offering significant advantages in large-scale computing scenarios. Traditional distributed systems are usually composed of task schedulers and processing nodes. Their design often has the following limitations in practical applications: First, the task lifecycle (submission, download, processing, and upload) is executed serially with files as the core. The time required for file transfer before and after processing is heavily dependent on network bandwidth. When the task file is large or the node load is high, the processing time will be compounded by the transmission delay, and the overall efficiency is limited by network performance. Second, when a task fails, most systems directly reset or reallocate the task to a new node, requiring repeated downloading of the input file and reprocessing, resulting in a waste of bandwidth resources. In particular, network congestion is exacerbated when processing large files. In addition, the fixed number of nodes makes it difficult to respond to load fluctuations in a timely manner. Tasks are easily accumulated under high load, and redundant nodes are idle under low load, resulting in unbalanced resource utilization. Finally, when a node exits unexpectedly or a task is interrupted, the system lacks fine-grained state tracking and recovery capabilities, often requiring manual intervention or a full rollback, which affects the robustness of the system. Summary of the Invention

[0003] Purpose

[0004] The present invention addresses the limitations of existing technologies and provides a distributed processing method and system. In this distributed processing system, by decoupling task data transmission from the actual task processing portion and adopting a highly parallel processing flow architecture, the waiting time for actual task processing is shortened, thereby shortening the average task processing time and improving the resource utilization of task processing nodes. At the same time, a task recovery and reallocation mechanism based on task status is introduced into the system. When a problem occurs in task processing, the task status is restored based on the task status and the number of errors, and the task is rolled back to the most recent task status and continued processing, reducing the negative impact of problems such as node failures and task processing errors. Finally, when the current online node task load is high, the task processing nodes are dynamically scaled based on IP addresses and historical registration information, shortening task waiting time, improving task processing efficiency, and also saving computing power and energy.

[0005] Technical Solution

[0006] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0007] A distributed task processing system includes a task center and at least one task processing node. The task center includes a task receiving module, a task scheduling module, and a node monitoring module; the task processing node includes a node management module, a task monitoring module, a task downloading module, a task processing module, and a task uploading module.

[0008] The task center also includes a pending task queue, data storage space, and multiple assigned task queues. The pending task queue stores all tasks waiting to be assigned; the data storage space is responsible for storing the task files required for task processing and provides storage space for task processing results; the assigned task queue is generated after the task processing node is started and successfully registered, and each assigned task queue corresponds to a task processing node. The task center receives tasks uniformly through the task receiving module, generates task information, and sequentially adds the task information to the end of the pending task queue. The task scheduling module removes tasks from the head of the pending task queue, modifies the task status, and assigns them one by one to different task processing nodes, adding the information to the corresponding assigned task queue. The task center periodically monitors the status of processing nodes through the node monitoring module.

[0009] The task processing node contains three task information queues: a download queue, a pending queue, and an upload queue. These queues store task information in different task states within the node. Three semaphores are also set with initial values of zero: a download semaphore, a pending semaphore, and an upload semaphore, each representing the number of tasks in the corresponding task information queue.

[0010] In the present invention, the submission, downloading, processing, and uploading of tasks are all implemented in the form of files. A task can be understood as a complete process, including submission, downloading, processing, and uploading, performing a series of processing operations on a specific task file and its data. When a task is submitted, the relevant data file is uploaded to the task center, and the task center assigns the task to the appropriate processing node. After querying the task, the processing node first downloads the corresponding task file and then processes the data in the file. After the processing node processes the task file and generates a result, it uploads the generated result file back to the task center.

[0011] The task monitoring module of the task processing node periodically monitors the tasks generated by the task center in the task queue corresponding to the node, obtains the task information at the head of the queue and adds the task information to the download queue of the node, waits for the task data to be downloaded, and adds 1 to the download semaphore and makes a judgment. If the download semaphore is 0, the task download module is awakened.

[0012] The task download module of the task processing node decrements the value of the download semaphore by 1 and makes a judgment. If the value is less than zero, the task enters a blocked state and waits to be awakened. Otherwise, an item is immediately taken out from the head of the download queue, the task information is obtained and the download starts. After the task file is downloaded, the task information is added to the pending queue, the pending semaphore is increased by 1 and a judgment is made. If the pending semaphore is less than or equal to 0, the task processing module is awakened. Finally, the next task download operation is continued.

[0013] The task processing module of the task processing node decrements the value of the pending semaphore by 1 and makes a judgment. If the value is less than 0, it enters a blocked state and waits to be awakened. Otherwise, it immediately takes out an item from the head of the pending queue, obtains the task information and starts processing. After the task file is processed, the task processing result information is added to the upload queue, the upload semaphore is increased by 1 and judged. If the upload semaphore is less than or equal to 0, the task upload module is awakened. Finally, the next task processing operation is continued.

[0014] The task upload module of the task processing node decrements the upload semaphore by 1 and checks if the value is less than 0. If the value is less than 0, the node enters a blocked state and waits to be awakened. Otherwise, the node immediately takes an item from the head of the upload queue, obtains the task information, and uploads the task processing result. Finally, the node continues to execute the next task upload operation.

[0015] Furthermore, the method further comprises:

[0016] The task processing node uses dynamic fault tolerance and hierarchical retry mechanism to effectively solve the problem of task failure by combining state-level recovery and task-level recovery. During the task processing process, if an error or execution timeout occurs in the task download module, task processing module or task upload module, the processing node will record the number of stage failures. If the number of stage failures is less than or equal to the preset maximum number of attempts, the task will be restored to the starting point of the stage and re-executed according to the task status; otherwise, the task will be transferred to task-level recovery, and the task information will be sent to the task center. The task center receives the task and records the number of task failures, and then makes a judgment. If the number of task failures does not exceed the preset maximum processing times, the task will be added to the task queue to be assigned and the task will be assigned. Otherwise, the task will be deemed to be an unfinishable task and will no longer be executed.

[0017] When the task processing node is started, it registers with the task center, provides information such as the IP address and node name, and also provides its own resource information. The task center configures the task upper limit of the node.

[0018] The node management module of the task processing node periodically sends heartbeat signals to the task center. The processing node monitoring module of the task center is responsible for receiving and recording these heartbeat signals, and periodically checking the heartbeat signals of all task processing nodes. When it is detected that the heartbeat signal of a task processing node is lost, the task center will immediately start the node exit process. First, the task center sends a forced exit instruction to the lost node, requiring it to stop all current task processing and enter the exit state; at the same time, the node is marked as unavailable, and new tasks are stopped from being assigned to it, and all tasks in its assigned task queue are reallocated to other available nodes. The task center will also remove the status information of the node from the list of available nodes and record relevant information about the exit event.

[0019] The node management module of the task processing node continuously monitors the running status of the task download module. If the time required to download a task file exceeds a set threshold, the task is deemed to have timed out, the download has failed, the number of download failures is recorded, and the task download module is restarted. If the number of download failures does not reach the maximum number of attempts, the task download is attempted again; otherwise, the task download failure is reported to the task center, and the task status and failure count are modified.

[0020] The node management module of the task processing node continuously monitors the operational status of the task processing module. If the processing time for a task exceeds a set threshold, the task is deemed to have timed out, the number of failed attempts is recorded, and the task processing module is restarted. If the number of failed attempts does not reach the maximum number of attempts, the task is processed again; otherwise, the task processing failure is reported to the task center, and the task status and failure count are modified.

[0021] The node management module of the task processing node continuously monitors the operational status of the task upload module. If the time required to upload a task result file exceeds a set threshold, the upload is deemed to have timed out and failed. The number of failed uploads is recorded and the task upload module is restarted. If the number of failed uploads does not reach the maximum number of attempts, the task is uploaded again. Otherwise, the task upload failure is reported to the task center, and the task status and failure count are modified.

[0022] When assigning tasks, if the Task Center fails to find a matching node or the number of pending tasks on all available nodes exceeds the upper limit, the system will dynamically start an inactive task processing node based on historical registration information and IP addresses. If the new node is successfully started, it will be added to the list of available nodes and assigned the task. If the startup fails, other candidate nodes will be tried until the task is successfully assigned or all nodes have been tried. Furthermore, if too many idle nodes are detected, the system will automatically initiate a dynamic shutdown process to shut down idle nodes and free up resources.

[0023] The present invention provides a distributed task processing method and system with the following significant features: First, by modularizing the design of task acquisition, transmission and processing, parallel pipeline processing of tasks is achieved, significantly improving the system processing efficiency. Secondly, the system has a complete task status monitoring and recovery mechanism, which can effectively deal with abnormal situations such as node failures and task failures, and improve the task success rate through the task retry mechanism. In addition, the system adopts a dynamic node management strategy, which can automatically start and stop processing nodes according to the task load, optimize task allocation and shorten waiting time. At the same time, through the task queue mechanism, the system achieves precise control of the number of concurrent tasks, ensuring the reasonable allocation of bandwidth resources and system stability. Finally, the system also provides real-time monitoring and dynamic adjustment functions, which can flexibly adjust the task processing priority according to the node status and task load, and further balance the system throughput and response time. Compared with the existing technology, the present invention has significantly improved task processing efficiency, system stability and resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The provided figures illustrate the abstract concepts of the invention and the systems, methods, or processes used.

[0025] Figure 1 Shown is a schematic diagram of the distributed processing system of the present invention.

[0026] Figure 2 Shown is a task state transition diagram of the distributed processing system of the present invention.

[0027] Figure 3 Shown is a flow chart of task allocation in the distributed processing system of the present invention.

[0028] Figure 4 Shown is a flow chart of the node management module in the distributed processing system of the present invention.

[0029] Figure 5 Shown is a schematic diagram of a task monitoring module in the distributed processing system of the present invention.

[0030] Figure 6 Shown is a schematic diagram of a task download module in the distributed processing system of the present invention.

[0031] Figure 7 Shown is a schematic diagram of a task processing module in the distributed processing system of the present invention.

[0032] Figure 8 Shown is a schematic diagram of a task uploading module in the distributed processing system of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be described in detail below with reference to the accompanying drawings.

[0034] Figure 1Figure 2 shows a schematic diagram of the distributed processing system of the present invention. The system includes a task center and at least one task processing node. The diagram illustrates the structure of the task center and task processing nodes, as well as the task processing process. The task center node includes a task receiving module, a task scheduling module, and a node monitoring module. The task processing node includes a node management module, a task monitoring module, a task downloading module, a task processing module, and a task uploading module.

[0035] When a task processing node starts up, it first registers with the task center, registering its machine and node information. It then starts the task monitoring module. After the node starts up, it periodically reports heartbeat information to the task center.

[0036] The task center's node monitoring module monitors the status of all task processing nodes registered with the node and determines the node's health status based on the node's heartbeat information. If a node loses heartbeat connection, it is considered to have failed and enters node failure processing. The task center sends a forced exit command to the lost node, requiring it to stop all current task processing and enter the exit state. At the same time, the task center marks the node as unavailable, stops assigning new tasks to it, and performs task-level recovery on all tasks in its assigned task queue, reallocating them to other available nodes. Finally, the task center removes the node's registration information from the list of available nodes and records the relevant information about the node's exit.

[0037] Task status design

[0038] Figure 2 The figure shows the task state transition diagram of the distributed processing system of the present invention. According to the process in which the task is located, the task status is divided into 7 types: pending, downloading, processing, uploading, processing completed, pending retry, and processing terminated. Among them, pending means that the task has not been processed yet and has been assigned to the task processing node; downloading means that the task processing node has added it to the download queue and entered the download process; processing means that the task processing node is processing the task; uploading means that the task processing node has completed the processing and added the task processing result to the upload queue, waiting for upload; processing completed means that the task processing result has been uploaded and the task processing has ended; pending retry means that an error occurred during the processing of the task and it is waiting for the task center to schedule it again; processing terminated means that the task execution times have reached the maximum processing times and the task is deemed to be an unfinished task.

[0039] Task scheduling process

[0040] The task receiving module of the task center is responsible for receiving tasks, storing the content that needs to be processed in the task locally, generating task-related information and adding it to the task queue. The task scheduling module then performs the initial assignment of the task. Figure 3The figure shows the task allocation flow chart of the distributed processing system of the present invention: first find the node number with the least current task number and normal heartbeat record. If one or more nodes that meet the conditions are found, select the node with the smallest number, assign the task to the node, and add the task to the corresponding assigned task queue; otherwise, start a new node and assign it through the elastic node scaling mechanism. If there is no new node available, actively block the preset time interval and execute the task scheduling process again until the task is successfully assigned. In addition, in order to improve the efficiency of task processing and reduce repeated calculations, when the system detects the same or similar tasks, its priority is increased according to the similarity of the tasks, and it is preferentially assigned to the node that has processed the task before, so as to reuse the intermediate data and reduce repeated calculations.

[0041] Finally, the task status is set to "pending" and waits for the corresponding task processing node to proceed to the next step.

[0042] Elastic node scaling mechanism

[0043] The task center flexibly manages task processing nodes through a dynamic start-stop mechanism to cope with changes in task load. When the task scheduling module fails to find a qualified node during the task allocation process or the number of pending tasks for all available nodes exceeds the upper limit, the node monitoring module will filter out spare nodes that can be used for task execution based on historical registration information, IP addresses and other information, and try to start these task processing nodes in turn. During the startup process, the node monitoring module sends a magic packet to the target node through WOL (Wake-on-LAN) technology, and monitors the heartbeat information and registration information returned by the node to ensure that the node is online normally. After the new node is successfully started, it will immediately register the node in the task center, be added to the list of available nodes and wait for tasks to be assigned. If the node fails to start, the system will continue to try other candidate nodes until the task is successfully assigned or all nodes have been tried.

[0044] At the same time, the node monitoring module of the task center will continuously monitor the node load. When it detects that there are too many idle nodes (the number of pending tasks is continuously lower than the set threshold), it will automatically start the dynamic shutdown process: the system will first safely migrate the unprocessed tasks on the idle nodes to other nodes, and then send an exit signal to the idle node, requiring the node to exit and release resources after the remaining tasks are processed, thereby achieving optimal resource utilization.

[0045] Task dynamic fault tolerance and hierarchical retry mechanism

[0046] This mechanism is divided into two levels: state-level recovery based on the task stage status and task-level recovery based on the overall number of attempts:

[0047] 1. State-level recovery: If an error occurs when the task status is "downloading", "processing", or "uploading", or if the node management module finds that a certain stage of the task has timed out when monitoring the running status of the task download module, task upload module, and task processing module, the task processing node will determine that the stage has failed, restart the abnormal module, and restore the task to the starting point of the stage according to the current task status and re-execute it, while recording the number of stage failures; if failure still occurs and the number of stage failures exceeds the preset maximum number of attempts, the task will be transferred to task-level recovery, the task status will be set to "pending retry", and the task information will be sent to the task center.

[0048] 2. Task-level recovery: The task receiving module of the task center receives task information with a status of "pending retry" sent by all online nodes. At the same time, it regularly scans all tasks with a status of "downloading", "processing", or "uploading". If the task processing node to which the task belongs is offline, the task status is changed to "pending retry". After receiving the task, the task center records the number of task-level failures of the task and determines whether the number has reached the preset maximum processing number. If it has, the task status is set to "processing terminated" and an error message is returned. If it has not, the task information is added to the pending task queue. The task is reallocated to an available task processing node through the task scheduling process, and the task status is reset to "pending". At the same time, the number of stage failure records is cleared.

[0049] Task processing based on parallel pipeline

[0050] The task processing node consists of five modules: node management module, task monitoring module, task download module, task upload module and task processing module.

[0051] Figure 4 The flowchart of the node management module in the distributed processing system of the present invention is shown. The node management module is responsible for continuously sending heartbeat signals to the task center, maintaining the node online state, and ensuring that the task center can assign tasks to this node. It is also responsible for monitoring the operating status of the task monitoring module, task download module, task upload module and task processing module. When it is detected that the above modules have failed or the node receives a shutdown instruction, a termination instruction is sent to the four modules. If the task download module, task upload module or task processing module continues to download, upload or process a task beyond the set deadline, it is determined that the download, upload or processing has failed, the number of failures of the task in the module is recorded, and the module where the task is located is restarted according to the task status at the time of the error. If the number of failures of the same task in the same module does not reach the maximum number of attempts, the module is restarted and the download, upload or processing of the task is performed again. Otherwise, the task processing failure information is reported to the task center, the task status is modified to "pending retry" and the number of times processed is increased.

[0052] Figure 5 The figure shows a schematic diagram of the task monitoring module in the distributed processing system of the present invention. It periodically monitors the tasks in the assigned task queue corresponding to the current node, generates tasks from the task center, obtains task information from the head of the queue, adds the task information to the current node's download queue, waits for the task file to be downloaded, sets the task status to "downloading," increments the download semaphore, and performs a check. If the download semaphore is zero, the task download module is awakened.

[0053] Figure 6 Figure 1 shows a schematic diagram of the task download module in the distributed processing system of the present invention. The task download module is responsible for completing the download of task files. The specific workflow is as follows: The task download module decrements the value of the download semaphore by 1 and makes an evaluation. If the value is less than 0, it indicates that there are no tasks to be downloaded in the current download queue, and the task download module enters a blocked state, waiting to be awakened. Otherwise, it immediately removes an item from the head of the download queue, obtains the task information, and begins downloading. If the download is successful, the task status is changed to "processing," and the task information is added to the pending queue, awaiting processing by the task processing module. If the download fails, the number of download failures for this task is recorded. If the number of download failures does not exceed the maximum number of attempts, the task file download for this task is retried. Otherwise, the task status is changed to "pending retry," and the task information is sent to the task center for task reallocation. After the task file download is complete, the task status is changed to "processing," the task information is added to the pending queue, the pending semaphore is incremented by 1, and an evaluation is performed. If the pending semaphore is less than or equal to 0, the task processing module is awakened. Finally, the task processing module reenters the idle state, preparing for the next task download operation.

[0054] Figure 7Figure 1 shows a schematic diagram of the task processing module in the distributed processing system of the present invention. The task processing module is responsible for completing the specific content of task processing. The specific workflow is as follows: The task processing module decrements the value of the pending semaphore by 1 and makes an evaluation. If the value is less than 0, it indicates that there are no tasks to be processed in the current pending queue, and the task processing module enters a blocked state, waiting to be awakened. Otherwise, it immediately removes an item from the head of the pending queue, obtains the task information, and processes it according to the task requirements. If the processing is successful, the task status is changed to "Uploading" and the task result information is added to the upload queue, waiting for the task result to be uploaded. If the processing fails, the number of processing failures for this task is recorded. If the number of processing failures does not exceed the maximum number of attempts, the task processing is attempted again. Otherwise, the task status is changed to "Waiting for Retry" and the task information is sent to the task center for task reallocation. After the task file is processed, the task processing result information is added to the upload queue, the upload semaphore is incremented by 1, and an evaluation is performed. If the upload semaphore is less than or equal to 0, the task upload module is awakened. Finally, the task upload module re-enters the idle state, preparing for the next task processing operation.

[0055] Figure 8 The figure shows a schematic diagram of the task upload module in the distributed processing system of the present invention. The task upload module is responsible for uploading the task processing result file after the task file is processed back to the task center. The task upload module reduces the value of the upload semaphore by 1 and judges. If the value is less than 0, it means that there is no task that needs to be uploaded in the current upload queue, and the task upload module enters a blocked state and waits to be awakened; otherwise, it immediately takes out an item from the head of the upload queue, obtains the task processing result information and uploads the task processing result. If the upload is successful, the task status is modified to "uploading", and the task information is added to the upload queue, waiting for the task processing module to process the task; if the upload fails, the number of upload failures of this task is recorded. If the number of upload failures does not exceed the maximum number of attempts, the task processing result file upload of the task is attempted to be re-executed. Otherwise, the task status is modified to "pending retry" and the task information is sent to the task center for task reallocation. After the task file upload is completed, the task status is modified to "processing completed". Finally, the idle state is re-entered to prepare for the next task upload operation.

[0056] Program exit protection

[0057] There are four exit methods for task processing nodes:

[0058] 1. The task processing node exits immediately.

[0059] 2. The task processing node exits after completing the current task.

[0060] 3. Force exit the task center.

[0061] 4. The task processing node exits illegally.

[0062] When exiting under Method 1, the task processing node immediately ceases all task transmission and processing related content and logs out of the task center. Tasks assigned to this node will be redistributed by the task center during the task-level recovery process. Simultaneously, the task processing node adds an exit signal to the end of the download queue, pending queue, and upload queue, and increments the download semaphore, pending semaphore, and upload semaphore values by one. When the task download module, task processing module, and task upload module receive the exit signal from the queue, they immediately exit the current module.

[0063] When exiting in Method 2, the node management module of the task processing node sends an exit signal to each module. Upon receiving the exit signal, the task monitoring module stops monitoring new tasks for this node and exits immediately. The task downloading module exits after the current task is processed. The task processing module and the task upload module exit after ensuring that all tasks in the task processing queue and task upload queue have been processed and uploaded.

[0064] When exiting under Method 3, the task center first sends an exit command to the task processing node. Upon receiving the command, the node management module of the task processing node immediately enters the exit process described in Method 1. If the node has unfinished tasks, the task center will redistribute the remaining tasks according to the task-level recovery process and assign them to other task processing nodes for further processing.

[0065] When exiting under mode 4, the task processing node will retain the currently processed task status information, task queue information, and node status information in the local storage. After the task processing node is restarted, it will first read the information in the local storage, perform state-level recovery based on the task status and continue processing. At the same time, it will restore the task queue information and node status information, and re-register the node with the task center. The task processing node will report its recovery status and unfinished task information to the task center. The task center will decide whether the unfinished task will continue to be processed by the node based on the node report and the operating status of each node, the number of tasks, and other information provided by the node management module. If the task center decides that the original node will continue to process the unfinished task, the node will continue to process it; otherwise, the original node will clean up the relevant task information, and the unfinished task will be reallocated by the task center through the task-level recovery process.

Claims

1. A distributed processing system, characterized in that: It includes a task center and at least one task processing node, where: a. The mission center includes: The task receiving module is used to receive external tasks and generate task information. The task scheduling module is used to obtain tasks from the head of the task queue to be assigned and assign them to the task processing node. The node monitoring module is used to periodically monitor the heartbeat signals of task processing nodes, and dynamically adjust according to the heartbeat status, promptly mark the node with lost heartbeat connection as unavailable, and perform task-level recovery on the tasks assigned to the node. A to-be-assigned task queue, data storage space, and multiple assigned task queues, where each assigned task queue corresponds to a registered task processing node. b. The task processing node includes: The node management module is used to register node information with the task center, send heartbeat signals, and monitor the operating status of other modules of this node. The task monitoring module is used to obtain task information from the assigned task queue of the task center. The task download module manages task file downloads and downloads the task files required for task processing from the task center. The task processing module is used to perform local task processing and generate results. The task upload module manages task file uploads and uploads the result files generated by task processing back to the task center. A download queue, a pending queue, and an upload queue store task information in different states. The download semaphore, pending semaphore, and upload semaphore represent the number of tasks in the download queue, pending queue, and upload queue, respectively.

2. The system according to claim 1, wherein: The exit methods of the task processing node include: a. Exit immediately: Stop all tasks and log out of the node. The task center will reallocate unfinished tasks. b. Exit after task completion: Exit after the current task and upload queue tasks are completed. c. Forced exit from the Task Center: Terminates the operation immediately after receiving the exit command. d. Restart after illegal exit: Read the unfinished task information stored locally and resume processing from the breakpoint.

3. A distributed task processing method, characterized in that: The following steps are involved: a. The task center receives external tasks, generates task information and stores it in the task queue to be assigned. b. The task scheduling module assigns tasks to the task processing nodes with normal heartbeat and minimum load, and stores them in the corresponding assigned task queue. c. The task processing node obtains the task from the assigned task queue, adds it to the download queue, and downloads the task file through the task download module. d. After the task download is completed, the task enters the processing queue and is processed by the task processing module. e. After the task processing is completed, the task is added to the upload queue and the processing results are uploaded to the task center through the task upload module. g. Hierarchical retry mechanism handles task failures: State-level recovery: When a task fails during the download, processing, or upload phase, it can be retraced to the starting point of the previous phase and re-executed. Task-level recovery: When a task fails to be processed on any node, the number of failures is recorded. When the number of failures exceeds the preset threshold, the task is marked as failed. Otherwise, it is reallocated to other nodes.

4. The method according to claim 3, characterized in that Step b further comprises: a. When the same or similar tasks are detected, they are preferentially assigned to the node that has previously processed the task to reuse the intermediate data. b. Select the optimal node based on the number of tasks to be processed and the historical load of the node.

5. The method according to claim 3, characterized in that The method further comprises: a. When a task processing node loses heartbeat connection, the task center marks it as unavailable and reallocates all unfinished tasks to the node. b. The task center dynamically starts and stops task processing nodes according to the task load, starts new nodes when available nodes are insufficient, and releases idle nodes when the idle node threshold exceeds.

6. The method according to claim 5, characterized in that The method further comprises: a. When the number of pending tasks on available nodes exceeds the upper limit, new nodes are started based on historical registration information. b. After the new node is successfully started, it is added to the list of available nodes and assigned tasks. c. When it is detected that the number of pending tasks of a node is continuously lower than the threshold, its tasks are migrated to other nodes and an exit instruction is sent.

7. The method according to claim 3, characterized in that When errors occur during task downloading, processing, or uploading, the following recovery strategies are implemented: a. The hierarchical retry mechanism includes state-level recovery and task-level recovery, and the recovery method is selected based on the cumulative number of task failures. b. State-level recovery: When a task fails during the download, processing, or upload phase and the number of failures does not exceed the maximum number of attempts, the task is restored to the starting point of the phase and re-executed. c. Task-level recovery: When the number of failures of a task at any stage exceeds the preset threshold for the stage but does not exceed the maximum number of processing times, the task will be reallocated to other nodes for execution, and the number of failures will be recorded. d. Task termination: If the number of failures at any stage of a task exceeds the maximum number of processing times, the task will be marked as unfinishable, the error information will be reported to the task center, and processing will be terminated.

Images