Distributed task processing method and terminal

By decomposing distributed tasks into subtasks for parallel processing and recording execution data in a cache, clearing the cache upon success and correcting or migrating upon failure, the problem of low execution efficiency of distributed tasks is solved, and efficient task execution is achieved.

CN118573678BActive Publication Date: 2025-11-28福建天晴在线互动科技有限公司
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410637863.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-22
Publication Date
2025-11-28
Estimated Expiration
2044-05-22

AI Technical Summary

Technical Problem

In existing technologies, distributed task execution is inefficient. When a task fails, all steps need to be re-executed, resulting in wasted resources and time consumption.

Method used

The distributed task is decomposed into subtasks for parallel processing. On success, the cached data is deleted, and on failure, the execution data is read from the cache and executed again. The failure type is corrected or migrated.

Benefits of technology

It improves the execution efficiency of distributed tasks, reduces redundant data transmission and processing, and optimizes the task execution process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118573678B_ABST
    Figure CN118573678B_ABST
Patent Text Reader

Abstract

The application discloses a distributed task processing method and a terminal. The method comprises the following steps: obtaining a distributed task to be processed; distributing the distributed task to each target node for processing to obtain execution data; storing all the execution data into a preset cache data set; judging whether the distributed task is successfully executed according to the execution data; if yes, deleting the execution data in the cache data set; otherwise, reading the execution data from the cache data set and executing the distributed task again according to the execution data. The application can reduce the transmission and processing of useless data when the task is executed again, and improve the overall execution efficiency of the task.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed task, and particularly relates to a distributed task processing method and a terminal. BACKGROUND

[0002] At present, there are various types of massive data in the Internet system, in order to execute the related tasks of the massive data, a plurality of distributed tasks are provided to perform scheduling, so as to reduce the task execution time. Generally, the task execution time is related to the resources possessed by the task, the execution data volume, the execution steps and the like. When any one of the conditions does not meet the task execution requirement, the task will fail. When the task fails, the task can only be restarted according to a certain re-execution strategy, that is, the task will be re-executed. If the task has multiple steps, all the steps will be re-executed. If the task still does not meet the execution requirement when re-executed, the task continues to fail. And the task processing mode causes low task execution efficiency. SUMMARY

[0003] The present application solves the technical problem of providing a distributed task processing method and a terminal, and improving the execution efficiency of the distributed task.

[0004] In order to solve the above technical problem, the technical scheme adopted by the present application is as follows:

[0005] A distributed task processing method, comprising:

[0006] acquiring a distributed task to be processed;

[0007] distributing the distributed task to each target node for processing to obtain execution data;

[0008] storing all the execution data into a preset cache data set;

[0009] judging whether the distributed task is executed successfully according to the execution data;

[0010] if yes, deleting the execution data in the cache data set;

[0011] otherwise, reading the execution data from the cache data set and re-executing the distributed task according to the execution data.

[0012] In order to solve the above technical problem, another technical scheme adopted by the present application is as follows:

[0013] A distributed task processing terminal, comprising a memory, a processor and a computer program stored in the memory and running on the processor, and the processor implements each step in the above distributed task processing method when executing the computer program.

[0014] The application has the beneficial effect that in the process of executing the distributed task at the target node, all execution data corresponding to the distributed task is recorded in the cache data set, when the distributed task is executed successfully, the execution data in the current cache data set is deleted, reducing the consumption of system resources. When the distributed task fails, the last obtained execution data is read from the cache data set, and the distributed task is executed again according to the backup execution data, thereby reducing the data transmission process, and since the execution data is the data obtained by executing the distributed task, when the distributed task is executed again, the relevant data can be directly obtained based on the execution data, without the need to re-execute all steps of the distributed task, effectively improving the overall execution efficiency of the distributed task. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 A step flow chart of a distributed task processing method provided for the embodiment of the application;

[0016] Figure 2 A structure schematic diagram of a distributed task processing terminal provided for the embodiment of the application;

[0017] Label explanation:

[0018] 100, a distributed task processing terminal; 101, a memory; 102, a processor. DETAILED DESCRIPTION

[0019] To explain the technical content, the achieved purposes and effects of the application in detail, the following will be explained in combination with the embodiments and the accompanying drawings.

[0020] The embodiment of the application provides a distributed task processing method, comprising:

[0021] obtaining a distributed task to be processed;

[0022] distributing the distributed task to each target node for processing to obtain execution data;

[0023] storing all the execution data to a preset cache data set;

[0024] judging whether the distributed task is executed successfully according to the execution data;

[0025] if yes, deleting the execution data in the cache data set;

[0026] otherwise, reading the execution data from the cache data set, and executing the distributed task again according to the execution data.

[0027] From the above description, the beneficial effects of the present application are that in the process of executing the distributed task at the target node, all execution data corresponding to the distributed task is recorded in the cache data set, when the distributed task is executed successfully, the execution data in the current cache data set is deleted, reducing the consumption of system resources. When the distributed task fails, the last obtained execution data is read from the cache data set, and the distributed task is executed again according to the backup execution data, thereby reducing the data transmission process, and since the execution data is the data obtained by executing the distributed task, the relevant data can be directly obtained based on the execution data when the distributed task is executed again, without the need to re-execute all steps of the distributed task, thereby effectively improving the overall execution efficiency of the distributed task.

[0028] Further, the distributing the distributed task to each target node for processing to obtain execution data comprises:

[0029] dividing the distributed task into a plurality of subtasks, each of the subtasks comprising a plurality of task steps;

[0030] distributing the plurality of subtasks to the target nodes respectively, and processing the task steps corresponding to the subtasks at the target nodes to obtain execution data of each of the task steps.

[0031] From the above description, the distributed task is divided into a plurality of subtasks and distributed to each target node for parallel processing, thereby improving the task processing efficiency.

[0032] Further, the execution data comprises cache data and result data;

[0033] storing all the execution data to a preset cache data set comprises:

[0034] determining whether the task step corresponding to the subtask at the target node is executed successfully according to the result data;

[0035] if yes, storing the cache data and the result data corresponding to the task step to the preset cache data set according to the execution order of the task step.

[0036] As can be known from the above description, since the distributed task is divided into multiple subtasks, and the execution of the entire distributed task fails when there is a failed subtask, it is necessary to determine whether the subtask under the current target node is executed successfully according to the result data. Meanwhile, since the processing process of the distributed task has a sequence of execution, that is, the result data generated by a previous task step can be used as input data of a subsequent task step, the execution data corresponding to each task step also has a corresponding sequence, so as to facilitate subsequent direct query and acquisition of the result data and the cache data of each task step. Therefore, the execution data is stored in the cache data set according to the execution sequence of the task steps, thereby improving the data reading efficiency.

[0037] Further, the determining whether the distributed task is executed successfully according to the execution data comprises:

[0038] determining whether each subtask is executed successfully according to the result data;

[0039] if yes, the distributed task is executed successfully;

[0040] otherwise, the distributed task is executed unsuccessfully;

[0041] the deleting the execution data in the cache data set comprises:

[0042] deleting all the cache data in the cache data set.

[0043] As can be known from the above description, when there is a failed subtask in the distributed task, the distributed task is executed unsuccessfully. When all the subtasks are executed successfully, the entire distributed task is executed successfully, and the cache data in the cache data set is deleted synchronously, thereby reducing the data amount in the data set and improving the data query and reading efficiency.

[0044] Further, before the reading the execution data from the cache data set and re-executing the distributed task according to the execution data, the method further comprises:

[0045] determining a type of the distributed task execution failure according to the cache data;

[0046] if the type is data error, re-executing the distributed task according to the original task execution strategy;

[0047] if the type is conditional exception, reading the execution data from the cache data set and re-executing the distributed task according to the execution data.

[0048] From the above description, if the current distributed task execution failure is caused by data error, it indicates that the execution data in the current cache data set may have errors, so the distributed task cannot be executed by the previously backed up execution data, and the entire distributed task can only be re-executed to solve the data error problem. If the current distributed task execution failure is caused by conditional exception, it indicates that the execution data in the current cache data set is correct, and the configuration condition has a problem, so the execution data can be directly obtained to execute the distributed task again, thereby reducing the repeated transmission and processing of related data and improving the execution efficiency of the task.

[0049] Further, the reading of the execution data from the cache data set and the re-execution of the distributed task according to the execution data comprises:

[0050] reading result data of a task step corresponding to the subtask under the target node from the cache data set;

[0051] judging whether the task step is executed successfully according to the result data;

[0052] if yes, marking the result data of the task step as input data of a next task step, the execution order of the next task step being located after the task step;

[0053] otherwise, executing the task step according to the result data of a previous task step, the execution order of the previous task step being located before the task step.

[0054] From the above description, for the task step executed successfully, the result data in the cache data set can be directly read as the input data of the next task step. In this way, the previously executed task step does not need to be executed again, but is directly obtained from the previously backed up result data. Only the previously executed task step needs to be re-executed, thereby avoiding the execution of all steps in the distributed task, thereby reducing the time consumed by the task execution and improving the overall execution speed of the task.

[0055] Further, the storing of the execution data to the preset cache data set further comprises:

[0056] if the task step corresponding to the subtask under the target node fails, determining the type of the task step execution failure according to the cache data of the task step;

[0057] storing the cache data, the result data and the type corresponding to the task step to the cache data set according to the execution order.

[0058] From the above description, for the task step of task execution failure, the corresponding cache data and the type of execution failure can be added to the cache data set, so that when the distributed task is executed again, the execution data in the cache data set can be used for debugging and modification, to ensure the successful execution of the distributed task. Thus, the abnormal data is quickly determined and located through the execution data, the success rate of distributed task execution is improved, and the execution efficiency of the distributed task is improved.

[0059] Further, the type includes data error and conditional exception.

[0060] The execution data is read from the cache data set, and the distributed task is executed again according to the execution data.

[0061] The result data of the task step corresponding to the subtask under the target node is read from the cache data set.

[0062] It is judged whether the task step is executed successfully according to the result data.

[0063] If yes, the result data of the task step is marked as the input data of the next task step, and the execution order of the next task step is located after the task step.

[0064] Otherwise, the type of execution failure of the task step is read from the cache data set.

[0065] If the type is data error, the cache data of the task step in the cache data set is read, and the error data in the cache data is automatically corrected, and the task step is executed according to the corrected cache data.

[0066] If the type is conditional exception, the task step is executed according to the result data of the previous task step, and the execution order of the previous task step is located before the task step.

[0067] From the above description, for the task step of task execution failure, the corresponding cache data and the type of execution failure can be added to the cache data set, so that when the distributed task is executed again, the execution data in the cache data set can be used for debugging and modification, to ensure the successful execution of the distributed task. Thus, the abnormal data is quickly determined and located through the execution data, the success rate of distributed task execution is improved, and the execution efficiency of the distributed task is improved.

[0068] Further, the performing the task step according to the result data of the previous task step comprises:

[0069] migrating the sub-task corresponding to the task step from the target node to another target node;

[0070] reading the result data of the previous task step in the cache data set on the another target node, and performing the task step according to the result data of the previous task step.

[0071] As can be seen from the above description, when the task step fails to be executed, it is possible that the current target node does not meet the configuration condition, so the sub-task corresponding to the task step can be directly migrated from the current target node to another target node for execution, so as to ensure the success rate of the distributed task execution, and further reduce the number of repeated execution of the distributed task, thereby improving the task execution efficiency.

[0072] Another embodiment of the present application provides a distributed task processing terminal, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements each step in the above distributed task processing method when executing the computer program.

[0073] As can be seen from the above description, the present application has the following advantages: during the execution of the distributed task on the target node, all execution data corresponding to the distributed task is recorded in the cache data set, when the distributed task is successfully executed, the execution data in the current cache data set is deleted, so as to reduce the consumption of system resources. When the distributed task fails to be executed, the execution data obtained last time is read from the cache data set, and the distributed task is executed again according to the backup execution data, so that the data transmission process is reduced, and since the execution data is the data obtained by executing the distributed task, the relevant data can be directly obtained based on the execution data when the distributed task is executed again, without the need to execute all steps of the distributed task again, so that the overall execution efficiency of the distributed task is effectively improved.

[0074] Embodiments of the present application provide a distributed task processing method and terminal, which can be applied to the scheduling and processing of distributed tasks, and can effectively improve the execution efficiency of the distributed task. The following will be described through specific embodiments:

[0075] Please refer to Figure 1 An embodiment of the present application is:

[0076] A distributed task processing method comprises:

[0077] S110, obtaining a distributed task to be processed.

[0078] S120, distribute the distributed task to each target node for processing to obtain execution data.

[0079] Specifically, the step S120 includes:

[0080] S1201, divide the distributed task into a plurality of subtasks, each of which includes a plurality of task steps.

[0081] In some embodiments, a distributed task is obtained, and the distributed task is divided into three subtasks according to a preset splitting rule, each of which includes five task steps, wherein the task steps include pulling data, obtaining data, and processing data, and the like internal processing logic.

[0082] S1202, distribute the plurality of subtasks to the target nodes respectively, and process the task steps corresponding to the subtasks on the target nodes to obtain execution data of each task step.

[0083] In some embodiments, there are currently a plurality of target nodes, and the plurality of subtasks are distributed to each target node, that is, one-to-one correspondence between the subtasks and the target nodes. Specifically, there are currently target nodes A, B, C, and D, and the three subtasks divided in the step S1201 are allocated to the three target nodes A, B, and C, and each subtask will execute the corresponding task step on the corresponding target node.

[0084] S130, store all the execution data to a preset cache data set.

[0085] The execution data includes cache data and result data.

[0086] Specifically, the step S130 includes:

[0087] S1301, determine whether the task step corresponding to the subtask on the target node is executed successfully according to the result data.

[0088] In some embodiments, on the target node, whenever a task step is executed, a notification message is synchronously sent, wherein the notification message includes the result data corresponding to the task step. When the notification message is received, it is determined whether the current task step is executed successfully.

[0089] S1302, if yes, store the cache data and the result data corresponding to the task step to the preset cache data set according to the execution order of the task step.

[0090] In some embodiments, the cache dataset is a distributed cache cluster that can be accessed by all target nodes and their corresponding subtasks. Therefore, the data in the cache dataset is shared data, and the data after the task steps in each subtask are completed can be stored in the cache dataset.

[0091] S140. Determine whether the distributed task was executed successfully based on the execution data.

[0092] Step S140 includes:

[0093] S1401. Determine whether each of the subtasks was executed successfully based on the result data.

[0094] S1402. If so, the distributed task is executed successfully, and the following step S150 is executed.

[0095] S1403. Otherwise, the distributed task fails to execute, and the following step S160 is executed.

[0096] In some embodiments, the failure of one of the subtasks will cause the entire distributed task to fail.

[0097] S150. Delete the execution data from the cached dataset.

[0098] Specifically, step S150 includes deleting all the cached data in the cached dataset.

[0099] In some embodiments, after all subtasks have executed successfully and the distributed task has received notifications of successful execution from all subtasks, it will send an asynchronous message to delete all cached data related to this distributed task in the cached dataset, thereby reducing the memory usage of the data.

[0100] S160. Read the execution data from the cached dataset and execute the distributed task again based on the execution data.

[0101] In an optional implementation, the cache dataset stores only cached data and result data of task steps that have been executed and successfully executed. Therefore, before step S160, the method further includes:

[0102] S161. Determine the type of failure of the distributed task based on the cached data.

[0103] In some embodiments, after a distributed task fails to execute, the current distributed task is monitored in real time to see if it meets the conditions for re-execution. If it does, step S161 is executed to select whether to re-execute the distributed task or to re-execute the distributed task based on the execution data.

[0104] S162, if the type is a data error, re-executing the distributed task according to the original task execution strategy.

[0105] In some embodiments, the data error includes a program error and a debugging error, etc., wherein the debugging error can be determined according to the result data obtained in the debugging process. Specifically, if a program error occurs or the result data is directly determined to have an error in the debugging process, the distributed task needs to be re-executed.

[0106] S163, if the type is a conditional exception, reading the execution data from the cache data set and re-executing the distributed task according to the execution data.

[0107] In some embodiments, the conditional exception includes a resource shortage, a data acquisition timeout, an execution timeout, etc. Specifically, if the target node has a resource shortage problem in the process of executing the subtask, the subtask can be migrated from the current target node to other target nodes with sufficient resources for execution to solve the problem of conditional exception.

[0108] In an optional implementation, the notification information can be generated based on the type of the failed execution of the distributed task to prompt the user of the reason for the failed execution of the distributed task, and the user can choose to re-execute the distributed task according to the original task execution strategy or execute the distributed task according to the execution data in the cache data set according to the reason for the failed execution.

[0109] In an optional implementation, only the cache data and result data of the task steps that have been executed and successfully executed are stored in the cache data set, and the step S160 includes:

[0110] S1601a, reading the result data of the task step corresponding to the subtask under the target node from the cache data set.

[0111] In some embodiments, when the distributed task is re-executed according to the execution data, the distributed task to be executed needs to be obtained first, and then the distributed task is divided into three subtasks according to the previous division rule, and each subtask is distributed to each target node.

[0112] S1602a, determining whether the task step is successfully executed according to the result data.

[0113] In some embodiments, when each target node executes a subtask, the target node reads corresponding result data from the cache dataset according to a task step in which the current subtask is being executed. Since the cache dataset only stores result data of task steps that have been executed successfully, if the target node does not obtain the result data, it indicates that the task step has failed to be executed; if the target node obtains the result data, it indicates that the task step has been executed successfully.

[0114] S1603a, if yes, the result data of the task step is marked as input data of a next task step, which is in a sequence after the task step.

[0115] S1604a, otherwise, the task step is executed according to result data of a previous task step, which is in a sequence before the task step.

[0116] In some embodiments, if the task step has been executed successfully in the last execution process, the task step does not need to be executed again, and the result data of the task step is directly obtained as input of a subsequent task step, thereby realizing fast execution of the task step. If the task step has failed to be executed in the last execution process, the task step needs to be executed again according to the result data of the previous task step. Specifically, the current subtask includes a task step a, a task step b and a task step c, and the execution sequence of the task steps is a->b->c. In this case, if the current task step b has been executed successfully in the last execution process, the result data of the task step b in the cache dataset is directly obtained as input data of the task step c; if the current task step b has failed to be executed in the last execution process, the result data of the task step a in the cache dataset is obtained, and the task step b is executed again according to the result data of the task step a.

[0117] In an optional implementation, the step S1604 includes: migrating the subtask corresponding to the task step from the target node to another target node; reading the result data of the previous task step in the cache dataset on the another target node, and executing the task step according to the result data of the previous task step.

[0118] In some embodiments, the subtask 3 in the target node C fails to be executed due to insufficient resources of the target node C, the target node D is currently in an idle state, and the resources of the target node D are greater than those of the target node C, so the subtask 3 is migrated from the target node C to the target node D, and the target node D accesses the cache dataset to obtain the result data of the previous task step to execute the subtask 3 again.

[0119] Embodiment two of the present application is:

[0120] A distributed task processing method, which is different from embodiment one in that the step S130 further comprises:

[0121] S1303, if the task step corresponding to the subtask under the target node fails to execute, determining the type of the execution failure of the task step according to the cache data of the task step.

[0122] S1304, storing the cache data, result data and type corresponding to the task step into the cache data set according to the execution order.

[0123] At this time, the cache data and result data of all task steps are stored in the cache data set. Whether the task step is successfully executed or not, its cache data and result data are stored in the cache data set, and the type of the execution failure of the task step needs to be additionally stored.

[0124] In an optional embodiment, the cache data and result data of all task steps are stored in the cache data set, and the step S160 comprises:

[0125] S1601b, reading the result data of the task step corresponding to the subtask under the target node from the cache data set.

[0126] S1602b, judging whether the task step is successfully executed according to the result data.

[0127] S1603b, if yes, marking the result data of the task step as the input data of the next task step, the execution order of which is located after the task step.

[0128] S1604b, otherwise, reading the type of the execution failure of the task step from the cache data set.

[0129] S1605b, if the type is data error, reading the cache data of the task step in the cache data set, automatically correcting the error data in the cache data, and executing the task step according to the corrected cache data.

[0130] In some embodiments, when the task step fails to execute due to the business data error in the cache data, the business data can be automatically corrected, so that the task step is executed again based on the corrected business data. If the task step is the first time that the business data error occurs, the business data is manually corrected by the operation and maintenance personnel, and the corresponding processing scheme is saved. In the subsequent execution process, if the same type of data error occurs, the error data can be automatically repaired based on the previous processing method and the operation and maintenance scene, so as to ensure the successful execution of the task step.

[0131] S1606b, if the type is a conditional exception, then executing the task step according to the result data of the previous task step, which is in the execution order before the task step.

[0132] In some embodiments, if a task step fails to execute due to insufficient resources, the exception information in the cache dataset can be directly ignored.

[0133] Referring to Figure 2 Embodiment three of the present application is:

[0134] A distributed task processing terminal 100, comprising a memory 101, a processor 102, and a computer program stored on the memory 101 and running on the processor 102, wherein the processor 102 implements each step in the distributed task processing method of the above-mentioned embodiment one or embodiment two when executing the computer program.

[0135] In summary, the present application provides a distributed task processing method and terminal, which splits a distributed task into multiple sub-tasks and distributes them to different target nodes for execution, achieving parallel processing of the distributed task. At the same time, during the execution of the sub-tasks by the target nodes, the cache data and result data generated for each task step are stored in the cache dataset using an asynchronous recording cache method. When the distributed task is successfully executed, the useless temporary data in the cache dataset is deleted. When the distributed task fails to execute due to conditional exceptions such as timeout or insufficient resources, the result data already backed up in the cache dataset is read during the subsequent task execution or restart, thereby quickly executing the task steps that have already been successfully executed, and the task steps that have failed to execute need to be executed again. When the distributed task fails to execute due to data errors such as program errors, the error data of the task step that has failed to execute can be automatically corrected during the subsequent task execution or restart, thereby executing again. In this way, when the distributed task needs to be executed again, there is no need to re-execute all the steps of the distributed task, reducing the transmission and processing of useless data, and effectively improving the overall execution efficiency of the distributed task.

[0136] The above-mentioned embodiments are merely examples of the present application, and do not limit the patent scope of the present application. Any equivalent transformation or direct or indirect application in related technical fields based on the content of the present application specification and drawings is also included in the patent protection scope of the present application.

Claims

1. A distributed task processing method, characterized in that, include: Retrieve distributed tasks to be processed; The distributed task is distributed to each target node for processing to obtain execution data; All execution data is stored in a preset cache dataset; Determine whether the distributed task was executed successfully based on the execution data; If so, then delete the execution data from the cached dataset; Otherwise, the execution data is read from the cached dataset, and the distributed task is executed again based on the execution data; The step of distributing the distributed task to each target node for processing to obtain execution data includes: The distributed task is divided into multiple subtasks, and each subtask includes multiple task steps. The multiple subtasks are distributed to target nodes respectively, and the task steps corresponding to the subtasks are processed on the target nodes to obtain the execution data of each task step; The execution data includes cached data and result data; The step of storing all the execution data in a preset cache dataset includes: Based on the result data, determine whether the task steps corresponding to the subtask under the target node were executed successfully; If so, the cached data and result data corresponding to the task steps are stored in a preset cache dataset according to the execution order of the task steps; Before reading the execution data from the cached dataset and re-executing the distributed task based on the execution data, the method further includes: The type of failure in the distributed task is determined based on the cached data; If the type is a data error, the distributed task will be re-executed according to the native task execution strategy; If the type is a conditional exception, the execution data is read from the cached dataset, and the distributed task is executed again based on the execution data.

2. The distributed task processing method according to claim 1, characterized in that, The step of determining whether the distributed task was executed successfully based on the execution data includes: Based on the results data, determine whether each of the sub-tasks was executed successfully; If so, the distributed task was executed successfully; Otherwise, the distributed task will fail to execute; The execution data to be deleted from the cached dataset includes: Delete all cached data in the cached dataset.

3. The distributed task processing method according to claim 1, characterized in that, The step of reading the execution data from the cached dataset and re-executing the distributed task based on the execution data includes: Read the result data of the task steps corresponding to the subtask under the target node from the cached dataset; Determine whether the task steps were executed successfully based on the result data; If so, the result data of the task step is marked as the input data of the next task step, and the execution order of the next task step is after the task step; Otherwise, the task step is executed based on the result data of the previous task step, and the execution order of the previous task step is before the task step.

4. The distributed task processing method according to claim 1, characterized in that, Storing the execution data in a preset cache dataset further includes: If the task step corresponding to the subtask under the target node fails to execute, the type of failure of the task step is determined according to the cached data of the task step. The cached data, result data, and types corresponding to the task steps are stored in the cached dataset according to the execution order.

5. A distributed task processing method according to claim 4, characterized in that, The types include data errors and conditional anomalies; The step of reading the execution data from the cached dataset and re-executing the distributed task based on the execution data includes: Read the result data of the task steps corresponding to the subtask under the target node from the cached dataset; Determine whether the task steps were executed successfully based on the result data; If so, the result data of the task step is marked as the input data of the next task step, and the execution order of the next task step is after the task step; Otherwise, read the type of task step failure from the cached dataset; If the type is a data error, then read the cached data of the task step in the cached dataset, automatically correct the erroneous data in the cached data, and execute the task step according to the corrected cached data; If the type is a conditional exception, then the task step is executed based on the result data of the previous task step, and the execution order of the previous task step is before the task step.

6. A distributed task processing method according to claim 3 or 5, characterized in that, The step of performing the task based on the result data of the previous task step includes: The subtasks corresponding to the task steps are moved from the target node to other target nodes; Read the result data of the previous task step from the cached dataset on the other target nodes, and execute the task step based on the result data of the previous task step.

7. A distributed task processing terminal, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements each step of the distributed task processing method as described in any one of claims 1-6.

Citation Information

Patent Citations

  • A caching method for a query intermediate result set of a distributed database system

    CN109947796A

  • Data processing method, device and system and computer readable storage medium

    CN111078423A