Task processing method, apparatus, device, and computer-readable storage medium

Abstract

This application provides a task processing method, apparatus, device, and computer-readable storage medium. The task processing process includes a Map task processing stage and a Reduce task processing stage. The Reduce task processing stage includes multiple sub-stages. The method includes: estimating the time consumption percentage of each sub-stage's Reduce task execution; for any i-th Reduce task among the M currently executing Reduce tasks, determining the execution progress of the i-th Reduce task based on the sub-stage in which the i-th Reduce task is located and the time consumption percentage of at least one sub-stage; obtaining the average execution progress based on the execution progress of the M Reduce tasks; and determining a target task whose execution progress is less than or equal to the average execution progress based on the execution progress of each Reduce task and the average execution progress. This application can improve the accuracy of slow task prediction.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a task processing method, apparatus, device and computer-readable storage medium. Background Technology

[0002] In a distributed cluster environment, such as during computation on the Hadoop platform, due to reasons such as user programs, unbalanced load, or uneven resource distribution, the running speed of multiple tasks in the same job may be inconsistent, with some tasks running significantly slower than others, thus slowing down the overall execution speed of the job.

[0003] To improve the overall processing speed of tasks, it is a common practice to start a backup task for slower tasks. The aim is to complete the task as quickly as possible by starting a backup task and using a competitive task competition (the backup task competes with the original task).

[0004] However, the inventors of this application have found that the current solution has low accuracy in evaluating slow tasks. Summary of the Invention

[0005] This application provides a task processing method, apparatus, device, and computer-readable storage medium that can improve the accuracy of slow task evaluation.

[0006] In a first aspect, embodiments of this application provide a task processing method. The task processing process includes a Map task processing stage and a Reduce task processing stage. The Reduce task processing stage includes multiple sub-stages. The method includes: estimating the time consumption percentage of each sub-stage in the multiple sub-stages for executing Reduce tasks; for any i-th Reduce task among the M currently executing Reduce tasks, determining the execution progress of the i-th Reduce task based on the sub-stage in which the i-th Reduce task is located and the time consumption percentage of at least one sub-stage, where i and M are positive integers; obtaining the average execution progress based on the execution progress of the M Reduce tasks; and determining a target task whose execution progress is less than or equal to the average execution progress based on the execution progress of each Reduce task and the average execution progress.

[0007] According to an embodiment of the first aspect of this application, before determining the target task whose execution progress is less than or equal to the average execution progress based on the execution progress and average execution progress of each Reduce task, the method may further include: obtaining a first evaluation parameter, the first evaluation parameter being used to correct the average execution progress; determining the target task whose execution progress is less than or equal to the average execution progress based on the execution progress and average execution progress of each Reduce task, specifically including: for any j-th Reduce task among the M currently executed Reduce tasks, when the execution progress of the j-th Reduce task is less than or equal to the product of a first difference and the average execution progress, the j-th Reduce task is determined as the target task, where j is a positive integer; the first difference is the difference between the value 1 and the first evaluation parameter.

[0008] According to any of the foregoing embodiments of the first aspect of this application, the multiple sub-stages include a shuffle sub-stage, a sort sub-stage, and a reduce sub-stage; estimating the time percentage of each sub-stage in executing the reduce task can specifically include: estimating the time spent executing the reduce task in the shuffle sub-stage, the sort sub-stage, and the reduce sub-stage; and obtaining the time percentage of the reduce task executed in the shuffle sub-stage, the time percentage of the reduce task executed in the sort sub-stage, and the time percentage of the reduce task executed in the reduce sub-stage based on the sum of the time spent executing the reduce task in the shuffle sub-stage, the sort sub-stage, and the reduce sub-stage, and the time percentage of the reduce task executed in each sub-stage.

[0009] According to any of the foregoing embodiments of the first aspect of this application, the execution progress of the i-th Reduce task is determined based on the sub-stage in which the i-th Reduce task is located and the time consumption percentage of at least one sub-stage. Specifically, this may include: determining the task progress based on the data already processed by the i-th Reduce task and the total data that the i-th Reduce task needs to process; when the i-th Reduce task is in the shuffle sub-stage, determining the execution progress of the i-th Reduce task based on the time consumption percentage of the Reduce task in the shuffle sub-stage and the task progress; when the i-th Reduce task is in the sort sub-stage, determining the execution progress of the i-th Reduce task based on the time consumption percentage of the Reduce task in the shuffle sub-stage, the time consumption percentage of the Reduce task in the sort sub-stage, and the task progress; when the i-th Reduce task is in the reduce sub-stage, determining the execution progress of the i-th Reduce task based on the time consumption percentage of the Reduce task in the shuffle sub-stage, the time consumption percentage of the Reduce task in the sort sub-stage, the time consumption percentage of the Reduce task in the reduce sub-stage, and the task progress.

[0010] According to any of the foregoing embodiments of the first aspect of this application, the method may further include: for N nodes executing Map tasks, calculating a first average execution efficiency for each node executing multiple Map tasks; calculating the average of the N first average execution efficiencies to obtain a second average execution efficiency, wherein the N first average execution efficiencies correspond one-to-one with the N nodes; for any p-th node among the N nodes, when the first average execution efficiency of the p-th node is less than the product of the second difference and the second average execution efficiency, determining the p-th node as a first target node, where p is a positive integer; the second difference is the difference between the value 1 and a preset first adjustment coefficient; and removing the first target node.

[0011] According to any of the foregoing embodiments of the first aspect of this application, the method may further include: for Q nodes executing Reduce tasks, calculating the third average execution efficiency of each node executing multiple Reduce tasks; calculating the average of the Q third average execution efficiencies to obtain a fourth average execution efficiency, wherein the Q third average execution efficiencies correspond one-to-one with the Q nodes; for any q-th node among the Q nodes, when the third average execution efficiency of the q-th node is less than the product of the third difference and the fourth average execution efficiency, determining the q-th node as a second target node, where q is a positive integer; the third difference is the difference between the value 1 and a preset second adjustment coefficient; and eliminating the second target node.

[0012] According to any of the foregoing embodiments of the first aspect of this application, after determining the target task whose execution progress is less than or equal to the average execution progress based on the execution progress and average execution progress of each Reduce task, the method may further include: selecting a target backup node; and executing a backup task of the target task based on the target backup node.

[0013] According to any of the foregoing embodiments of the first aspect of this application, selecting a target backup node may specifically include: obtaining basic information of multiple nodes that are communicatively connected to the x-th node and bandwidth information between the multiple nodes and the x-th node, wherein the basic information includes the number of node cores, frequency, and core clock cycle, and x is a positive integer; calculating backup task validity evaluation parameters between the x-th node and the multiple nodes respectively based on the basic information and bandwidth information; and determining the target backup node from the multiple nodes based on the backup task validity evaluation parameters between the x-th node and the multiple nodes when the x-th node performs the target task.

[0014] According to any of the foregoing embodiments of the first aspect of this application, before determining the target backup node from multiple nodes based on the backup task validity evaluation parameters between the x-th node and multiple nodes, the method may further include: for any y-th node, obtaining the probability values ​​of the n nodes communicating with the y-th node appearing in the target task and the backup task validity evaluation parameters between the y-th node and the n nodes respectively, where y and n are both positive integers; determining the potential contribution of the y-th node to the global backup task based on the backup task validity evaluation parameters between the y-th node and the n nodes and the probability values ​​of the n nodes appearing in the target task; determining the target backup node from multiple nodes based on the backup task validity evaluation parameters between the x-th node and multiple nodes may specifically include: determining the target backup node from multiple nodes based on the backup task validity evaluation parameters between the x-th node and multiple nodes and the potential contribution of each node in the multiple nodes to the global backup task.

[0015] Secondly, embodiments of this application provide a task processing apparatus. The task processing process includes a Map task processing stage and a Reduce task processing stage. The Reduce task processing stage includes multiple sub-stages. The apparatus includes: an estimation module for estimating the time consumption percentage of each sub-stage in the multiple sub-stages for executing Reduce tasks; a first determination module for determining the execution progress of any i-th Reduce task among the M currently executing Reduce tasks, based on the sub-stage in which the i-th Reduce task is located and the time consumption percentage of at least one sub-stage, where i and M are positive integers; a second determination module for obtaining the average execution progress based on the execution progress of the M Reduce tasks; and a third determination module for determining a target task whose execution progress is less than or equal to the average execution progress based on the execution progress of each Reduce task and the average execution progress.

[0016] Thirdly, embodiments of this application provide an electronic device, which includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the task processing method provided in the first aspect.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the task processing method provided in the first aspect.

[0018] The task processing method, apparatus, device, and computer-readable storage medium of this application embodiment include a task processing process comprising a Map task processing stage and a Reduce task processing stage. The Reduce task processing stage comprises multiple sub-stages. The method includes: estimating the time consumption percentage of each sub-stage in the multiple sub-stages for executing Reduce tasks; for any i-th Reduce task among the M currently executing Reduce tasks, determining the execution progress of the i-th Reduce task based on the sub-stage in which the i-th Reduce task is located and the time consumption percentage of at least one sub-stage, where i and M are positive integers; obtaining the average execution progress based on the execution progress of the M Reduce tasks; and determining a target task whose execution progress is less than or equal to the average execution progress based on the execution progress of each Reduce task and the average execution progress. This application estimates and divides the Reduce task processing stage, dynamically and accurately determines the execution time ratio of each sub-stage in the Reduce task processing stage, and precisely determines the average execution progress based on the execution time ratio of each sub-stage in the Reduce task processing stage. This is used to evaluate and judge target tasks with slow execution progress (i.e., slow tasks), thereby improving the accuracy of slow task prediction and avoiding redundant creation of slow tasks and oversights in slow task judgment. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a task processing method provided in an embodiment of this application;

[0021] Figure 2 for Figure 1 A flowchart illustrating step S101 in the task processing method shown;

[0022] Figure 3 for Figure 1 A flowchart illustrating step S102 in the task processing method shown;

[0023] Figure 4 Another flowchart illustrating the task processing method provided in this application embodiment;

[0024] Figure 5 Another flowchart illustrating the task processing method provided in this application embodiment;

[0025] Figure 6Another flowchart illustrating the task processing method provided in this application embodiment;

[0026] Figure 7 Another flowchart illustrating the task processing method provided in this application embodiment;

[0027] Figure 8 for Figure 7 A flowchart illustrating step S701 in the task processing method shown;

[0028] Figure 9 This is a schematic diagram illustrating an application scenario of the task processing method provided in the embodiments of this application;

[0029] Figure 10 for Figure 7 Another flowchart of step S701 in the task processing method shown;

[0030] Figure 11 This is a schematic diagram of the structure of the task processing device provided in the embodiments of this application;

[0031] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0033] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0034] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0035] Before describing the technical solutions provided in the embodiments of this application, in order to facilitate understanding of the embodiments of this application, this application first specifically explains the problems existing in the prior art:

[0036] The task in this embodiment can specifically be a MapReduce task, which can also be called a map-reduce task. A MapReduce task can include Map tasks and Reduce tasks. Accordingly, the processing (or scheduling) of a MapReduce task is generally divided into a Map task processing stage and a Reduce task processing stage. The Reduce task processing stage includes multiple sub-stages. For example, the Reduce task processing stage can include a shuffle sub-stage (also known as a shuffling sub-stage), a sort sub-stage (also known as a sorting sub-stage), and a reduce sub-stage (also known as a reduction sub-stage).

[0037] In the shuffle sub-stage, each Map task remotely copies the data it needs to process from the nodes where the Map tasks reside. The shuffle sub-stage ends when all data has been copied. Then, the sort sub-stage begins, which rearranges the input data according to its original order and uses this rearranged data as input for the reduce sub-stage. The reduce sub-stage then passes the sorted data to the Reduce() function for further processing, saving the final results to the Hadoop Distributed File System (HDFS). Once all Reduce tasks in the job are completed, the entire MapReduce job scheduling process is finished.

[0038] In the management of backup tasks, the identification and scheduling of slow tasks are crucial for improving overall computing speed. Slow task retrieval is a key step in the execution of backup tasks on the Hadoop platform. In related technologies, the standard and commonly used algorithms provided by the Hadoop platform divide the Reduce phase into three standard sub-phases (shuffle, sort, and reduce), each accounting for 1 / 3 of the total time, used to identify slow tasks. These algorithms only assume that each sub-phase has the same time commitment; they use a three-segment division for slow task estimation without considering the actual processing time of each sub-phase, leading to significant errors in slow task identification.

[0039] After reviewing existing technical solutions, the inventors of this application have optimized the algorithms for the Reduce task processing stage in existing MapReduce, providing a task processing method, apparatus, device, and computer-readable storage medium that can solve the technical problem of low accuracy in slow task evaluation in existing technologies.

[0040] The technical concept of this application embodiment is as follows: by predicting and dividing the Reduce task processing stage, the execution time ratio of each sub-stage in the Reduce task processing stage is dynamically and accurately determined, and the average execution progress is accurately determined based on the execution time ratio of each sub-stage in the Reduce task processing stage, so as to evaluate and judge the target task with a slow execution progress (i.e., slow task), thereby improving the accuracy of slow task prediction and avoiding redundant creation of slow tasks and omissions in slow task judgment.

[0041] The task processing method provided in the embodiments of this application will be described below.

[0042] Figure 1 This is a flowchart illustrating a task processing method provided in an embodiment of this application. In this embodiment, the task processing includes a Map task processing stage and a Reduce task processing stage, with the Reduce task processing stage comprising multiple sub-stages. For example... Figure 1 As shown, the method may include the following steps:

[0043] S101. Estimate the time percentage of each sub-stage in which the Reduce task is executed;

[0044] S102. For any i-th Reduce task among the M Reduce tasks currently being executed, determine the execution progress of the i-th Reduce task based on the sub-stage in which the i-th Reduce task is located and the time consumption ratio of at least one sub-stage, where i and M are positive integers.

[0045] S103. Based on the execution progress of the M Reduce tasks, obtain the average execution progress;

[0046] S104. Based on the execution progress and average execution progress of each Reduce task, determine the target task whose execution progress is less than or equal to the average execution progress.

[0047] The specific implementation methods of the above steps will be described in detail below.

[0048] The task processing method in this application estimates and divides the Reduce task processing stage, dynamically and accurately determining the execution time ratio of each sub-stage within the Reduce task processing stage. This means it fully considers the actual time ratio of each sub-stage and precisely determines the average execution progress based on this ratio. This average execution progress is then used to evaluate and judge slow-performing target tasks (i.e., slow tasks). Compared to related technologies, the average execution progress determined in this application is closer to the actual average execution progress, meaning it is more accurate. This improves the accuracy of slow task prediction and avoids redundant creation of slow tasks and oversights in slow task identification.

[0049] The specific implementation methods for each of the above steps are described below.

[0050] First, let's introduce S101 and estimate the time spent executing Reduce tasks in each of the multiple sub-stages.

[0051] As described above, in some specific embodiments, the multiple sub-stages of the Reduce task processing phase may include a shuffle sub-stage, a sort sub-stage, and a reduce sub-stage. For example... Figure 2 As shown, optionally, S101 may specifically include the following steps S201 and S202.

[0052] S201. Estimate the time required for the shuffle, sort, and reduce sub-stages to execute the Reduce task.

[0053] For example, in the shuffle sub-stage, the duration of the Reduce task execution in the shuffle sub-stage can be estimated based on the following expression (1):

[0054] T sh =S / R*V net (1)

[0055] Among them, T shThis indicates the time taken to execute Reduce tasks in the shuffle sub-stage; S represents the total amount of data in the Reduce tasks, R represents the total number of Reduce tasks, and V represents the total number of Reduce tasks. net This indicates network bandwidth.

[0056] In the sort sub-stage, the time taken to execute the Reduce task in the sort sub-stage can be estimated based on the following expression (2):

[0057] T so =t*(S / R)*log2(S / R) (2)

[0058] Among them, T so This indicates the time taken to execute the Reduce task in the sort sub-stage, where nlog2n is the algorithm's time complexity, and the size of the data to be merged in each Reduce node is S / R, meaning n can be S / R; t represents the time required to sort 1 byte of data.

[0059] In the reduce sub-stage, the time required to execute the reduce task in the reduce sub-stage can be estimated based on the following expression (3):

[0060] T se =t pre *(S / S pre (3)

[0061] Among them, T se S represents the time taken to execute the Reduce task in the Reduce sub-stage; S represents the total amount of data in the Reduce task; S pre t represents the total amount of data from the previous task of the current node; pre This indicates the time taken for the previous task at the current node.

[0062] S202. Based on the sum of the time spent executing Reduce tasks in the shuffle sub-stage, sort sub-stage, and reduce sub-stage, and the time spent executing Reduce tasks in each sub-stage, obtain the time percentage spent executing Reduce tasks in the shuffle sub-stage, the time percentage spent executing Reduce tasks in the sort sub-stage, and the time percentage spent executing Reduce tasks in the reduce sub-stage.

[0063] For example, for the shuffle sub-stage, the percentage of time spent executing the Reduce task in the shuffle sub-stage can be calculated based on the following expression (4):

[0064]

[0065] Among them, Prosh This indicates the percentage of time spent executing Reduce tasks in the shuffle sub-stage.

[0066] For the sort sub-stage, the time spent executing the Reduce task in the sort sub-stage can be calculated according to the following expression (5):

[0067]

[0068] Among them, Pro so This indicates the percentage of time spent executing Reduce tasks in the sort sub-stage.

[0069] For the reduce sub-stage, the time spent executing the reduce task in the reduce sub-stage can be calculated according to the following expression (6):

[0070]

[0071] Among them, Pro se This indicates the percentage of time spent executing the Reduce task in the Reduce sub-stage.

[0072] This application's embodiments dynamically and accurately determine the execution time ratio of each sub-stage in the Reduce task processing stage, that is, fully consider the actual time ratio of each sub-stage. The average execution progress determined in this way is closer to the average execution progress in the actual situation, that is, the average execution progress is more accurate, thereby improving the accuracy of slow task prediction and avoiding redundant creation of slow tasks and omissions in slow task judgment.

[0073] The above describes the specific implementation of S101. The following describes the specific implementation of S102.

[0074] S102. For any i-th Reduce task among the M Reduce tasks currently being executed, determine the execution progress of the i-th Reduce task based on the sub-stage in which the i-th Reduce task is located and the time consumption ratio of at least one sub-stage.

[0075] like Figure 3 As shown, according to some embodiments of this application, optionally, S102 may specifically include the following steps S301 to S304.

[0076] S301. Determine the task progress based on the data already processed by the i-th Reduce task and the total data that the i-th Reduce task needs to process.

[0077] Specifically, when calculating the execution progress of a Reduce task, we assume that the number of Reduce tasks is M, the task is currently in the s'th sub-stage, the data that the Reduce task currently needs to process (or the total data that the Reduce task needs to process) is B, the data that the Reduce task has already processed is C, the task progress is P, and the execution progress of the Reduce task (or the current task progress as a percentage of the overall task progress) is SP.

[0078] In S301, the task progress can be calculated based on the following expression (7):

[0079]

[0080] Where P represents the progress of the i-th Reduce task, C represents the data that the i-th Reduce task has completed processing, and B represents the total data that the i-th Reduce task needs to process.

[0081] S302. When the i-th Reduce task is in the shuffle sub-stage, determine the execution progress of the i-th Reduce task based on the time consumption of the Reduce task in the shuffle sub-stage and the task progress.

[0082] For example, when the i-th Reduce task is in the shuffle sub-stage, i.e., s' = 1, the execution progress of the i-th Reduce task can be calculated according to the following expression (8):

[0083] SP = Pro sh ·P (8)

[0084] S303. When the i-th Reduce task is in the sort sub-stage, determine the execution progress of the i-th Reduce task based on the time spent executing Reduce tasks in the shuffle sub-stage, the time spent executing Reduce tasks in the sort sub-stage, and the task progress.

[0085] For example, when the i-th Reduce task is in the sort sub-stage, i.e., s' = 2, the execution progress of the i-th Reduce task can be calculated according to the following expression (9):

[0086] SP = Pro sh +Pro so ·P (9)

[0087] S304. When the i-th Reduce task is in the reduce sub-stage, determine the execution progress of the i-th Reduce task based on the time spent executing Reduce tasks in the shuffle sub-stage, the time spent executing Reduce tasks in the sort sub-stage, the time spent executing Reduce tasks in the reduce sub-stage, and the task progress.

[0088] For example, when the i-th Reduce task is in the reduce sub-stage, i.e., s' = 3, the execution progress of the i-th Reduce task can be calculated according to the following expression (10):

[0089] SP = Pro sh +Pro so +Pro se ·P (10)

[0090] The above describes the specific implementation of S102. The following describes the specific implementation of S103.

[0091] S103. Based on the execution progress of the M Reduce tasks, obtain the average execution progress.

[0092] Specifically, the average execution progress of M Reduce tasks can be calculated to obtain the average execution progress.

[0093] For example, the average execution progress can be calculated based on the following expression (11):

[0094]

[0095] Here, AvgSP represents the average execution progress of M Reduce tasks.

[0096] The above describes the specific implementation of S103. The following describes the specific implementation of S104.

[0097] S104. Based on the execution progress and average execution progress of each Reduce task, determine the target task whose execution progress is less than or equal to the average execution progress.

[0098] Specifically, in some examples, the execution progress of each Reduce task can be compared with the average execution progress to identify target tasks whose execution progress is less than or equal to the average execution progress.

[0099] In other examples, the average execution progress can be corrected using a first evaluation parameter, thereby making the final slow task prediction result more accurate. Specifically, before S104, a first evaluation parameter can be obtained, which is used to correct the average execution progress. The specific value of the first evaluation parameter can be flexibly adjusted according to the actual situation. For example, the first evaluation parameter can be determined based on experience or through simulation training using historical data; this embodiment does not limit this. Accordingly, S104 may specifically include the following steps: For any j-th Reduce task among the M currently executing Reduce tasks, when the execution progress of the j-th Reduce task is less than or equal to the product of the first difference and the average execution progress, the j-th Reduce task is determined as the target task. j is a positive integer, and the first difference is the difference between the value 1 and the first evaluation parameter.

[0100] That is, when SP≤(1-Slow) task When performing AvgSP, the Reduce task is identified as the target task (i.e., the slow task). Among these, Slow... task This represents the first evaluation parameter.

[0101] To facilitate understanding, the following will be combined with... Figure 4 The specific application examples shown are illustrated below.

[0102] like Figure 4 As shown, in some specific application embodiments, optionally, the task processing method of this application embodiment may include the following steps S401 to S409.

[0103] S401, The task manager obtains local information about the Reduce tasks. The task manager is also known as Task-management.

[0104] S402. Estimate the time percentage of each sub-stage in executing Reduce tasks, and obtain the average execution progress based on the execution progress of the M Reduce tasks. Step S402 can be understood as steps S101 to S103 above, and will not be elaborated further for the sake of brevity.

[0105] S403, Identify lagging Reduce tasks.

[0106] S404. For a lagging Reduce task, determine whether the Reduce task is the task with the longest remaining time in the queue. If so, proceed to step S405.

[0107] S405. Determine whether the Reduce task meets the slow task evaluation criteria. If so, proceed to step S406. Step S405 can be understood as step S104 above, which compares the execution progress of the Reduce task with the average execution progress to determine whether the execution progress of the Reduce task is less than or equal to the average execution progress.

[0108] S406. Determine whether the running time of the Reduce task exceeds the predetermined threshold. If so, proceed to step S407.

[0109] S407. Determine whether the system backup task has reached the upper limit standard. If not, proceed to step S408.

[0110] S408, Task Manager starts backup task.

[0111] S409. Delete the information of the lagging Reduce task from the slow task queue.

[0112] Considering the differences in execution between the Map task processing stage and the Reduce task processing stage, slow nodes (i.e., nodes that are slow in executing Map or Reduce tasks) can be removed when starting the backup task to speed up the overall execution efficiency. Therefore, slow nodes can be distinguished and retrieved.

[0113] Specifically, such as Figure 5 As shown, according to some embodiments of this application, optionally, the task processing method of this application embodiment may further include the following steps S501 to S504.

[0114] S501. For N nodes executing Map tasks, calculate the first average execution efficiency of each node executing multiple Map tasks.

[0115] For example, the first average execution efficiency of the i-th node performing multiple Map tasks can be calculated according to the following expression (12):

[0116]

[0117] Among them, ProgressRateMap i CountMap represents the first average execution efficiency of multiple Map tasks executed by the i-th node; ProgressRate represents the execution efficiency of each Map task.

[0118] S502. Calculate the average of the N first average execution efficiencies to obtain the second average execution efficiency. The N first average execution efficiencies correspond one-to-one with the N nodes.

[0119] Specifically, the second average execution efficiency can be calculated based on the following expression (13):

[0120]

[0121] Here, AvgProgressRateMap represents the second average execution efficiency.

[0122] S503. For any p-th node among N nodes, when the first average execution efficiency of the p-th node is less than the product of the second difference and the second average execution efficiency, the p-th node is determined as the first target node, where p is a positive integer; the second difference is the difference between the value 1 and the preset first adjustment coefficient.

[0123] Specifically, for example, when ProgressRateMap < (1 - SlowParameter) * Avg ProgressRateMap, the node is determined as the first target node (i.e., the slow node). Here, SlowParameter represents the first adjustment coefficient. The specific value of the first adjustment coefficient can be flexibly adjusted according to the actual situation. For example, the first adjustment coefficient can be determined based on experience or through simulation training using historical data; this embodiment does not limit this.

[0124] S504, Remove the first target node.

[0125] Therefore, by eliminating slow nodes that execute Map tasks, the overall execution efficiency is accelerated.

[0126] Similarly, such as Figure 6 As shown, according to some embodiments of this application, optionally, the task processing method of this application embodiment may further include the following steps S601 to S604.

[0127] S601. For Q nodes executing Reduce tasks, calculate the third average execution efficiency of each node executing multiple Reduce tasks. Where Q is a positive integer.

[0128] S602. Calculate the average of the Q third average execution efficiencies to obtain the fourth average execution efficiency. The Q third average execution efficiencies correspond one-to-one with the Q nodes.

[0129] S603. For any q-th node among the Q nodes, when the third average execution efficiency of the q-th node is less than the product of the third difference and the fourth average execution efficiency, the q-th node is determined as the second target node, where q is a positive integer; the third difference is the difference between the value 1 and the preset second adjustment coefficient. The specific value of the second adjustment coefficient can be flexibly adjusted according to the actual situation. For example, the second adjustment coefficient can be determined based on experience or through simulation training using historical data; this embodiment does not limit this.

[0130] S604, Remove the second target node.

[0131] It should be noted that the specific implementation process of steps S601 to S604 is similar to that of steps S501 to S504, and will not be repeated here for the sake of brevity.

[0132] Therefore, by eliminating slow nodes that execute Reduce tasks, the overall execution efficiency can be further accelerated.

[0133] The inventors of this application further discovered that although slow nodes have been removed in the previous step, the Hadoop system consists of multiple racks and even different data centers. Due to the influence of cross-regional and multi-network factors, the data transmission rate within a rack is much higher than the data transmission rate between racks. Therefore, it is necessary to further select the execution nodes for backup tasks and schedule backup tasks. When creating backup tasks, all nodes in the Hadoop system cannot be treated the same without distinction. Instead, based on the judgment of slow task nodes, differences in distance and bandwidth between nodes need to be considered. Therefore, this application's embodiments create node backup task validity evaluation parameters to evaluate the scheduling cost of tasks between nodes.

[0134] Specifically, such as Figure 7 As shown, according to some embodiments of this application, optionally, the task processing method of this application embodiment may further include the following steps S701 and S702.

[0135] S701, Select the target backup node.

[0136] S702, Execute the backup task of the target task based on the target backup node.

[0137] like Figure 8 As shown, in some specific embodiments, optionally, S701, selecting the target backup node, may specifically include the following steps S801 to S803.

[0138] S801. Obtain basic information of multiple nodes that are connected to the x-th node and bandwidth information between the multiple nodes and the x-th node. The basic information includes the number of node cores, frequency and core clock cycle, where x is a positive integer.

[0139] S802. Based on the basic information and bandwidth information, calculate the backup task validity evaluation parameters between the x-th node and multiple nodes.

[0140] like Figure 9 As shown, the backup task validity evaluation parameter K nm This is used to describe the validity of backup task scheduling between node n and node m. For any node among multiple nodes, the validity evaluation parameters of the backup task between the x-th node and the node can be calculated based on the basic information of the node and the bandwidth information between the node and the x-th node.

[0141] For example, for any node among multiple nodes, the backup task validity evaluation parameters between the x-th node and any node can be calculated according to the following expressions (14) and (15):

[0142] K nm =V net ·C al (14)

[0143] C al =w·λ·μ (15)

[0144] Among them, K nm V represents the parameter for evaluating the validity of the backup task between the x-th node and any i-th node; net C represents the bandwidth between the x-th node and any i-th node; al λ represents the computing power of the i-th node; w represents the number of node cores of the i-th node; λ represents the frequency of the i-th node; μ represents the core clock cycle of the i-th node.

[0145] It should be noted that K obtained from expression (14) nm This is the initial value used to initialize and measure the task backup efficiency between different nodes. In actual operation, it will be dynamically replaced by the unit data processing efficiency. Where S represents the total amount of data in the task, and T represents the execution time.

[0146] S803. When the target task is executed at the x-th node, the target backup node is determined from the multiple nodes based on the backup task validity evaluation parameters between the x-th node and multiple nodes.

[0147] For example, from multiple nodes corresponding to multiple backup task validity evaluation parameters, the node corresponding to the largest backup task validity evaluation parameter is selected as the target backup node.

[0148] like Figure 10 As shown, according to some embodiments of this application, optionally, before determining the target backup node from multiple nodes based on the backup task validity evaluation parameters between the xth node and multiple nodes in S803, the method may further include the following steps S901 and S902.

[0149] S901. For any node y, obtain the probability values ​​of the target task appearing in the n nodes that are communicatively connected to node y, and the backup task validity evaluation parameters between node y and the n nodes respectively, where y and n are both positive integers. Here, node y can be understood as any node.

[0150] S902. Based on the backup task validity evaluation parameters between the y-th node and the n-th node, and the probability values ​​of the n-th node appearing in the target task, determine the potential contribution of the y-th node to the global backup task.

[0151] For example, the potential contribution of the y-th node to the global backup task can be calculated based on the following expression (16):

[0152]

[0153] Where M represents the potential contribution of the y-th node to the global backup task; K nmi Z represents the parameter for evaluating the validity of the backup task between the y-th node and the i-th node; i This represents the probability value of the target task appearing at the i-th node.

[0154] Accordingly, S803, based on the backup task validity evaluation parameters between the x-th node and multiple nodes, determines the target backup node from the multiple nodes, which may specifically include:

[0155] Based on the backup task validity evaluation parameters between the x-th node and multiple nodes, and the potential contribution of each node in the multiple nodes to the global backup task, the target backup node is determined from the multiple nodes.

[0156] For example, a multi-objective solution can be performed to calculate the backup task validity evaluation parameters and potential contributions to determine the target backup node. For instance, the node corresponding to the highest backup task validity evaluation parameter and / or the node with the lowest potential contribution to the global backup task can be selected as the target backup node. For example, in some examples, among nodes A, B, and C, node A has the highest backup task validity evaluation parameter and the lowest potential contribution to the global backup task; therefore, node A is selected as the target backup node. As another example, when node A has the highest backup task validity evaluation parameter but node B has the lowest potential contribution to the global backup task, either node A or node B can be selected as the target backup node.

[0157] The task processing method provided in this application embodiment, based on general MapReduce, estimates and divides the Reduce task processing stage, which can accurately estimate the execution time ratio of the three sub-stages of shuffle, sort and reduce. Then, based on the estimated time and the global task evaluation execution progress, slow tasks are identified. When starting the backup task, slow tasks are distinguished, retrieved and removed, which can speed up the overall execution efficiency.

[0158] The task processing method provided in this application, in addition to considering slow task nodes in the backup task scheduling, also creates node backup task validity evaluation parameters to evaluate the scheduling between nodes. When any node experiences a task anomaly, it solves the multi-objective problem based on the current node's optimal backup node and the globally optimal backup node, thereby completing the task backup strategy execution. This can effectively improve the backup task creation efficiency and satisfy global optimality.

[0159] Compared with the existing technology that randomly selects task nodes to execute backup tasks during the slow task scheduling process, the embodiments of this application introduce backup task validity evaluation parameters to evaluate the scheduling between nodes, taking into account factors such as cluster heterogeneity during the backup task creation and scheduling process. This can effectively improve the efficiency of backup task creation and meet global optimality.

[0160] Based on the task processing method provided in the above embodiments, this application also provides specific implementations of the task processing apparatus. Please refer to the following embodiments.

[0161] The task processing includes a Map task processing phase and a Reduce task processing phase, with the Reduce task processing phase comprising multiple sub-phases. For example... Figure 11 As shown, the task processing device 100 provided in this application embodiment includes the following modules:

[0162] The estimation module 1001 is used to estimate the time consumption of each sub-stage in executing the Reduce task in multiple sub-stages;

[0163] The first determining module 1002 is used to determine the execution progress of any i-th Reduce task among the M Reduce tasks currently being executed, based on the sub-stage in which the i-th Reduce task is located and the time consumption ratio of at least one sub-stage, where i and M are positive integers.

[0164] The second determining module 1003 is used to obtain the average execution progress based on the execution progress of the M Reduce tasks;

[0165] The third determination module 1004 is used to determine the target task whose execution progress is less than or equal to the average execution progress based on the execution progress and average execution progress of each Reduce task.

[0166] The task processing apparatus of this application embodiment estimates and divides the Reduce task processing stage, dynamically and accurately determines the execution time ratio of each sub-stage in the Reduce task processing stage, and accurately determines the average execution progress based on the execution time ratio of each sub-stage in the Reduce task processing stage. This is used for the evaluation and judgment of target tasks with slow execution progress (i.e., slow tasks), thereby improving the accuracy of slow task estimation and avoiding redundant creation of slow tasks and oversight in slow task judgment.

[0167] In some embodiments, the third determining module 1004 is specifically used to: for any j-th Reduce task among the M Reduce tasks currently being executed, when the execution progress of the j-th Reduce task is less than or equal to the product of the first difference and the average execution progress, determine the j-th Reduce task as the target task, where j is a positive integer; the first difference is the difference between the value 1 and the first evaluation parameter.

[0168] In some embodiments, the multiple sub-stages include a shuffle sub-stage, a sort sub-stage, and a reduce sub-stage; the estimation module 1001 is specifically used to: estimate the time consumed by the shuffle sub-stage, the sort sub-stage, and the reduce sub-stage in executing the reduce task; and obtain the time consumption percentage of the shuffle sub-stage in executing the reduce task, the time consumption percentage of the sort sub-stage in executing the reduce task, and the time consumption percentage of the reduce task in executing the reduce task in the reduce sub-stage based on the sum of the time consumed by the shuffle sub-stage, the sort sub-stage, and the reduce sub-stage.

[0169] In some embodiments, the first determining module 1002 is specifically used to: determine the task progress based on the data already processed by the i-th Reduce task and the total data that the i-th Reduce task needs to process; when the i-th Reduce task is in the shuffle sub-stage, determine the execution progress of the i-th Reduce task based on the time consumption percentage of the Reduce task in the shuffle sub-stage and the task progress; when the i-th Reduce task is in the sort sub-stage, determine the execution progress of the i-th Reduce task based on the time consumption percentage of the Reduce task in the shuffle sub-stage, the time consumption percentage of the Reduce task in the sort sub-stage, and the task progress; when the i-th Reduce task is in the reduce sub-stage, determine the execution progress of the i-th Reduce task based on the time consumption percentage of the Reduce task in the shuffle sub-stage, the time consumption percentage of the Reduce task in the sort sub-stage, the time consumption percentage of the Reduce task in the reduce sub-stage, and the task progress.

[0170] In some embodiments, the task processing apparatus 100 provided in this application further includes a node elimination module, which is used to calculate the first average execution efficiency of each node executing multiple Map tasks for N nodes executing Map tasks; calculate the average of the N first average execution efficiencies to obtain a second average execution efficiency, wherein the N first average execution efficiencies correspond one-to-one with the N nodes; for any p-th node among the N nodes, when the first average execution efficiency of the p-th node is less than the product of the second difference and the second average execution efficiency, determine the p-th node as a first target node, where p is a positive integer; the second difference is the difference between the value 1 and a preset first adjustment coefficient; and eliminate the first target node.

[0171] In some embodiments, the node elimination module is further configured to: for Q nodes executing Reduce tasks, calculate the third average execution efficiency of each node executing multiple Reduce tasks; calculate the average of the Q third average execution efficiencies to obtain the fourth average execution efficiency, wherein the Q third average execution efficiencies correspond one-to-one with the Q nodes; for any q-th node among the Q nodes, when the third average execution efficiency of the q-th node is less than the product of the third difference and the fourth average execution efficiency, determine the q-th node as the second target node, where q is a positive integer; the third difference is the difference between the value 1 and the preset second adjustment coefficient; and eliminate the second target node.

[0172] In some embodiments, the task processing device 100 provided in this application further includes an execution module for selecting a target backup node and executing a backup task of the target task based on the target backup node.

[0173] In some embodiments, the execution module is specifically used to obtain basic information of multiple nodes that are communicatively connected to the x-th node and bandwidth information between the multiple nodes and the x-th node. The basic information includes the number of node cores, frequency, and core clock cycle, where x is a positive integer. Based on the basic information and bandwidth information, the module calculates the backup task validity evaluation parameters between the x-th node and the multiple nodes respectively. When the x-th node performs the target task, the module determines the target backup node from the multiple nodes based on the backup task validity evaluation parameters between the x-th node and the multiple nodes.

[0174] In some embodiments, the execution module is specifically used to, for any y-th node, obtain the probability values ​​of the n nodes communicating with the y-th node appearing in the target task and the backup task validity evaluation parameters between the y-th node and the n nodes respectively, where y and n are both positive integers; determine the potential contribution of the y-th node to the global backup task based on the backup task validity evaluation parameters between the y-th node and the n nodes and the probability values ​​of the n nodes appearing in the target task; and determine the target backup node from the multiple nodes based on the backup task validity evaluation parameters between the x-th node and multiple nodes and the potential contribution of each of the multiple nodes to the global backup task.

[0175] Figure 11 Each module / unit in the illustrated device has the ability to implement Figure 1 The functions of each step in the process and their corresponding technical effects are explained in detail here for the sake of brevity.

[0176] Based on the task processing method provided in the above embodiments, this application also provides specific implementation methods for electronic devices. Please refer to the following embodiments.

[0177] Figure 12 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0178] The electronic device may include a processor 1101 and a memory 1102 storing computer program instructions.

[0179] Specifically, the processor 1101 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0180] Memory 1102 may include mass storage for data or instructions. For example, and not limitingly, memory 1102 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one example, memory 1102 may include removable or non-removable (or fixed) media, or memory 1102 may be non-volatile solid-state memory. Memory 1102 may be internal or external to the integrated gateway disaster recovery device.

[0181] In one example, memory 1102 may be read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0182] Memory 1102 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this application.

[0183] The processor 1101 reads and executes computer program instructions stored in the memory 1102 to achieve... Figure 1 The method / steps S101 to S104 in the illustrated embodiment achieve the following: Figure 1 The technical effects achieved by executing the methods / steps shown in the examples are not elaborated here for the sake of brevity.

[0184] In one example, the electronic device may also include a communication interface 1103 and a bus 1110. Wherein, for example... Figure 12 As shown, the processor 1101, memory 1102, and communication interface 1103 are connected through bus 1110 and communicate with each other.

[0185] The communication interface 1103 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0186] Bus 1110 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 1110 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0187] Furthermore, in conjunction with the task processing methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the task processing methods in the above embodiments. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as electronic circuits, semiconductor memory devices, ROM, random access memory, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, and hard disks.

[0188] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0189] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet or intranets.

[0190] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0191] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0192] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A task processing method, characterized in that, The task processing includes a Map task processing stage and a Reduce task processing stage, wherein the Reduce task processing stage includes multiple sub-stages, and the method includes: Estimate the percentage of time spent executing Reduce tasks in each of the multiple sub-stages; For any i-th Reduce task among the M Reduce tasks currently being executed, the execution progress of the i-th Reduce task is determined based on the sub-stage in which the i-th Reduce task is located and the time consumption ratio of at least one of the sub-stages, where i and M are positive integers. The average execution progress is obtained based on the execution progress of the M Reduce tasks; Based on the execution progress of each Reduce task and the average execution progress, determine the target task whose execution progress is less than or equal to the average execution progress; Before determining the target task whose execution progress is less than or equal to the average execution progress based on the execution progress of each Reduce task and the average execution progress, the method further includes: Obtain a first evaluation parameter, which is used to correct the average execution progress; The step of determining the target task whose execution progress is less than or equal to the average execution progress based on the execution progress of each Reduce task and the average execution progress includes: For any j-th Reduce task among the M Reduce tasks currently being executed, when the execution progress of the j-th Reduce task is less than or equal to the product of the first difference and the average execution progress, the j-th Reduce task is determined to be the target task, where j is a positive integer; the first difference is the difference between the value 1 and the first evaluation parameter. After determining the target task whose execution progress is less than or equal to the average execution progress based on the execution progress of each Reduce task and the average execution progress, the method further includes: Select the target backup node; The backup task of the target task is executed based on the target backup node; The selection of the target backup node specifically includes: Obtain basic information about multiple nodes that are communicatively connected to the x-th node and bandwidth information between the multiple nodes and the x-th node. The basic information includes the number of node cores, frequency, and core clock cycle, where x is a positive integer. Based on the basic information and the bandwidth information, calculate the backup task validity evaluation parameters between the x-th node and the multiple nodes respectively; When the target task is executed at the x-th node, the target backup node is determined from the multiple nodes based on the backup task validity evaluation parameters between the x-th node and the multiple nodes. Before determining the target backup node from the plurality of nodes based on the backup task validity evaluation parameters between the x-th node and the plurality of nodes, the method further includes: For any y-th node, obtain the probability values ​​of the n nodes that are communicatively connected to the y-th node appearing in the target task, as well as the backup task validity evaluation parameters between the y-th node and the n nodes respectively, where y and n are both positive integers; Based on the backup task validity evaluation parameters between the y-th node and the n-th node, and the probability values ​​of the n-th node appearing in the target task, the potential contribution of the y-th node to the global backup task is determined. The step of determining the target backup node from the plurality of nodes based on the backup task validity evaluation parameters between the x-th node and the plurality of nodes specifically includes: The target backup node is determined from the plurality of nodes based on the backup task validity evaluation parameters between the x-th node and the plurality of nodes, and the potential contribution of each of the plurality of nodes to the global backup task.

2. The method according to claim 1, characterized in that, The multiple sub-stages include the shuffle sub-stage, the sort sub-stage, and the reduce sub-stage; The estimated time percentage for each of the multiple sub-stages in executing the Reduce task specifically includes: Estimate the time required for the shuffle sub-stage, the sort sub-stage, and the reduce sub-stage to execute the Reduce task; Based on the sum of the time spent executing Reduce tasks in the shuffle sub-stage, the sort sub-stage, and the reduce sub-stage, as well as the time spent executing Reduce tasks in each sub-stage, the percentage of time spent executing Reduce tasks in the shuffle sub-stage, the percentage of time spent executing Reduce tasks in the sort sub-stage, and the percentage of time spent executing Reduce tasks in the reduce sub-stage are obtained.

3. The method according to claim 2, characterized in that, The execution progress of the i-th Reduce task is determined based on the sub-stage in which the i-th Reduce task is located and the time consumption percentage of at least one of the sub-stages, specifically including: The task progress is determined based on the data already processed by the i-th Reduce task and the total data that the i-th Reduce task needs to process. When the i-th Reduce task is in the shuffle sub-stage, the execution progress of the i-th Reduce task is determined based on the proportion of time spent executing Reduce tasks in the shuffle sub-stage and the task progress. When the i-th Reduce task is in the sort sub-stage, the execution progress of the i-th Reduce task is determined based on the time consumption percentage of the Reduce task in the shuffle sub-stage, the time consumption percentage of the Reduce task in the sort sub-stage, and the task progress. When the i-th Reduce task is in the reduce sub-stage, the execution progress of the i-th Reduce task is determined based on the time spent executing Reduce tasks in the shuffle sub-stage, the time spent executing Reduce tasks in the sort sub-stage, the time spent executing Reduce tasks in the reduce sub-stage, and the task progress.

4. The method according to claim 1, characterized in that, The method further includes: For N nodes executing Map tasks, calculate the first average execution efficiency of each node executing multiple Map tasks; Calculate the average of the N first average execution efficiencies to obtain the second average execution efficiency, and the N first average execution efficiencies correspond one-to-one with the N nodes; For any p-th node among the N nodes, when the first average execution efficiency of the p-th node is less than the product of the second difference and the second average execution efficiency, the p-th node is determined as the first target node, where p is a positive integer; the second difference is the difference between the value 1 and the preset first adjustment coefficient. Remove the first target node.

5. The method according to claim 1, characterized in that, The method further includes: For Q nodes executing Reduce tasks, calculate the third average execution efficiency of each node executing multiple Reduce tasks; Calculate the average of the Q third average execution efficiencies to obtain the fourth average execution efficiency, and the Q third average execution efficiencies correspond one-to-one with the Q nodes; For any q-th node among the Q nodes, when the third average execution efficiency of the q-th node is less than the product of the third difference and the fourth average execution efficiency, the q-th node is determined to be the second target node, where q is a positive integer; the third difference is the difference between the value 1 and the preset second adjustment coefficient; Remove the second target node.

6. A task processing device, characterized in that, The task processing includes a Map task processing stage and a Reduce task processing stage, wherein the Reduce task processing stage includes multiple sub-stages, and the apparatus includes: The estimation module is used to estimate the time consumption percentage of each of the multiple sub-stages in executing the Reduce task; The first determining module is used to determine the execution progress of any i-th Reduce task among the M currently executing Reduce tasks, based on the sub-stage in which the i-th Reduce task is located and the time consumption ratio of at least one of the sub-stages, where i and M are positive integers. The second determining module is used to obtain the average execution progress based on the execution progress of the M Reduce tasks; The third determining module is used to determine the target task whose execution progress is less than or equal to the average execution progress based on the execution progress of each Reduce task and the average execution progress. The third determining module is further configured to obtain a first evaluation parameter, which is used to correct the average execution progress; for any j-th Reduce task among the M currently executed Reduce tasks, when the execution progress of the j-th Reduce task is less than or equal to the product of the first difference and the average execution progress, the j-th Reduce task is determined as the target task, where j is a positive integer; the first difference is the difference between the value 1 and the first evaluation parameter; The device further includes an execution module for selecting a target backup node and executing a backup task based on the target backup node. The execution module is further configured to acquire basic information of multiple nodes that are communicatively connected to the x-th node and bandwidth information between the multiple nodes and the x-th node. The basic information includes the number of node cores, frequency, and core clock cycle, where x is a positive integer. Based on the basic information and bandwidth information, the module calculates the backup task validity evaluation parameters between the x-th node and the multiple nodes respectively. When the x-th node executes the target task, the module determines the target backup node from the multiple nodes based on the backup task validity evaluation parameters between the x-th node and the multiple nodes. The execution module is further configured to, for any y-th node, obtain the probability values ​​of the n nodes communicating with the y-th node appearing in the target task and the backup task validity evaluation parameters between the y-th node and the n nodes respectively, where y and n are both positive integers; determine the potential contribution of the y-th node to the global backup task based on the backup task validity evaluation parameters between the y-th node and the n nodes and the probability values ​​of the n nodes appearing in the target task; and determine the target backup node from the multiple nodes based on the backup task validity evaluation parameters between the x-th node and multiple nodes, and the potential contribution of each of the multiple nodes to the global backup task.

7. An electronic device, characterized in that, The electronic device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the task processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the task processing method as described in any one of claims 1 to 5.

Images