Target node selection method and apparatus, computer device, and medium

Abstract

The application relates to the technical field of computers and discloses a target node selection method and device, computer equipment and a medium, the method comprising the following steps: when detecting that an executor in a Spark distributed job fails, checkpoint information corresponding to the executor is obtained from a preset storage position; the checkpoint information is scheduled to an original node on which the executor runs before the failure; when it is determined that the remaining resources on the original node are insufficient to start a recovery executor, the cost of transferring resources on the original node is evaluated to obtain a first evaluation result; the cost of assigning the executor to other nodes to start recovery is evaluated to obtain a second evaluation result; and the target node for performing a start-recovery operation on the executor is determined according to the first evaluation result and the second evaluation result. In this way, the speed of the executor performing the start-recovery operation is improved, and time consumption is reduced.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a method, apparatus, computer equipment, and medium for selecting target nodes. Background Technology

[0002] Spark is a member of the big data ecosystem and is an in-memory computing framework. Spark distributed jobs can run on Yarn clusters. A Spark job consists of a driver and an executor. The driver coordinates the job and result collection. The executor is the workload for executing the job. Spark supports automatic recovery when an executor encounters a failure. To recover to a previous state after a failure, the executor periodically extracts its runtime state as checkpoint information and saves it to storage. During failure recovery, the executor reads the most recently successfully saved checkpoint information, thus restoring to the state closest to the moment before the failure.

[0003] However, during the executor failure recovery process, it is necessary to configure the target node to complete the executor failure recovery operation. How to select the target node is the technical problem to be solved in this application. Summary of the Invention

[0004] In view of this, the present invention provides a target node selection method, apparatus, computer equipment and medium to solve the problem of how to select a target node to complete the executor fault recovery operation during the executor fault recovery process.

[0005] In a first aspect, the present invention provides a target node selection method, which is applied to Spark distributed jobs distributed across different nodes. The method includes:

[0006] When a failure is detected in the executor of a Spark distributed job, the checkpoint information corresponding to the executor is retrieved from the preset storage location.

[0007] The checkpoint information is scheduled to the original node where the actuator was running before the failure, so that the start-up and recovery operation of the actuator can be completed on the original node;

[0008] When it is determined that the remaining resources on the original node are insufficient to start the recovery executor, the cost of transferring the resources on the original node is evaluated, and a first evaluation result is obtained;

[0009] In addition, the cost of assigning executors to other nodes to initiate recovery is evaluated, and a second evaluation result is obtained;

[0010] Based on the results of the first and second assessments, the target node for performing the start-up recovery operation on the actuator is determined.

[0011] The target node selection method provided by this invention has the following advantages:

[0012] When an executor failure is detected in a Spark distributed job, checkpoint information corresponding to the executor is retrieved from a preset storage location. This checkpoint information is then scheduled to the original node where the executor was running before the failure, thus completing the executor's startup and recovery operation. The reason for scheduling the checkpoint information to the original node to complete the executor startup and recovery operation is that this eliminates the need to execute Yarn's traditional scheduling process, reducing operational steps and improving efficiency.

[0013] However, in certain special cases, such as when the remaining resources on the original node are insufficient to start the recovery executor, it may be considered to transfer the job from the original node to other nodes so that the executor's startup and recovery operations can continue on the original node. Alternatively, the executor's checkpoint information can be distributed to other nodes to complete the startup and recovery operations. Specifically, the execution costs of these two scenarios can be evaluated, and the target node for the executor's recovery operation can be determined based on the evaluation results. The purpose of this process is also to maximize the speed of the executor's startup and recovery operations and minimize time consumption.

[0014] In one optional implementation, when an executor failure is detected in a Spark distributed job, checkpoint information corresponding to the executor is retrieved from a preset storage location, specifically including:

[0015] Detect whether there are reused nodes in Spark distributed jobs and distributed file systems;

[0016] When it is determined that there are reuse nodes between the Spark distributed job and the distributed file system, and the checkpoint information corresponding to the executor is located on the reuse node, the checkpoint information is retrieved from the reuse node;

[0017] or,

[0018] If it is determined that there is no reuse node between the Spark distributed job and the distributed file system, or that the checkpoint information corresponding to the executor is not stored on the reuse node, check whether checkpoint information exists in the local environment where the Spark distributed job is located.

[0019] When it is determined that checkpoint information is stored in the local environment, the checkpoint information is read from the local environment.

[0020] or,

[0021] If it is determined that there is no reuse node between the Spark distributed job and the distributed file system, or that the checkpoint information corresponding to the executor is not stored on the reuse node, and the checkpoint information is not stored in the local environment, then the checkpoint information is read from the distributed file system.

[0022] Specifically, checkpoint information is stored separately in the distributed file system and locally for safety. The reason for storing it locally is that during the later startup and recovery operations of the executor, the checkpoint information can be retrieved directly from the local machine without needing to access the distributed file system, saving data scheduling time and improving fault recovery efficiency. However, there is a concern that the locally stored checkpoint information might fail, so a backup is needed in the distributed file system. However, when there is a shared node between the Spark distributed job and the distributed file system, and the checkpoint information corresponding to the executor is located on the shared node, the checkpoint information can be directly stored on that shared node, satisfying all the above requirements. Therefore, when there is a shared node between the Spark distributed job and the distributed file system, and the checkpoint information corresponding to the executor is located on the shared node, the checkpoint information can be read directly from the shared node. Similarly, if it is determined that there is no shared node between the Spark distributed job and the distributed file system, or that the checkpoint information corresponding to the executor is not stored on a shared node, then it is further determined whether the checkpoint information is stored in the local environment. If the checkpoint information is stored in the local environment, it is read directly from the local environment. Only when neither of the first two conditions is met will checkpoint information be read from the distributed file system.

[0023] In one alternative implementation, when it is determined that the remaining resources on the original node are sufficient to start the recovery executor, but the number of failures of the original node in starting and recovering the executor still exceeds a preset threshold, the checkpoint information is directly transferred to other nodes, and the start and recovery operation of the executor is completed on other nodes.

[0024] Specifically, if it is determined that the remaining resources on the original node are sufficient to start the recovery executor, but the number of failures of the original node in starting and recovering the executor still exceeds the preset threshold, it indicates that the original node itself has failed. Therefore, the checkpoint information needs to be transferred to other nodes, and the start and recovery operation of the executor needs to be completed on other nodes.

[0025] In one optional implementation, when it is determined that the remaining resources on the original node are insufficient to start the recovery executor, the cost of transferring resources from the original node is evaluated to obtain a first evaluation result, specifically including:

[0026] The time consumed when the executor starts, the time consumed when transferring the job in the original node, the time consumed when saving the checkpoint information to the local environment, and the time consumed when saving to the local environment, calculated by the exponential moving average algorithm corresponding to the previous evaluation.

[0027] The first evaluation result is determined based on the first consumption time, the second consumption time, the third consumption time, and the fourth consumption time.

[0028] Specifically, the first time consumption is a fixed amount of time, which needs to be calculated regardless of whether the data is stored locally or on other nodes. Transferring jobs from the original node ensures sufficient local resources to complete the executor's startup and recovery operation; this transfer takes time, which is the second time consumption. When the executor's startup and recovery operation needs to be completed on the original node, checkpoint information needs to be retrieved, which also consumes some storage time. The time consumed to read checkpoint information is positively correlated with the time consumed to save checkpoint information to the local environment. Therefore, the time required to read checkpoint information can be estimated by the time it takes to save checkpoint information to the local environment. The fourth time consumption, calculated using the exponential moving average algorithm from the previous evaluation, can be used as a reference to help evaluate the time consumed in completing the executor's startup and recovery operation on the original node this time. Therefore, the first evaluation result needs to be determined based on the first to fourth time consumptions.

[0029] In one optional implementation, the first evaluation result is determined based on the first consumption time, the second consumption time, the third consumption time, and the fourth consumption time, specifically calculated using the following formula:

[0030] costQ t =EMA(tl) n costA t-1 )+texecutor+costB

[0031] Among them, costQ t The first evaluation result is the total time required for recovery at the local node. EMA() is an exponential moving average algorithm. n For the third time consumed, costA t-1 CostB represents the fourth time consumed, while costB represents the second time consumed.

[0032] In one optional implementation, the second time consumption is determined based on the priority of each of the preset number of jobs to be transferred and the total scheduling time for rescheduling the preset number of jobs to be transferred.

[0033] Specifically, to ensure that the remaining resources after the professional tasks on the original node can complete the executor startup and recovery operation, the tasks in the original node need to be transferred. However, the order of the transferred tasks also has a certain priority, with the aim of transferring as few tasks as possible to achieve the effect of the remaining resources meeting the above conditions. Therefore, a priority needs to be configured for each task to be transferred. Finally, the second consumption time is determined based on the priority of each transferred task and the total scheduling time for rescheduling a preset number of tasks to be transferred.

[0034] In one alternative implementation, the second consumption time is determined according to the following formula:

[0035] costB = g(p1 + p2 + ... + pn) + nt

[0036] Where costB is the second consumption time, g is the correction coefficient, p1, p2, ..., pn are the priorities corresponding to the preset number of jobs to be transferred, and nt is the total scheduling time for rescheduling the preset number of jobs to be transferred.

[0037] Specifically, transferring higher-priority jobs incurs a greater cost than transferring lower-priority jobs, but the number of lower-priority jobs transferred is greater than the number of higher-priority jobs transferred. This results in a longer transfer time. Therefore, a trade-off needs to be struck between high-priority and low-priority jobs to achieve the minimum transfer cost, which is the optimal transfer time.

[0038] In one optional implementation, the cost of assigning executors to other nodes to initiate recovery is evaluated to obtain a second evaluation result, specifically including:

[0039] The time consumed when the executor starts, the time consumed when saving the checkpoint information to the distributed file system, the time consumed when saving to the distributed file system (calculated using the exponential moving average algorithm corresponding to the previous evaluation), and the time consumed when scheduling the checkpoint information from the distributed file system.

[0040] The second evaluation result is determined based on the first consumption time, the fifth consumption time, the sixth consumption time, and the seventh consumption time.

[0041] Specifically, the reason for needing to obtain the first time consumption when obtaining the second evaluation result is explained above and will not be repeated here. When transferring to other nodes, it cannot be guaranteed that checkpoint information is stored in the local environment. Therefore, what is calculated here is the maximum transfer cost to other nodes, that is, the time required to read checkpoint information from the distributed file system. This time consumption is also positively correlated with saving checkpoint information to the distributed file system. Therefore, the fifth time consumption corresponding to saving checkpoint information to the distributed file system can be used to measure the time required to read checkpoint information from the distributed file system. Similarly, the sixth time consumption is the time consumption for saving to the distributed file system calculated by the exponential moving average algorithm corresponding to the previous evaluation. It can also be used as a reference data to help evaluate the time consumption for completing the executor startup and recovery operation on other nodes this time. The seventh time consumption is the time required to schedule checkpoint information from the distributed file system to other nodes this time. Therefore, the second evaluation result needs to be determined based on the first time consumption, the fifth time consumption, the sixth time consumption, and the seventh time consumption.

[0042] In one optional implementation, the second evaluation result is determined based on the first consumption time, the fifth consumption time, the sixth consumption time, and the seventh consumption time, specifically expressed by the following formula:

[0043] costP = EMA(t) rn costC t-1 )+texecutor+t h

[0044] Where costP is the second evaluation result, which is the total time taken to complete the actuator startup and recovery operation after the checkpoint information is transferred to other nodes, EMA() is the exponential moving average algorithm, and tr n For the fifth time consumed, costC t-1 The sixth time consumed is texecutor, and the first time consumed is t. h This is the seventh time consumed.

[0045] In a second aspect, the present invention provides a target node selection device, the device comprising:

[0046] The detection module is used to detect whether the executor in a Spark distributed job has failed.

[0047] The acquisition module is used to retrieve checkpoint information corresponding to the executor from a preset storage location when the detection module detects a failure in the executor of the Spark distributed job.

[0048] The scheduling module is used to schedule checkpoint information to the original node where the actuator was running before the failure, so that the start-up and recovery operation of the actuator can be completed on the original node;

[0049] The evaluation module is used to evaluate the cost of transferring resources from the original node when it is determined that the remaining resources on the original node are insufficient to start the recovery executor, and obtain a first evaluation result; and to evaluate the cost of allocating the executor to other nodes to start recovery, and obtain a second evaluation result.

[0050] The determination module is used to determine the target node for performing the start-up recovery operation on the actuator based on the first evaluation result and the second evaluation result.

[0051] The target node selection device provided by the present invention has the following advantages:

[0052] When an executor failure is detected in a Spark distributed job, checkpoint information corresponding to the executor is retrieved from a preset storage location. This checkpoint information is then scheduled to the original node where the executor was running before the failure, thus completing the executor's startup and recovery operation. The reason for scheduling the checkpoint information to the original node to complete the executor startup and recovery operation is that this eliminates the need to execute Yarn's traditional scheduling process, reducing operational steps and improving efficiency.

[0053] However, in certain special cases, such as when the remaining resources on the original node are insufficient to start the recovery executor, it may be considered to transfer the job from the original node to other nodes so that the executor's startup and recovery operations can continue on the original node. Alternatively, the executor's checkpoint information can be distributed to other nodes to complete the startup and recovery operations. Specifically, the execution costs of these two scenarios can be evaluated, and the target node for the executor's recovery operation can be determined based on the evaluation results. The purpose of this process is also to maximize the speed of the executor's startup and recovery operations and minimize time consumption.

[0054] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the target node selection method described in the first aspect or any corresponding embodiment thereof.

[0055] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the target node selection method described in the first aspect or any of its corresponding embodiments. Attached Figure Description

[0056] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0057] Figure 1 This is a flowchart illustrating a target node selection method provided in an embodiment of the present invention;

[0058] Figure 2 This is a flowchart illustrating another target node selection method provided in an embodiment of the present invention;

[0059] Figure 3 It is a simplified structural diagram consisting of a distributed file system, an executor, and a local environment;

[0060] Figure 4 This is a schematic diagram of the process provided by the present invention for evaluating the cost of initiating recovery by allocating executors to other nodes and obtaining a second evaluation result;

[0061] Figure 5 This is a schematic diagram of a fault recovery scheduling process for an actuator provided in an embodiment of the present invention;

[0062] Figure 6 This is a schematic diagram of an operation process for generating and saving checkpoint information according to an embodiment of the present invention;

[0063] Figure 7 This is a simplified operation flowchart illustrating the recovery operation of actuator scheduling checkpoint information provided in an embodiment of the present invention.

[0064] Figure 8 This is a structural block diagram of a target node selection device according to an embodiment of the present invention;

[0065] Figure 9 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Spark is an in-memory computing framework and a type of distributed computing engine. Spark distributed jobs can run on a Yarn cluster. A Spark job consists of driver and executor roles. The driver coordinates the job and collects results; it can schedule jobs for executors and monitor the executor's running status. Executors are the workload for executing the job. Spark supports both batch processing and stream processing jobs. Spark supports automatic recovery when an executor encounters a failure. To recover to a previous state after a failure, the executor periodically extracts its running state as checkpoint information and saves it to storage. During failure recovery, the executor reads the most recently successfully saved checkpoint information, thus restoring to the state closest to the moment before the failure.

[0068] Yarn is a scheduling framework for the Hadoop big data system. It is responsible for the unified management of resources across the entire cluster. It allows different types of jobs to share cluster resources. Yarn supports queue scheduling strategies, which can allocate resources reasonably when multiple jobs coexist according to rules, while ensuring a relatively balanced load across all nodes in the cluster, and providing fault detection capabilities for cluster nodes. The smallest scheduling unit for a Yarn job is called a container. Spark jobs running on a Yarn cluster use both the Spark driver and the Spark executor as Yarn containers.

[0069] The Spark job failure recovery process currently faces the following technical issues:

[0070] During executor failure recovery, a target node needs to be configured to complete the executor failure recovery operation. However, there is no better way in the prior art to select a suitable node to complete the executor failure recovery operation. To solve the above problem, this invention provides a target node selection embodiment. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system (computer device) including a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0071] This embodiment provides a target node selection method, which can be used in the aforementioned terminal devices, such as mobile phones and tablet computers. Figure 1 This is a flowchart of a target node selection method according to an embodiment of the present invention. This method is applied to Spark distributed jobs, where the Spark distributed jobs are distributed across different nodes, such as... Figure 1As shown, the process includes the following steps:

[0072] Step S101: When a failure is detected in the executor of the Spark distributed job, the checkpoint information corresponding to the executor is obtained from the preset storage location.

[0073] Specifically, during normal operation, the Executor can be configured to periodically extract its running state as checkpoint information and save it to a preset location. This information is used to retrieve the most recently saved checkpoint information and perform fault recovery in the event of an Executor failure, restoring the Executor to the state of the most recent checkpoint before the failure.

[0074] Step S102: Schedule the checkpoint information to the original node where the actuator was running before the failure occurred.

[0075] Specifically, scheduling checkpoint information to the original node where the executor was running before the failure facilitates the start-up and recovery of the executor on that original node. The reason for prioritizing scheduling information to the original node is that this step eliminates the need to configure other nodes in Yarn, which can improve the executor's recovery speed to some extent.

[0076] Step S103: When it is determined that the remaining resources on the original node are insufficient to start the recovery executor, the cost of transferring the resources on the original node is evaluated, and a first evaluation result is obtained.

[0077] Step S104: Evaluate the cost of assigning executors to other nodes to initiate recovery and obtain a second evaluation result.

[0078] Step S105: Based on the first evaluation result and the second evaluation result, determine the target node for performing the start-up recovery operation on the actuator.

[0079] Specifically, in certain special cases, the original checkpoint may not be suitable for completing the executor startup and recovery operation. Alternatively, the resources at the current checkpoint may be insufficient to support the executor startup and recovery operation. In this case, it is necessary to evaluate the cost of transferring resources from the original node to obtain a first evaluation result. Furthermore, the cost of allocating the executor to another node for startup and recovery needs to be evaluated to obtain a second evaluation result. Finally, based on the first and second evaluation results, the target node for performing the executor startup and recovery operation is determined. That is, the node with the lower cost is selected as the target node.

[0080] The target node selection method provided in this embodiment retrieves checkpoint information corresponding to the executor from a preset storage location when a failure is detected in an executor in a Spark distributed job. Then, the checkpoint information is scheduled to the original node where the executor was running before the failure to complete the executor's startup and recovery operation. The reason for scheduling the checkpoint information to the original node to complete the executor's startup and recovery operation is that this operation eliminates the need to execute Yarn's traditional scheduling process, omitting processes such as resource allocation, scheduler policy execution, and execution node allocation, thereby improving the speed of fault recovery.

[0081] However, in certain special cases, such as when the remaining resources on the original node are insufficient to start the recovery executor, it may be considered to transfer the job from the original node to other nodes so that the executor's startup and recovery operations can continue on the original node. Alternatively, the executor's checkpoint information can be distributed to other nodes to complete the startup and recovery operations. Specifically, the execution costs of these two scenarios can be evaluated, and the target node for the executor's recovery operation can be determined based on the evaluation results. The purpose of this process is also to maximize the speed of the executor's startup and recovery operations and minimize time consumption.

[0082] This embodiment provides a target node selection method, which can be used in the aforementioned mobile terminals, such as mobile phones and tablets. Figure 2 This is a flowchart of another target node selection method provided by an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0083] Step S201: When a failure is detected in the executor of the Spark distributed job, the checkpoint information corresponding to the executor is obtained from the preset storage location.

[0084] In one optional implementation, checkpoint information can be stored in a distributed file system or backed up locally. Storing it locally is chosen because transferring the checkpoint information from the distributed file system to the executor's local environment and then reading it locally can be time-consuming, especially if the checkpoint information is large, impacting fault recovery efficiency. Therefore, a local backup is faster than retrieving checkpoint information from the distributed file system to start and restore the executor. Furthermore, the locally stored checkpoint information only contains the checkpoint information for that single executor, unlike the checkpoint information stored on the distributed file system, which contains checkpoint information for all Spark executors. Finding and retrieving checkpoint information locally is also faster. Of course, if the stored checkpoint information is not found locally, it can be transferred from the distributed file system and the execution state restored.

[0085] Based on the above considerations, this embodiment of the invention also discloses another method for storing checkpoints, namely, storing them in a reuse node.

[0086] Therefore, the specific steps for obtaining the checkpoint information corresponding to the actuator from the preset position can be found in the following steps:

[0087] Step a1: Check if there are reused nodes between the Spark distributed job and the distributed file system.

[0088] Specifically, both Spark distributed jobs and distributed file systems are configured with corresponding node clusters. When there are nodes shared by both node clusters, these nodes are called reused nodes.

[0089] Step a2: When it is determined that there are reuse nodes between the Spark distributed job and the distributed file system, and the checkpoint information corresponding to the executor is located on the reuse node, the checkpoint information is retrieved from the reuse node.

[0090] or,

[0091] Step a3: When it is determined that there is no reuse node between the Spark distributed job and the distributed file system, or that the checkpoint information corresponding to the executor is not stored on the reuse node, check whether checkpoint information exists in the local environment where the Spark distributed job is located.

[0092] Step a4: When it is determined that checkpoint information is stored in the local environment, read the checkpoint information from the local environment;

[0093] or,

[0094] Step a5: When it is determined that there is no reuse node between the Spark distributed job and the distributed file system, or the checkpoint information corresponding to the executor is not stored on the reuse node, and the checkpoint information is not stored in the local environment, read the checkpoint information from the distributed file system.

[0095] Specifically, when it is determined that there are reuse nodes between the Spark distributed job and the distributed file system, the reuse nodes are both local storage and distributed file system storage. Therefore, checkpoint information can be directly stored in the reuse nodes. Thus, when retrieving checkpoint information, the first step is to determine if a reuse node exists. If it does, further determine if the checkpoint information corresponding to the executor is located on the reuse node. If it is on the reuse node, read the checkpoint information directly from the reuse node. If the checkpoint information does not exist on the reuse node, or there is no reuse node, proceed to step a3. If the checkpoint information is stored in the local environment, read the checkpoint information directly from the local environment. Alternatively, if it is determined that there are no reuse nodes between the Spark distributed job and the distributed file system, or the checkpoint information corresponding to the executor is not stored on the reuse node, and the checkpoint information is not stored in the local environment, then read the checkpoint information from the distributed file system. The specific reasons have been explained in detail above and will not be repeated here.

[0096] Figure 3 The diagram illustrates a simplified structure consisting of a distributed file system, executors running on each node, and local environments. Each node's local environment stores checkpoint information corresponding to the executors running on that node.

[0097] Step S202: The checkpoint information is scheduled to the original node where the actuator was running before the failure, so that the start-up and recovery operation of the actuator can be completed on the original node.

[0098] For a detailed description of steps S201 and S202, please refer to [link / reference needed]. Figure 1 Steps S101 and S102 of the illustrated embodiment will not be described again here.

[0099] Step S203: When it is determined that the remaining resources on the original node are insufficient to start the recovery executor, the cost of transferring the resources on the original node is evaluated, and a first evaluation result is obtained.

[0100] In an optional implementation, step 203 can be implemented in the following manner, specifically including:

[0101] Step S2031: Obtain the first time consumed when the executor starts, the second time consumed when transferring the job in the original node, the third time consumed when saving the checkpoint information to the local environment, and the fourth time consumed when saving to the local environment, calculated by the exponential moving average algorithm corresponding to the previous evaluation.

[0102] The first and third consumption times can be obtained through statistical analysis of experimental data. The fourth consumption time is calculated using an exponential moving average algorithm based on the previous evaluation. If this is the first time, the fourth consumption time can be the default value.

[0103] As for the second time consumption, it can be determined based on the priority of each of the preset number of jobs to be transferred and the total scheduling time for rescheduling the preset number of jobs to be transferred.

[0104] In an optional implementation, the second consumption time can be determined as follows:

[0105] costB = g(p1 + p2 + ... + pn) + nt (Formula 1)

[0106] Where costB is the second consumption time, g is the correction coefficient, p1, p2, ..., pn are the priorities corresponding to the preset number of jobs to be transferred, and nt is the total scheduling time for rescheduling the preset number of jobs to be transferred.

[0107] In other words, to ensure that the remaining resources after the professional tasks on the original node can complete the executor startup and recovery operation, the tasks in the original node need to be transferred. However, the order of the transferred tasks also has a certain priority, the purpose of which is to transfer as few tasks as possible to achieve the effect of the remaining resources meeting the above conditions. Therefore, a priority needs to be configured for each task to be transferred. Finally, the second consumption time is determined based on the priority of each transferred task and the total scheduling time for rescheduling a preset number of tasks to be transferred. Among them, transferring tasks with higher priority has a higher transfer cost than transferring tasks with lower priority, but the number of tasks with lower priority is greater than the number of tasks with higher priority. The time consumed is also greater. Therefore, a trade-off needs to be made between high-priority and low-priority tasks to achieve the minimum transfer cost, that is, the optimal time for transferring tasks.

[0108] Step S2032: Determine the first evaluation result based on the first consumption time, the second consumption time, the third consumption time, and the fourth consumption time.

[0109] Specifically, the first time consumption is a fixed amount of time, which needs to be calculated regardless of whether the data is stored locally or on other nodes. Transferring jobs from the original node ensures sufficient local resources to complete the executor's startup and recovery operation; this transfer takes time, which is the second time consumption. When the executor's startup and recovery operation needs to be completed on the original node, checkpoint information needs to be retrieved, which also consumes some storage time. The time consumed to read checkpoint information is positively correlated with the time consumed to save checkpoint information to the local environment. Therefore, the time required to read checkpoint information can be estimated by the time it takes to save checkpoint information to the local environment. The fourth time consumption, calculated using the exponential moving average algorithm from the previous evaluation, can be used as a reference to help evaluate the time consumed in completing the executor's startup and recovery operation on the original node this time. Therefore, the first evaluation result needs to be determined based on the first to fourth time consumptions.

[0110] In an optional implementation, the first evaluation result, determined based on the first consumption time, the second consumption time, the third consumption time, and the fourth consumption time, can be calculated as follows:

[0111] costQ t =EMA(tl) n costA t-1 )+texecutor+costB (Formula 2)

[0112] Among them, costQ t The first evaluation result is the total time required for recovery at the local node. EMA() is an exponential moving average algorithm. n For the third time consumed, costA t-1 CostB represents the fourth time consumed, while costB represents the second time consumed.

[0113] Step S204: Evaluate the cost of assigning executors to other nodes to initiate recovery and obtain a second evaluation result.

[0114] In an optional implementation, the specific process of evaluating the cost of assigning executors to other nodes to initiate recovery and obtaining a second evaluation result can be found in [link to relevant documentation]. Figure 4 As shown, the method includes the following steps:

[0115] Step S2041: Obtain the first time consumed when the executor starts, the fifth time consumed when saving the checkpoint information to the distributed file system, the sixth time consumed when saving to the distributed file system (calculated using the exponential moving average algorithm corresponding to the previous evaluation), and the seventh time consumed when scheduling the checkpoint information from the distributed file system.

[0116] Specifically, the first, fifth, and seventh time consumption periods can all be obtained through statistical analysis of experimental data. The sixth time consumption period is calculated using an exponential moving average algorithm based on the previous evaluation. Similarly, if this is the first time, the sixth time consumption period will be the default value.

[0117] Step S2042: Determine the second evaluation result based on the first consumption time, the fifth consumption time, the sixth consumption time, and the seventh consumption time.

[0118] The reason for needing to obtain the first time consumption when obtaining the second evaluation result is explained above and will not be repeated here. When migrating to other nodes, it cannot be guaranteed that checkpoint information is stored in the local environment. Therefore, this calculation focuses on the maximum migration cost to other nodes, which is the time required to read checkpoint information from the distributed file system. This time consumption is also positively correlated with saving checkpoint information to the distributed file system. Therefore, the fifth time consumption, corresponding to saving checkpoint information to the distributed file system, can be used to measure the time required to read checkpoint information from the distributed file system. Similarly, the sixth time consumption, calculated using the exponential moving average algorithm, is the time consumption for saving to the distributed file system in the previous evaluation. This can also serve as a reference to help evaluate the time consumption for completing the executor startup and recovery operations on other nodes. The seventh time consumption is the time required to schedule checkpoint information from the distributed file system to other nodes. Therefore, the second evaluation result needs to be determined based on the first, fifth, sixth, and seventh time consumptions.

[0119] In an optional implementation, the second evaluation result is determined based on the first consumption time, the fifth consumption time, the sixth consumption time, and the seventh consumption time, and can be expressed by the following formula:

[0120] costP = EMA(t) rn costC t-1 )+texecutor+t h (Formula 3)

[0121] Where costP is the second evaluation result, which is the total time taken to complete the actuator startup and recovery operation after the checkpoint information is transferred to other nodes, EMA() is the exponential moving average algorithm, and tr n For the fifth time consumed, costC t-1 The sixth time consumed is texecutor, and the first time consumed is t. h This is the seventh time consumed.

[0122] Step S205: Based on the first evaluation result and the second evaluation result, determine the target node for performing the start-up recovery operation on the actuator.

[0123] In one alternative implementation, besides the possibility that the actuator may need to be transferred to another node to complete the actuator startup and recovery operation due to insufficient resources, based on the first and second evaluation results, there is also a situation where the actuator inevitably needs to be transferred to another node to perform the recovery startup operation.

[0124] For example, if it is determined that the remaining resources on the original node are sufficient to start the recovery executor, but the number of failures of the original node's start recovery operation for the executor still exceeds the preset threshold, the checkpoint information is directly transferred to other nodes, and the start recovery operation for the executor is completed on other nodes.

[0125] In other words, if it is determined that the remaining resources on the original node are sufficient to start the recovery executor, but the number of failures of the original node in starting and recovering the executor still exceeds the preset threshold, it indicates that the original node itself has failed. Therefore, the checkpoint information needs to be transferred to other nodes, and the start and recovery operation of the executor needs to be completed on other nodes.

[0126] This embodiment also provides a target node selection device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0127] Figure 5 The diagram illustrates the actuator fault recovery scheduling process. This includes:

[0128] 1. Begin;

[0129] 2. Determine whether to execute the Spark job failure recovery process;

[0130] 3. If no, then it can be determined that it belongs to the normal YARN scheduling process. The YARN scheduling logic is used to complete resource scheduling and policy scheduling operations, and then the process jumps to step 16.

[0131] 4. If yes, then attempt to perform the recovery locally.

[0132] 5. Determine if the previously running node still exists. If yes, proceed to step 6; otherwise, proceed to step 14.

[0133] 6. If the previously running node exists, it will be restored and started locally;

[0134] 7. Has the startup been successful? If yes, proceed to step 7; if no, proceed to step 10.

[0135] 8. The actuator ran successfully.

[0136] 9. Does the program fail to run after startup for a period of time? If yes, proceed to step 10; otherwise, proceed to step 15.

[0137] 10. Startup failed, try starting again.

[0138] 11. Will it succeed within the retries? Yes, proceed to step 8. No, proceed to step 12.

[0139] 12. Is the inversion due to insufficient resources preventing the operation? If yes, proceed to step 13; otherwise, proceed to step 14.

[0140] 13. Calculate the first and second evaluation results to determine the target node.

[0141] 14. Schedule to other nodes using the YARN scheduling strategy;

[0142] 15. Actuator fault recovery complete.

[0143] 16. End operation.

[0144] Figure 6 The diagram illustrates the process of generating and saving checkpoint information. This includes:

[0145] 1. Begin.

[0146] 2. Save checkpoint information to the distributed file system;

[0147] 3. Determine if the save was successful? If no, proceed to step 4; if yes, proceed to step 5.

[0148] 4. Retry saving the checkpoint information to the distributed file system, and then jump to step 6.

[0149] 5. Are the distributed file system nodes reused with the Spark compute nodes? Is a local storage strategy available? If yes, skip to step 17; if no, skip to step 7.

[0150] 6. Was the save successful? If yes, proceed to step 8; otherwise, proceed to step 7.

[0151] 7. Failed to save checkpoint information, job exited, proceed to step 17.

[0152] 8. Is there a failure flag indicating that the current node failed to save checkpoint information? If yes, proceed to step 9; otherwise, proceed to step 10.

[0153] 9. Has the time interval exceeded since the last attempt to save locally? Yes, proceed to step 8; no, continue waiting.

[0154] 10. Try saving checkpoint information to the local environment.

[0155] 11. Was the save successful? If yes, proceed to step 14; otherwise, proceed to step 12.

[0156] 12. Retry saving checkpoint information to local storage.

[0157] 13. Within the retry limit, can the data be saved successfully? If yes, proceed to step 14; otherwise, proceed to step 16.

[0158] 14. Checkpoint information saved successfully.

[0159] 15. Clear the node's local checkpoint save failure flag.

[0160] 16. Failed to mark the current node as having failed to save checkpoint information locally.

[0161] 17. End.

[0162] Figure 7 The diagram illustrates a simplified operation procedure for completing the recovery operation using actuator scheduling checkpoint information, including:

[0163] 1. Begin.

[0164] 2. Determine if the distributed file system data storage node is reused with the Spark compute node? Is the checkpoint information stored locally? If no, proceed to step 3; if yes, proceed to step 9.

[0165] 3. Read the checkpoint information saved in the local environment.

[0166] 4. Does the local checkpoint information exist? If yes, proceed to step 5; otherwise, proceed to step 8.

[0167] 5. Try to recover from local checkpoint information.

[0168] 6. Was the recovery successful? If no, proceed to step 7; otherwise, proceed to step 11.

[0169] 7. Delete locally saved checkpoint information.

[0170] 8. Read checkpoint information from the distributed file system.

[0171] 9. Bypass the distributed file system protocol and read checkpoint information directly from its data storage location.

[0172] 10. Can the recovery be successful within the retry limit? If yes, proceed to step 11; otherwise, proceed to step 12.

[0173] 11. Actuator malfunction has been resolved.

[0174] 12. Unable to recover from checkpoint; startup recovery failed.

[0175] 13. End.

[0176] Figures 5 to 7 The overall operation process has already been described above, so it will not be repeated here.

[0177] This embodiment provides a target node selection device, such as... Figure 8 The system includes: a detection module 801, an acquisition module 802, a scheduling module 803, an evaluation module 804, and a determination module 805.

[0178] Detection module 801 is used to detect whether the executor in a Spark distributed job has failed;

[0179] The acquisition module 802 is used to obtain checkpoint information corresponding to the executor from a preset storage location when the detection module 801 detects a failure in the executor of the Spark distributed job.

[0180] The scheduling module 803 is used to schedule checkpoint information to the original node where the actuator was running before the failure, so that the start-up and recovery operation of the actuator can be completed on the original node.

[0181] The evaluation module 804 is used to evaluate the cost of transferring resources from the original node and obtain a first evaluation result when it is determined that the remaining resources on the original node are insufficient to start the recovery executor; and to evaluate the cost of allocating the executor to other nodes to start recovery and obtain a second evaluation result.

[0182] The determination module 805 is used to determine the target node for performing the start-up recovery operation on the actuator based on the first evaluation result and the second evaluation result.

[0183] In an optional implementation, the detection module 801 is further configured to detect whether there are reused nodes in the Spark distributed job and the distributed file system;

[0184] The module 802 is specifically used to retrieve checkpoint information from the reuse node when it is determined that there is a reuse node between the Spark distributed job and the distributed file system, and the checkpoint information corresponding to the executor is located on the reuse node.

[0185] or,

[0186] If it is determined that there is no reuse node between the Spark distributed job and the distributed file system, or that the checkpoint information corresponding to the executor is not stored on the reuse node, check whether checkpoint information exists in the local environment where the Spark distributed job is located.

[0187] When it is determined that checkpoint information is stored in the local environment, the checkpoint information is read from the local environment.

[0188] or,

[0189] If it is determined that there is no reuse node between the Spark distributed job and the distributed file system, or that the checkpoint information corresponding to the executor is not stored on the reuse node, and the checkpoint information is not stored in the local environment, then the checkpoint information is read from the distributed file system.

[0190] In an optional implementation, the determining module 805 is further configured to, when it is determined that the remaining resources on the original node are sufficient to start the recovery executor, but the number of failures of the original node in starting and recovering the executor still exceeds a preset threshold, directly transfer the checkpoint information to other nodes and complete the start and recovery operation of the executor on other nodes.

[0191] In an optional implementation, the evaluation module 804 is specifically used to obtain the first time consumed when the executor starts, the second time consumed when transferring the job in the original node, the third time consumed when saving the checkpoint information to the local environment, and the fourth time consumed when saving to the local environment, calculated by the exponential moving average algorithm corresponding to the previous evaluation.

[0192] The first evaluation result is determined based on the first consumption time, the second consumption time, the third consumption time, and the fourth consumption time.

[0193] In an optional implementation, the evaluation module 804 determines the first evaluation result based on the first consumption time, the second consumption time, the third consumption time, and the fourth consumption time, specifically calculated using the following formula:

[0194] costQ t =EMA(tl) n costA t-1 )+texecutor+costB (Formula 4)

[0195] Among them, costQ t The first evaluation result is the total time required for recovery at the local node. EMA() is an exponential moving average algorithm. n For the third time consumed, costA t-1 CostB represents the fourth time consumed, while costB represents the second time consumed.

[0196] In one optional implementation, the second time consumption is determined based on the priority of each of the preset number of jobs to be transferred and the total scheduling time for rescheduling the preset number of jobs to be transferred.

[0197] In an optional implementation, the second consumption time is determined according to the following formula:

[0198] costB = g(p1 + p2 + ... + pn) + nt (Formula 5)

[0199] Where costB is the second consumption time, g is the correction coefficient, p1, p2, ..., pn are the priorities corresponding to the preset number of jobs to be transferred, and nt is the total scheduling time for rescheduling the preset number of jobs to be transferred.

[0200] In an optional implementation, the evaluation module 804 is specifically used to obtain the first time consumed when the executor starts, the fifth time consumed when saving the checkpoint information to the distributed file system, the sixth time consumed when saving to the distributed file system calculated by the exponential moving average algorithm corresponding to the previous evaluation, and the seventh time consumed when scheduling the checkpoint information from the distributed file system.

[0201] The second evaluation result is determined based on the first consumption time, the fifth consumption time, the sixth consumption time, and the seventh consumption time.

[0202] In an optional implementation, the evaluation module 804 determines the second evaluation result based on the first consumption time, the fifth consumption time, the sixth consumption time, and the seventh consumption time, specifically expressed by the following formula:

[0203] costP = EMA(t) rn costC t-1 )+texecutro+t h (Formula 6)

[0204] Where costP is the second evaluation result, which is the total time taken to complete the actuator startup and recovery operation after the checkpoint information is transferred to other nodes, EMA() is the exponential moving average algorithm, and tr n For the fifth time consumed, costC t-1 The sixth time consumed is texecutor, and the first time consumed is t. h This is the seventh time consumed.

[0205] In this embodiment, the target node selection device is presented in the form of a functional module. Here, a module refers to an application-specific integrated circuit (ASIC), a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0206] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0207] This invention provides a target node selection device that, when an executor in a Spark distributed job is detected to have failed, retrieves checkpoint information corresponding to the executor from a preset storage location. Then, it schedules the checkpoint information to the original node where the executor was running before the failure, thus completing the executor's startup and recovery operation. The reason for scheduling the checkpoint information to the original node to complete the executor's startup and recovery operation is that this eliminates the need to execute Yarn's traditional scheduling process, reducing operational steps and improving work efficiency.

[0208] However, in certain special cases, such as when the remaining resources on the original node are insufficient to start the recovery executor, it may be considered to transfer the job from the original node to other nodes so that the executor's startup and recovery operations can continue on the original node. Alternatively, the executor's checkpoint information can be distributed to other nodes to complete the startup and recovery operations. Specifically, the execution costs of these two scenarios can be evaluated, and the target node for the executor's recovery operation can be determined based on the evaluation results. The purpose of this process is also to maximize the speed of the executor's startup and recovery operations and minimize time consumption.

[0209] This invention also provides a computer device having the above-described features. Figure 8 The target node selection device is shown.

[0210] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 9As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 9 Take a processor 10 as an example.

[0211] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0212] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0213] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device as shown by a landing page for an app. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0214] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0215] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0216] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0217] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0218] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A target node selection method, characterized in that, The method is applied to Spark distributed jobs, which are distributed across different nodes. The method includes: When a failure is detected in the executor of the Spark distributed job, the checkpoint information corresponding to the executor is obtained from a preset storage location. The checkpoint information is scheduled to the original node where the actuator was running before the failure, so that the start-up and recovery operation of the actuator can be completed on the original node; When it is determined that the remaining resources on the original node are insufficient to start and restore the executor, the cost of transferring the resources on the original node is evaluated, and a first evaluation result is obtained; In addition, the cost of assigning the actuator to other nodes to initiate recovery is evaluated, and a second evaluation result is obtained; Based on the first evaluation result and the second evaluation result, the target node for performing the start-up recovery operation on the actuator is determined.

2. The method according to claim 1, characterized in that, When a failure is detected in the executor of the Spark distributed job, the checkpoint information corresponding to the executor is retrieved from a preset storage location, specifically including: Detect whether there are reused nodes between the Spark distributed job and the distributed file system; When it is determined that there is a reuse node between the Spark distributed job and the distributed file system, and the checkpoint information corresponding to the executor is located on the reuse node, the checkpoint information is retrieved from the reuse node; or, When it is determined that there is no reuse node between the Spark distributed job and the distributed file system, or that the checkpoint information corresponding to the executor is not stored in the reuse node, check whether the checkpoint information exists in the local environment where the Spark distributed job is located. When it is determined that the checkpoint information is stored in the local environment, the checkpoint information is read from the local environment; or, When it is determined that there is no reuse node between the Spark distributed job and the distributed file system, or the checkpoint information corresponding to the executor is not stored on the reuse node, and the checkpoint information is not stored in the local environment, the checkpoint information is read from the distributed file system.

3. The method according to claim 1 or 2, characterized in that, When it is determined that the remaining resources on the original node are sufficient to start and restore the executor, but the number of failures of the original node in starting and restoring the executor still exceeds a preset threshold, the checkpoint information is directly transferred to other nodes, and the start and restore operation of the executor is completed on other nodes.

4. The method according to claim 2, characterized in that, When it is determined that the remaining resources on the original node are insufficient to start and restore the executor, the cost of transferring the resources on the original node is evaluated, and a first evaluation result is obtained, specifically including: The system acquires the first time consumed when the executor starts, the second time consumed when transferring the job in the original node, the third time consumed when saving the checkpoint information to the local environment, and the fourth time consumed when saving to the local environment, calculated by the exponential moving average algorithm corresponding to the previous evaluation. The first evaluation result is determined based on the first consumption time, the second consumption time, the third consumption time, and the fourth consumption time.

5. The method according to claim 4, characterized in that, The first evaluation result, determined based on the first consumption time, the second consumption time, the third consumption time, and the fourth consumption time, is specifically calculated using the following formula: costQ t =EMA(tl n ,costA t-1 )+texecutor+costB Among them, costQ t The first evaluation result represents the total time required for recovery at the local node. EMA() is an exponential moving average algorithm. n For the third time consumed, costA t-1 The fourth consumption time is given, and costB is the second consumption time.

6. The method according to claim 5, characterized in that, The second time consumption is determined based on the priority of each of the preset number of jobs to be transferred and the total scheduling time for rescheduling the preset number of jobs to be transferred.

7. The method according to claim 6, characterized in that, The second time consumption is determined according to the following formula: costB = g(p1 + p2 + ... + pn) + nt Where costB is the second consumption time, g is the correction coefficient, p1, p2, ..., pn are the priorities corresponding to the preset number of jobs to be transferred, and nt is the total scheduling time for rescheduling the preset number of jobs to be transferred.

8. The method according to any one of claims 2 or 4-7, characterized in that, The cost of assigning the actuator to other nodes to initiate recovery is evaluated to obtain a second evaluation result, which specifically includes: The following data are obtained: the first time consumed when the executor starts, the fifth time consumed when saving the checkpoint information to the distributed file system, the sixth time consumed when saving to the distributed file system (calculated using the exponential moving average algorithm corresponding to the previous evaluation), and the seventh time consumed when scheduling the checkpoint information from the distributed file system. The second evaluation result is determined based on the first consumption time, the fifth consumption time, the sixth consumption time, and the seventh consumption time.

9. The method according to claim 8, characterized in that, The determination of the second evaluation result based on the first consumption time, the fifth consumption time, the sixth consumption time, and the seventh consumption time is specifically expressed by the following formula: costP=EMA(t rn ,costC t-1 )+texecutor+t h Where costP is the second evaluation result, which is the total time taken to complete the start-up and recovery operation of the actuator after the checkpoint information is transferred to other nodes, EMA() is the exponential moving average algorithm, and tr n For the fifth consumption time, costC t-1 The sixth time consumed, texecutor, is the first time consumed, t h This refers to the seventh time consumed.

10. A target node selection device, characterized in that, The device includes: The detection module is used to detect whether the executor in a Spark distributed job has failed. The acquisition module is used to acquire checkpoint information corresponding to the executor from a preset storage location when the detection module detects a failure in the executor of the Spark distributed job. The scheduling module is used to schedule the checkpoint information to the original node where the actuator was running before the failure occurred, so as to complete the start-up and recovery operation of the actuator on the original node; The evaluation module is used to evaluate the cost of transferring resources from the original node and obtain a first evaluation result when it is determined that the remaining resources on the original node are insufficient to start the recovery of the executor; and to evaluate the cost of allocating the executor to other nodes to start the recovery and obtain a second evaluation result. The determination module is used to determine the target node for performing the start-up recovery operation on the actuator based on the first evaluation result and the second evaluation result.

11. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the target node selection method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the target node selection method according to any one of claims 1 to 9.

Images