Checkpoint recovery method, device and equipment for data processing task and medium

Abstract

The application discloses a checkpoint recovery method, device and equipment for data processing tasks and a medium. When it is detected that there is a fault point in each task processed by a system, the method determines an expected recovery speed of a target checkpoint of at least one system that has performed an operation of saving an application state snapshot, determines an execution task set of the target checkpoint recovered to the fault point according to an execution dependency relationship between each task, further determines a task execution order of each task in the execution task set according to the expected recovery speed of the target checkpoint, and processes each task in the execution task set according to the task execution order, so that the sorting of each task that needs to be executed from the checkpoint to the fault point is realized, the recovery time of all tasks in the execution task set is the shortest, the recovery efficiency of the checkpoint is improved, and the technical problem of long checkpoint recovery time in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data processing, and particularly relates to a checkpoint recovery method and device for data processing tasks, equipment and a medium. BACKGROUND

[0002] Taking Hadoop as an example, the process of data processing task scheduling is introduced. The early main purpose of Hadoop is to run large batch processing jobs, such as web indexing and log mining. Users submit jobs to a queue, and then the cluster runs them in order. As enterprise organizations put more data in their Hadoop clusters and develop more computations they want to run, another use case becomes attractive: sharing a MapReduce cluster among multiple users. The benefits of sharing are huge: putting all data in one place, users can run queries that would otherwise be impossible to execute, and costs are reduced because system utilization is higher than building separate Hadoop clusters for each group.

[0003] However, sharing requires the support of a Hadoop job scheduler to guarantee the capacity of production jobs and good response time for interactive jobs, while fairly allocating resources among users. When a job is submitted to the management node of Hadoop, it is divided into several stages (such as map stage and reduce stage), and each stage is divided into several tasks. The scheduling of tasks is based on priority and available resources of each processing node in the cluster, including first-in-first-out, fair scheduling and capacity-related scheduling strategies.

[0004] When a fault point occurs in the process of processing batch tasks, the task is usually re-executed, that is, the blood relationship between jobs in the execution plan is maintained in the master control node, and the input data of the upstream is retained, when the fault occurs, the processing node with input data of the upstream is found through the blood relationship, and the calculation is re-calculated to recover. Alternatively, a serial checkpoint writing and recovery method is used, that is, when a high-priority task is encountered, the low-priority task is usually handled by killing or waiting. Alternatively, a checkpoint random scheduling recovery method is used, when a fault occurs, the task is randomly recovered from the checkpoint, and when all tasks are recovered, it is recovered to the fault point.

[0005] However, in the above three methods, re-executing the task is based on the recalculation of the blood relationship, and for some stateful operations, such as a session with a long time window, the full recalculation according to the blood relationship is time-consuming and may exceed the requirements of the task response time when the task fails. The method of serial checkpoint writing and recovery is too slow in checkpoint writing and fault recovery, and sometimes exceeds the requirements of the job execution time. The method of checkpoint random scheduling recovery cannot make the total task recovery time shortest because it does not consider the existing order and recovery time of task recovery. SUMMARY

[0006] The application provides a checkpoint recovery method, device, equipment and medium for a data processing task, to improve the checkpoint recovery efficiency of the data processing task and solve the technical problem of long checkpoint recovery time in the prior art.

[0007] According to an aspect of the application, a checkpoint recovery method for a data processing task is provided, the method comprising:

[0008] If a fault point is detected in each task, the expected recovery speed of at least two target checkpoints is determined, wherein the fault point is a current fault task processed by the system, and the target checkpoint is a historical processing task corresponding to the system when performing an operation of saving an application state snapshot;

[0009] An execution dependency relationship corresponding to each task is obtained, and an execution task set corresponding to the target checkpoint is determined based on the execution dependency relationship.

[0010] Based on the expected recovery speed of each target checkpoint, a task execution order corresponding to the execution task set is determined, and the tasks in the execution task set are processed in the task execution order.

[0011] Optionally, the expected recovery speed of at least two target checkpoints is determined by:

[0012] The historical processing speed of each task is obtained.

[0013] The expected recovery speed of at least two target checkpoints is determined based on the historical processing speed.

[0014] Optionally, the execution dependency relationship corresponding to each task is obtained by:

[0015] A directed acyclic graph corresponding to each task is obtained, wherein the directed acyclic graph comprises connection direction information between the tasks.

[0016] The execution dependency relationship corresponding to each task is determined based on the connection direction information.

[0017] Optionally, the obtaining of the execution dependency relationship corresponding to each task comprises:

[0018] The execution dependency relationship corresponding to each task is obtained based on the scheduler corresponding to each task.

[0019] Optionally, the obtaining of the execution dependency relationship corresponding to each task based on the scheduler corresponding to each task comprises:

[0020] The upstream and downstream relationship between jobs is determined based on the scheduler, wherein the job is composed of multiple stages, and the stage is composed of multiple tasks executed in parallel;

[0021] The timing information of the stage of each job is obtained based on the scheduler;

[0022] The execution dependency relationship corresponding to each task is determined based on the upstream and downstream relationship and the timing information by the scheduler.

[0023] Optionally, the processing of the tasks in the execution task set according to the task execution order comprises:

[0024] The task execution order is sent to the scheduler;

[0025] The tasks in the execution task set are scheduled according to the task execution order by the scheduler.

[0026] Optionally, the scheduling of the tasks in the execution task set according to the task execution order comprises:

[0027] A to-be-restarted checkpoint corresponding to a task executed first in the task execution order is determined;

[0028] The to-be-restarted checkpoint is restarted, and the tasks in the execution task set are scheduled according to the task execution order.

[0029] According to another aspect of the present application, a checkpoint recovery device for data processing tasks is provided, and the device comprises:

[0030] A recovery speed prediction module is configured to determine the predicted recovery speed of at least two target checkpoints if a fault point is detected in each task, wherein the fault point is a current fault task processed by the system, and the target checkpoint is a historical processing task corresponding to the system when performing an operation of saving an application state snapshot.

[0031] A task set determination module is configured to obtain an execution dependency relationship corresponding to each task, and determine an execution task set of the target checkpoint recovered to the fault point based on the execution dependency relationship.

[0032] The execution order determination module is used to determine the task execution order corresponding to the execution task set based on the expected recovery speed of each target checkpoint, and process the tasks in the execution task set according to the task execution order.

[0033] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0034] At least one processor; and

[0035] A memory communicatively connected to the at least one processor; wherein,

[0036] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the checkpoint recovery method for the data processing task described in any embodiment of the present invention.

[0037] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the checkpoint recovery method for a data processing task according to any embodiment of the present invention.

[0038] The technical solution of this invention, when a fault point is detected in each task processed by the system, determines the estimated recovery speed of at least one target checkpoint where the system has performed the operation of saving an application state snapshot. Based on the obtained execution dependencies between each task, a set of execution tasks for restoring the target checkpoint to the fault point is determined. Then, the task execution order of each task in the execution task set is determined by the estimated recovery speed of the target checkpoint, and each task in the execution task set is processed according to the task execution order. This achieves the sorting of each task that needs to be executed from the checkpoint to the fault point, so that the recovery time of all tasks in the execution task set is minimized, improving the recovery efficiency of the checkpoint and solving the technical problem of excessively long checkpoint recovery time in the prior art.

[0039] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating a checkpoint recovery method for a data processing task provided in Embodiment 1 of the present invention;

[0042] Figure 2A This is a flowchart illustrating a checkpoint recovery method for a data processing task provided in Embodiment 2 of the present invention;

[0043] Figure 2B This is a schematic diagram of the operation of a scheduler provided in Embodiment 2 of the present invention;

[0044] Figure 3 This is a schematic diagram of the structure of a checkpoint recovery device for a data processing task provided in Embodiment 3 of the present invention;

[0045] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0046] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0047] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0048] Example 1

[0049] Figure 1 This is a flowchart illustrating a checkpoint recovery method for data processing tasks according to Embodiment 1 of the present invention. This embodiment is applicable to situations where a fault point occurs during the execution of various data processing tasks in a data processing system. It determines a set of tasks that can recover to the fault point from the checkpoint, and arranges the execution order of the tasks in the task set according to the expected recovery speed of the checkpoint. The tasks are then executed sequentially in the arranged order to achieve checkpoint recovery. This method can be executed by a checkpoint recovery device for data processing tasks, which can be implemented in hardware and / or software and can be configured in electronic devices such as mobile phones, computers, and tablets. Figure 1 As shown, the method includes:

[0050] S110. If a fault point is detected in each task, the expected recovery speed of at least two target checkpoints is determined, wherein the fault point is the currently faulty task being processed by the system, and the target checkpoint is the historical processing task corresponding to the system when performing the operation of saving an application state snapshot.

[0051] In this embodiment, a task can be a data processing task executed in the system, such as a big data job submitted for data processing in a big data system. In a big data processing system, a job is a collection of parallel tasks; a processing job can be divided into several processing stages (e.g., map stage and reduce stage), and each stage can include several identical processing tasks executed in parallel; furthermore, the execution of different stages of the same job has a temporal order.

[0052] Here, a fault point can be a task that causes a failure while the system is processing a task, i.e., the currently faulty task; the number of currently faulty tasks can be one or more. A checkpoint can be a historical processing task executed by the system at the moment it performs the operation of saving a snapshot of the application state.

[0053] Specifically, checkpoints provide fault tolerance for data processing systems. The system saves a snapshot of the application's state at each checkpoint, allowing the application to restart from the checkpoint when a failure occurs—that is, when a failure point is created. This is especially important for long-running applications running on computing systems prone to failure points. As batch processing applications begin processing tens to hundreds of thousands of records, the application needs to be restartable at some point without having to rerun the entire job from scratch. Therefore, the system can set multiple checkpoints during the execution of individual tasks. If the application fails before the next checkpoint, it can be restarted by providing checkpoint information and successfully completing the last position in the checkpoint file. The application can then restart at that point without starting from scratch.

[0054] In this embodiment, checkpoint recovery can be a restart from the checkpoint, and each task that needs to be processed to recover from the checkpoint to the fault point can be processed sequentially.

[0055] Specifically, this embodiment can monitor the processing status of the system's applications in real time. If a fault is detected in the processing status of the applications, i.e., there is a fault point in each task, the expected recovery speed of each target checkpoint is further determined.

[0056] The target checkpoint can be the checkpoint closest to the current failure point, or the latest checkpoint in the checkpoint file. The target checkpoint can also be a checkpoint with an execution dependency on the failure point; that is, to recover to the failure point, the task corresponding to the target checkpoint must be executed. Of course, considering that tasks are often executed in parallel, the system can create a checkpoint for each of the multiple parallel tasks. Therefore, the distance between the checkpoints corresponding to multiple parallel historical processing tasks and the failure point can be the same. In this embodiment, the number of target checkpoints can be at least two.

[0057] In this embodiment, the estimated recovery speed of the target checkpoint can be the estimated speed of restarting the target checkpoint and executing each task. The estimated recovery speed can be expressed in time, such as 30 minutes, 120 minutes, etc. Specifically, the estimated recovery speed of each target checkpoint can be estimated based on the system's speed in processing historical tasks.

[0058] For example, determining the expected recovery speed of at least two target checkpoints includes: obtaining the historical processing speed of each task; and determining the expected recovery speed of at least two target checkpoints based on the historical processing speed.

[0059] Specifically, by offline extraction of historical processing speeds for all system tasks, the tolerable time for checkpoint writes and recoveries—that is, the expected recovery speed—can be estimated. Historical processing speeds at the task level can also be extracted offline to estimate the recovery time for jobs and tasks at each current target checkpoint. More specifically, by obtaining information such as the system's computational throughput for all historical tasks and system resources, the historical processing speeds of all system tasks can be determined, and an index can be created at the job level to facilitate querying the historical processing speeds of jobs or tasks.

[0060] Of course, in this embodiment, the expected recovery speed of the target checkpoint can also be estimated in real time based on the workload between the target checkpoint and the fault point, as well as the currently available resources of the system.

[0061] S120. Obtain the execution dependency relationship corresponding to each task, and determine the set of execution tasks to restore the target checkpoint to the fault point based on the execution dependency relationship.

[0062] Specifically, after determining the expected recovery speed of each target checkpoint, the execution dependencies between the various tasks that the system needs to process can be further obtained. These execution dependencies can be the relationships between tasks during execution; specifically, a task can only start after its upstream task has been completed. Since each job also has execution dependencies, tasks within different jobs also have execution dependencies.

[0063] For example, the execution dependency between task 1 and task 2 is that task 1 is executed first and then task 2 is executed. Task 1 is divided into tasks 11 and 12 to be executed sequentially, and task 2 is divided into tasks 21 and 22 to be executed in parallel. Then the execution dependency between tasks 11, 12, 21 and 22 is: task 11 → task 12 → task 21 and task 22.

[0064] In one implementation, obtaining the execution dependencies corresponding to each task includes: obtaining a directed acyclic graph corresponding to each task, wherein the directed acyclic graph includes connection direction information between each task; and determining the execution dependencies corresponding to each task based on the connection direction information.

[0065] Specifically, a directed acyclic graph (DAG) can be a directed graph without loops, containing information about the connection directions between tasks. The execution dependencies between tasks can be determined based on the connection direction information in the DAG. For example, in a DAG, tasks with execution dependencies can be connected by directed lines; the task pointed to by a line can only be executed after the task pointing to it has been completed.

[0066] In this embodiment, based on the execution dependencies of each task, the set of tasks required to restore the target checkpoint to the fault point can be determined, i.e., the execution task set. Specifically, the target checkpoint and the fault point can be located in the execution dependencies. Furthermore, all tasks between the target checkpoint and the fault point, the target checkpoint, and the fault point are determined as the execution task set.

[0067] Using the previous example, if the target checkpoint is task 11 and the fault point is task 12, then the set of tasks to be executed includes task 11 and task 12.

[0068] It should be noted that in this embodiment, there are multiple target checkpoints, and the set of execution tasks restored to the fault point can be the same for different target checkpoints. For example, if two target checkpoints are two historical tasks executed in parallel within the same job, then the set of execution tasks restored to the fault point for the two target checkpoints can be the same.

[0069] S130. Based on the expected recovery speed of each target checkpoint, determine the task execution order corresponding to the task set, and process the tasks in the task set according to the task execution order.

[0070] Specifically, the execution order of tasks corresponding to the task set can be determined based on the expected recovery speed of the identified target checkpoints. For example, the expected recovery speeds of each target checkpoint can be compared, and the target checkpoint with the fastest expected recovery speed can be identified as the optimal execution checkpoint. Tasks corresponding to this optimal execution checkpoint are then executed first, and tasks in the task set that have execution dependencies on this optimal execution checkpoint are sorted. Then, tasks corresponding to target checkpoints with expected recovery speeds lower than the optimal execution checkpoint are executed, and tasks in the task set that have execution dependencies on this optimal execution checkpoint are sorted again, until all tasks in the task set are sorted.

[0071] For example, the target checkpoints include tasks 24 and 27. The expected recovery time for task 24 is 55 minutes, and the expected recovery time for task 27 is 63 minutes. Therefore, the target checkpoint corresponding to task 24 can be taken as the optimal execution checkpoint. The optimal execution checkpoint is prioritized for recovery, that is, the optimal execution checkpoint is restarted, and the downstream tasks of task 24 are executed. After the optimal execution checkpoint is recovered, the other target checkpoint is recovered, that is, the target checkpoint is restarted, and the downstream tasks of the task corresponding to the target checkpoint are executed.

[0072] In this embodiment, the purpose of sorting the set of execution tasks according to the expected recovery speed of each target checkpoint is to prioritize the completion of tasks with faster expected recovery speeds. This allows the system to release the system resources corresponding to the faster-recovering tasks after completion, thereby accelerating the processing of tasks with slower expected recovery speeds and minimizing the total execution time, thus significantly improving checkpoint recovery efficiency.

[0073] Furthermore, if an upstream task is a bottleneck on the checkpoint recovery path, then by sorting it by execution dependencies, the upstream task can be scheduled first, that is, the tasks on the preceding dependent nodes have higher execution priority, thereby minimizing the total execution time.

[0074] Furthermore, once the execution order of the tasks in the task set is determined, each task can be processed according to that execution order.

[0075] Specifically, scheduling the tasks in the task set according to the task execution order may involve: determining the checkpoint to be restarted corresponding to the task executed first in the task execution order; restarting the checkpoint to be restarted; and scheduling the tasks in the task set according to the task execution order.

[0076] Among them, the checkpoint to be restarted can be the target checkpoint with the fastest expected recovery speed. In this embodiment, the checkpoint to be restarted can be restarted, and the tasks in the task set can be scheduled in sequence according to the task execution order.

[0077] The technical solution of this embodiment determines the estimated recovery speed of at least one target checkpoint where the system has performed an operation to save an application state snapshot when a fault point is detected in each task being processed by the system. Based on the obtained execution dependencies between each task, a set of execution tasks for restoring the target checkpoint to the fault point is determined. Then, the execution order of each task in the execution task set is determined by the estimated recovery speed of the target checkpoint, and each task in the execution task set is processed according to the execution order. This achieves the sorting of each task that needs to be executed from the checkpoint to the fault point, so that the recovery time of all tasks in the execution task set is minimized, thereby improving the recovery efficiency of the checkpoint and solving the technical problem of excessively long checkpoint recovery time in the prior art.

[0078] Example 2

[0079] Figure 2AThis is a flowchart illustrating a checkpoint recovery method for a data processing task according to Embodiment 2 of the present invention. Based on the above embodiments, optionally, obtaining the execution dependencies corresponding to each task includes: obtaining the execution dependencies corresponding to each task based on the scheduler corresponding to each task. Figure 2A As shown, the method includes:

[0080] S210. If a fault point is detected in each task, the expected recovery speed of at least two target checkpoints is determined, wherein the fault point is the currently faulty task being processed by the system, and the target checkpoint is the historical processing task corresponding to the system when performing the operation of saving an application state snapshot.

[0081] S220. Based on the scheduler corresponding to each task, obtain the execution dependency relationship corresponding to each task, and determine the set of execution tasks to restore the target checkpoint to the fault point based on the execution dependency relationship.

[0082] The scheduler can be the unit in the data processing system responsible for scheduling tasks. Specifically, the scheduler can automatically determine the execution dependencies of each task based on the execution dependencies between jobs and the tasks contained in each job.

[0083] For example, obtaining the execution dependency relationship of each task based on the scheduler corresponding to each task includes: determining the upstream and downstream relationship between each job based on the scheduler, wherein each job consists of multiple stages, and each stage consists of multiple tasks executed in parallel; obtaining the timing information of the stages of each job based on the scheduler; and determining the execution dependency relationship of each task based on the upstream and downstream relationship and the timing information through the scheduler.

[0084] The timing information of a stage can be the execution order of each stage. After determining the upstream and downstream relationships between jobs, the scheduler can further determine the execution dependencies of all tasks based on the timing information of the stages contained within each job. In this way, the scheduler can quickly determine execution dependencies, further improving the efficiency of checkpoint recovery.

[0085] S230. Based on the expected recovery speed of each target checkpoint, determine the task execution order corresponding to the task set, and process the tasks in the task set according to the task execution order.

[0086] Specifically, processing the tasks in the task set according to the task execution order can be achieved by: sending the task execution order to the scheduler; and scheduling the tasks in the task set according to the task execution order through the scheduler.

[0087] For example, such as Figure 2B The diagram shown illustrates the operation of a scheduler. (See also...) Figure 2B The scheduler can be set in the main processing node (master server). The scheduler can schedule each task, and each task is executed by the slave processing nodes (slave servers), namely execution node 1, execution node 2, and execution node 3.

[0088] After the scheduler determines the execution order of the tasks corresponding to the task set, each execution node executes the tasks in the task set according to the scheduler's scheduling. For example, execution node 1 includes tasks 11, 12, and 13 of job 1, and tasks 21 and 22 of job 2; execution node 2 includes tasks 14, 15, and 16 of job 1, and tasks 23 and 24 of job 2; execution node 3 includes tasks 17, 18, and 19 of job 1, and tasks 25 and 26 of job 2. Tasks belonging to the same job in execution nodes 1, 2, and 3 can be in the same stage or in different stages. For example, tasks 21 and 22 belong to one stage, while tasks 23 and 24 belong to another stage.

[0089] exist Figure 2B In this process, the scheduler can determine the execution dependencies between jobs and between tasks, i.e., a directed acyclic graph. The scheduler can also read information such as the amount of data and computational throughput of historical tasks from the job execution history catalog, and estimate the expected recovery speed of each target checkpoint of the system through the cost estimation module.

[0090] Furthermore, the scheduler can determine the set of execution tasks that the target checkpoint can be restored to the fault point based on the directed acyclic graph, and sort each task in the execution task set according to the expected recovery speed calculated by the cost estimation module, and schedule each task in the execution task set according to the sorting.

[0091] Of course, the scheduler can also schedule each task in the execution task set based on the current resources of each execution node, that is, taking into account the resource limitations or data processing speed of each execution node, so as to minimize the total recovery time.

[0092] The technical solution of this embodiment obtains the execution dependencies of each task through the scheduler in the data processing system. Then, based on the obtained execution dependencies between each task, it determines the set of execution tasks to be performed to restore the target checkpoint to the fault point. Then, it determines the execution order of each task in the execution task set based on the expected recovery speed of the target checkpoint, and processes each task in the execution task set according to the execution order. This realizes the sorting of each task that needs to be performed to restore the checkpoint to the fault point, so that the recovery time of all tasks in the execution task set is minimized, the recovery efficiency of the checkpoint is improved, and the technical problem of excessively long checkpoint recovery time in the prior art is solved.

[0093] Example 3

[0094] Figure 3 This is a schematic diagram of a checkpoint recovery device for a data processing task provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a recovery speed prediction module 310, a task set determination module 320, and an execution order determination module 330.

[0095] The recovery speed prediction module 310 is used to determine the expected recovery speed of at least two target checkpoints if a fault point is detected in each task, wherein the fault point is the currently faulty task being processed by the system, and the target checkpoint is the historical processing task corresponding to the system when performing the operation of saving an application state snapshot;

[0096] The task set determination module 320 is used to obtain the execution dependency relationship corresponding to each task, and determine the set of execution tasks to restore the target checkpoint to the fault point based on the execution dependency relationship;

[0097] The execution order determination module 330 is used to determine the task execution order corresponding to the execution task set based on the expected recovery speed of each target checkpoint, and process the tasks in the execution task set according to the task execution order.

[0098] Optionally, the recovery speed prediction module 310 includes a historical speed prediction unit, which is used to obtain the historical processing speed of each task and determine the expected recovery speed of at least two target checkpoints based on the historical processing speed.

[0099] Optionally, the task set determination module 320 includes a directed acyclic graph acquisition unit, which is used to acquire the directed acyclic graph corresponding to each task, wherein the directed acyclic graph includes connection direction information between each task, and the execution dependency relationship corresponding to each task is determined based on the connection direction information.

[0100] Optionally, the task set determination module 320 includes a scheduler acquisition unit, which is used to acquire the execution dependency relationship of each task based on the scheduler corresponding to each task.

[0101] Optionally, the scheduler acquisition unit is specifically used for:

[0102] The scheduler determines the upstream and downstream relationships between each job, wherein each job consists of multiple stages, and each stage consists of multiple tasks executed in parallel.

[0103] The scheduler obtains the timing information of each stage of the job.

[0104] The scheduler determines the execution dependencies of each task based on the upstream and downstream relationships and the timing information.

[0105] Optionally, the execution order determination module 330 includes a task execution unit, which is used to send the task execution order to the scheduler, and the scheduler schedules the tasks in the execution task set according to the task execution order.

[0106] Optionally, the execution order determination module 330 includes a restart execution unit, which is used to determine the checkpoint to be restarted corresponding to the first task executed in the task execution order, restart the checkpoint to be restarted, and schedule the tasks in the set of execution tasks according to the task execution order.

[0107] In this embodiment, the recovery speed prediction module determines the estimated recovery speed of at least one target checkpoint where the system has performed an operation to save an application state snapshot when a fault point is detected in the tasks processed by the system. The task set determination module determines the set of tasks to be executed from the target checkpoint to the fault point based on the execution dependencies between the tasks. Then, the execution order determination module determines the execution order of each task in the execution task set based on the estimated recovery speed of the target checkpoint, and processes each task in the execution task set according to the execution order. This sorts the tasks that need to be executed from the checkpoint to the fault point, so that the recovery time of all tasks in the execution task set is minimized, the recovery efficiency of the checkpoint is improved, and the technical problem of excessively long checkpoint recovery time in the prior art is solved.

[0108] The checkpoint recovery device for data processing tasks provided in this embodiment of the invention can execute the checkpoint recovery method for data processing tasks provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0109] Example 4

[0110] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0111] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0112] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0113] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the checkpoint recovery method for data processing tasks.

[0114] In some embodiments, the checkpoint recovery method for a data processing task may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the checkpoint recovery method for the data processing task described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the checkpoint recovery method for the data processing task by any other suitable means (e.g., by means of firmware).

[0115] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0117] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0118] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0119] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0120] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0121] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0122] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A checkpoint recovery method for a data processing task, characterized in that, include: If a fault point is detected in any task, the expected recovery speed of at least two target checkpoints is determined, wherein the fault point is the currently faulty task being processed by the system, and the target checkpoint is the historical processing task corresponding to the system when performing the operation of saving an application state snapshot; Obtain the execution dependencies corresponding to each task, and determine the set of execution tasks to restore the target checkpoint to the fault point based on the execution dependencies; Based on the expected recovery speed of each target checkpoint, the task execution order corresponding to the task set is determined, and the tasks in the task set are processed according to the task execution order; The step of determining the task execution order corresponding to the set of execution tasks based on the expected recovery speed of each target checkpoint includes: setting the task corresponding to the target checkpoint with the fastest expected recovery speed as the first execution position; continuing to sort the tasks in the set of execution tasks that have an execution dependency relationship with the target checkpoint with the fastest expected recovery speed; then arranging the tasks corresponding to the target checkpoint with the second fastest expected recovery speed; continuing to sort the tasks in the set of execution tasks that have an execution dependency relationship with the target checkpoint with the second fastest expected recovery speed; until all tasks in the set of execution tasks are sorted.

2. The method according to claim 1, characterized in that, Determining the expected recovery speed for at least two target checkpoints includes: Obtain the historical processing speed of each task; The expected recovery speed for at least two target checkpoints is determined based on the historical processing speed.

3. The method according to claim 1, characterized in that, The step of obtaining the execution dependencies corresponding to each task includes: Obtain the directed acyclic graph corresponding to each task, wherein the directed acyclic graph includes the connection direction information between each task; The execution dependencies of each task are determined based on the connection direction information.

4. The method according to claim 1, characterized in that, The step of obtaining the execution dependencies corresponding to each task includes: Based on the scheduler corresponding to each task, the execution dependencies of each task are obtained.

5. The method according to claim 4, characterized in that, The step of obtaining the execution dependencies of each task based on the scheduler corresponding to each task includes: The scheduler determines the upstream and downstream relationships between each job, wherein each job consists of multiple stages, and each stage consists of multiple tasks executed in parallel. The scheduler obtains the timing information of each stage of the job. The scheduler determines the execution dependencies of each task based on the upstream and downstream relationships and the timing information.

6. The method according to claim 4, characterized in that, The step of processing the tasks in the task set according to the task execution order includes: The task execution order is sent to the scheduler; The scheduler schedules the tasks in the task set according to the task execution order.

7. The method according to claim 1, characterized in that, The step of scheduling tasks in the task set according to the task execution order includes: Determine the checkpoint to be restarted for the task executed first in the task execution sequence; Restart the checkpoint to be restarted, and schedule the tasks in the task set to be executed according to the task execution order.

8. A checkpoint recovery device for a data processing task, characterized in that, The device includes: The recovery speed prediction module is used to determine the expected recovery speed of at least two target checkpoints if a fault point is detected in each task, wherein the fault point is the currently faulty task being processed by the system, and the target checkpoint is the historical processing task corresponding to the system when performing the operation of saving an application state snapshot; The task set determination module is used to obtain the execution dependencies corresponding to each task, and determine the set of execution tasks to restore the target checkpoint to the fault point based on the execution dependencies; The execution order determination module is used to determine the task execution order corresponding to the execution task set based on the expected recovery speed of each target checkpoint, and to process the tasks in the execution task set according to the task execution order; The step of determining the task execution order corresponding to the set of execution tasks based on the expected recovery speed of each target checkpoint includes: setting the task corresponding to the target checkpoint with the fastest expected recovery speed as the first execution position; continuing to sort the tasks in the set of execution tasks that have an execution dependency relationship with the target checkpoint with the fastest expected recovery speed; then arranging the tasks corresponding to the target checkpoint with the second fastest expected recovery speed; continuing to sort the tasks in the set of execution tasks that have an execution dependency relationship with the target checkpoint with the second fastest expected recovery speed; until all tasks in the set of execution tasks are sorted.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the checkpoint recovery method for the data processing task according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the checkpoint recovery method for the data processing task according to any one of claims 1-7.

Images