Backtracking task instance recovery method, device, equipment and medium

Abstract

The present application discloses a backtracking task instance recovery method, apparatus, device and medium, which are applied to the field of computer technology to solve the problem of low efficiency in identifying abnormal backtracking task instances in the prior art, including generating a candidate backtracking task instance list based on the core instance identification field of each backtracking task instance of a target backtracking task and associated backtracking tasks that have a job dependency relationship with the target backtracking task within a set time range; marking the current execution status based on the most recent historical execution status of each backtracking task instance; deleting the backtracking task instance whose current execution status is waiting, dependent ready or running from the candidate backtracking task instance list to obtain a target backtracking task instance list; and re-executing each backtracking task instance in the target backtracking task instance list, so as to quickly and accurately identify and recover failed, terminated, and lost backtracking task instances, thereby improving the efficiency and accuracy of backtracking task instance recovery.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and medium for recovering a backtracking task instance. Background Art

[0002] Task backtracking is a core requirement of data processing and scheduling systems. When a fault occurs in the data processing and scheduling system and causes a task failure, it is necessary to quickly and accurately backtrack and recover these failed tasks to avoid greater data asset losses. Currently, when a task fails, operation and maintenance personnel are required to identify and determine the task instances that need to be backtracked and recovered, which not only increases the workload of manual intervention, but also is prone to omissions, and cannot guarantee that all affected tasks can be recovered in a timely manner. Summary of the invention

[0003] The present application provides a backtracking task instance recovery method, apparatus, device and medium to solve the problem that the recognition efficiency and accuracy of task instances in abnormal states in the prior art are low, thereby resulting in poor task recovery effect. The technical solution provided by the present application is as follows:

[0004] On the one hand, the present application provides a backtracking task instance recovery method, comprising:

[0005] Generate a list of candidate backtracking task instances based on the core instance identification field of each backtracking task instance within a set time range of the target backtracking task and each associated backtracking task of the target backtracking task; wherein the associated backtracking task is a backtracking task that has a job dependency relationship with the target backtracking task;

[0006] Based on the query results of the recent historical execution status of each backtracking task instance, mark the current execution status of each backtracking task instance; wherein the current execution status is one of waiting, dependency ready, running, success, failure, termination and loss;

[0007] Based on the current execution status of each backtracking task instance, the core instance identification field of the backtracking task instance whose current execution status is waiting, dependency ready, running or successful is deleted from the candidate backtracking task instance list, and an abnormal backtracking task instance list including the core instance identification field of the backtracking task instance whose current execution status is failed, terminated or lost is obtained;

[0008] Based on the core instance identification field of each backtracing task instance in the abnormal backtracing task instance list, each backtracing task instance in the abnormal backtracing task instance list is re-executed to restore each backtracing task instance in the abnormal backtracing task instance list.

[0009] In a possible implementation, based on the core instance identification field of each backtracking task instance of the target backtracking task and each associated backtracking task of the target backtracking task within a set time range, a candidate backtracking task instance list is generated, including:

[0010] Based on the job dependency relationship between the backtracking tasks in the target job to which the target backtracking task belongs, the backtracking task having a job dependency relationship with the target backtracking task is determined as the job dependent task of the target backtracking task;

[0011] Determine the job dependent tasks whose task creation time and task triggering time are within the set time range as the associated backtracking tasks of the target backtracking task;

[0012] A list of candidate backtrace task instances is generated based on the instance identifier, instance name, execution queue identifier, and instance creation time of each backtrace task instance within a set time range of the target backtrace task and each associated backtrace task.

[0013] In a possible implementation, when generating a list of candidate backtracking task instances based on the core instance identification field of each backtracking task instance within a set time range of the target backtracking task and each associated backtracking task of the target backtracking task, the following is further included:

[0014] The core task identification field of the target backtracking task and the core task identification fields of each associated backtracking task are added to the candidate backtracking task instance list; wherein the core task identification field includes at least a task identification, a task name and a task type.

[0015] In a possible implementation, based on the query result of the most recent historical execution status of each backtracking task instance, marking the current execution status of each backtracking task instance includes:

[0016] For each backtracking task instance, based on the task ID and instance ID of the backtracking task instance, query the previous execution time of the backtracking task instance and the previous execution status corresponding to the previous execution time from the task instance execution record database; wherein the previous execution status is one of waiting, dependency ready, running, success, failure, and termination;

[0017] If the previous execution time of the backtracking task instance and the previous execution status corresponding to the previous execution time are not queried from the task instance execution record database, the current execution status of the backtracking task instance is marked as a lost status;

[0018] If the previous execution times of the backtracking task instance and the previous execution states corresponding to the previous execution times are queried from the task instance execution record database, then the execution state corresponding to the most recent execution time is obtained from the previous execution states of the backtracking task instance as the most recent historical execution state of the backtracking task instance, and the most recent historical execution state of the backtracking task instance is marked as the current execution state of the backtracking task instance.

[0019] In a possible implementation, re-executing each backtracking task instance in the abnormal backtracking task instance list based on the core instance identification field of each backtracking task instance in the abnormal backtracking task instance list includes:

[0020] Display a list of exception backtracking task instances;

[0021] In response to a screening operation on each backtracking task instance in the abnormal backtracking task instance list, determining a core instance identification field of each backtracking task instance after screening;

[0022] Based on the core instance identification field of each filtered backtracking task instance in the abnormal backtracking task instance list, each filtered backtracking task instance in the abnormal backtracking task instance list is re-executed.

[0023] In a possible implementation, based on the core instance identification field of each backtracking task instance in the abnormal backtracking task instance list, when re-executing each backtracking task instance in the abnormal backtracking task instance list, it also includes:

[0024] The current execution status of each backtracking task instance in the abnormal backtracking task instance list is updated to being executed.

[0025] In a possible implementation, based on the core instance identification field of each backtracking task instance in the abnormal backtracking task instance list, after re-executing each backtracking task instance in the abnormal backtracking task instance list, the method further includes:

[0026] Based on the re-execution result of each backtracking task instance in the abnormal backtracking task instance list, the most recent historical execution status of each backtracking task instance in the abnormal backtracking task instance list is updated.

[0027] On the other hand, the present application provides a backtracking task instance recovery device, comprising:

[0028] A list generating unit, for generating a list of candidate backtracking task instances based on the core instance identification field of each backtracking task instance within a set time range of the target backtracking task and each associated backtracking task of the target backtracking task; wherein the associated backtracking task is a backtracking task having a job dependency relationship with the target backtracking task;

[0029] A status marking unit, used to mark the current execution status of each backtracking task instance based on the most recent historical execution status query result of each backtracking task instance; wherein the current execution status is one of waiting, dependency ready, running, success, failure, termination and loss;

[0030] An instance locating unit is used to delete the core instance identification field of the backtracking task instance whose current execution status is waiting, dependency ready, running or successful from the candidate backtracking task instance list based on the current execution status of each backtracking task instance, and obtain an abnormal backtracking task instance list including the core instance identification field of the backtracking task instance whose current execution status is failed, terminated or lost;

[0031] The instance recovery unit is used to re-execute each backtracing task instance in the abnormal backtracing task instance list based on the core instance identification field of each backtracing task instance in the abnormal backtracing task instance list to recover each backtracing task instance in the abnormal backtracing task instance list.

[0032] On the other hand, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned backtracking task instance recovery method when executing the computer program.

[0033] On the other hand, the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the above-mentioned backtracking task instance recovery method is implemented.

[0034] The beneficial effects of this application are as follows:

[0035] The present application generates a list of candidate backtracing task instances based on the core instance identification field of each backtracing task instance within a set time range of the target backtracing task and each associated backtracing task that has an operation dependency relationship with the target backtracing task, and marks the current execution status of each backtracing task instance based on the most recent historical execution status of each backtracing task instance. According to the current execution status of each backtracing task instance, the present application can realize automatic identification and screening of abnormal backtracing task instances of each backtracing task with complex dependencies, thereby realizing backtracing and recovery of abnormal backtracing task instances of each backtracing task with complex dependencies, which not only reduces the workload of manual intervention and reduces the problem of missing backtracing instances, but also ensures that all affected downstream tasks can be restored in a timely manner.

[0036] Other features and advantages of the present application will be described in the subsequent description, and in part, will become apparent from the description, or will be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The schematic diagram and description of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0038] Figure 1 A schematic diagram of an overview of the method for recovering a backtracking task instance in an embodiment of the present application;

[0039] Figure 2 A schematic diagram of the historical execution status of a backtracking task instance in an embodiment of the present application;

[0040] Figure 3 This is a functional structure diagram of a backtracking task instance recovery device in an embodiment of the present application;

[0041] Figure 4 Schematic diagram of the hardware structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and beneficial effects of this application clearer, the technical solution of this application will be clearly and completely described below in combination with the embodiments and drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] The offline task system is the core link in the entire data production process. The offline task system needs to generate tens of thousands or even hundreds of thousands of task instances every day, supporting the data ecology of the entire business chain. Its upper-level data application systems (such as BI intelligent data analysis system, AB testing system, abnormal analysis system, etc.) are all strongly dependent on the stability of the offline task system. Therefore, in the offline task system, the task backtracking capability is very important. How to effectively solve the task backtracking problem is of great significance to ensure the continuity and stability of data production and ensure data consistency. The demand for task backtracking mainly comes from the following aspects:

[0044] Cluster issues: Due to hardware failure, network problems, or other reasons, some nodes in the cluster may fail, causing task execution failure.

[0045] Data issues: The data source may contain errors or incomplete data, causing the task execution to fail.

[0046] Production logic issues: As the business continues to develop and change, the production logic of the task may need to be changed and expanded, and new fields need to be added, which creates a need for data backtracking.

[0047] Scheduling system issues: Potential bugs or design flaws in the scheduling system may lead to task scheduling failures.

[0048] In these cases, task backtracking is required to correct the data to ensure the stability and reliability of the overall data. Currently, one solution is to monitor the status and health of task execution through logging, monitoring, and alarm mechanisms. When a task failure or abnormality is detected, an alarm is triggered to notify the operation and maintenance personnel to intervene manually, and provide task retry and manual task recovery functions. However, these functions often require manual operation by operation and maintenance personnel, which is inefficient and prone to errors. Another solution is to correct the data through automatic task backtracking. There are two common methods:

[0049] Backtracking of a single task instance: This backtracking method focuses on the backtracking of a single task instance, that is, the task backtracking system backtracks a single task instance and tries to recover or re-execute the task instance. This backtracking method is effective when a single task instance has a problem, but in a complex task chain, this backtracking method has great limitations.

[0050] Backtracking of a single task: This backtracking method backtracks a single task within a specified time range, that is, the task backtracking system backtracks all task instances of a single task within a specified time period, checks the status of all task instances of a single task within a specified time period, and performs necessary recovery operations. This backtracking method is suitable for quickly repairing data within a specified time period after completing the logical repair of a single task. However, this backtracking method is still limited to the scope of a single task and cannot handle complex dependencies between tasks.

[0051] Although the above two backtracking methods can solve the problem of task failure to a certain extent, they both have significant limitations:

[0052] Limited coverage: Both backtracking methods focus only on a single task or a single task instance, and cannot fully handle the dependency chain problem across tasks. In data processing and scheduling systems, there are often complex dependencies between tasks, and the failure of a task may affect multiple downstream tasks. In this case, backtracking only a single task or task instance is not enough.

[0053] High demand for manual intervention: Since it only covers a single task scenario, the task backtracking system cannot automatically identify and process the dependencies between tasks. When a task failure affects multiple downstream tasks it depends on, the operation and maintenance personnel need to manually sort out and analyze the entire dependency chain to determine the tasks that need to be backtracked and restored. This not only increases the workload of manual intervention, but also makes it easy to miss problems and cannot guarantee that all affected tasks can be restored in a timely manner.

[0054] Inefficiency: When dealing with large-scale task failures, it is impossible to quickly and comprehensively perform recovery operations. Complex task chains may involve hundreds or even thousands of tasks. The process of manually combing and backtracking these tasks is not only time-consuming, but may also cause long-term interruptions in data processing, affecting business continuity.

[0055] These limitations indicate that traditional task backtracking methods are difficult to provide efficient and comprehensive solutions in the face of complex, dynamic and large-scale data processing environments. To overcome these problems, a more flexible and automated abnormal task instance identification and backtracking mechanism is needed to effectively manage and recover all task instances in complex task chains.

[0056] In order to solve the problem of abnormal task instance identification and backtracking in the entire task chain, this application divides the task backtracking scenarios into two categories according to the different needs of task backtracking: all task backtracking and abnormal task backtracking. Each scenario corresponds to different fault causes and processing strategies. Among them, all task backtracking is mainly aimed at task chain failures caused by production logic problems or data problems. Such problems usually involve errors in business logic or data source problems, and all task instances in the task chain need to be backtracked and re-executed. Based on this, the task backtracking system will traverse all task instances from the root node to the leaf node and mark their current execution status. Even if some task instances succeeded during the first execution, the task backtracking system will re-execute these task instances during the backtracking process to ensure the consistency of the entire task chain and the correctness of the data. In this case, the user can select a specific time range, and the task backtracking system will backtrack all task instances within this time range, which can minimize data loss caused by data problems or logical errors; failed (lost) task backtracking is mainly used to deal with task backtracking requirements caused by cluster problems (such as hardware failure, network interruption, etc.) or scheduling system problems (such as system bugs, task scheduling failures, etc.). In this case, only failed, terminated or lost task instances need to be restored. Based on this, the task backtracking system will first scan all task instances in the candidate backtracking task instance list, and focus on marking the task instances with the current execution status of failure, termination or loss for recovery. In the above two task backtracking scenarios, a list of candidate backtracking task instances is generated based on the core instance identification field of each backtracking task instance within the set time range of the target backtracking task and each associated backtracking task that has a job dependency relationship with the target backtracking task, and the current execution status of each backtracking task instance is marked based on the most recent historical execution status of each backtracking task instance. According to the current execution status of each backtracking task instance, automatic identification and screening of abnormal backtracking task instances of each backtracking task with complex dependencies can be achieved, so that backtracking and recovery of abnormal backtracking task instances of each backtracking task with complex dependencies can be achieved, thereby reducing the workload of manual intervention, reducing the problem of missing backtracking instances, and ensuring that all affected downstream tasks can be restored in a timely manner.

[0057] After introducing the application scenarios and design ideas of the present application, the technical solutions provided by the present application are described in detail below.

[0058] This application embodiment provides a method for recovering a backtracking task instance. Figure 1 As shown, the overview process of the backtracking task instance recovery method provided in the embodiment of the present application is as follows:

[0059] Step 101: Generate a list of candidate backtracing task instances based on the core instance identification field of each backtracing task instance within a set time range of the target backtracing task and each associated backtracing task of the target backtracing task; wherein the associated backtracing task is a backtracing task that has an operation dependency relationship with the target backtracing task.

[0060] In the embodiments of the present application, a task instance refers to the execution entity of a specific task at a certain point in time. For example, a daily data processing task will generate an execution instance for that day, and the execution instance is the task instance. When the task instance is generated, the execution status of the task instance is marked as the historical execution status called in the subsequent backtracking task instance recovery process. In actual applications, refer to Figure 2 As shown, the complete life cycle of a task instance includes the following execution states: waiting, dependency ready, running, termination, failure, success; the execution states of each task instance are stored in the task instance execution record database, in which waiting represents waiting, ready represents dependency ready, running represents running, failed represents failure, succss represents success, and stop represents termination. In this way, when a task failure is detected, a prompt message representing the task failure is sent to the user, and in response to the root node, leaf node and set time range configured by the user for the failed task, the failed task is determined as the target backtracking task, and according to the root node, leaf node and set time range of the target backtracking task, each backtracking task instance of the target backtracking task and each associated backtracking task with which the target backtracking task has a job dependency relationship within the set time range is determined, and based on the core instance identification field of each backtracking task instance of the target backtracking task and each associated backtracking task with which the target backtracking task has a job dependency relationship within the set time range, a candidate backtracking task instance list is generated. In specific implementation, when generating a list of candidate backtracking task instances based on the core instance identification field of each backtracking task instance of the target backtracking task and each associated backtracking task having a job dependency relationship with the target backtracking task within a set time range, the following methods may be used but are not limited to:

[0061] First, based on the job dependency relationship between the backtracking tasks in the target job to which the target backtracking task belongs, the backtracking tasks that have a job dependency relationship with the target backtracking task are determined as job dependent tasks of the target backtracking task.

[0062] Then, the job dependent tasks whose task creation time and task triggering time are within the set time range among the job dependent tasks are determined as associated backtracking tasks of the target backtracking task.

[0063] Finally, based on the core instance identification fields such as instance identification, instance name, execution queue identification and instance creation time of each backtracking task instance of the target backtracking task and each associated backtracking task within the set time range, a list of candidate backtracking task instances is generated. Specifically, according to the root node and leaf node input by the user and the job dependency between each business on the business chain, a directed acyclic graph consisting of each backtracking task instance of the target backtracking task and each associated backtracking task within the set time range is generated, and based on the core instance identification fields such as instance identification, instance name, execution queue identification and instance creation time of the backtracking task instance represented by each node in the directed acyclic graph, a list of candidate backtracking task instances is generated; wherein, the root node represents the starting point of the task instance link, that is, the first task instance to be executed, and the user needs to explicitly specify this node so that the business backtracking system can start the backtracking analysis from here; the leaf node represents the end point of the task instance link, that is, the last executed task instance. The task backtracking system determines each backtracking task instance by analyzing all task instances from the root node to the leaf node.

[0064] In an embodiment of the present application, a directed acyclic graph is constructed and used to represent the dependency relationship between task instances. Since a directed acyclic graph has the characteristic of no cycles, all paths and dependency chains of each task instance from the root node to the leaf node can be clearly displayed. The task backtracking system automatically parses the root node and leaf node provided by the user, and constructs a complete task instance link through a directed acyclic graph. In this way, the position and role of each task instance in the entire task instance link can be clearly understood, thereby improving the accuracy of task backtracking.

[0065] In actual applications, when generating a list of candidate backtracking task instances based on the core instance identification fields of each backtracking task instance of the target backtracking task and each associated backtracking task that has a job dependency relationship with the target backtracking task within a set time range, the core task identification field of the target backtracking task and the core task identification field of each associated backtracking task can also be added to the list of candidate backtracking task instances. Specifically, the core task identification fields such as the task identification, task name, task type, and task business line identification of the target backtracking task and the core task identification fields such as the task identification, task name, task type, and task business line identification of each associated backtracking task are added to the list of candidate backtracking task instances.

[0066] Step 102: Based on the recent historical execution status query results of each backtracking task instance, mark the current execution status of each backtracking task instance; wherein the current execution status is one of waiting, dependency ready, running, success, failure, termination and lost.

[0067] In the embodiment of the present application, when marking the current execution status of each backtracking task instance based on the query result of the most recent historical execution status of each backtracking task instance, the following methods may be used but are not limited to:

[0068] First, for each backtracking task instance, based on the task ID and instance ID of the backtracking task instance, the previous execution time of the backtracking task instance and the previous execution status corresponding to the previous execution time are queried from the task instance execution record database; among which, the previous execution status is one of waiting, dependency ready, running, success, failure, and termination.

[0069] Then, if the previous execution time of the backtracking task instance and the previous execution states corresponding to the previous execution time are not queried from the task instance execution record database, the current execution state of the backtracking task instance is marked as a lost state; if the previous execution time of the backtracking task instance and the previous execution states corresponding to the previous execution time are queried from the task instance execution record database, the execution state corresponding to the most recent execution time is obtained from the previous execution states of the backtracking task instance as the most recent historical execution state of the backtracking task instance, and the most recent historical execution state of the backtracking task instance is marked as the current execution state of the backtracking task instance. In the embodiment of the present application, the current execution state is one of waiting, dependency ready, running, failure, success, loss and termination; wherein, waiting represents waiting, ready represents dependency ready, running represents running, failed represents failure, succss represents success, stop represents termination, and lost represents loss.

[0070] Step 103: Based on the current execution status of each backtracing task instance, delete the core instance identification field of the backtracing task instance whose current execution status is waiting, dependency ready, running or successful from the candidate backtracing task instance list, and obtain an abnormal backtracing task instance list containing the core instance identification field of the backtracing task instance whose current execution status is failed, lost or terminated.

[0071] In an embodiment of the present application, based on the current execution status of each backtracing task instance, the core instance identification field of the backtracing task instance whose current execution status is waiting, dependent ready, running or successful is deleted from the candidate backtracing task instance list, and after obtaining the abnormal backtracing task instance list containing the core instance identification field of the backtracing task instance whose current execution status is failed, lost or terminated, the abnormal backtracing task instance list can also be displayed, and in response to the filtering operation on each backtracing task instance in the abnormal backtracing task instance list, the core instance identification field of each backtracing task instance after filtering is determined to perform subsequent recovery operations.

[0072] Step 104: Based on the core instance identification field of each backtracing task instance in the abnormal backtracing task instance list, re-execute each backtracing task instance in the abnormal backtracing task instance list to restore each backtracing task instance in the abnormal backtracing task instance list.

[0073] In an embodiment of the present application, if the user performs a screening operation on each backtracking task instance in the abnormal backtracking task instance list, after determining the core instance identification field of each backtracking task instance after screening, based on the core instance identification field of each backtracking task instance after screening in the abnormal backtracking task instance list, re-execute each backtracking task instance after screening in the abnormal backtracking task instance list, and during the re-execution process, update the current execution status of each backtracking task instance after screening in the abnormal backtracking task instance list to being executed. Further, after re-executing each backtracking task instance after screening in the abnormal backtracking task instance list, the recent historical execution status of each backtracking task instance in the abnormal backtracking task instance list can also be updated based on the re-execution result of each backtracking task instance in the abnormal backtracking task instance list.

[0074] In summary, by backtracking task instances based on the mark-sweep algorithm, we can quickly and accurately find and recover task instances in the three states of failure, termination, and loss. The specific workflow is as follows:

[0075] Build instance list: Generate a list of all instances within a time interval based on the backtracking time range provided by the user, combined with the task creation time (less than or equal to the maximum value of the backtracking time range) and the trigger time (that is, the start time).

[0076] Execution status marking: In the task backtracking process, execution status marking is a key step to identify the task instance to be restored. The task backtracking system determines the current execution status of each task instance based on the historical execution status of the task instance. The specific steps are as follows:

[0077] Retrieve historical execution status: The task backtracking system retrieves the historical execution status of each task instance from the database. The database stores the execution records of each task instance. Each execution record contains information such as the task ID, execution status, and update time. The task backtracking system searches for the latest update record of each task instance based on the task ID. The latest record is determined by obtaining the "Update Date" field, thereby obtaining the most recent execution status of the task instance.

[0078] Current marking status:

[0079] Failed tasks: If the latest record shows that the status of the task instance is "failed", the task instance is marked as failed. This means that the task instance encountered an error during execution and failed to complete successfully.

[0080] Terminated Task: If the latest record shows that the status of the task instance is "Terminated", the task instance is marked as terminated. This means that the task instance was stopped manually or automatically during execution and did not complete the scheduled work.

[0081] Lost Task: If a task instance is in the generated candidate backtracking task instance list, but fails to find any history in the database or does not have the latest status update, the task instance is marked as lost, which means that the task instance is lost or failed to start during execution.

[0082] Update instance list: By storing task instances with marked status in the newly generated candidate backtracking task instance list, the current execution status of each task instance is accurately identified. Task instances with current execution status of failure, termination, and loss will be screened out to prepare for further backtracking and recovery operations.

[0083] In this way, the task backtracking system can quickly and accurately identify the task instances that need to be backtracked, ensuring that the entire task chain can be repaired in time after a failure occurs, maintaining the continuity and stability of data processing.

[0084] Task instance cleanup:

[0085] Based on the current execution status of the marker, the task backtracking system will perform the following operations:

[0086] Clear the task instances whose current execution status is waiting, dependency ready, successful, or running. These task instances are considered to be restartable or reschedulable and do not need to be retained during the backtracking process. Therefore, the task backtracking system will clear the task instances in these execution states and remove them from the candidate backtracking task instance list.

[0087] Keep the task instances whose current execution status is failed, terminated or lost. These task instances represent the objects that need to be focused on during the backtracking process. The task backtracking system will keep these task instances and prepare for further recovery operations.

[0088] Return the cleared abnormal backtracking task instance list: After clearing unnecessary task instances, the task backtracking system will return an abnormal backtracking task instance list that only contains task instances whose current execution status is failed, terminated, or lost. This abnormal backtracking task instance list contains key task instances that need to be backtracked and restored.

[0089] User confirmation and re-execution: The list of abnormal backtracking task instances will be presented to the user, and the user can view and confirm the task instances that need to be restored. By filtering the list of abnormal backtracking task instances, different types of task backtracking can be freely selected. After confirmation, the task backtracking system will re-execute these task instances in order to recover and reprocess the failed or lost task instances.

[0090] The task backtracking system based on the mark-sweep algorithm has the following advantages:

[0091] Automation: The task backtracking system can automatically identify and mark failed, terminated, or lost task instances without manual intervention by operators.

[0092] Fast recovery: The task backtracking system can quickly and accurately locate and recover failed, terminated, or lost task instances, greatly improving the efficiency and accuracy of task recovery.

[0093] Flexibility: Users can flexibly set the time range and filtering conditions according to their needs to meet the task backtracking requirements in different scenarios.

[0094] Memory optimization: In addition to implementing basic functions, the task backtracking system also implements memory optimization. The two parts that consume the most memory are the task list and the instance list. If the task backtracking system implements a complete task list and instance list, the memory consumption of a single task is about 600 bytes; the memory consumption of adding a single instance is about 250 bytes.

[0095] If there are 100,000 task objects and each task object has an average of 1.5 instance objects, the total memory consumption is approximately:

[0096] 100,000 (task objects) * (600 bytes + 1.5 * 250 bytes) ≈ 100,000 * 1,250bytes ≈ 125 MB

[0097] For more efficient storage, the task backtracking system only retains the core fields of the task object and instance object when generating a list of candidate backtracking task instances; among them:

[0098] Memory consumption calculation of the core fields of the task object:

[0099] Task number; id (Integer): 4 bytes;

[0100] Task name; jobName (String): 100 bytes;

[0101] Task business line number; businessId (Integer): 4 bytes;

[0102] Task type; jobType (Enum): 4 bytes;

[0103] Task creation time; createTime (LocalDateTime): 16 bytes (object overhead) + 8 bytes (long type timestamp);

[0104] Total 136 bytes.

[0105] Memory consumption calculation of core fields of instance objects:

[0106] Instance number; id (Integer): 4 bytes;

[0107] Job ID; jobId (Integer): 4 bytes;

[0108] Task name; jobName (String): 100 bytes;

[0109] Task type; jobType (Enum): 4 bytes;

[0110] Instance name; instanceName (String): 100 bytes;

[0111] Execute queue; executeAccount(String):100 bytes;

[0112] Instance creation time; createTime (LocalDateTime): 16 bytes (object overhead) + 8 bytes (long type timestamp);

[0113] Total: 336 bytes.

[0114] In the embodiment of the present application, only the core fields of the task object and the instance object are retained when generating the candidate backtracking task instance list, which can save a lot of memory, and the memory occupied is 80.8MB, which is a significant optimization compared with the previous memory consumption. The task backtracking method based on the mark-clearing algorithm can effectively solve the task backtracking problem in the task scheduling system, improve the continuity and stability of data production, and provide strong support for the data ecological construction of enterprises.

[0115] Based on the above embodiments, the present application embodiment provides a backtracking task instance recovery device, see Figure 3 As shown, the backtracking task instance recovery device 300 provided in the embodiment of the present application at least includes:

[0116] A list generating unit 301 is used to generate a list of candidate backtracking task instances based on the core instance identification field of each backtracking task instance within a set time range of the target backtracking task and each associated backtracking task of the target backtracking task; wherein the associated backtracking task is a backtracking task that has a job dependency relationship with the target backtracking task;

[0117] A status marking unit 302 is used to mark the current execution status of each backtracking task instance based on the most recent historical execution status query result of each backtracking task instance; wherein the current execution status is one of waiting, dependency ready, running, success, failure, termination and lost;

[0118] The instance locating unit 303 is used to delete the core instance identification field of the backtracking task instance whose current execution status is waiting, dependent ready, running or successful from the candidate backtracking task instance list based on the current execution status of each backtracking task instance, and obtain the abnormal backtracking task instance list including the core instance identification field of the backtracking task instance whose current execution status is failed, lost or terminated;

[0119] The instance recovery unit 304 is used to re-execute each backtracing task instance in the abnormal backtracing task instance list based on the core instance identification field of each backtracing task instance in the abnormal backtracing task instance list to recover each backtracing task instance in the abnormal backtracing task instance list.

[0120] In one possible implementation, the list generation unit 301 is used to determine the backtracing tasks that have a job dependency relationship with the target backtracing task as the job dependent tasks of the target backtracing task based on the job dependency relationship between the various backtracing tasks in the target job to which the target backtracing task belongs; determine the job dependent tasks whose task creation time and task trigger time in each job dependent task are within a set time range as the associated backtracing tasks of the target backtracing task; and generate a list of candidate backtracing task instances based on the instance identifier, instance name, execution queue identifier and instance creation time of each backtracing task instance within the set time range of the target backtracing task and each associated backtracing task.

[0121] In a possible implementation, the list generation unit 301 is also used to add the core task identification field of the target backtracking task and the core task identification field of each associated backtracking task to the candidate backtracking task instance list; wherein the core task identification field includes at least a task identification, a task name and a task type.

[0122] In one possible implementation, the state marking unit 302 is used to query the previous execution time of the backtracing task instance and the previous execution status corresponding to the previous execution time from the task instance execution record database for each backtracing task instance based on the task identifier and instance identifier of the backtracing task instance; wherein the previous execution status is one of waiting, dependency ready, running, success, failure, and termination; if the previous execution time of the backtracing task instance and the previous execution status corresponding to the previous execution time are not queried from the task instance execution record database, the current execution state of the backtracing task instance is marked as a lost state; if the previous execution time of the backtracing task instance and the previous execution status corresponding to the previous execution time are queried from the task instance execution record database, the execution state corresponding to the most recent execution time is obtained from the previous execution status of the backtracing task instance as the most recent historical execution state of the backtracing task instance, and the most recent historical execution state of the backtracing task instance is marked as the current execution state of the backtracing task instance.

[0123] In a possible implementation, the backtracking task instance recovery device 300 provided in the embodiment of the present application further includes:

[0124] The instance screening unit 305 is used to display the abnormal backtracking task instance list; in response to the screening operation on each backtracking task instance in the abnormal backtracking task instance list, obtain the core instance identification field of each backtracking task instance after screening.

[0125] In a possible implementation, the instance recovery unit 304 is configured to re-execute each backtracing task instance screened in the abnormal backtracing task instance list based on the core instance identification field of each backtracing task instance screened in the abnormal backtracing task instance list.

[0126] In a possible implementation, the list generating unit 301 is further configured to update the current execution state of each backtracing task instance in the abnormal backtracing task instance list to being executed.

[0127] In a possible implementation, the backtracking task instance recovery device 300 provided in the embodiment of the present application further includes:

[0128] The state updating unit 306 is used to update the most recent historical execution state of each backtracing task instance in the abnormal backtracing task instance list based on the re-execution result of each backtracing task instance in the abnormal backtracing task instance list.

[0129] It should be noted that the principle of solving the technical problem by the backtracking task instance recovery device 300 provided in the embodiment of the present application is similar to the backtracking task instance recovery method provided in the embodiment of the present application. Therefore, the implementation of the backtracking task instance recovery device 300 provided in the embodiment of the present application can refer to the implementation of the backtracking task instance recovery method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0130] After introducing the backtracking task instance recovery method and device provided in the embodiment of the present application, next, the electronic device provided in the embodiment of the present application is briefly introduced.

[0131] See also Figure 4 As shown, the electronic device 400 provided in the embodiment of the present application includes at least a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program, the backtracking task instance recovery method provided in the embodiment of the present application is implemented.

[0132] The electronic device 400 provided in the embodiment of the present application may further include a bus 403 connecting different components (including the processor 401 and the memory 402). The bus 403 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0133] The memory 402 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 4021 and / or a cache memory 4022, and may further include a read-only memory (ROM) 4023. The memory 402 may also include a program tool 4025 having a group (at least one) of program modules 4024, and the program modules 4024 include but are not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each of which or some combination may include the implementation of a network environment.

[0134] Processor 401 may be a processing element or a collective term for multiple processing elements. For example, processor 401 may be a central processing unit (CPU) or one or more integrated circuits configured to implement the backtracking task instance recovery method provided in the embodiment of the present application. Specifically, processor 401 may be a general-purpose processor, including but not limited to a CPU, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.

[0135] The electronic device 400 may also communicate with one or more external devices 404 (e.g., keyboards, remote controls, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 400 (e.g., mobile phones, computers, etc.), and / or communicate with any device that enables the electronic device 400 to communicate with one or more other electronic devices 400 (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 405. Furthermore, the electronic device 400 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 406. Figure 4 As shown, the network adapter 406 communicates with other modules of the electronic device 400 via the bus 403. It should be understood that although Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 400, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0136] It should be noted that Figure 4 The electronic device 400 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0137] The computer-readable storage medium provided by the embodiment of the present application is introduced below. The computer-readable storage medium provided by the embodiment of the present application stores computer instructions, and when the computer instructions are executed by the processor, the backtracking task instance recovery method provided by the embodiment of the present application is implemented. Specifically, the computer instructions can be built-in or installed in the processor, so that the processor can implement the backtracking task instance recovery method provided by the embodiment of the present application by executing the built-in or installed computer instructions.

[0138] In addition, the backtracking task instance recovery method provided in the embodiment of the present application can also be implemented as a computer program product, which includes a program code. When the program code is run on a processor, it implements the backtracking task instance recovery method provided in the embodiment of the present application.

[0139] The computer program product provided in the embodiments of the present application may adopt one or more computer-readable storage media, and the computer-readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any suitable combination of the above. Specifically, more specific examples of computer-readable storage media (a non-exhaustive list) include an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

[0140] The computer program product provided in the embodiment of the present application may adopt a CD-ROM and include program code, and may also be run on electronic devices such as computers and servers. However, the computer program product provided in the embodiment of the present application is not limited thereto. In the embodiment of the present application, the computer-readable storage medium may be any tangible medium containing or storing program code, and the program code may be used by or in combination with an instruction execution system, apparatus, or device.

[0141] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0142] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0143] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0144] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A backtracking task instance recovery method, characterized in that: include: In response to the root node, leaf node and set time range configured for the failed task, the failed task is determined as a target backtracking task, and according to the root node, the leaf node and the set time range, each backtracking task instance of the target backtracking task and each associated backtracking task of the target backtracking task within the set time range is determined; wherein the root node represents the task instance that starts to be executed first, and the leaf node represents the task instance that is executed last; the backtracking task instance is an execution entity of the target backtracking task or the associated backtracking task at a time point within the set time range; Generate a list of candidate backtracking task instances based on the core instance identification field of each of the backtracking task instances of the target backtracking task and each of the associated backtracking tasks of the target backtracking task within the set time range; wherein the associated backtracking tasks are backtracking tasks that have a job dependency relationship with the target backtracking task; Based on the recent historical execution status query results of each of the backtracking task instances, mark the current execution status of each of the backtracking task instances; wherein the current execution status is one of waiting, dependency ready, running, success, failure, termination and loss; Based on the current execution status of each of the backtracking task instances, the core instance identification field of the backtracking task instance whose current execution status is waiting, dependent ready, running or successful is deleted from the candidate backtracking task instance list, and an abnormal backtracking task instance list containing the core instance identification field of the backtracking task instance whose current execution status is failed, terminated or lost is obtained; Based on the core instance identification field of each of the backtracing task instances in the abnormal backtracing task instance list, re-execute each of the backtracing task instances in the abnormal backtracing task instance list to restore each of the backtracing task instances in the abnormal backtracing task instance list; Wherein, based on the query result of the most recent historical execution status of each of the backtracking task instances, marking the current execution status of each of the backtracking task instances includes: For each of the backtracking task instances, based on the task identifier and instance identifier of the backtracking task instance, query the previous execution time of the backtracking task instance and the previous execution status corresponding to the previous execution time from the task instance execution record database; wherein the previous execution status is one of waiting, dependency ready, running, success, failure, and termination; If the previous execution time of the backtracking task instance and the previous execution states corresponding to the previous execution times are not found in the task instance execution record database, the current execution state of the backtracking task instance is marked as a lost state; If the previous execution times of the backtracing task instance and the previous execution states corresponding to the previous execution times are queried from the task instance execution record database, then the execution state corresponding to the most recent execution time is obtained from the previous execution states of the backtracing task instance as the most recent historical execution state of the backtracing task instance, and the most recent historical execution state of the backtracing task instance is marked as the current execution state of the backtracing task instance.

2. The backtracking task instance recovery method according to claim 1, characterized in that: Based on the core instance identification field of each backtracking task instance of the target backtracking task and each associated backtracking task of the target backtracking task within a set time range, a candidate backtracking task instance list is generated, including: Based on the job dependency relationship between the backtracking tasks in the target job to which the target backtracking task belongs, determining the backtracking task that has a job dependency relationship with the target backtracking task as the job dependent task of the target backtracking task; Determine, among the job-dependent tasks, the job-dependent tasks whose task creation time and task trigger time are within the set time range as associated backtracking tasks of the target backtracking task; The candidate backtracing task instance list is generated based on the instance identifier, instance name, execution queue identifier and instance creation time of each backtracing task instance within a set time range of the target backtracing task and each of the associated backtracing tasks.

3. The backtracking task instance recovery method according to claim 1, characterized in that: When generating a list of candidate backtracking task instances based on the core instance identification field of each backtracking task instance of the target backtracking task and each associated backtracking task of the target backtracking task within a set time range, the following is also included: The core task identification field of the target backtracking task and the core task identification fields of each of the associated backtracking tasks are added to the candidate backtracking task instance list; wherein the core task identification field includes at least a task identification, a task name and a task type.

4. The backtracking task instance recovery method according to claim 1, characterized in that: Re-executing each of the backtracing task instances in the abnormal backtracing task instance list based on the core instance identification field of each of the backtracing task instances in the abnormal backtracing task instance list includes: Display the exception backtracking task instance list; In response to a screening operation on each of the backtracing task instances in the abnormal backtracing task instance list, determining a core instance identification field of each backtracing task instance after screening; Based on the core instance identification field of each filtered backtracing task instance in the abnormal backtracing task instance list, each filtered backtracing task instance in the abnormal backtracing task instance list is re-executed.

5. The backtracking task instance recovery method according to claim 1, characterized in that: When re-executing each of the backtracing task instances in the abnormal backtracing task instance list based on the core instance identification field of each of the backtracing task instances in the abnormal backtracing task instance list, the method further includes: The current execution status of each of the backtracing task instances in the abnormal backtracing task instance list is updated to being executed.

6. The backtracking task instance recovery method according to any one of claims 1 to 5, characterized in that: After re-executing each of the backtracing task instances in the abnormal backtracing task instance list based on the core instance identification field of each of the backtracing task instances in the abnormal backtracing task instance list, the method further includes: Based on the re-execution result of each of the backtracing task instances in the abnormal backtracing task instance list, the most recent historical execution status of each of the backtracing task instances in the abnormal backtracing task instance list is updated.

7. A backtracking task instance recovery device, characterized in that: include: An instance determination unit is used to determine the failed task as a target backtracking task in response to the root node, leaf node and set time range configured for the failed task, and determine each backtracking task instance of the target backtracking task and each associated backtracking task of the target backtracking task within the set time range according to the root node, the leaf node and the set time range; wherein the root node represents the task instance that starts to be executed first, and the leaf node represents the task instance that is executed last; the backtracking task instance is an execution entity of the target backtracking task or the associated backtracking task at a time point within the set time range; A list generating unit, configured to generate a list of candidate backtracing task instances based on the core instance identification field of each of the backtracing task instances within the set time range of the target backtracing task and each of the associated backtracing tasks of the target backtracing task; wherein the associated backtracing tasks are backtracing tasks having a job dependency relationship with the target backtracing task; A status marking unit, used to mark the current execution status of each of the backtracking task instances based on the most recent historical execution status query results of each of the backtracking task instances; wherein the current execution status is one of waiting, dependency ready, running, success, failure, termination and loss; An instance locating unit is used to delete the core instance identification field of the backtracing task instance whose current execution status is waiting, dependent ready, running or successful from the candidate backtracing task instance list based on the current execution status of each backtracing task instance, and obtain an abnormal backtracing task instance list including the core instance identification field of the backtracing task instance whose current execution status is failed, terminated or lost; An instance recovery unit, configured to re-execute each of the backtracing task instances in the abnormal backtracing task instance list based on the core instance identification field of each of the backtracing task instances in the abnormal backtracing task instance list, so as to recover each of the backtracing task instances in the abnormal backtracing task instance list; Among them, the state marking unit is used to query the previous execution time of the backtracing task instance and the previous execution status corresponding to the previous execution time from the task instance execution record database for each of the backtracing task instances based on the task identifier and instance identifier of the backtracing task instance; wherein the previous execution statuses are one of waiting, dependency ready, running, success, failure, and termination; if the previous execution time of the backtracing task instance and the previous execution status corresponding to the previous execution time are not queried from the task instance execution record database, the current execution state of the backtracing task instance is marked as a lost state; if the previous execution time of the backtracing task instance and the previous execution status corresponding to the previous execution time are queried from the task instance execution record database, the execution state corresponding to the most recent execution time is obtained from the previous execution statuses of the backtracing task instance as the most recent historical execution state of the backtracing task instance, and the most recent historical execution state of the backtracing task instance is marked as the current execution state of the backtracing task instance.

8. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for recovering a backtracking task instance as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the backtracking task instance recovery method according to any one of claims 1 to 6 is implemented.

Images