Backtracking instance bloodline graph construction method, task backtracking method and system
By constructing a blood relationship map of backtracking instances and performing batch backtracking, the problem of poor backtracking effect is solved, the accuracy and efficiency of backtracking are improved, and data correction and historical data analysis are supported.
Patent Information
- Application Number
- CN202410889451.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-04
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-04
AI Technical Summary
The task backtracking effect in the prior art is poor, and it is difficult to effectively support data correction and historical data analysis, resulting in inaccurate backtracking and low efficiency.
By constructing a backtracking instance blood relationship map, based on the data output time and job dependency relationship of the target backtracking task, the dependency link construction operation is performed cycleically, the backtracking instance blood relationship map is generated, and the backtracking task is added according to the backtracking priority and scheduling initiation time. The task queue is backtracked and batch backtracking is performed.
It improves the accuracy and efficiency of task backtracking, ensures the orderly task backtracking, and enhances the support capabilities of data correction and historical data analysis.
Smart Images

Figure CN118467139B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method for constructing a backtracking instance bloodline graph, a task backtracking method and a system. Background Art
[0002] The task backtracking function is a very important function of the data middle platform. For example, when data anomalies occur and need to be corrected, that is, when completing the processing scenario, or when re-analyzing and mining historical data, that is, when re-running the scenario, the task backtracking function is required. However, the task backtracking effect of the current task backtracking scheme is often poor. Summary of the invention
[0003] The present application provides a backtracking instance lineage graph construction method, a task backtracking method and a system to solve the problem of poor task backtracking effect in the prior art. The technical solution provided by the present application is as follows:
[0004] On the one hand, the present application provides a method for constructing a backtracking instance bloodline graph, comprising:
[0005] Determine each backtracking instance of the target backtracking task within the data backtracking time period and the data output time of each backtracking instance corresponding to the target backtracking task;
[0006] Based on the data output time of each backtracking instance corresponding to the target backtracking task, a backtracking instance lineage graph between each target backtracking instance is constructed according to the job dependency relationship between each task in the target job to which the target backtracking task belongs; wherein the job dependency relationship includes the data dependency relationship and the data dependency time; the target backtracking instance includes each backtracking instance corresponding to the target backtracking task and each backtracking instance of the associated backtracking task corresponding to the target backtracking task in the target job within the instance backtracking time period; the associated backtracking tasks are each task in the target job that has a direct or indirect chain reaction based on the data output of each backtracking instance corresponding to the target backtracking task.
[0007] In a possible implementation, based on the data production time of each backtracking instance corresponding to the target backtracking task and according to the job dependency relationship between each task in the target job to which the target backtracking task belongs, a backtracking instance lineage graph between each target backtracking instance is constructed, including:
[0008] Based on the data output time of each backtracking instance corresponding to the target backtracking task, according to the job dependency relationship between each task in the target job to which the target backtracking task belongs, the dependency link building operation is cyclically executed until it is determined that the instance backtracking end condition is met, and based on the dependency link between the current upstream instance node and the target downstream instance node constructed each time the dependency link building operation is executed, a backtracking instance lineage graph is generated; wherein the dependency link building operation includes:
[0009] Determine the current upstream instance node; wherein, when the dependency link building operation is executed for the first time, the current upstream instance node is the root node; when the dependency link building operation is not executed for the first time, the current upstream instance node is the target downstream instance node determined when the dependency link building operation was last executed; the root node is the backtracking instance that does not have a job dependency relationship in each backtracking instance corresponding to the target backtracking task;
[0010] Based on the data dependency relationship between the tasks in the target job, determine the backtracking instance that depends on the current upstream instance node as the initial downstream instance node;
[0011] Based on the data dependency time of the initial downstream instance node on the current upstream instance node and the timing cycle size relationship between the data output time of the current upstream instance node, the target downstream instance node of the current upstream instance node is determined with the initial downstream instance node as the benchmark;
[0012] Build a dependency link from the current upstream instance node to the target downstream instance node.
[0013] In a possible implementation, based on the data dependency time of the initial downstream instance node on the current upstream instance node and the timing cycle size relationship between the data production time of the current upstream instance node, determining the target downstream instance node of the current upstream instance node with the initial downstream instance node as a reference includes:
[0014] If it is determined that the data dependency time of the initial downstream instance node on the current upstream instance node is the same as the timing cycle between the data production time of the current upstream instance node, the initial downstream instance node is determined as the target downstream instance node of the current upstream instance node;
[0015] If it is determined that the data dependency time of the initial downstream instance node on the current upstream instance node is a small period, and the data output time of the current upstream instance node is a large period, then the initial downstream instance node is expanded to obtain the target downstream instance node of the current upstream instance node;
[0016] If it is determined that the data dependency time of the initial downstream instance node on the current upstream instance node is a large cycle and the data production time of the current upstream instance node is a small cycle, the initial downstream instance node is shrunk to obtain the target downstream instance node of the current upstream instance node.
[0017] In a possible implementation manner, if it is determined that the data dependency time of the initial downstream instance node on the current upstream instance node is the same as the timing period between the data production time of the current upstream instance node, the method further includes:
[0018] When it is detected that the data dependency time of the initial downstream instance node on the current upstream instance node is greater than the data production time of the current upstream instance node, it is determined that the upstream instance node of the initial downstream instance node is missing in the current upstream instance node;
[0019] Based on the time difference between the data dependency time of the initial downstream instance node on the current upstream instance node and the data production time of the current upstream instance node, the upstream instance node of the missing initial downstream instance node is completed to obtain the subsequent upstream instance node;
[0020] Build a dependency link from the backup upstream instance node to the target downstream instance node.
[0021] In a possible implementation, when building a dependency link from the current upstream instance node to the target downstream instance node, the following is further included:
[0022] When it is detected that the dependency link from the current upstream instance node to the target downstream instance node is a circular dependency link, the dependency link from the current upstream instance node to the target downstream instance node is deleted.
[0023] On the other hand, the present application provides a task backtracking method, comprising:
[0024] Obtaining task backtracking information; wherein the task backtracking information at least includes a target backtracking task, a data backtracking time period, and an instance backtracking time period;
[0025] Based on the task backtracking information, the backtracking instance lineage graph of the target backtracking task is constructed using the backtracking instance lineage graph construction method described above;
[0026] According to the backtracking priority and scheduling initiation time of the target backtracking task, the target backtracking task is added to the queue of tasks to be backtracked as a task to be backtracked;
[0027] When it is determined that the number of tasks to be backtraced in the task queue to be backtraced reaches the queue capacity of the task queue to be backtraced, batch backtraces are performed on each task to be backtraced in the task queue to be backtraced according to the arrangement order of each task to be backtraced in the task queue to be backtraced and based on the backtracing instance lineage graph of each task to be backtraced.
[0028] In a possible implementation, according to the arrangement order of each task to be backtracked in the task queue to be backtracked, based on the backtracking instance lineage graph of each task to be backtracked, the process of batch backtracking each task to be backtracked in the task queue to be backtracked also includes:
[0029] When it is determined that the number of backtracking tasks in the backtracking task queue reaches an alarm threshold, and / or when it is determined that there are backtracking instances in which backtracking fails or backtracking is terminated in each backtracking task, an alarm message is initiated.
[0030] In a possible implementation, the task backtracking method provided by the present application further includes:
[0031] When it is determined that the backtracking instance in the backtracking task that failed or was terminated meets the rerunning conditions, the backtracking task is continued to be backtracked starting from the backtracking instance in the backtracking task that failed or was terminated.
[0032] In a possible implementation, batch backtracking is performed on each backtracking task in the backtracking task queue according to the arrangement order of each backtracking task in the backtracking task queue, including:
[0033] According to the arrangement order of each backtracking task in the backtracking task queue, each backtracking task in the backtracking task queue is sequentially determined as the current backtracking task;
[0034] Determine the launch time of the downstream instance node based on the launch time of the upstream instance node in the current backtracking task, the upstream instance output cycle offset, and the downstream instance dependency cycle offset.
[0035] Backtrack the current backtracking task according to the initiation time of each instance node in the current backtracking task.
[0036] On the other hand, the present application provides a task backtracking system, comprising:
[0037] The information configuration subsystem is used to obtain task backtracking information; wherein the task backtracking information at least includes a target backtracking task, a data backtracking time period, and an instance backtracking time period;
[0038] A lineage graph construction subsystem is used to construct a backtracking instance lineage graph of a target backtracking task based on task backtracking information and using the backtracking instance lineage graph construction method described above;
[0039] The task backtracking subsystem is used to add the target backtracking task as a to-be-backtracked task to the to-be-backtracked task queue according to the backtracking priority and scheduling initiation time of the target backtracking task; when it is determined that the number of to-be-backtracked tasks in the to-be-backtracked task queue reaches the queue capacity of the to-be-backtracked task queue, batch backtracking is performed on each to-be-backtracked task in the to-be-backtracked task queue according to the arrangement order of each to-be-backtracked task in the to-be-backtracked task queue and based on the backtracking instance lineage graph of each to-be-backtracked task.
[0040] The beneficial effects of this application are as follows:
[0041] This application constructs a backtracking instance lineage graph between each backtracking instance based on the data output time of each backtracking instance and the job dependency between the tasks to which each backtracking instance belongs, and backtracks the target backtracking task based on the backtracking instance lineage graph between each backtracking instance. Each backtracking instance of the target backtracking task can be traversed during the backtracking process, thereby improving the backtracking effect of the target task. Moreover, by adding the target backtracking task as a task to be backtracked to the queue of tasks to be backtracked according to the backtracking priority and scheduling initiation time of the target backtracking task, when the number of tasks to be backtracked in the queue of tasks to be backtracked reaches the queue capacity of the queue of tasks to be backtracked, batch backtracking is performed on each task to be backtracked in the queue of tasks to be backtracked, which can improve the efficiency of task backtracking while ensuring the orderliness of task backtracking.
[0042] Other features and advantages of the present application will be described in the subsequent description, and in part, will become apparent from the description, or will be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The schematic diagram and description of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0044] Figure 1 A schematic diagram of the general flow of the method for constructing a backtracking instance bloodline graph in the embodiments of the present application;
[0045] Figure 2 A schematic diagram of a backtracking instance bloodline diagram based on a continuous dependency relationship in an embodiment of the present application;
[0046] Figure 3 A schematic diagram of a backtracking instance bloodline diagram based on multi-day dependencies in an embodiment of the present application;
[0047] Figure 4 A schematic diagram of a backtracking instance bloodline graph based on cross-cycle dependencies in an embodiment of the present application;
[0048] Figure 5 A schematic diagram of a backtracking instance bloodline diagram based on cross-dependency in an embodiment of the present application;
[0049] Figure 6 A schematic diagram of a backtracking instance bloodline diagram based on a circular dependency relationship in an embodiment of the present application;
[0050] Figure 7 The following is a schematic diagram of the task backtracking method in the embodiment of the present application;
[0051] Figure 8 This is a schematic diagram of the target backtracking task initiation process in an embodiment of the present application;
[0052] Fig. 9 This is a schematic diagram of the backtracking instance scheduling process in the embodiment of the present application;
[0053] Fig.10 Schematic diagram of the estimation process of the backtracking completion time in the embodiment of the present application;
[0054] Fig.11 This is a functional structure diagram of the task backtracking system in the embodiment of the present application;
[0055] Fig.12 Schematic diagram of the hardware structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solution and beneficial effects of this application clearer, the technical solution of this application will be clearly and completely described below in combination with the embodiments and drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0057] The present application embodiment provides a method for constructing a backtracking instance bloodline graph, which can be applied to electronic devices such as computers and servers. Figure 1 As shown, the general process of the method for constructing a backtracking instance bloodline graph provided in the embodiment of the present application is as follows:
[0058] Step 101: Determine each backtracking instance of a target backtracking task within a data backtracking time period and the data production time of each backtracking instance corresponding to the target backtracking task.
[0059] In an embodiment of the present application, a user can set a target backtracking task, an instance backtracking time period, and a data backtracking time period according to actual needs; the target backtracking task is a task that meets the backtracking conditions determined from various tasks corresponding to the target job, wherein the backtracking conditions may be, but are not limited to, data output anomalies occurring during task execution, analysis or mining of historical data output by the task, etc.; the instance backtracking time period is used to determine the time interval for backtracking instances, and instances that are not in the instance backtracking time period are not backtracked; the data backtracking time period is used to determine the time interval for backtracking data, and data that is not in the data backtracking time period is not backtracked.
[0060] A task instance refers to the execution entity of a specific task at a certain point in time. For example, a daily data processing task will generate an operation instance for that day. Usually, during the execution of each task contained in the target job, the instance operation data of each task instance will be generated and stored in the database as historical operation data. The instance operation data of each task instance includes at least the instance identifier of the task instance, instance operation data, data output time (that is, instance operation time), etc. Based on this, after the user sets the target backtracking task, instance backtracking time period, and data backtracking time period, the target backtracking task, instance backtracking time period, and data backtracking time period set by the user can be used to obtain each backtracking instance of the target backtracking task within the data backtracking time period from the database, and determine the data output time of each backtracking instance corresponding to the target backtracking task.
[0061] Step 102: Based on the data output time of each backtracking instance corresponding to the target backtracking task, and in accordance with the job dependency between each task in the target job to which the target backtracking task belongs, a backtracking instance lineage graph between each target backtracking instance is constructed; wherein the job dependency includes the data dependency and the data dependency time; the target backtracking instance includes each backtracking instance corresponding to the target backtracking task and each backtracking instance of the associated backtracking task corresponding to the target backtracking task in the target job within the instance backtracking time period; the associated backtracking task is a task in the target job that has a direct or indirect chain reaction based on the data output of each backtracking instance corresponding to the target backtracking task.
[0062] In the embodiment of the present application, based on the data output time of each backtracking instance corresponding to the target backtracking task, according to the job dependency relationship between each task in the target job to which the target backtracking task belongs, a backtracking instance lineage graph can be obtained by cyclically executing the dependency link construction operation; wherein, the backtracking instance nodes in the backtracking instance lineage graph are represented by task identifiers and data output time; the lineage relationship between each backtracking instance node in the backtracking instance lineage graph is represented by directed arrows; specifically, the backtracking instance lineage graph can be constructed in the following manners but is not limited to:
[0063] Step 1021: Determine the current upstream instance node; wherein, when the dependency link building operation is executed for the first time, the current upstream instance node is the root node; when the dependency link building operation is not executed for the first time, the current upstream instance node is the target downstream instance node determined when the dependency link building operation was last executed; the root node is the backtracing instance for which there is no job dependency in each backtracing instance corresponding to the target backtracing task.
[0064] For example, suppose the target backtracking task is a day-level A task. The day-level A task includes backtracking instance a within the data backtracking period (for example, 3 days). 1 , Backtracking Example a 2 and backtracking instance a 3 ; Backtracking instance a 1 There is no dependency, output a 1(T-1) Data; Backtracking Example a 2 Backtracking instance a that depends on the day-level A task 1 The output of a 1(T-1) data, and output a 2(T-2) Data; Backtracking Example a 3 Backtracking instance a that depends on the day-level A task 2 The output of a 2(T-2) data, and output a 3(T-3) Data, the root node is the backtracking instance a 1 .
[0065] Step 1022: Based on the data dependency relationship between the tasks in the target job, determine the backtracking instance that depends on the current upstream instance node as the initial downstream instance node.
[0066] For example, suppose the current upstream instance node is the backtracking instance a of task A at the day level (timing period is "day"). 1 , which produces a 1(T-1) Data; backtracking instance a of the day-level A task in each task of the target job 2 Backtracking instance a that depends on the day-level A task 1 The output of a 1(T-1) Data; then according to the data dependency relationship, the initial downstream instance node of the current upstream instance node is determined to be the backtracking instance a of the day-level A task 2 .
[0067] For example, suppose the current upstream instance node is the backtracking instance a of the day-level A task 3 , which produces a 3(T-3) Data; backtracking instance b of the day-level B task in each task of the target job 1 Backtracking instance a that depends on the day-level A task 3 The output of a 3(T-3)Data; then according to the data dependency relationship, the initial downstream instance node of the current upstream instance node is determined to be the backtracking instance b of the day-level B task 1 .
[0068] Step 1023: Based on the data dependency time of the initial downstream instance node on the current upstream instance node and the timing cycle size relationship between the data production time of the current upstream instance node, determine the target downstream instance node of the current upstream instance node with the initial downstream instance node as the benchmark; and construct a dependency link from the current upstream instance node to the target downstream instance node.
[0069] In the embodiment of the present application, based on the data dependency time of the initial downstream instance node on the current upstream instance node and the timing cycle size relationship between the data production time of the current upstream instance node, when the target downstream instance node of the current upstream instance node is determined based on the initial downstream instance node, there may be but are not limited to the following two situations:
[0070] The first case: the data dependency time of the initial downstream instance node on the current upstream instance node is the same as the timing cycle between the data production time of the current upstream instance node.
[0071] In this case, the initial downstream instance node may be determined as the target downstream instance node of the current upstream instance node.
[0072] For example, suppose that the day-level A task is the root node, there is no job dependency, and the output is a (T-1) Data; Day-level B tasks depend on a (T-1) Data; day-level C task dependency b (T-1) Data; the time period of the day-level A task, day-level B task and day-level C task is the same, all of which are "days"; the data backtracking period is 2023-01-02; the specific dependency relationship is as follows Figure 2As shown, first, the backtracking instance a_20230102 of the day-level A task is constructed. Due to the data change (that is, data output) of the backtracking instance a_20230102 of the day-level A task, according to the job dependency, the target downstream instance node of the backtracking instance a_20230102 of the day-level A task is determined to be the backtracking instance b_20231102 of the day-level B task, and the backtracking instance b_20231102 from the backtracking instance a_20230102 of the day-level A task to the day-level B task is constructed. 02; further, due to the data change of the backtracking instance b_20231102 of the day-level B task, according to the job dependency, it is determined that the target downstream instance node of the backtracking instance b_20231102 of the day-level B task is the backtracking instance c_20230102 of the day-level C task, and a dependency link is constructed from the backtracking instance b_20231102 of the day-level B task to the backtracking instance c_20230102 of the day-level C task. This is the simplest backtracking instance lineage graph.
[0073] It is worth mentioning that in the embodiment of the present application, if it is determined that the data dependency time of the initial downstream instance node on the current upstream instance node is the same as the timing period between the data production time of the current upstream instance node, then it also includes:
[0074] When it is detected that the data dependency time of the initial downstream instance node on the current upstream instance node is greater than the data production time of the current upstream instance node, it is determined that the upstream instance node of the initial downstream instance node is missing in the current upstream instance node;
[0075] Based on the time difference between the data dependency time of the initial downstream instance node on the current upstream instance node and the data production time of the current upstream instance node, the upstream instance node of the missing initial downstream instance node is completed to obtain the subsequent upstream instance node;
[0076] Build a dependency link from the backup upstream instance node to the target downstream instance node.
[0077] For example, suppose that the day-level A task is the root node, there is no job dependency, and the output is a (T-1) Data; the day-level B task depends on the day-level A task a (T-1) Data, output (T-1) Data; Day-level D tasks depend on the day-level A tasks (T-1) Data, output (T-1) Data; the day-level C task depends on the b of the day-level B task (T-1,T-3) Data and day-level D tasks (T-1) Data; the time period of the day-level A task, day-level B task, day-level C task and day-level D task is the same, all of which are "days"; the data backtracking period is 2023-01-03; the specific dependency relationship is as follows Figure 3 As shown, first create the backtracking instance a_20230103 of the day-level A task. Due to the data change (that is, data output) of the backtracking instance a_20230103 of the day-level A task, according to the job dependency, determine that the target downstream instance node of the backtracking instance a_20230103 of the day-level A task is the backtracking instance b_20231103 of the day-level B task and the backtracking instance d_20231103 of the day-level D task, and construct the backtracking instance a_20230103 from the day-level A task. 230103 to the backtracking instance b_20231103 of the day-level B task and the dependent link from the backtracking instance a_20230103 of the day-level A task to the backtracking instance d_20231103 of the day-level D task; further, due to the data changes of the backtracking instance b_20231103 of the day-level B task and the backtracking instance d_20231103 of the day-level D task, according to the job dependency, the backtracking instance b_20231103 of the day-level B task and the backtracking instance d_20231103 of the day-level D task are determined. The target downstream instance nodes of the backtracking instance d_20231103 of the D task are all the backtracking instance c_20230103 of the day-level C task. A dependency link is constructed from the backtracking instance b_20231103 of the day-level B task to the backtracking instance c_20230103 of the day-level C task, and from the backtracking instance d_20231103 of the day-level D task to the backtracking instance c_20230103 of the day-level C task. At the same time, the backtracking instance c_20231103 of the day-level C task is The missing upstream instance nodes of 230103 are completed, that is, the backtracing instance b_20231102 of the day-level B task and the backtracing instance b_20231101 of the day-level B task are completed, and the dependency link from the backtracing instance b_20231102 of the day-level B task to the backtracing instance c_20230103 of the day-level C task and the dependency link from the backtracing instance b_20231101 of the day-level B task to the backtracing instance c_20230103 of the day-level C task are constructed.
[0078] In the above example, the day-level C task depends on the b of the day-level B task. (T-1,T-3) The data shows that the daily C task depends on the b produced by the daily B task on day T-1. (T-1) Data, b produced on day T-2 (T-2) Data and b produced on T-3 day (T-3) data, that is, there is a multi-day dependency problem, and according to the a of the day-level A task (T-1) Data, according to the job dependency, can only generate the b of the day-level B task (T-1) Data, b of the day-level B task (T-2) Data and b (T-3) Data is missing, in this case, you can add the missing day-level B task b (T-2) Data and b (T-3)The data is supplemented and the b of the day-level B task is constructed (T-2) Data to day-level C task c (T-1) Data dependency links and b from day-level B tasks (T-3) Data to day-level C task c (T-1) The data dependency link can effectively avoid the b (T-2) Data and b (T-3) The data does not exist, resulting in the failure of the backtracking instance of the day-level C task to be ready, affecting the backtracking progress and backtracking effect of the target backtracking task. It is worth mentioning that for the upstream instance node of the missing initial downstream instance node, in addition to automatic completion, the user can also be reminded of the missing backtracking instance and manually complete it.
[0079] The second situation: the timing cycle between the data dependency time of the initial downstream instance node on the current upstream instance node and the data production time of the current upstream instance node is different.
[0080] In this case, if it is determined that the data dependency time of the initial downstream instance node on the current upstream instance node is a small cycle, and the data output time of the current upstream instance node is a large cycle, the initial downstream instance node is expanded to obtain the target downstream instance node of the current upstream instance node; if it is determined that the data dependency time of the initial downstream instance node on the current upstream instance node is a large cycle, and the data output time of the current upstream instance node is a small cycle, the initial downstream instance node is contracted to obtain the target downstream instance node of the current upstream instance node.
[0081] For example, suppose that the weekly (timing period is "week") task A is the root node, there is no job dependency, and the output is a (T-1) Data; day-level (timing period is "day") B task depends on the weekly A task a (T-1) Data, output (T-1) Data; hour-level (timing period is "hour") C task depends on day-level B task b (T-1) Data, output c (T-1) Data; the data backtracking period is the first week of January 2023, that is, 2023-01-02 to 2023-01-08; since the timing cycles of weekly A tasks, daily B tasks, and hourly C tasks are different, there is a cross-cycle dependency problem, and cross-cycle dependency will inevitably cause changes in the number of downstream instance nodes. The specific dependency relationship is as follows Figure 4As shown, first construct the backtracking instance A(week)_a_202301 of the weekly task A. Due to the data changes (i.e., data output) of the backtracking instance A(week)_a_202301 of the weekly task A, according to the job dependency, determine that the target downstream instance node of the backtracking instance A(week)_a_202301 of the weekly task A is the task instance corresponding to each day of the daily task B in the first week of January 2023, that is, the daily task B corresponds to 7 backtracking instances: backtracking instance B(Monday)_b_20230102, backtracking instance B(Tuesday)_b_20230103, backtracking instance B(Wednesday)_b_20230104...backtracking instance B(Sunday)_b_20230108; similarly, due to the daily task B The data changes (that is, data output) of the 7 backtracking instances are determined according to the job dependency relationship. The target downstream instance node of each backtracking instance of the day-level B task is determined to be the task instance corresponding to each hour of the hour-level C task on the day to which the backtracking instance belongs. That is, the hour-level C task corresponds to 24 backtracking instances for each backtracking instance of the day-level B task. Taking the backtracking instance B(Monday)_b_20230102 of the day-level B task as an example, its downstream backtracking instances include the backtracking instance C(00 o'clock)_b_2023010200, backtracking instance C(01 o'clock)_b_2023010201, backtracking instance C(02 o'clock)_b_2023010202...backtracking instance C(23 o'clock)_b_2023010223 of the hour-level C task.
[0082] In an embodiment of the present application, if the downstream instance node is a small cycle (for example, days) and depends on the large cycle of the upstream instance node (for example, weeks), the number of downstream instance nodes is expanded; similarly, if the downstream instance node is a large cycle (for example, weeks) and depends on the small cycle of the upstream instance node (for example, days), the number of downstream instance nodes is shrunk; wherein, the expansion or contraction number of the downstream instance nodes and the timing cycle of the instance node last week have a mapping relationship as shown in Table 1; in Table 1, the first row represents the timing cycle of the upstream instance node, and the first column represents the timing cycle of the downstream instance node, and the number of the intersection of the two is the number of instance nodes generated theoretically (factors such as the super time domain and the downstream task scheduling time are not considered here).
[0083] Table 1.
[0084]
[0085] In an embodiment of the present application, the multi-day dependency problem can be solved by completing the missing upstream instance nodes, and the cross-cycle dependency problem can be solved by expanding or shrinking the downstream instance nodes, thereby improving the comprehensiveness and accuracy of the backtracking instance lineage graph. When solving the multi-day dependency problem and the cross-cycle dependency problem, the problem of the downstream instance node exceeding the time domain may occur in the process of inferring the downstream instance node through the upstream instance node. For example, the downstream instance node is a task instance that will be produced in the future, which obviously does not meet the task backtracking requirements and expectations. Based on this, in order to solve the super-time domain problem, in the process of inferring the downstream instance node through the upstream instance node, the end time of the data backtracking time period (that is, the current time to initiate the backtracking) is used as the end time, and the initiation time of any downstream instance node shall not exceed the end time, and the downstream instance node of the super-time domain is trimmed to control the downstream instance node within the instance backtracking time period.
[0086] It is worth mentioning that in order to solve the cross-instance node problem, after building the dependency link from the current upstream instance node to the target downstream instance node, it also includes:
[0087] According to the task identifier and data output time of each backtracking instance node on each dependent link, when the same backtracking instance node is detected, the same backtracking instance node is merged into a cross-instance node. That is, each backtracking instance node in the backtracking instance lineage graph is provided with a unique instance identifier (for example, the unique instance identifier is represented by the task identifier and the data output time). In the process of constructing the backtracking instance lineage graph, the cross-instance node problem can be solved by merging the backtracking instance nodes with the same instance identifier, thereby improving the accuracy of the backtracking instance lineage graph.
[0088] For example, suppose that the day-level A task is the root node, there is no job dependency, and the output is a (T-1) Data; Day-level B tasks depend on a (T-1) Data, output (T-1) Data; Day-level D tasks depend on a (T-1) Data, output (T-1) Data; day-level C task dependency b (T-1) Data and d (T-1) Data, output c (T-1) Data; the time period of the day-level A task, day-level B task, day-level C task and day-level D task is the same, all of which are "days"; the data backtracking period is 2023-01-02; the specific dependency relationship is as follows Figure 5As shown, through recursive operation, two dependency links are generated, namely, the backtracing instance a_20230102 of the day-level A task -> the backtracing instance b_20230102 of the day-level B task -> the backtracing instance c_20230102 of the day-level C task and the backtracing instance a_20230102 of the day-level A task -> the backtracing instance d_20230102 of the day-level D task -> the backtracing instance c_20230102 of the day-level C task. According to the task identifier and data output time of each backtracing instance node on the two dependency links, it is detected that there are two identical backtracing instances a_20230102 of the day-level A task and two identical backtracing instances c_20230102 of the day-level C task. The two identical backtracing instances a_20230102 of the day-level A task are merged into a cross-instance node and the two identical backtracing instances c_20230102 of the day-level C task are merged into one cross-instance node.
[0089] In addition, in the embodiment of the present application, after building a dependency link from the current upstream instance node to the target downstream instance node, it also includes:
[0090] When it is detected that the dependency link from the current upstream instance node to the target downstream instance node is a circular dependency link, the dependency link from the current upstream instance node to the target downstream instance node is deleted.
[0091] For example, suppose that the backtracking instance a_20230101 of the day-level A task depends on the data produced by the backtracking instance c_20230101 of the day-level C task; the backtracking instance b_20230101 of the day-level B task depends on the data produced by the backtracking instance a_20230101 of the day-level A task; the backtracking instance c_20230101 of the day-level C task depends on the data produced by the backtracking instance b_20230101 of the day-level B task; the timing periods of the day-level A task, the day-level B task and the day-level C task are the same, all "days"; then the specific dependency relationship is as follows Figure 6As shown in the figure, a circular dependency is formed between the backtracking instance a_20230101 of the day-level A task, the backtracking instance b_20230101 of the day-level B task, and the backtracking instance c_20230101 of the dependent day-level C task. In this case, since the entire target backtracking task lacks an initial condition, the target backtracking task cannot be backtracked. In order to solve the problem of circular dependency, the backtracking instance a_20230101 from the day-level C task to the day-level A task can be detected. 30101 is a circular dependency link. When it is detected that the dependency link between the backtracking instance c_20230101 of the day-level C task and the backtracking instance a_20230101 of the day-level A task is a circular dependency link, the dependency link from the backtracking instance c_20230101 of the day-level C task to the backtracking instance a_20230101 of the day-level A task is deleted, thereby solving the circular dependency problem by constructing a directed acyclic graph (DAG) between each backtracking instance node. Specifically, when a circular dependency link is detected, the circular dependency link can be automatically deleted, or the user can be notified so that the user can better optimize the target backtracking task and the target job to which the target backtracking task belongs, thereby solving the circular dependency problem caused by user misconfiguration.
[0092] Step 1024: Determine whether the instance backtracking end condition is met; wherein, the instance backtracking end condition is that there is no task in the target job that produces a direct or indirect chain reaction based on the data output of each backtracking instance corresponding to the target backtracking task; if so, execute step 1025; if not, return to step 1021.
[0093] Step 1025: Generate a backtracking instance lineage graph based on the dependency link from the current upstream instance node to the target downstream instance node constructed each time the dependency link construction operation shown in steps 1021 to 1023 is executed.
[0094] In the embodiment of the present application, after the backtracking instance lineage graph is constructed through the above steps 1021-1025, in actual applications, the backtracking requirements are constantly changing. Although the backtracking instance lineage graph has been generated and the core backtracking requirements can be solved, there are often some usage scenarios that cannot be covered. For example, the upstream instance node has been corrected by other means, and now only the downstream instance node needs to be backtracked. Therefore, the user hopes to remove the root node from the backtracking instance lineage graph, or remove certain backtracking instance nodes from the backtracking instance lineage graph, etc., and only the remaining backtracking instance nodes are backtracked. At this time, the instance node filtering can be implemented through the filtering function, that is, according to the instance identifier to be filtered and the instance filtering type in the instance filtering instruction, the corresponding backtracking instance node and its corresponding dependent link are deleted from the backtracking instance lineage graph, that is, the user determines whether to filter out certain backtracking instance nodes, whether to cascade filter, etc. If the user chooses cascade filtering, the target backtracking instance node and its downstream instance node are cascade filtered, so that the special node filtering problem can be solved through the instance filtering function, and then the actual backtracking requirements can be further adapted.
[0095] Based on the same inventive concept, the present application embodiment provides a task backtracking method, which can be applied to electronic devices such as computers and servers, see Figure 7 As shown, the overview process of the task backtracking method provided in the embodiment of the present application is as follows:
[0096] Step 701: Obtain task backtracking information; wherein the task backtracking information at least includes a target backtracking task, a data backtracking time period, and an instance backtracking time period.
[0097] Step 702: Based on the task backtracking information, the backtracking instance lineage graph of the target backtracking task is constructed using the backtracking instance lineage graph construction method described above.
[0098] Step 703: according to the backtracking priority and scheduling initiation time of the target backtracking task, add the target backtracking task as a task to be backtracked into a queue of tasks to be backtracked.
[0099] Step 704: When it is determined that the number of tasks to be backtraced in the task queue to be backtraced reaches the queue capacity of the task queue to be backtraced, batch backtraces are performed on each task to be backtraced in the task queue according to the arrangement order of each task to be backtraced in the task queue to be backtraced and based on the backtracing instance lineage graph of each task to be backtraced.
[0100] In specific implementation, according to the arrangement order of each task to be backtracked in the task queue to be backtracked, based on the backtracking instance lineage graph of each task to be backtracked, when batch backtracking each task to be backtracked in the task queue to be backtracked, the following methods may be used but are not limited to:
[0101] First, according to the arrangement order of each backtracking task in the backtracking task queue, each backtracking task in the backtracking task queue is determined as the current backtracking task in sequence.
[0102] Then, based on the initiation time of the upstream instance node in the current backtracking task, the upstream instance output cycle offset, and the downstream instance dependency cycle offset, the initiation time of the downstream instance node is determined. Specifically, the initiation time of the downstream instance node can be calculated using the following formula 2:
[0103] Upstream instance launch time (upstream_launch_time) + upstream task output cycle offset (upstream_out_offset) = downstream instance launch time (downstream_launch_time) + downstream task dependency cycle offset (downstream_dependency_offset) ... Formula 1
[0104] By converting the above formula 1, the formula for calculating the initiation time of the downstream instance node can be obtained as follows:
[0105] Downstream instance launch time (downstream_launch_time) = upstream instance launch time (upstream_launch_time) + upstream task output cycle offset (upstream_out_offset) - downstream task dependency cycle offset (downstream_dependency_offset) ... Formula 2
[0106] The above formula 2 can be used to calculate the initiation time of the downstream instance node by using the initiation time of the upstream instance node, the upstream instance output cycle offset and the downstream instance dependency cycle offset.
[0107] Finally, the current backtracking task is backtracked according to the initiation time of each instance node in the current backtracking task.
[0108] In the embodiment of the present application, according to the arrangement order of each task to be backtracked in the task queue to be backtracked, based on the backtracking instance lineage graph of each task to be backtracked, the process of batch backtracking each task to be backtracked in the task queue to be backtracked also includes:
[0109] When it is determined that the number of backtracking tasks in the backtracking task queue reaches an alarm threshold, and / or when it is determined that there are backtracking instances in which backtracking fails or backtracking is terminated in each backtracking task, an alarm message is initiated.
[0110] It is worth mentioning that after determining that there are backtracing instances that have failed or terminated backtracing in each backtracing task, the correction status of the backtracing instances that have failed or terminated backtracing can also be monitored. When it is determined that the backtracing instances that have failed or terminated backtracing in the backtracing task have been corrected, it is determined that the backtracing instances that have failed or terminated backtracing meet the re-running conditions. At this time, the backtracing tasks are continued starting with the backtracing instances that have failed or terminated backtracing in the backtracing task.
[0111] In the task backtracking method provided in the embodiment of the present application, refer to Figure 8 As shown in the figure, the target backtracking task initiation process includes:
[0112] 1) Fill in the task backtracking information: (1) Instance backtracking time period: used to control the time interval for submitting backtracking instances. Backtracking instances that exceed this interval will not be submitted; (2) Data backtracking period: used to control the time interval of data to be backtracked; (3) Backtracking reason: optional information, used for the approval of the target backtracking task, optional information; (4) Root node: used to determine the target backtracking task; Leaf node: optional information, used to determine other backtracking tasks except the target backtracking task. Among them, when the root node does not have a leaf node, it is a link backtracking; when the root node has a leaf node, it is a cascade backtracking; when the root node and the leaf node are the same, it is a single task backtracking; when the root node and the leaf node are not related, it is a failed task backtracking.
[0113] 2) Obtain the backtracking instance list in real time based on the task backtracking information: Based on the task hydrogen information in step 1, dynamically generate the backtracking instance list for providing to the user for secondary confirmation.
[0114] 3) Custom tailoring instances: supports custom tailoring based on multiple combinations such as task level and / or instance level, project group, task type and / or instance type, task owner and / or instance owner, etc.
[0115] 4) Generate a backtracking instance lineage graph and perform topological sorting: Perform topological sorting on the backtracking instance lineage graph to form an instance topological graph.
[0116] 5) Estimated backtracking completion time: The instance running time of the backtracking instance node (for example, the time taken to run P95) can be obtained from the historical running data (except for special instances). Based on the instance running time of each backtracking instance node in the backtracking instance lineage graph obtained from the historical running data, the corresponding backtracking instance node in the backtracking instance lineage graph is marked with a running time label. The longest time-consuming path of the backtracking instance lineage graph is obtained through path calculation, and the sum of the instance running times of each backtracking instance node on the longest time-consuming path is determined as the backtracking completion time of the target backtracking task.
[0117] 6) Automated operations: One-click batch backtracking, one-click intermittent rerunning, one-click termination, one-click pause, one-click resolution of conflicting instances, and other automated operations.
[0118] 7) Control strategy of the task queue to be backtracked: submit tasks gradually according to the queue capacity to ensure the stability of the scheduler, and optimize the task submission strategy in combination with task concurrency.
[0119] 8) Failed backtracking alarm: For the backtracking instances that failed during the backtracking process, the user is notified through system messages, SMS, public account messages, etc. After the user repairs the backtracking instance that failed, by clicking on intermittent rerun, the backtracking task will continue to be backtracked starting from the backtracking instance that failed. There is no need to re-initiate a new run of the backtracking task, which can save backtracking time and improve backtracking efficiency.
[0120] 9) Backtracking ends: When the entire backtracking task is completed, the backtracking status of the backtracking task is marked.
[0121] In the task backtracking method provided in the embodiment of the present application, refer to Fig. 9 As shown, the backtracking instance scheduling process includes:
[0122] 1) Construct a backtracking instance lineage graph and perform topological sorting on the backtracking instance lineage graph to obtain an instance topology graph;
[0123] 2) Store the instance topology map in the TiDB database and wait for backtracking scheduling.
[0124] 3) Backtracking task scheduling strategy: scheduling priority level > scheduling initiation time; Among them, supplementary scheduling strategy: control the number of single task backtracking submissions in combination with task concurrency to prevent too many instances of the same task from being submitted at one time, resulting in abnormal occupation submission exceeding task concurrency.
[0125] 4) The backtracking instances in the same backtracking task are submitted to the queue of tasks to be backtracked in sequence.
[0126] 5) Check the capacity of the queue of tasks to be backtracked: the maximum number of instances allowed to be scheduled. When the capacity of the queue of tasks to be backtracked is exceeded, submission will be suspended to wait for the capacity to be released.
[0127] 6) Check whether there is a conflict in the backtracking instance, that is, check whether the backtracking instance has been running in other backtracking tasks. If a conflict is found, notify the user to resolve the conflict.
[0128] 7) Submit the backtracking instance to the scheduler queue and wait for scheduling.
[0129] 8) Determine whether the backtracking instance dependencies are ready.
[0130] 9) Initiate heap sorting for the backtracking instances in the same backtracking task at the same time according to the task priority.
[0131] 10) Submit to the dispatcher execution queue for backtracking.
[0132] In the task backtracking method provided in the embodiment of the present application, refer to Fig.10 As shown in Figure 1, the estimation process of the backtracking completion time of the target backtracking task includes:
[0133] Periodically (for example, every day), the average instance running time (for example, P95 running time) of each instance in the most recent set time period (for example, within 30 days) is calculated according to the job granularity through the ETL task. After excluding special time-consuming data, the instance running data is synchronized to the TiDB database through the data synchronization job. In the backtracking instance lineage graph, the running time label is marked for each backtracking instance node based on the average instance running time of each backtracking instance node in the set time period. The longest time-consuming path in the backtracking instance lineage graph is the backtracking completion time of the backtracking task.
[0134] In the task backtracking method provided in the embodiment of the present application, the automated backtracking function includes:
[0135] 4.1. Intermittent re-run function
[0136] In actual applications, we often encounter backtracking instances that fail to backtrack. For the backtracking instances that fail to backtrack, we prompt the user to make corrections and monitor the correction status of the backtracking instances that fail to backtrack. After confirming that the backtracking instance that fails to backtrack has been manually repaired by the user, we can use one-click to pull up the intermittent rerun function, starting from the backtracking instance that fails to backtrack, and continue to backtrack the backtracking task, that is, submit the backtracking instance with a failed running status in the backtracking task to the scheduler again for backtracking. Among them, the intermittent rerun function is also applicable to the backtracking instance that has terminated the backtracking.
[0137] 4.2. Instance conflict handling function
[0138] During the backtracking process, when it is determined that the backtracking instance has been backtracked in other backtracking tasks, the backtracking of the backtracking instance is suspended, and an instance conflict notification is initiated, wherein the conflict notification includes the backtracking instances of each conflicting backtracking task, and the backtracking instance to be continued is determined to be backtracked according to the user's selection operation in the backtracking instances of each conflicting backtracking task. When there are many conflicting backtracking instances, a list of conflicting backtracking instances can be provided to the user so that the user can solve the conflict problem through batch operations, reducing the time for the user to deal with the conflict problem. That is, periodically check whether there is a conflict in each backtracking instance corresponding to the backtracking task. When a conflict is detected in each backtracking instance corresponding to the backtracking task, obtain and display the list of conflicting instances, and the user can solve the conflicting instance with one click.
[0139] In the task backtracking method provided in the embodiment of the present application, the monitoring alarm function includes:
[0140] 5.1. Monitoring and alarm of the task queue to be backtracked: Implement the alarm strategy by configuring the capacity threshold.
[0141] 5.2. Backtracking task status alarm: Real-time monitoring of the backtracking status of the backtracking instance in the backtracking task. If the backtracking status is detected to be failed or terminated, an alarm message is sent to the backtracking initiator, who will then resolve the backtracking instance that failed or terminated.
[0142] Based on the above embodiments, the present application provides a task backtracking system. Fig.11 As shown, the task backtracking system 100 provided in the embodiment of the present application at least includes:
[0143] The information configuration subsystem 11 is used to obtain task backtracking information; wherein the task backtracking information at least includes a target backtracking task, a data backtracking time period, and an instance backtracking time period;
[0144] A lineage graph construction subsystem 12 is used to construct a backtracking instance lineage graph of a target backtracking task based on the task backtracking information and using the backtracking instance lineage graph construction method described above;
[0145] The task backtracking subsystem 13 is used to add the target backtracking task as a task to be backtracked to the task queue to be backtracked according to the backtracking priority and scheduling initiation time of the target backtracking task; when it is determined that the number of tasks to be backtracked in the task queue to be backtracked reaches the queue capacity of the task queue to be backtracked, batch backtracking is performed on each task to be backtracked in the task queue to be backtracked according to the arrangement order of each task to be backtracked in the task queue to be backtracked and based on the backtracking instance lineage graph of each task to be backtracked.
[0146] It should be noted that the principle of solving technical problems by the task backtracing system 100 provided in the embodiment of the present application is similar to the task backtracing method provided in the embodiment of the present application. Therefore, the implementation of the task backtracing system 100 provided in the embodiment of the present application can refer to the implementation of the task backtracing method provided in the embodiment of the present application, and the repeated parts will not be repeated.
[0147] After introducing the backtracking instance bloodline graph construction method, task backtracking method and system provided in the embodiments of the present application, next, a brief introduction to the electronic device provided in the embodiments of the present application is given.
[0148] See also Fig.12As shown, the electronic device 200 provided in the embodiment of the present application includes at least a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the backtracking instance bloodline graph construction method and / or the task backtracking method provided in the embodiment of the present application are implemented.
[0149] The electronic device 200 provided in the embodiment of the present application may further include a bus 23 connecting different components (including the processor 21 and the memory 22). The bus 23 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.
[0150] The memory 22 may include a readable storage medium in the form of a volatile memory, such as a random access memory (RAM) 221 and / or a cache memory 222, and may further include a read-only memory (ROM) 223. The memory 22 may also include a program tool 225 having a set (at least one) of program modules 224, the program modules 224 including but not limited to an operating subsystem, one or more application programs, other program modules, and program data, each of which or some combination may include the implementation of a network environment.
[0151] The processor 21 may be a processing element or a collective term for multiple processing elements. For example, the processor 21 may be a central processing unit (CPU), or one or more integrated circuits configured to implement the backtracking instance lineage graph construction method and / or task backtracking method provided in the embodiment of the present application. Specifically, the processor 21 may be a general-purpose processor, including but not limited to a CPU, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.
[0152] The electronic device 200 can communicate with one or more external devices 24 (e.g., keyboards, remote controls, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 200 (e.g., mobile phones, computers, etc.), and / or, communicate with devices that enable the electronic device 200 to communicate with one or more other electronic devices 200 (e.g., routers, modems, etc.). Such communication can be performed through an input / output (I / O) interface 25. In addition, the electronic device 200 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 26. Fig.12 As shown, the network adapter 26 communicates with other modules of the electronic device 200 via the bus 23. It should be understood that although Fig.12 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 200, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.
[0153] It should be noted that Fig.12 The electronic device 200 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0154] The computer-readable storage medium provided in the embodiment of the present application is introduced below. The computer-readable storage medium provided in the embodiment of the present application stores computer instructions, and when the computer instructions are executed by the processor, the backtracking instance bloodline graph construction method and / or the task backtracking method provided in the embodiment of the present application are implemented. Specifically, the computer instructions can be built-in or installed in the processor, so that the processor can implement the backtracking instance bloodline graph construction method and / or the task backtracking method provided in the embodiment of the present application by executing the built-in or installed computer instructions.
[0155] In addition, the backtracing instance lineage graph construction method and / or task backtracing method provided in the embodiments of the present application can also be implemented as a computer program product, which includes program code, and when the program code is run on a processor, it implements the backtracing instance lineage graph construction method and / or task backtracing method provided in the embodiments of the present application.
[0156] The computer program product provided in the embodiments of the present application may adopt one or more computer-readable storage media, and the computer-readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any suitable combination of the above. Specifically, more specific examples of computer-readable storage media (a non-exhaustive list) include an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
[0157] The computer program product provided in the embodiment of the present application may adopt a CD-ROM and include program code, and may also be run on electronic devices such as computers and servers. However, the computer program product provided in the embodiment of the present application is not limited thereto. In the embodiment of the present application, the computer-readable storage medium may be any tangible medium containing or storing program code, and the program code may be used by or in combination with an instruction execution system, apparatus, or device.
[0158] It should be noted that the "and / or" mentioned in this application describes the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.
[0159] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.
[0160] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0161] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0162] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A method for constructing a backtracking instance bloodline graph, characterized in that: include: Determine each backtracking instance of the target backtracking task within the data backtracking time period and the data output time of each backtracking instance corresponding to the target backtracking task; Based on the data output time of each backtracking instance corresponding to the target backtracking task, and in accordance with the job dependency relationship between each task in the target job to which the target backtracking task belongs, a backtracking instance lineage graph between each target backtracking instance is constructed; wherein the job dependency relationship includes data dependency relationship and data dependency time; the target backtracking instance includes each backtracking instance corresponding to the target backtracking task and each backtracking instance of the associated backtracking task corresponding to the target backtracking task in the target job within the instance backtracking time period; the associated backtracking task is each task in the target job that has a direct or indirect chain reaction based on the data output of each backtracking instance corresponding to the target backtracking task; Among them, based on the data output time of each backtracking instance corresponding to the target backtracking task, according to the job dependency relationship between each task in the target job to which the target backtracking task belongs, a backtracking instance lineage graph between each target backtracking instance is constructed, including: Based on the data output time of each backtracking instance corresponding to the target backtracking task, according to the job dependency relationship between each task in the target job to which the target backtracking task belongs, the dependency link construction operation is cyclically executed until it is determined that the instance backtracking end condition is met, and based on the dependency link between the current upstream instance node and the target downstream instance node constructed each time the dependency link construction operation is executed, the backtracking instance lineage graph is generated; wherein the dependency link construction operation includes: Determine the current upstream instance node; wherein, when the dependency link building operation is executed for the first time, the current upstream instance node is the root node, and when the dependency link building operation is not executed for the first time, the current upstream instance node is the target downstream instance node determined when the dependency link building operation was last executed; the root node is the backtracking instance for which there is no job dependency relationship among the backtracking instances corresponding to the target backtracking task; Based on the data dependency relationship between the tasks in the target job, determine the backtracking instance that depends on the current upstream instance node as the initial downstream instance node; Based on the relationship between the data dependency time of the initial downstream instance node on the current upstream instance node and the data production time of the current upstream instance node, determine the target downstream instance node of the current upstream instance node with the initial downstream instance node as a reference; A dependency link is constructed from the current upstream instance node to the target downstream instance node.
2. The method for constructing a backtracking instance bloodline graph according to claim 1, characterized in that: Based on the data dependency time of the initial downstream instance node on the current upstream instance node and the timing cycle size relationship between the data production time of the current upstream instance node, determining the target downstream instance node of the current upstream instance node with the initial downstream instance node as a reference includes: If it is determined that the data dependency time of the initial downstream instance node on the current upstream instance node is the same as the timing period between the data production time of the current upstream instance node, the initial downstream instance node is determined as the target downstream instance node of the current upstream instance node; If it is determined that the data dependency time of the initial downstream instance node on the current upstream instance node is a small period, and the data production time of the current upstream instance node is a large period, the initial downstream instance node is expanded to obtain the target downstream instance node of the current upstream instance node; If it is determined that the data dependency time of the initial downstream instance node on the current upstream instance node is a large cycle and the data production time of the current upstream instance node is a small cycle, the initial downstream instance node is shrunk to obtain the target downstream instance node of the current upstream instance node.
3. The method for constructing a backtracking instance bloodline graph according to claim 2, characterized in that: If it is determined that the data dependency time of the initial downstream instance node on the current upstream instance node is the same as the timing period between the data production time of the current upstream instance node, the method further includes: When it is detected that the data dependency time of the initial downstream instance node on the current upstream instance node is greater than the data production time of the current upstream instance node, it is determined that the upstream instance node of the initial downstream instance node is missing in the current upstream instance node; Based on the time difference between the data dependency time of the initial downstream instance node on the current upstream instance node and the data production time of the current upstream instance node, the missing upstream instance node of the initial downstream instance node is supplemented to obtain a supplementary upstream instance node; A dependency link is constructed from the backup upstream instance node to the target downstream instance node.
4. The method for constructing a retrospective instance bloodline graph according to any one of claims 1 to 3, characterized in that: When building a dependency link from the current upstream instance node to the target downstream instance node, the method further includes: When it is detected that the dependency link from the current upstream instance node to the target downstream instance node is a circular dependency link, the dependency link from the current upstream instance node to the target downstream instance node is deleted.
5. A task backtracking method, characterized in that: include: Obtaining task backtracking information; wherein the task backtracking information at least includes a target backtracking task, a data backtracking time period, and an instance backtracking time period; Based on the task backtracking information, a backtracking instance lineage graph of the target backtracking task is constructed by using the backtracking instance lineage graph construction method according to any one of claims 1 to 4; According to the backtracking priority and scheduling initiation time of the target backtracking task, the target backtracking task is added to the queue of tasks to be backtracked as a task to be backtracked; When it is determined that the number of tasks to be backtraced in the task queue to be backtraced reaches the queue capacity of the task queue to be backtraced, batch backtracing is performed on each task to be backtraced in the task queue to be backtraced according to the arrangement order of each task to be backtraced in the task queue to be backtraced and based on the backtracing instance lineage graph of each task to be backtraced.
6. The task backtracking method according to claim 5, characterized in that: In the process of batch backtracking the tasks to be backtracked in the task queue according to the arrangement order of the tasks to be backtracked in the task queue and based on the backtracking instance lineage graph of the tasks to be backtracked, the process further includes: When it is determined that the number of backtrace tasks in the backtrace task queue reaches an alarm threshold, and / or when it is determined that there is a backtrace instance in which backtrace fails or backtrace is terminated in each of the backtrace tasks, an alarm message is initiated.
7. The task backtracking method according to claim 6, characterized in that: Also includes: When it is determined that the backtracing instance in the backtracing task that fails or is terminated meets the rerun condition, the backtracing task is continued to be backtraced starting from the backtracing instance in the backtracing task that fails or is terminated.
8. The task backtracking method according to any one of claims 5 to 7, characterized in that: The method of batch-backtracing the backtracing tasks in the backtracing task queue according to the arrangement order of the backtracing tasks in the backtracing task queue includes: According to the arrangement order of each of the backtracking tasks in the backtracking task queue, each of the backtracking tasks in the backtracking task queue is sequentially determined as a current backtracking task; Determine the initiation time of the downstream instance node based on the initiation time of the upstream instance node in the current backtracking task, the upstream instance output cycle offset, and the downstream instance dependency cycle offset shift; The current backtracking task is backtracked according to the initiation time of each instance node in the current backtracking task.
9. A task backtracking system, characterized in that: include: An information configuration subsystem, used to obtain task backtracking information; wherein the task backtracking information at least includes a target backtracking task, a data backtracking time period, and an instance backtracking time period; A lineage graph construction subsystem, configured to construct a backtracking instance lineage graph of the target backtracking task based on the task backtracking information and using the backtracking instance lineage graph construction method as described in any one of claims 1 to 4; The task backtracing subsystem is used to add the target backtracing task as a to-be-backtraced task to the queue of tasks to be backtraced according to the backtracing priority and scheduling initiation time of the target backtracing task; when it is determined that the number of to-be-backtraced tasks in the queue of tasks to be backtraced reaches the queue capacity of the queue of tasks to be backtraced, batch backtraces are performed on each of the to-be-backtraced tasks in the queue of tasks to be backtraced according to the arrangement order of each of the to-be-backtraced tasks in the queue of tasks to be backtraced and based on the backtracing instance lineage graph of each of the to-be-backtraced tasks.
