Backup-based cross-data center DAG operation and task scheduling method

Abstract

A backup-based cross-data-center DAG job and task scheduling method comprises the steps that a backup-based cross-data-center DAG job set is acquired, input data of all jobs in the job set are distributed on cross-regional data centers, and the input data are backed up and stored in the multiple data centers; calculating approximate completion time of each job in the job set; sorting the jobs in the job set according to the approximate completion time of the jobs and the shortest job priority principle to obtain a sorting queue; combining the jobs of which the approximate completion time difference is smaller than a completion time difference threshold value in the sorting queue to obtain a new scheduling job sequence; and in order to minimize the average response time of all the jobs, constructing a backup-based cross-data center DAG job completion time optimization model, and determining an optimal task placement and scheduling strategy of all the jobs in the scheduling job sequence. According to the method, the average response time of cross-regional DAG type data analysis operation can be effectively shortened.

Description

Technical Field

[0001] The present invention mainly relates to the field of data processing technology, and in particular to a backup-based cross-data center DAG job and task scheduling method. Background Art

[0002] In order to provide low-latency services and improve user experience, companies such as Facebook and Google have established a large number of data centers around the world to provide instant services to users. A large amount of user activity or log data is constantly generated and stored in different data centers. Analyzing and mining these data distributed across data centers can obtain a lot of valuable information, such as querying user logs for business decisions, querying network logs for network attack detection (such as DoS attacks, information tampering, etc.). In order to ensure data reliability, major organizations usually back up the collected data to multiple data centers for storage. According to a Forrester survey of 106 large organizations, the vast majority (77%) of the surveyed organizations backed up data or applications at three or more sites.

[0003] In parallel data analysis, a job is usually constructed as a directed acyclic graph (DAG), where each node represents a task. Each task consists of many subtasks that perform the same calculation. A task without a predecessor node is called an entry task, and a task without a successor node is called an exit task. The execution process of a task cannot violate the dependencies between tasks. Usually, a job contains multiple entry tasks and one exit task. In order to ensure the availability and robustness of data, many companies and institutions periodically copy and move multiple copies of the same data to data centers in different regions. Therefore, the input data of the entry task can be obtained from multiple sites (i.e., data centers) distributed across regions.

[0004] Figure 1 This diagram shows an example of a multi-copy cross-data center DAG-style data analytics job (GeodistributedDAG-style Data Analytics, GDDA for short). Figure 1 In the example, the input data D1 of task 1 is stored on both site 1 and site 3, so its input data can be obtained from any of them. However, current research on cross-data center data analysis assumes that all data is stored on only one site. Therefore, in traditional methods, the entry task is usually placed only on the site where its input data is generated, while ignoring the impact of other backups.

[0005] Data backup (replication) across multiple data centers provides new opportunities for cross-data center data analysis. Cross-data center data backup is one of the important ways to prevent data on data centers from being affected by disasters. First, cross-data center data backup improves the flexibility of task placement. For any GDDA job, each entry task can be executed on any site that stores its input data. By selecting a suitable data source for each entry task, the completion time of the job can be reduced. In contrast, traditional methods ignore the role of data backup, and each entry task can only be placed on the data center where its input data is generated. Second, cross-data center data backup can provide better load balancing for cross-data center data analysis systems. For multiple GDDA jobs that share input data, their entry tasks can be deployed to multiple sites to achieve simultaneous execution. Unfortunately, in traditional methods, these jobs all point to a single data center and can only be executed sequentially on the data center where their input data is generated, which prolongs the completion time of the job. Third, cross-data center data backup can reduce data transmission executed across regional data centers. The placement of subsequent tasks affects the amount of data transmission between cross-regional data centers. Since WNA bandwidth is scarce, when an ingress task has multiple copies on multiple sites distributed across regions, it increases the possibility of placing tasks with dependencies on the same site, avoiding unnecessary data transmission.

[0006] Despite the above opportunities, an important feature and challenge of cross-data center data analysis is that the hardware resource capacity and data distribution between different sites are highly heterogeneous. Due to the heterogeneity and data backup of cross-regional distributed data centers, solving the calling problem of cross-regional DAG jobs is extremely challenging. On the one hand, in a DAG job, all dependencies and other constraints (such as resource capacity, etc.) between tasks must be satisfied. In GDDA, the data distribution, WAN bandwidth, and computing resources of cross-regional distributed sites are all heterogeneous. Therefore, it is difficult to solve the scheduling solution for the DAG job scheduling problem in polynomial time.

[0007] On the other hand, for multiple data replicas, each entry task of a DAG job can be deployed to any site with its input data, which further complicates the scheduling problem of DAG jobs. However, existing methods ignore the impact of the input data of the entry task, and the current task placement and scheduling strategies are not applicable in GDDA considering replicas. Therefore, designing an efficient and low-complexity DAG job scheduling solution that can fully utilize replicas is an urgent and challenging task. Summary of the invention

[0008] In view of the problems existing in the prior art, the present invention proposes a cross-data center DAG job and task scheduling method based on backup.

[0009] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0010] On the one hand, the present invention provides a backup-based cross-data center DAG job and task scheduling method, comprising:

[0011] Obtain a cross-data center DAG job set based on backup, where input data of each job in the cross-data center DAG job set is distributed across data centers across regions, and backups of the input data are stored in multiple data centers;

[0012] Calculate the approximate completion time of each job in the cross-data center DAG job set;

[0013] According to the approximate completion time of each job, the jobs in the job set are sorted according to the shortest job first principle to obtain a sorted queue;

[0014] Set a completion time difference threshold, merge the jobs in the sorting queue whose approximate completion time difference is less than the completion time difference threshold into one job, and obtain a new scheduling job sequence;

[0015] With the goal of minimizing the average response time of all jobs, a cross-data center DAG job completion time optimization model based on backup is constructed to determine the optimal task placement and scheduling strategy for all jobs in the scheduling job sequence.

[0016] On the one hand, the present invention provides a cross-data center DAG job and task scheduling device based on backup, comprising:

[0017] The first module is used to obtain a cross-data center DAG job set based on backup, where input data of each job in the cross-data center DAG job set is distributed on data centers across regions, and backups of the input data are stored in multiple data centers;

[0018] The second module is used to calculate the approximate completion time of each job in the cross-data center DAG job set;

[0019] The third module is used to sort the jobs in the job set according to the approximate completion time of each job and the principle of shortest job first to obtain a sorted queue;

[0020] The fourth module is used to set a completion time difference threshold, merge the jobs in the sorting queue whose approximate completion time difference is less than the completion time difference threshold into one job, and obtain a new scheduling job sequence;

[0021] The fifth module is used to build a backup-based cross-data center DAG job completion time optimization model with the goal of minimizing the average response time of all jobs, and to determine the optimal task placement and scheduling strategy for all jobs in the scheduling job sequence.

[0022] On the other hand, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned backup-based cross-data center DAG job and task scheduling method are implemented.

[0023] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned backup-based cross-data center DAG job and task scheduling method when the computer program is executed by a processor.

[0024] On the other hand, the present invention provides a computer program product, which is stored on a computer-readable storage medium and includes computer instructions, which, when executed by a processor, enable a computer device to implement the steps of the above-mentioned backup-based cross-data center DAG job and task scheduling method.

[0025] Compared with the prior art, the technical effects of the present invention are:

[0026] The present invention provides a cross-data center DAG job and task scheduling method based on backup, with the goal of minimizing the average response time of all jobs, constructing a cross-data center DAG job completion time optimization model based on backup, and determining the optimal task placement and scheduling strategy for all jobs in the scheduling job sequence. The present invention can make full use of the replicas distributed across regions, calculate appropriate tasks and job scheduling strategies to minimize the average job completion time of GDDA jobs, and can effectively improve the performance of cross-regional DAG type data analysis jobs. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0028] Figure 1 An example diagram showing a multi-copy cross-data center DAG-style data analysis job (GeodistributedDAG-style Data Analytics, GDDA for short);

[0029] Figure 2A flowchart of a backup-based cross-data center DAG job and task scheduling method provided by an embodiment is shown;

[0030] Figure 3 Schematic diagram of the task aggregation strategy;

[0031] Figure 4 Schematic diagram of the structural type of entry tasks, where (a) is an incast type entry task, (b) is a mixed type entry task, and (c) is a multicast type entry task;

[0032] Figure 5 The performance comparison chart of DAMC-bl with HEFT, TDCA, Pixida and Geode is shown in Figure 2. Figure 5 (a) is a comparison chart of the average job completion time of DAMC-bl, HEFT, TDCA, Pixida and Geode. Figure 5 (b) is a comparison of the reduction in average job completion time between DAMC-bl and HEFT, TDCA, Pixida, and Geode. Figure 5 (c) is the CDF comparison chart of DAMC-bl, HEFT, TDCA, Pixida and Geode. Figure 5 (d) is the running time comparison chart of DAMC-bl, HEFT, TDCA, Pixida and Geode;

[0033] Figure 6 The comparison chart of the impact of resource heterogeneity and the number of replicas on the job completion time of DAMC-bl, HEFT, TDCA, Pixida, and Geode is shown in the figure. Figure 6 (a) shows the average job completion time comparison between DAMC-bl and HEFT, TDCA, Pixida and Geode under different resource difference rates. Figure 6 (b) shows the comparison of the average job completion time of DAMC-bl under different numbers of backups;

[0034] Figure 7 The performance comparison of DAMC and four comparison methods, HEFT, TDCA, Pixida and Geode, and the impact of the number of replicas on job completion time are shown in the figure. Figure 7 (a) is a comparison chart of the average job completion time of DAMC and four comparison methods: HEFT, TDCA, Pixida and Geode. Figure 7 (b) is the CDF comparison chart of the job completion time reduction rate of DAMC and the four comparison methods of HEFT, TDCA, Pixida and Geode; Figure 7(c) is a comparison chart obtained by combining the two job scheduling methods SJF and FCFS with the task scheduling method DAMC-bl proposed in the present invention to test the average job completion time under different job scheduling methods. Figure 7 (d) is a comparison chart of the impact of the number of replicas on job completion time among DAMC and the four comparison methods: HEFT, TDCA, Pixida, and Geode;

[0035] Figure 8 This is a comparison chart of the impact of time interval and job arrival frequency on job completion time, where Figure 8 (a) is a comparison chart showing the impact of the time interval between DAMC algorithm calls on job completion time. Figure 8 (b) is a comparison chart showing the impact of job arrival frequency on job completion time in the DAMC algorithm. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] In one embodiment of the present invention, referring to Figure 2 , provides a backup-based cross-data center DAG job and task scheduling method, including:

[0038] Obtain a cross-data center DAG job set based on backup, where input data of each job in the cross-data center DAG job set is distributed across data centers across regions, and backups of the input data are stored in multiple data centers;

[0039] Calculate the approximate completion time of each job in the cross-data center DAG job set;

[0040] According to the approximate completion time of each job, the jobs in the job set are sorted according to the shortest job first principle to obtain a sorted queue;

[0041] Set a completion time difference threshold, merge the jobs in the sorting queue whose approximate completion time difference is less than the completion time difference threshold into one job, and obtain a new scheduling job sequence;

[0042] With the goal of minimizing the average response time of all jobs, a cross-data center DAG job completion time optimization model based on backup is constructed to determine the optimal task placement and scheduling strategy for all jobs in the scheduling job sequence.

[0043] The detailed description of the present invention is shown in Algorithm 3.1.

[0044]

[0045] Job k in the GDDA job set Q is represented as a directed acyclic graph (N k , E k ), where N k and E k They represent the task set and task dependency set of job k, respectively, and N k Each task in N consists of a series of similar subtasks. k The subtasks of each task in are executed concurrently in the data center where the input data is located; each edge in the task dependency set represents the dependency constraint between tasks, and the edge (u, v)∈E k It means that task v cannot be started before task u is completed; in the directed acyclic graph (N k , E k ), pred(u, k) represents the set of direct predecessor nodes of task u of job k, succ(u, k) represents the set of direct successor nodes of task u of job k, a task without a direct predecessor node is called an entry task, and a task without a direct successor node is called an exit task; the input data of each job in the GDDA job set Q is backed up multiple times, and the backup data is stored in multiple data centers. Each node in the directed acyclic graph of job k represents a task in job k.

[0046] The input data of the jobs in the GDDA job set Q are distributed across cross-regional data centers (i.e., sites), represented by the data center set S. In order to ensure the performance of the system under various failures (such as hacker attacks, site or link failures), various organizations will make multiple backups of the job input data and store the backup data in multiple data centers. In parallel jobs, a task usually consists of multiple subtasks. Due to insufficient computing resources at the site, these subtasks are usually divided into different batches for execution. Therefore, assuming that a site can only execute one task at a time, the resources allocated to the task will not be allocated to other tasks until the task is completed. In order to accurately determine the duration of network transmission, it is assumed that no two data transmissions can be performed simultaneously between tasks using the same link.

[0047] Specifically, the present invention calculates the approximate completion time of each job in the GDDA job set Q, including:

[0048] Calculate the rank value of task u for job k in the GDDA job set Q The calculation is as follows:

[0049]

[0050] Among them, OCT(k,u,i) represents the shortest completion time of job k from task u to export task in data center i, and |S| represents the number of data centers in the data center set S.

[0051] Specifically, the calculation method of OCT(k,u,i) is as follows:

[0052]

[0053] Among them, OCT(k, v, j) represents the shortest completion time of job k from task v to egress task in data center j; represents the computation time of task v of job k in data center j; represents the data transfer time from task u to task v of job k.

[0054] The entry task of job k can only be executed on the data center that stores its input data. If data center i does not store the input data of task u of job k, OCT(k,u,i) will be set to null. For each DAG job, an exit task will be added for it. k |, Export Task|N k | will be placed at the submission site of job k, whose computation time is 0, and the completion time of job k is equal to the exit task |N k |Completion time; for the exit task, the value of OCT(k, exit, i) is equal to 0.

[0055] In one embodiment of the present invention, with the goal of minimizing the average response time of all jobs, a backup-based cross-data center DAG job completion time optimization model is constructed as follows:

[0056]

[0057] Where |Q| represents the number of jobs in the GDDA job set Q, T k represents the completion time of job k. For each GDDA job, the completion time of the GDDA job is determined by the last task in the GDDA job. The completion time of job k is T k The calculation is as follows:

[0058]

[0059] Indicates the submission time of job k, using a binary variable Indicates whether task u of job k is assigned to data center i. If task u is executed in data center i, then let otherwise, and They represent the start time and calculation time of task u of job k in data center i respectively.

[0060] The constraints of the above backup-based cross-data center DAG job completion time optimization model include:

[0061] For the entry task u of job k, use the binary variable Indicates whether data center i stores the input data of task u; indicates that data center i stores its input data; otherwise, Indicates that data center i does not store its input data; using binary variables Indicates whether task u of job k is assigned to data center i. If task u of job k is executed in data center i, then let otherwise, The task u of query job k can only be executed by one data center; the entry task u of job k can only be executed on the data center containing its input data; let Indicates whether task u is the entry task of job k. If task u is the entry task, then set Otherwise, set Thus we get:

[0062]

[0063] For each task of job k, the immediate successor of the task cannot start execution until the task is completed and the required intermediate results are transmitted, with:

[0064]

[0065] in and They represent the start time and calculation time of task u of job k in data center i, respectively. represents the data transmission time required to transfer the intermediate data generated by task u executed in data center i in job k to task v executed in data center j. During the transmission process of task u in job k to task V, tasks u and v are executed in data center i and data center j respectively. represents the start time of task v of job k in data center j, indicates whether task V of job k is assigned to data center j. If task v of job k is executed in data center j, then otherwise,

[0066] If the tasks of different jobs are assigned to the same data center, it is necessary to determine the execution order between the tasks of different jobs. Indicates whether task u of job k in data center i is executed before task v of job q in data center i, that is, Indicates whether task u of job k on data center i takes precedence over task v of job q on data center i. If task u of job k on data center i takes precedence over task v of job q on data center i, then otherwise have:

[0067]

[0068] in, indicates whether task v of job q is assigned to data center i. If task V of job q is executed in data center i, then otherwise,

[0069] When multiple data sets need to be transmitted on the same link, the order of transmission needs to be determined; Indicates whether the transmission of job k from task u to task v on the link from data center i to data center j takes precedence over the transmission of job q from task m to task n. When the transmission of job k from task u to task v on the link from data center i to data center j takes precedence over the transmission of job q from task m to task n, otherwise The starting time of data transmission from task u to task v of job k is:

[0070]

[0071] in indicates whether task m of job q is assigned to data center i. If task m of job q is executed in data center i, then set otherwise, indicates whether task n of job q is assigned to data center j. If task n of job q is executed in data center j, then set otherwise,

[0072] Due to the limited computing resources of the site, it is usually impossible to complete all subtasks of a task at once. Therefore, the subtasks of the task need to be executed in batches. The computing time of task u of job k in data center i is and the data transfer time required to transfer the intermediate data generated by task u in job k executed in data center i to task v executed in data center j It is expressed as follows:

[0073]

[0074] Among them B i，j represents the bandwidth between data center i and data center j, s i represents the number of slots in data center i, It represents the ratio of the amount of intermediate data of task u in job k to the amount of input data of task u. represents the duration of the subtask of task u, Represents the amount of input data for task u of job k.

[0075] In one embodiment, determining a scheduling strategy for each job in all jobs in a scheduling job sequence includes:

[0076] Among the rank values ​​of all tasks of job k, the maximum rank value is used as the approximate completion time of job k.

[0077] According to the approximate completion time of each job, the jobs are sorted based on the shortest job first principle (SJF) to obtain the sorted queue Q sch Therefore, when there is competition for the same resources, jobs with shorter completion time can get resources first. In addition, following the insertion policy principle, this method considers the possibility of inserting a task in the earliest idle time interval between two scheduled tasks on a site.

[0078] For jobs with the same or similar approximate completion time, merging these jobs into a new composite job can make full use of the site's resources, which reduces the average job completion time of these jobs. For example, if jobs A, B, and C have similar completion times, merge jobs A, B, and C into a new composite job and add a common exit node. Therefore, jobs A, B, and C are scheduled as a single job, and their tasks can be scheduled together.

[0079] In one embodiment, determining the optimal task placement and scheduling strategy for all jobs in the scheduled job sequence includes:

[0080] Get the rank values ​​of all tasks of job k. The rank value of a task represents its priority. The higher the rank value of a task, the earlier it will be scheduled.

[0081] The DAG graph of job k is used to construct a new DAG graph through task aggregation, where the task aggregation method is: for a task with only one child task (i.e., child node) in the DAG graph of job k, the task and its child tasks are merged into a new task, where the child task is a task that uses the output data of the task as its input data, and the child task can only be executed after the task is completed.

[0082] For the scheduling of entry tasks in job k, the entry tasks are divided into different groups according to the structure of the entry tasks in the new DAG graph of job k. For each group of tasks, they are scheduled from large to small according to the rank values ​​of the tasks in the group;

[0083] For the scheduling of other tasks other than the entry task in job k, they are scheduled from large to small according to the rank value of the task.

[0084] Determine the priority of tasks based on PEFT. Specifically, for each task, a rank value is calculated to represent the priority of the task. The higher the rank value of a task, the earlier it will be scheduled. For job k in the GDDA job set Q, calculate the rank value of task u of job k The method has been clearly described in the previous article and will not be repeated here.

[0085] Generally, the complexity of the task scheduling algorithm increases with the number of nodes in the job DAG graph (nodes correspond to the tasks of the job). The task merging component merges multiple nodes to convert the DAG of the job into a new DAG with fewer nodes, thereby improving the performance of the algorithm.

[0086] Reference Figure 3 , Figure 3 This is a diagram of the task aggregation strategy; when a task has only one child node, the task and its child nodes are merged into a new task. This is because placing them in the same data center for execution can avoid data transmission between different data centers. Figure 3 As shown, Task 1 has only one child task (Task 3), and Task 3 also has only one child task (Task 6). Therefore, Tasks 1, 3, and 6 can be merged into a new task. Similarly, Tasks 2 and 4, Tasks 8 and 10 can be merged into Task 1 and Task 5 in the new DAG structure, respectively. Figure 3 The DAG structure of (a) is transformed into a new DAG with only 7 nodes, as shown in Figure 3 (b) as shown.

[0087] Merging multiple nodes into one node can not only reduce the complexity of the task scheduling algorithm, but also avoid jobs falling into bad task placement schemes. Figure 3In (a), the rank values ​​of Task 2 and 3 are close, and the rank value of Task 2 is greater than that of Task 3. According to the priority of the tasks, Task 1 will be scheduled first, followed by Task 2 and Task 3. That is, Task 1, Task 3, and Task 6 may be placed in different locations. This may greatly increase the transmission load of cross-regional sites. Since the bandwidth between cross-regional sites is scarce, this may result in a longer job completion time. However, scheduling jobs according to the new DAG structure can avoid this cross-site data transmission.

[0088] Since the input data of the entry task can be obtained from multiple sites, the backup in cross-data provides new opportunities for improving the performance of cross-regional data analysis. Assume that the entry task can only be executed on the site with its input data. Due to the heterogeneity of WAN bandwidth and computing resources, the site selection for executing the entry task has a great impact on the job completion time. For the scheduling of the entry task, the present invention divides the entry task into different groups according to the structure of the entry task in the new DAG graph of the job, and for each group of tasks, the tasks are scheduled from large to small according to the rank value of the tasks in the group.

[0089] like Figure 4 As shown, Figure 4 This is a schematic diagram of the structure type of the entry task, where Figure 4 (a) is an incast type entry task, Figure 4 (b) is a Mixed type entry task. Figure 4 (c) is a multicast type entry task.

[0090] (1) When an entry task has only one child task, it is called an incast type entry task. For an incast type entry task, the entry task and its sibling tasks and child tasks are grouped into the same group, and the shortest time of its child tasks is calculated. Figure 4 As shown in (a), tasks 1, 2, and 4 are entry tasks and have only one child node (i.e., only one child task). Tasks 1, 2, and 4 are all incast type entry tasks. Specifically, the child task of tasks 1 and 2 is task 3, and the child task of tasks 3 and 4 is task 5. Therefore, tasks 1, 2, and 3 are divided into group 1. Tasks 3, 4, and 5 are divided into group 2. Then, the shortest completion time of the child tasks in groups 1 and 2 in each group is calculated. Since the number of copies of each data set is small, usually 3, the places where each entry task can be executed are limited. Therefore, all task placement schemes for tasks in each group are traversed, and then the task scheme that can minimize the completion time of the child tasks in the group is selected as the final scheme.

[0091] (2) An entry task has multiple child tasks, and an entry task with multiple child tasks is also a sibling task of another entry task. Such an entry task is a Mixed type entry task. Figure 4 As shown in (b), both Task 1 and Task 2 are entry tasks, and Task 1 is the sibling task of Task 2. Task 1 also has two child tasks (Task 3 and 5), so Task 1 is a Mixed entry task. Mixed entry tasks are divided into two groups. For the two groups containing Mixed entry tasks, the maximum ranks of the tasks in the two groups are compared to determine the group with priority scheduling. The group with the larger maximum rank of the tasks in the group is the group with priority scheduling. Therefore, Figure 4 In (b), task 1 belongs to both group 1 and group 2. Assume that the maximum ranks of tasks in group 1 and group 2 are r1 and r2 respectively. If r1>r2, group 1 will be scheduled first. Otherwise, group 2 will be scheduled first.

[0092] (3) When an entry task has multiple child tasks, this type of entry task is called a multicast type entry task. In this case, in order to further utilize the advantages of multiple copies, task replication is used to improve execution efficiency. Specifically, the same entry task can be executed simultaneously in multiple data centers. Figure 4 As shown in (c), Task 1 is an entry task that contains three child tasks, that is, Task 1 is a multicast entry task. Task 1 has three copies of its input data. Figure 4 (c) lists four feasible task execution strategies for Task 1. To derive the optimal strategy, the execution order of Tasks 2, 3, and 4 is first determined in descending order of priority. For each of the three tasks, Tasks 2, 3, and 4, it selects the site that can achieve the shortest completion time, and the same is true for the placement of entry Task 1. Obviously, the site that executes Task 1 and provides intermediate data for Tasks 2, 3, and 4 can be the same or different.

[0093] For other tasks (non-entry tasks), they will be scheduled according to their rank values ​​from large to small. When task u is selected for scheduling, it will be scheduled according to the optimal shortest completion time (O EFT ) to determine the placement of task u. EFT It is equal to the sum of the earliest completion time to task u and the calculation time of the longest path to the exit task, and is defined as follows:

[0094] O EFT (k,u,i)=EFT(k,u,i)+OCT(k,u,i)

[0095] O EFT(k, u, i) represents the shortest completion time of task u of job k executed in data center i.

[0096] In data center i, the task with the shortest completion time will be selected for execution. EFT(k,u,i) represents the earliest completion time of task u of job k in data center i, and is defined as follows:

[0097]

[0098] In the above formula, if task v is a synthetic task, It is equal to the sum of the execution time of each task in data center j. In addition, EST(k,u,i) represents the earliest start time of task u of job k in data center i, which is defined as:

[0099]

[0100] In the above formula, assuming that task v is executed in data center j, AFT(k, v, j) represents the actual completion time of task V of job k in data center j.

[0101] In one embodiment, the optimal task placement and scheduling process for a single GDDA job is shown in Algorithm 3.3.

[0102] First, calculate the priority of all tasks in job k. The rank value of a task represents the priority of the task. The higher the rank value of a task, the higher the rank value of task u in job k. The method has been clearly described in the previous article and will not be repeated here.

[0103] According to the dependency relationship of tasks in the DAG graph of job k, the DAG graph structure of job k is simplified, and some tasks are merged into a synthetic task, that is, a new DAG graph is constructed by task aggregation, where the task aggregation method is: for a task with only one child task in the DAG graph of job k, the task and its child tasks are merged into a new task.

[0104] Construct the collection Sch, which records all scheduled tasks and its initial value is null.

[0105] Next, the task scheduling process begins: During the entire task scheduling process, first obtain the task u with the highest priority among the unscheduled tasks in job k. If task u is an entry task and has only one child task, the sibling task sib of task u will be obtained. u Then the brother task sib will be judged u Can it be scheduled or has been scheduled? If so, schedule task u and its sibling task sib. uTogether with the child tasks of task u, form a group and calculate an optimal task scheduling strategy for this group. If task u is an entry task and has multiple child tasks, group task u and its child tasks Chi u The child task v with the highest priority in the task is grouped together, and then the optimal task scheduling strategy for task u and task v is obtained. When task u is not an entry task, but the parent task f of task u is an entry task, the minimum O is calculated for task u and task f. EFT The optimal task placement strategy for (u, i, k). If task u is not an entry task and the parent task f of task u is not an entry task, the optimal task placement strategy for task i is calculated separately (step 17).

[0106] Since task replication can effectively reduce the completion time of DAG type jobs, the task replication strategy is also used for scheduling non-entry tasks. If task u is a non-entry task, obtain the direct predecessor task dp of task u. u , then, if you copy the direct predecessor task dp u Going to data center i can reduce the start time of task u, and task u will be placed in data center i for repeated execution.

[0107] After calculating the task scheduling strategy for task u or the group formed by task u, the scheduled tasks are placed in the set Sch. Next, the next task with the highest priority among the unscheduled tasks in job k is executed, and this iteration continues until all tasks in job k are scheduled, and the task scheduling solution Sch for job k is obtained. k .

[0108] After the above steps, the placement and scheduling time of each task have been determined. Next, a dynamic adjustment strategy will be used to further reduce the job completion time.

[0109] The dynamic adjustment strategy reduces the job completion time by changing the location of the critical task. Specifically, first find the job k in the current solution Sch k The critical path CP under n}. Among them, the critical path refers to the longest path from the entry task to the exit task. That is to say, on the critical path, the direct predecessor task determines the placement and scheduling of the subsequent tasks. If the length of the critical path can be shortened, the job completion time will also be reduced. Therefore, by changing the placement of tasks on the critical path, the job completion time can be reduced.

[0110] Algorithm 3.3 (i.e., DAMC-bl algorithm) describes the specific steps of the task scheduler. For each job, Algorithm 3.3 is called to calculate the appropriate task placement and task scheduling scheme to minimize the completion time of a single job.

[0111]

[0112] Algorithm 3.2 describes the dynamic adjustment strategy in detail. Specifically, in each iteration, the strategy considers the adjacent tasks on the critical path and traverses all task placement schemes (steps 4-15). For each task placement scheme, the new job completion time is obtained. With T k For comparison. Then the new task scheduling plan will be recorded. After traversing task v i and v i-1 After all the tasks are placed in the plan, if T k There has been a change, Assign to Sch k When changing the task scheduling scheme, the critical path may also change. Therefore, this algorithm iterates the process 4 times to reduce the job completion time while ensuring the performance of the algorithm.

[0113]

[0114] In order to prove the effectiveness of the cross-data center DAG job and task scheduling method based on backup proposed in the present invention, an experimental example is provided below to illustrate it. In this experiment, a large number of experiments were carried out based on Alibaba's data set to verify the performance of the cross-data center DAG job and task scheduling method based on backup (DAMC algorithm for short) proposed in the present invention.

[0115] Cluster settings: In this experiment, the number of data centers distributed across regions |S| is set to range from 10 to 90, and different data centers have different resource settings. The network bandwidth and computing resource capacity between data centers are set as follows: the bandwidth range is [0.1, 2] ~ GB / s, and the number of computing slots (corresponding to a certain amount of memory and cores) in each data center is [100, 1000]. The DAMC algorithm is executed periodically, and the default time window (T win ) is 500s. The default resource heterogeneity ratio and backup number are 1.0 and 3 respectively. The following table summarizes the configuration of the network used in the simulation experiment:

[0116]

[0117] Job load: The Alibaba dataset records the operation of millions of jobs running on 4,000 machines in 8 days. Specifically, for each job, the dataset records its arrival time, task number, the amount and distribution of input data, and the required resources. In addition, the dependencies between the tasks used to build the DAG of the job are also recorded. In order to simulate the environment of GDDA, the input data of the entry tasks of these jobs are backed up at multiple sites.

[0118] The comparison methods include:

[0119] 1) Geode: An advanced scheduler proposed in recent years that combines classic query planning with novel network-centric mechanisms to minimize the amount of data transmitted across sites.

[0120] 2) Pixida: In recent years, a scheduler designed to minimize data movement across regions has been proposed. It uses a new graph partitioning method to reduce traffic transmission.

[0121] 3) HEFT: A traditional task scheduling method used to minimize the job completion time of heterogeneous computing systems. It first sorts the tasks in non-descending order according to the B-level computing task priority and assigns the tasks to the processors so that they have the earliest initial assignment start time.

[0122] 4) TDCA: A state-of-the-art scheduler for heterogeneous computing systems in recent years. It replicates tasks on multiple machines so that the results of replicated tasks can be obtained on multiple computers to minimize the completion time of the job.

[0123] Evaluation indicators: The main evaluation indicators of this experiment are the average job completion time and the decrease rate of the average job completion time. Specifically, the decrease rate of the average job completion time is equal to the ratio of the difference between the job completion time of DAMC and the comparison method to the job completion time of the comparison method. In addition, the running time of the DAMC method and the comparison method is also compared.

[0124] This experiment first compares the average completion time of DAMC-bl (the task placement and scheduling algorithm for jobs provided by the present invention, namely Algorithm 3.3) and other task scheduling methods when the site scale increases from 10 to 90. Figure 5 The performance comparison chart of DAMC-bl with HEFT, TDCA, Pixida and Geode is shown in Figure 2. Figure 5 (a) is a comparison chart of the average job completion time of DAMC-bl, HEFT, TDCA, Pixida and Geode. Figure 5 (b) is a comparison of the reduction in average job completion time between DAMC-bl and HEFT, TDCA, Pixida, and Geode. Figure 5(c) is the CDF comparison chart of DAMC-bl, HEFT, TDCA, Pixida and Geode. Figure 5 (d) is a comparison of the running time of DAMC-bl, HEFT, TDCA, Pixida and Geode.

[0125] like Figure 5 As shown in (a), regardless of the site size, DAMC-bl can achieve the shortest average job completion time compared to other comparison methods. This also shows that DAMC-bl can effectively use cross-regional data backup to reduce the completion time of GDDA jobs. Among all the comparison methods, TDCA has the largest average job completion time gap with DAMC-bl. The reason is that TDCA ignores the heterogeneity and scarcity of WAN bandwidth, resulting in long transmission overhead. In addition, the number of tasks with multiple direct successor tasks in this dataset is limited. Therefore, GDDA cannot reduce the completion time of jobs through its task replication strategy. Geode and Pixida attempt to reduce data transmission between sites, but may perform large-scale data transmission on links with less bandwidth, or perform a large number of computing tasks on sites without sufficient computing resources. As a result, they usually produce longer job completion times compared to DAMC-bl. Since HEFT attempts to assign tasks to sites that can minimize their startup time, it can greatly reduce job completion time. Therefore, the job completion time gap between HEFT and DAMC-bl is relatively small, but HEFT still cannot utilize cross-regional data backup to further reduce the job completion time.

[0126] In order to further analyze the performance of DAMC-bl, the average decrease rate of average job completion time of DAMC-bl and four other comparison methods, HEFT, TDCA, Pixida and Geode, at different site sizes were compared. Figure 5 (b) shows the experimental results. The reduction rate of the average job completion time is between 40% and 90%. This result also further verifies the effectiveness of the DAMC-bl algorithm. In addition, the reduction rate that DAMCL-bl can achieve is stable under different site scales. This shows that DAMC-bl has good scalability and can be applied to large-scale cross-regional data analysis jobs. Figure 5(c) shows the CDF results of the average decrease rate of job completion time for DAMC-bl and four other comparison methods: HEFT, TDCA, Pixida, and Geode. The completion time of almost all jobs based on the DAMC-bl algorithm is less than the completion time of jobs based on other comparison methods, and when the TDCA method is used, the completion time of 20% of the jobs is 0.8 to 1 times higher than the completion time of jobs based on the DAMC-bl method. Therefore, TDCA has the worst performance in cross-regional data analysis with multiple backups.

[0127] Scheduling Load: In addition, the running time of DAMC-bl and four other comparison methods, HEFT, TDCA, Pixida and Geode, at different site sizes were tested, e.g. Figure 5 (d) When the site scale is between 10 and 50, the running time of DAMC-bl is very close to that of other algorithms. When the number of sites is greater than 50, the running time of DAMC-bl increases, but the average running time of DAMC-bl for each cross-region data analysis job is less than 8ms, which can meet the performance requirements.

[0128] Resource Heterogeneity: Next, we measured the impact of resource heterogeneity and the number of backups on the algorithm performance. Figure 6 The comparison chart of the impact of resource heterogeneity and the number of replicas on the job completion time of DAMC-bl, HEFT, TDCA, Pixida, and Geode is shown in the figure. Figure 6 (a) shows the average job completion time comparison of DAMC-bl and HEFT, TDCA, Pixida and Geode under different resource difference rates. Let s1 and s2 represent the maximum and minimum computing resource capacity of the site respectively. Then, the resource difference ratio of the system is (s1-s2) / 2. The higher the resource difference ratio, the greater the difference in resource capacity between different sites. Figure 6 (a) It can be found that as the resource difference rate increases, the gap in job completion time between DAMC-bl and other comparison methods widens. This shows that compared with other comparison methods, DAMC-bl can achieve better performance as the resource difference rate increases.

[0129] Number of backups: Figure 6(b) shows the average job completion time comparison of DAMC-bl under different numbers of backups. Obviously, as the number of backups increases, the job completion time decreases. The reason is that as the number of backups increases, the number of locations where the entry task can be placed will also increase. Therefore, resource conflicts will be reduced, and thus the job completion time based on DAMC-bl will also be reduced. On the contrary, when the input data of the entry task is only stored on one site, the job completion time varies greatly under different site sizes. This phenomenon shows that more data backups will give tasks placement options and improve the performance stability of the job.

[0130] Performance improvements: Figure 7 The performance comparison of DAMC and four comparison methods, HEFT, TDCA, Pixida and Geode, and the impact of the number of replicas on job completion time are shown in the figure. Figure 7 (a) is a comparison chart of the average job completion time of DAMC and four comparison methods: HEFT, TDCA, Pixida and Geode. Figure 7 (b) is the CDF comparison chart of the job completion time reduction rate of DAMC and the four comparison methods of HEFT, TDCA, Pixida and Geode; Figure 7 (c) is a comparison chart obtained by combining the two job scheduling methods SJF and FCFS with the task scheduling method DAMC-bl proposed in the present invention to test the average job completion time under different job scheduling methods. Figure 7 (d) is a comparison chart of the impact of the number of replicas on job completion time in DAMC and the four comparison methods: HEFT, TDCA, Pixida and Geode.

[0131] First, the performance of DAMC is compared with four comparison methods. Specifically, the task scheduling algorithms of the four comparison methods, HEFT, TDCA, Pixida and Geode, are Geode, Pixida, HEFT and TDCA, and their job scheduling algorithms are all SJF. The SJF scheduling method can effectively reduce the average JRT. Figure 7 (a) is a comparison chart of the average job completion time of DAMC and the four comparison methods of HEFT, TDCA, Pixida and Geode. It can be found from the figure that the average job completion time incurred by DAMC is the smallest, while TDCA has the worst performance. Compared with TDCA, Pixida and Geode, DAMC has a much lower average job completion time and a huge performance gain. The reason is that these methods ignore the impact of computing or wide area network bandwidth heterogeneity, which usually leads to longer transmission or computing time. The HEFT method takes into account the heterogeneity of resources and ignores the impact of multiple data backups. Therefore, its performance is between DAMC and the other three methods. Figure 7 (b) is a CDF comparison chart of the reduction rate of job completion time of DAMC and four comparison methods: HEFT, TDCA, Pixida and Geode. The results show that compared with other comparison methods, DAMC can enable most jobs to be completed in a shorter time. Therefore, DAMC can effectively reduce the average job completion time of multiple jobs in online scenarios.

[0132] In addition, the two job scheduling methods (SJF and FCFS) are combined with the task scheduling method DAMC-bl proposed in this invention to test the average job completion time under different job scheduling methods. The experimental results are as follows: Figure 7 (c) Compared with the other two job scheduling strategies, the job scheduling method proposed in the present invention can provide better performance and achieve the minimum job completion time. In addition, as the site scale increases, the performance gap between DAMC and the comparison method becomes larger. This shows that DAMC achieves better scalability.

[0133] Number of backups: Next, the impact of DAMC under different parameters is analyzed. Figure 7 (d) shows the job completion time of DAMC under different numbers of backups and site sizes. As the number of backups increases, the job completion time decreases. The reason is that when the number of backups increases, the location choices for placing the entry tasks will increase. Therefore, DAMC has more opportunities to reduce the job completion time by placing these tasks in appropriate locations. However, the gain of increasing backups decreases as the number of backups increases. Therefore, in different cross-regional data analysis scenarios, the system needs to set an appropriate number of backups to balance performance and storage overhead.

[0134] Time interval: Figure 8 This is a comparison chart of the impact of time interval and job arrival frequency on job completion time, where Figure 8 (a) is a comparison chart showing the impact of the time interval between DAMC algorithm calls on job completion time. Figure 8 (b) is a comparison chart showing the impact of job arrival frequency on job completion time in the DAMC algorithm.

[0135] Figure 8 (a) The effect of the time interval of DAMC algorithm calls on the job completion time is analyzed, and the time interval ranges from [50, 1000] seconds. As the time interval increases, the job completion time caused by DAMC also increases. The reason is that as the time interval increases, the job needs to wait longer. Therefore, if the user wants to achieve high performance, the system needs to set a lower algorithm execution time interval.

[0136] Frequency of job arrival: Finally, we tested the effect of the frequency of job arrival on job completion time. The results are as follows: Figure 8 (b). When p is set to 100 seconds, it means that the time interval between two adjacent job submission times is 0 to 100 seconds. In other words, the larger p is, the less frequent the job arrivals will be within a period of time. Figure 8 In (b), as p increases, the average job completion time decreases. This is because the number of jobs that the system needs to process decreases, and the competition between tasks also decreases.

[0137] In summary, the proposed cross-data center DAG job and task scheduling method based on backup can make full use of the cross-regional distributed replicas and calculate appropriate task and job scheduling strategies to minimize the average job completion time of GDDA jobs. A large number of experiments have shown that the present invention can effectively improve the performance of cross-regional DAG type data analysis jobs.

[0138] On the other hand, an embodiment provides a cross-data center DAG job and task scheduling device based on backup, including:

[0139] The first module is used to obtain a cross-data center DAG job set based on backup, where input data of each job in the cross-data center DAG job set is distributed on data centers across regions, and backups of the input data are stored in multiple data centers;

[0140] The second module is used to calculate the approximate completion time of each job in the cross-data center DAG job set;

[0141] The third module is used to sort the jobs in the job set according to the approximate completion time of each job and the principle of shortest job first to obtain a sorted queue;

[0142] The fourth module is used to set a completion time difference threshold, merge the jobs in the sorting queue whose approximate completion time difference is less than the completion time difference threshold into one job, and obtain a new scheduling job sequence;

[0143] The fifth module is used to build a backup-based cross-data center DAG job completion time optimization model with the goal of minimizing the average response time of all jobs, and to determine the optimal task placement and scheduling strategy for all jobs in the scheduling job sequence.

[0144] On the other hand, the present invention provides a computer device, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the backup-based cross-data center DAG job and task scheduling method provided in any of the above embodiments when executing the computer program. The computer device may be a server. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store sample data. The network interface of the computer device is used to communicate with an external terminal via a network connection.

[0145] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the backup-based cross-data center DAG job and task scheduling method provided in any of the above embodiments.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0147] Matters not covered by the present invention are known technologies.

[0148] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

[0150] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cross-data center DAG job and task scheduling method based on backup, characterized in that: include: Obtain a cross-data center DAG job set based on backup, where input data of each job in the cross-data center DAG job set is distributed across data centers across regions, and backups of the input data are stored in multiple data centers; Calculate the approximate completion time of each job in the cross-data center DAG job set; According to the approximate completion time of each job, the jobs in the job set are sorted according to the shortest job first principle to obtain a sorted queue; Set a completion time difference threshold, merge the jobs in the sorting queue whose approximate completion time difference is less than the completion time difference threshold into one job, and obtain a new scheduling job sequence; With the goal of minimizing the average response time of all jobs, a cross-data center DAG job completion time optimization model based on backup is constructed to determine the optimal task placement and scheduling strategy for all jobs in the scheduling job sequence.

2. The backup-based cross-data center DAG job and task scheduling method according to claim 1 is characterized in that: Job k in the GDDA job set Q is represented as a directed acyclic graph (N k ,E k ), where N k and E k They represent the task set and task dependency set of job k, respectively, and N k Each task in N consists of a series of similar subtasks. k The subtasks of each task in are executed concurrently in the data center where the input data is located; each edge in the task dependency set represents the dependency constraint between tasks, and the edge (u,v)∈E k It means that task v cannot be started before task u is completed; in the directed acyclic graph (N k ,E k ), pred(u,k) represents the set of direct predecessor nodes of task u of job k, succ(u,k) represents the set of direct successor nodes of task u of job k, a task without a direct predecessor node is called an entry task, and a task without a direct successor node is called an exit task; the input data of each job in the GDDA job set Q is backed up in multiple ways, and the backup data is stored in multiple data centers.

3. The backup-based cross-data center DAG job and task scheduling method according to claim 2 is characterized in that: Calculate the approximate completion time of each job in the GDDA job set Q, including: Calculate the rank value of task u for job k in the GDDA job set Q The calculation is as follows: Where OCT(k,u,i) represents the shortest completion time of job k executing task u in data center i, and |S| represents the number of data centers distributed across regions; Among the rank values ​​of all tasks of job k, the maximum rank value is used as the approximate completion time of job k.

4. The backup-based cross-data center DAG job and task scheduling method according to claim 3 is characterized in that: The calculation method of OCT(k,u,i) is as follows: Among them, OCT(k,v,j) represents the shortest completion time of job k from task v to the export task, and task v is executed in data center j; represents the computation time of task v of job k in data center j; represents the data transfer time from task u to task v of job k; The entry task of job k can only be executed on the data center that stores its input data. If data center i does not store the input data of task u of job k, OCT(k,u,i) will be set to null. For each job, an exit task |N will be added for it. k |, Export Task|N k | will be placed at the submission site of job k, whose computation time is 0, and the completion time of job k is equal to the exit task |N k |Completion time; for the exit task, the value of OCT(k,exit,i) is equal to 0.

5. The backup-based cross-data center DAG job and task scheduling method according to claim 3 or 4, characterized in that: With the goal of minimizing the average response time of all jobs, a cross-data center DAG job completion time optimization model based on backup is constructed as follows: Where |Q| represents the number of jobs in the GDDA job set Q, T k represents the completion time of job k. For each GDDA job, the completion time of the GDDA job is determined by the last task in the GDDA job. The completion time of job k is T k The calculation is as follows: Indicates the submission time of job k, using a binary variable Indicates whether task u of job k is assigned to data center i. If task u is executed in data center i, then let otherwise, and They represent the start time and computation time of task v executed by task u of job k in data center i.

6. The backup-based cross-data center DAG job and task scheduling method according to claim 5, characterized in that: The constraints of the backup-based cross-data center DAG job completion time optimization model include: For the entry task u of job k, use the binary variable Indicates whether data center i stores the input data of task u; indicates that data center i stores its input data; otherwise, Indicates that data center i does not store its input data; using binary variables Indicates whether task u of job k is assigned to data center i. If task u of job k is executed in data center i, then let otherwise, The query task u of job k can only be executed by one data center; the entry task u of job k can only be executed on the data center containing its input data; let Indicates whether task u is the entry task of job k. If task u is the entry task, then set Otherwise, set Thus we get: For each task of job k, the immediate successor of the task cannot start execution until the task is completed and the required intermediate results are transmitted, with: in and They represent the start time and calculation time of task u of job k in data center i, respectively. represents the data transmission time required to transfer the intermediate data generated by task u executed in data center i in job k to task v executed in data center j. During the transmission process from task u to task v in job k, tasks u and v are executed in data center i and data center j respectively. represents the start time of task v of job k in data center j, indicates whether task v of job k is assigned to data center j for execution. If task v of job k is executed in data center j, then set otherwise, 7. The backup-based cross-data center DAG job and task scheduling method according to claim 6, characterized in that: The constraints of the backup-based cross-data center DAG job completion time optimization model also include: in Indicates whether task u of job k on data center i is executed before job v of job q on data center i. If task u of job k on data center i is executed before job v of job q on data center i, then otherwise, indicates whether task v of job q is assigned to data center i. If task v of job q is executed in data center i, then set otherwise, 8. The backup-based cross-data center DAG job and task scheduling method according to claim 6, characterized in that: The constraints of the backup-based cross-data center DAG job completion time optimization model also include: Indicates whether the transmission of job k from task u to task v on the link from data center i to data center j takes precedence over the transmission of job q from task m to task n. When the transmission of job k from task u to task v on the link from data center i to data center j takes precedence over the transmission of job q from task m to task n, otherwise Indicates the start time of data transmission from task u to task v by job k: in Indicates whether task m of job q is assigned to data center i for execution. If task m of job q is executed in data center i, then let otherwise, indicates whether task n of job q is assigned to data center j. If task n of job q is executed in data center j, then set otherwise, 9. The backup-based cross-data center DAG job and task scheduling method according to claim 6, characterized in that: The computation time of task u of job k in data center i and the data transfer time required to transfer the intermediate data generated by task u in job k executed in data center i to task v executed in data center j It is expressed as follows: Among them B i,j represents the bandwidth between data center i and data center j, s i represents the number of slots in data center i, It represents the ratio of the amount of intermediate data of task u in job k to the amount of input data of task u. represents the duration of the subtask of task u, Represents the amount of input data for task u of job k.

10. The backup-based cross-data center DAG job and task scheduling method according to claim 6, characterized in that: Determine the optimal task placement and scheduling strategy for all jobs in the scheduled job sequence, including: Get the rank values ​​of all tasks of job k. The rank value of a task represents its priority. The higher the rank value of a task, the earlier it will be scheduled. The DAG graph of job k is used to construct a new DAG graph through task aggregation, where the task aggregation method is as follows: for a task in the DAG graph of job k that has only one child task, the task and its child tasks are merged into a new task; For the scheduling of entry tasks in job k, the entry tasks are divided into different groups according to the structure of the entry tasks in the new DAG graph of job k. For each group of tasks, they are scheduled from large to small according to the rank values ​​of the tasks in the group; For the scheduling of other tasks other than the entry task in job k, they are scheduled from large to small according to the rank value of the task.

Images