Cross-data-center multi-data analysis operation cost optimization method based on time delay constraint

Abstract

The invention provides a cross-data-center multi-data analysis operation cost optimization method based on time delay constraint, which comprises the following steps: acquiring a MapReduce operation set in a data center set, and dividing operations into short-delay operations and long-delay operations according to the deadline of each operation in the MapReduce operation set; for the short-delay operation, scheduling the short-delay operation by using an FIFO strategy, and minimizing the completion time of the short-delay operation by optimizing task placement of map and reduce stages of the short-delay operation; for long-delay jobs, the jobs are scheduled according to urgency and cost of the jobs, an appropriate resource allocation scheme is found, the scheme describes the quantity of resources allocated to each scheduling job on each site, and then a task placement strategy is calculated; the task placement strategy can minimize the job completion cost and meet the delay constraint of each job.

Description

Technical Field

[0001] The present invention mainly relates to the field of data processing technology, and in particular to a method for optimizing the cost of multi-data analysis operations across data centers based on delay constraints. Background Art

[0002] With the rapid development of edge computing, IoT, and 5G technologies, many Internet companies, research institutes, government agencies, and other institutions have deployed a large number of data centers around the world and deployed a large number of services on these data centers. The heterogeneity of resource capacity and prices between different data centers has brought huge challenges to the scheduling and task placement of MapReduce jobs across data centers.

[0003] In cross-data center data analysis, multiple sites are connected by a wide area network, and the bandwidth scarcity of the wide area network severely limits large-scale data transmission. General big data processing frameworks based on the MapReduce model (e.g., Spark and Hadoop) are often used to analyze cross-regional distributed data in parallel on multiple cross-regional distributed sites. These cross-data center data analysis jobs rely on the resources provided by the sites (e.g., clusters, data centers) that store the original input data. However, these sites may provide heterogeneous resources (bandwidth, computing, etc.) in terms of hardware capacity and resource prices, and the data distribution of these sites is also heterogeneous. This heterogeneity makes it impossible for general big data processing frameworks to effectively analyze cross-regional distributed data, and has a significant impact on the completion time and total cost of these cross-regional data analysis jobs. Existing scheduling strategies either only consider the total cost of the job or only optimize the completion time. Therefore, it is impossible to achieve an appropriate trade-off between the total completion time and the total cost. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention proposes a method for optimizing the cost of multi-data analysis operations across data centers based on delay constraints.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] On the one hand, the present invention provides a method for optimizing the cost of multi-data analysis operations across data centers based on latency constraints, comprising:

[0007] Get the MapReduce job set in the data center set;

[0008] According to the deadline of each job in the MapReduce job set, the jobs are divided into short-latency jobs and long-latency jobs. Each data center allocates a fixed ratio of resources to short-latency jobs and long-latency jobs.

[0009] For short-latency jobs, the FIFO strategy is used to schedule short-latency jobs and minimize the completion time of short-latency jobs by optimizing the task placement in the map and reduce phases of short-latency jobs.

[0010] For long-delay jobs, first obtain the current waiting long-delay jobs and currently available resources. The current waiting long-delay jobs include newly arrived long-delay jobs and unprocessed long-delay jobs; calculate the priority of each job based on the urgency of the job and the minimum required resource ratio; select the current scheduled job from the current waiting long-delay jobs based on the priority of the long-delay job; generate the resource configuration of the current scheduled job and the task placement strategy that meets the delay constraints and minimizes the cost under the corresponding resource configuration to achieve the lowest total cost, and continuously update the current waiting long-delay job until it ends.

[0011] On the one hand, the present invention provides a device for optimizing the cost of multi-data analysis operations across data centers based on latency constraints, comprising:

[0012] The first module is used to obtain a set of MapReduce jobs in a set of data centers;

[0013] The second module is used to classify jobs into short-latency jobs and long-latency jobs according to the deadline of each job in the MapReduce job set, and each data center allocates a fixed ratio of resources to short-latency jobs and long-latency jobs;

[0014] The third module is used to schedule short-latency jobs using the FIFO strategy and minimize the completion time of short-latency jobs by optimizing the task placement in the map and reduce phases of short-latency jobs.

[0015] The fourth module is used to first obtain the current waiting long-delay jobs and currently available resources for long-delay jobs. The current waiting long-delay jobs include newly arrived long-delay jobs and unprocessed long-delay jobs; calculate the priority of each job according to the urgency of the job and the minimum required resource ratio; select the current scheduling job from the current waiting long-delay jobs according to the priority of the long-delay job; generate the resource configuration of the current scheduling job and the task placement strategy that meets the delay constraints and minimizes the cost under the corresponding resource configuration to achieve the lowest total cost, and continuously update the current waiting long-delay job until it ends.

[0016] On the other hand, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned method for optimizing the cost of multi-data analysis operations across data centers based on delay constraints are implemented.

[0017] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for optimizing the cost of multi-data analysis operations across data centers based on latency constraints.

[0018] On the other hand, the present invention provides a computer program product, which is stored on a computer-readable storage medium and includes computer instructions, which, when executed by a processor, enable a computer device to implement the steps of the above-mentioned method for optimizing the cost of multi-data analysis operations across data centers based on latency constraints.

[0019] Compared with the prior art, the technical effects of the present invention are:

[0020] The present invention proposes an effective method to minimize the total cost of MapReduce jobs across data centers without violating their deadlines. The present invention schedules jobs according to their urgency and cost, and calculates the appropriate resource allocation ratio and task placement strategy for each job. Experimental results based on a real cluster operation data set show that compared with other methods, the method proposed in the present invention can effectively reduce the total cost and meet its latency constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0022] Figure 1 It is a flowchart of a method for optimizing the cost of multi-data analysis operations across data centers based on latency constraints provided by an embodiment;

[0023] Figure 2 This is an architecture diagram of a method for optimizing the cost of multi-data analysis operations across data centers based on latency constraints provided by an embodiment;

[0024] Figure 3 This is an example diagram of the resource sharing model;

[0025] Figure 4 This is a performance comparison chart based on different job scheduling and task placement methods based on the Google dataset. Figure 4(a) is a comparison chart of the number of legal jobs under different job scheduling and task placement methods based on the Google dataset, 4(b) is a comparison chart of the average job completion cost under different job scheduling and task placement methods based on the Google dataset, 4(c) is a comparison chart of the job completion cost of Tetrium under different job scheduling and task placement methods based on the Google dataset, 4(d) is a comparison chart of the job completion cost of MinCost under different job scheduling and task placement methods based on the Google dataset;

[0026] Figure 5 This is a performance comparison chart of different job scheduling and task placement methods based on the Alibaba dataset. Figure 5 (a) is a comparison chart of the number of legal jobs under different job scheduling and task placement methods based on the Alibaba dataset. (b) is a comparison chart of the average job completion cost under different job scheduling and task placement methods based on the Alibaba dataset. Figure 5 (c) is a comparison chart of Tetrium's job completion costs under different job scheduling and task placement methods based on the Alibaba dataset. Figure 5 (d) is a comparison chart of the job completion cost of MinCost under different job scheduling and task placement methods based on the Alibaba dataset;

[0027] Figure 6 This is a performance comparison chart of different methods under different latency constraints based on the Google dataset. Figure 6 (a) is a comparison chart of the percentage of legitimate jobs under different delay constraints based on the Google dataset. Figure 6 (b) is a comparison chart of average job completion costs under different delay constraints based on the Google dataset;

[0028] Figure 7 This is a performance comparison chart of different methods under different delay constraints based on the Alibaba dataset. Figure 7 (a) is a comparison chart of the percentage of legal jobs under different delay constraints based on the Alibaba dataset. Figure 7 (b) is a comparison chart of the average job completion cost under different delay constraints based on the Alibaba dataset;

[0029] Figure 8 The comparison chart of the number of legal jobs and the average job completion cost in different time windows is shown in Figure 2. Figure 8 (a) is a comparison chart of the number of legal jobs in different time windows. Figure 8 (b) is a comparison chart of the average job completion cost under different time windows;

[0030] Fig. 9is a comparison chart of the number of legal jobs and the average job completion cost under different θ values, Fig. 9 (a) is a comparison chart of the number of legal jobs under different θ values. Fig. 9 (b) is a comparison chart of the average job completion cost under different θ values;

[0031] Fig.10 This is a comparison chart of the number of legal jobs and the average job completion cost under different components, where Fig.10 (a) is a comparison chart of the number of legal jobs under different components. Fig.10 (b) is a comparison chart of the average job completion costs of legal jobs under different components. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] In one embodiment of the present invention, referring to Figure 1 , a method for optimizing the cost of multi-data analysis operations across data centers based on latency constraints is provided, including:

[0034] Get the MapReduce job set in the data center set;

[0035] Let D denote the set of data centers, D = {1, 2, ..., M}, and J denote the set of MapReduce jobs, J = {1, 2, ..., n}.

[0036] According to the deadline of each job in the MapReduce job set, the jobs are divided into short-latency jobs and long-latency jobs. Each data center allocates a fixed ratio of resources to short-latency jobs and long-latency jobs.

[0037] For short-latency jobs, the FIFO strategy is used to schedule short-latency jobs and minimize the completion time of short-latency jobs by optimizing the task placement in the map and reduce phases of short-latency jobs.

[0038] For long-delay jobs, first obtain the current waiting long-delay jobs and currently available resources. The current waiting long-delay jobs include newly arrived long-delay jobs and unprocessed long-delay jobs; calculate the priority of each job based on the urgency of the job and the minimum required resource ratio; select the current scheduled job from the current waiting long-delay jobs based on the priority of the long-delay job; generate the resource configuration of the current scheduled job and the task placement strategy that meets the delay constraints and minimizes the cost under the corresponding resource configuration to achieve the lowest total cost, and continuously update the current waiting long-delay job until it ends.

[0039] For the kth job in the MapReduce job set, let the kth job be at time was submitted and at the time Completion, duration of the kth job The corresponding delay constraint cannot be exceeded, that is, the kth job must be completed before its corresponding deadline T k Before completion,

[0040] Set the time window T win , if the deadline of the kth job is less than T win ×θ, then the kth job is a short-delay job, otherwise, the kth job is a delay-constrained job, where θ is a constant.

[0041] Figure 2 This is an architecture diagram of a delay-constrained cross-data center multi-data analysis job cost optimization method provided by an embodiment, including job classification, cost optimization for short-delay jobs, and cost optimization for long-delay jobs. The deadlines of data analysis jobs across data centers vary greatly. If all jobs are placed in the same queue and scheduled and a large job is being scheduled, the newly arrived job with a shorter delay may violate its delay constraint. In order to avoid this situation and improve scheduling efficiency, the jobs are divided into two queues according to their deadlines, namely short-delay jobs and long-delay jobs. A fixed proportion of cluster resources is allocated to short-delay jobs and long-delay jobs, respectively, by R s and R lIndicates that the proportion of resources allocated to each queue can be obtained based on the historical load. Short-delay jobs and long-delay jobs will be processed by the MCMG-S and MCMG-L modules respectively. For short-delay jobs, the cost of completing the job is usually relatively small, so the number of jobs that meet the delay constraints is increased as much as possible. Since the delays of short-delay jobs are similar, the FIFO strategy is used to schedule short-delay jobs. For each short-delay job, complete it as quickly as possible. Tetrium is an excellent task scheduling method in recent years. The present invention uses the Tetrium task scheduling method as a task placement method for short-delay jobs, while considering the impact of computing and network resources on data analysis across data centers, and minimizes the completion time of short-delay jobs by optimizing the task placement in the map and reduce stages of short-delay jobs.

[0042] For long-latency jobs, the MCMG-L module includes three components: job scheduling, resource allocation, and task placement. Job scheduling determines the priority of jobs in each time window. Resource allocation policy describes the amount of resources allocated to each job in each data center. Task placement generates the corresponding task placement policy, which determines the proportion of tasks allocated to each data center in the map and reduce phases of each job.

[0043] Regarding job scheduling, the present invention selects a currently scheduled job from currently waiting long-delayed jobs according to the priority of the long-delayed job, including:

[0044] Determine the minimum resource requirement ratio required for long-delay jobs;

[0045] Prioritize long-delay jobs based on their urgency and the ratio of the least resource requirements they require;

[0046] Long-delay jobs are sorted in descending order from high to low priority;

[0047] The current scheduled job is selected according to the priority and the minimum resource requirement ratio of the long-delay job. The current scheduled job is selected in descending order of priority until the currently available resources are saturated. All the current scheduled jobs constitute Q schedule gather.

[0048] Next, generate Q schedule The task placement strategy set TP corresponding to the set all , where TP all ={TP b ,b∈Q schedule}, TP b Indicates Q schedule The task placement strategy for the bth currently scheduled job in the set, TP bThe resource configuration allocated to the bth currently scheduled job and the task placement strategy that satisfies the delay constraint and minimizes the cost under the corresponding resource configuration are recorded.

[0049] The allocation of resources determines Q schedule The ratio of available resources allocated to each job in the set. The present invention proposes a resource sharing scheme to make full use of the resources of the data center (or site). A job can only be completed when it is allocated sufficient resources (network resources and computing resources). MapReduce-based jobs cannot occupy both resources at the same time. When network resources are occupied to transmit data, computing resources are idle, and vice versa. For each job being processed, you can consider trying to reallocate its idle resources to other jobs without interfering with the execution of other jobs. Figure 3 The following is an example diagram of the resource sharing mode; the blue and white squares represent the alternating use of network and computing resources, respectively. When Job 1 occupies computing resources in the map phase and releases network resources, Job 2 uses the idle network resources for its input data import process and completes this phase before the shuffle process of Job 1 begins. Similarly, when Job 1 temporarily releases computing resources, Job 2 reuses computing resources and is called the matching job of Job 1. Figure 3 The dashed box in represents the resource sharing between job 1 and job 2.

[0050] Under the resource sharing scheme, resources can be fully utilized, and matching jobs using idle resources will not affect the execution of original jobs. Figure 3 In this case, Job 2 will not affect the execution of Job 1. At the same time, this solution also takes into account the synchronization overhead caused by synchronizing and releasing the load when two jobs use resources alternately. For example, the total time of Job 2's input data import and resource synchronization cannot exceed the time of Job 1's map calculation process. Therefore, the execution of Job 1 will not be interrupted by Job 2.

[0051] Q schedule The set is selected based on priority and the least required resources. If resources are allocated to more jobs, it will cause Q schedule The jobs in Q cannot obtain the minimum required resources, thus timeout and waste the resources allocated to them. For this reason, when allocating resource configurations, the present invention also tries to find jobs that can share resources with the first batch of jobs. These jobs are called the second batch of jobs. The second batch of jobs can perform corresponding data transmission or computing operations when the network resources or computing resources used by the first batch of jobs are idle, making full use of system resources. It should be noted that not all jobs in Q schedule All jobs in the collection are given resources at the beginning. scheduleThe jobs in Q may also be divided into the second batch of jobs. For example, suppose there are 10 jobs (jobs 1, 2, ..., 10), sorted according to priority. schedule It starts with 5 jobs. If the available resources are allocated to Q schedule All the jobs in the task list may be able to complete each job before its deadline using the resource sharing mechanism. However, if only jobs 1, 2, and 3 are allocated resources, the resources obtained by jobs 1, 2, and 3 are greater than their minimum required resources, and jobs 4, 5, and 6 can also share resources with jobs 1, 2, and 3. Obviously, more jobs can be processed under the second strategy. The present invention will schedule The job with the lowest priority in the set is moved to the second batch of job queues until the number of jobs processed is maximized. For jobs not in the first batch of job queues, it is checked whether the job can use idle resources based on the job's priority.

[0052] Specifically, the present invention also includes: generating a corresponding job set P for each currently scheduled job match , in the task placement strategy set TP all In the included task placement strategy, when the i-th currently scheduled job performs a computation operation, the job set P match Each job in can effectively utilize the idle network resources of the i-th currently scheduled job for transmission; similarly, while one job is performing data transmission, another job uses the idle computing resources to perform computing operations.

[0053] In Q schedule The first batch of jobs that can be executed simultaneously are selected from the collection according to the priority and the minimum resource requirement ratio required by the long-delay jobs. At the same time, jobs that can share resources with the first batch of jobs when the first batch of jobs are executed are searched for, and the second batch of jobs with the highest idle resource utilization rate are searched for. The second batch of jobs can perform corresponding data transmission or computing operations when network resources or computing resources are idle when used by the first batch of jobs, making full use of system resources. The second batch of jobs constitutes the job set tempMatch, and the optimal job processing strategy match is calculated. The job processing strategy match records the processing strategies of all jobs included in the first batch of jobs and the second batch of jobs, including the placement plan of job tasks and the ratio of resources allocated to each job, as well as the number of jobs executed under the processing strategy tempNum and the cost consumed tempCost. The selection method of the first batch of jobs is: calculate Q schedule Each currently scheduled job in Q is added to the first batch of jobs in order of priority. The jobs with positive gains are added to the first batch of jobs. schedule Updated to the union of the first batch of jobs and the corresponding second batch of jobs.

[0054] The task placement strategy for the currently scheduled job determines the completion time and cost of the job. The task placement strategy for the job needs to determine the number of map and reduce tasks of the corresponding job placed in each data center, and determine which data center the input data for each task is obtained from. The present invention proposes that a task placement strategy that minimizes the cost can be generated while satisfying the latency constraint while taking the deadline into consideration.

[0055] For the kth job in the MapReduce job set, no matter it is a long-delay-constrained job or a short-delay-constrained job, it includes a map phase and a reduce phase. The map phase consists of two processes: input data import and map calculation. The reduce phase consists of a shuffle process and a reduce calculation process.

[0056] set up represents the amount of data stored in the ith data center by the kth job. In the map phase, let the amount of data transferred from the ith data center to the jth data center by the kth job be have Suppose the price of transmitting a unit of data is P i,j , then the cost of importing input data in the map phase of the kth job is

[0057] After the input data is imported, the data volume of the jth data center becomes The map task of the jth data center will generate intermediate data as the input data of the reduce task; represents the amount of intermediate data of the kth job in the jth data center, q k Represents the ratio of the intermediate data volume to the input data volume of the kth job, and we get:

[0058]

[0059] In the reduce phase, represents the proportion of reduce tasks executed by the k-th job in the i-th data center, and has Ensure that the reduce task of the kth job will be completed by all data centers. In order to complete the reduce task, all data centers need to exchange some intermediate data with other data centers that have related reduce tasks. Let represents the cost caused by intermediate data transmission in the reduce phase of the k-th job, Calculated by the following formula: in, represents the proportion of reduce tasks executed by the k-th job in the j-th data center, It represents the amount of intermediate data of the k-th job in the i-th data center.

[0060] The amount of input data for the map task of the kth job in the i-th data center is in represents the amount of data transferred from the jth data center to the ith data center by the kth job; after the shuffle process, the amount of data that the reduce task on the jth data center needs to process is make They represent the amount of data that each map and reduce task of the kth job can process, respectively. and They represent the number of map and reduce tasks on the i-th data center, respectively, as follows:

[0061]

[0062] Assume that each task requires a computation slot. The computation slots of the kth job in the map and reduce phases are and Let m i represents the price of a slot per unit time in the i-th data center, and Respectively represent the execution time of a map and a reduce task of the k-th job, and we get:

[0063]

[0064] In summary, for the kth job in the MapReduce job set, it includes the map phase and the reduce phase. The map phase consists of two processes: input data import and map calculation. The reduce phase consists of the shuffle process and the reduce calculation process. The total cost of the kth job is C k for:

[0065]

[0066] in They respectively represent the cost of importing input data in the map phase of the k-th job, the map computation cost in the map phase of the k-th job, the cost caused by intermediate data transmission in the reduce phase of the k-th job, and the reduce computation cost in the reduce phase of the k-th job.

[0067] For the kth job in the MapReduce job set, regardless of whether it is a long-delay-constrained job or a short-delay-constrained job, the completion time of the kth job is It is expressed as:

[0068]

[0069] in, They respectively represent the time used for importing input data in the map phase of the k-th job, the time used for map calculation in the map phase of the k-th job, the time used for intermediate data transmission in the reduce phase of the k-th job, the time used for reduce calculation in the reduce phase of the k-th job, and the waiting time of the k-th job.

[0070] Assume that the bandwidth and computing slot allocated to a job will not be preempted by other jobs before the job is completed; let represents the allocated bandwidth of the kth job from the i-th data center to the j-th data center at time t; therefore, in the map phase, the transmission time of the kth job from the i-th data center to the j-th data center is Represents the amount of data from the i-th data center to the j-th data center; in the reduce phase, the i-th data center needs to process reduce task, so the amount of data transferred from the i-th data center to the j-th data center is The transmission time from the i-th data center to the j-th data center is in and It is determined by the longest transmission time between any two data centers in the above process, and we get:

[0071]

[0072] And require The bandwidth allocated to all jobs from the i-th data center to the j-th data center cannot exceed the available bandwidth B i,j .

[0073] Since the number of tasks in a data center usually exceeds the number of available computing resources, they need to be executed in multiple batches. Suppose the computing slot allocated to the kth job in the i-th data center at time t is Then the i-th data center needs to spend and Complete map and reduce tasks in batches;

[0074] make They represent the time required for each map or reduce task of the k-th job. The map and reduce computing times of the k-th job in the ith data center are and make and represents the computation time of the kth job in the map and reduce phases, and Determined by the site that is executed last in the corresponding process, there are:

[0075]

[0076] And require At time t, the computing resources of all jobs in the ith data center cannot exceed the total computing resources S of the ith data center. i .

[0077] The problem of minimizing the cost of task placement strategy is transformed into two sub-problems, minimizing the cost of the map phase and minimizing the cost of the reduce phase. Both of these problems are linear programming problems that can be solved in polynomial time and are combined together as the minCost algorithm. The minCost algorithm is used to generate the task placement strategy with the optimal cost.

[0078] The task placement strategy that minimizes time is the task placement strategy that minimizes time, which aims to minimize the job completion time without considering the impact of resource price heterogeneity. Many research works are devoted to optimizing job completion time. Tetrium is one of the most advanced methods in this field. The present invention uses Tetrium to generate the task placement strategy that minimizes time for the job.

[0079] Assume that the kth job is judged to be a long-delay constrained job, let represents the completion time of the kth job under the minimum cost (minCost), It represents the lowest job completion time of the kth job obtained under Tetrium. Obviously, the completion time range of the kth job is

[0080] For the long-latency-constrained job selected as the current scheduled job, two sets of task placement strategies are generated by executing the Tetrium and minCost algorithms respectively. The decision variables of the task placement strategy problem for the long-latency-constrained job selected as the current scheduled job include the amount of data transferred from each data center to all other data centers during the map phase, and the proportion of reduce tasks assigned to each data center for execution.

[0081] In order to further determine the existence of a feasible solution to the job task allocation problem, check the time calculated by the Tetrium algorithm (i.e. ) has exceeded the given deadline T kIf it exceeds, it means that no feasible solution can be found for the job placement strategy problem. Then, the time for the minimization cost (minCost) algorithm to calculate is determined. Is it less than the given time T? k If it is less than, it will be returned directly. It shows that the job completion time of the feasible solution generated by the minimum cost algorithm meets the deadline T k .

[0082] If a feasible solution cannot be found through the above steps, its parameter settings must be adjusted heuristically to find a feasible solution that generates the optimal cost before the deadline. Specifically, the variable Assigned to Then, in the while loop, adjust the variable Until the job completion time is less than the deadline T k In order to quickly obtain a feasible solution, the present invention adopts a greedy strategy to update the decision variables. For example, if the current variable value is And the adjustment rate of the variable is β∈[0,1], then when or when The variable value is adjusted to Through multiple rounds of such adjustments, Eventually it will be less than T k . At the same time, the final feasible solution will be returned.

[0083] To verify the effectiveness of the method of the present invention, a large number of experiments were conducted based on real Google and Alibaba data sets to test the performance of the multi-data analysis operation cost optimization method (MCMG method) across data centers based on latency constraints provided by the present invention.

[0084] The delay-constrained cross-data center multi-data analysis job cost optimization method (MCMG method) provided by the present invention mainly includes the MCMG-S algorithm for short-delay jobs and the MCMG-L algorithm for long-delay jobs. The MCMG-L algorithm is more time-consuming than the MCMG-S algorithm. In the MCMG-L algorithm, the most time-consuming process is to calculate the task placement strategy under different resource quantities for each job that needs to be scheduled. The time complexity of the MCMG-L algorithm is O(n 3.5 L). Therefore, the time complexity of MCMG-L is Where k is the number of jobs to be scheduled, n is the number of variables, and L is the size of the problem. The number of times the task scheduling strategy is calculated for each job.

[0085] Cluster setup: This experiment considers |D|=10 cross-region data centers with different resource and price settings, and sets the capacity of network and computing resources as well as the prices of network and computing resources. The prices of network and computing resources are set according to Amazon EC2. Specifically, the bandwidth range is [0.5, 2.75] GB / s, and the bandwidth price is [0.02, 0.15] $ / GB. The number of computing slots is [100, 1000], and the price range of each slot is [4.5, 7.5] × 10-5 $ / s. The MCMG method proposed in this invention will be triggered periodically, and the default window size (T win ) is 10s. The parameter θ of job classification is set to 1.5, and the default resource adjustment ratio τ is 0.1.

[0086] Job load: The used job load is generated based on the job sizes and their location distribution in the cluster obtained from the Google cluster and Alibaba cluster job load datasets. The Google dataset collects machine, job, and task information for a cluster of 12,500 machines in a data center over a period of about one month. For each job recorded in the dataset, its arrival time, task number, input data size and distribution, and required resources are recorded. Alibaba Group released the Alibaba dataset in 2018, which contains job run records of about 4,000 machines in 8 days. Refer to Table 1 for the job input data size distribution table.

[0087] Table 1: Distribution of job input data size

[0088]

[0089] Both datasets contain a large number of jobs, and the input data volume of most jobs is relatively small. In addition, the frequency of job arrival in the Google and Alibaba datasets is analyzed, and the average interval between two consecutive jobs is about 157s and 17s, respectively. In other words, the job load on the Alibaba cluster job load dataset is much higher than that on the Google dataset.

[0090] According to the Tetrium algorithm and MinCost algorithm, the minimum completion time T of each job when using half of the cluster resources is obtained. time and the maximum completion time T cost , and according to T time and T cost To set the deadline for the job. The deadline for the job is equal to T time +((T cost -T time )×U), and U ≥ 0. U is the ratio of the cut-off times.

[0091] Comparison of job scheduling strategies:

[0092] 1) Varys: An advanced scheduler in recent years. It uses the minimum effective bottleneck first strategy to schedule coflow based on the bottleneck completion time heuristic to reduce the completion time of the job.

[0093] 2) SJF: The job with the shortest completion time will be scheduled first, and the scheduled job will occupy all resources.

[0094] 3) EDF (Earliest Deadline First): The job with the least remaining time will be executed first. Similarly, this job will occupy all resources.

[0095] 4) Capacity: A popular scheduling principle used in the YARN cluster resource manager. Each hierarchical queue is scheduled based on the FIFO principle. According to the task data of each job, this section divides the jobs into three queues (small job queue, medium-sized job queue, and large job queue), and each queue occupies a certain proportion of cluster resources.

[0096] 5) Fair: Idle resources are evenly distributed to the first few jobs in the waiting queue. The resources allocated to each job cannot be less than one-tenth of the site resources. Jobs are queued according to their arrival time.

[0097] Comparison of task placement strategies:

[0098] 1) Tetrium: An excellent task placement method in recent years. This method aims to optimize the completion time of a job by optimizing the placement of map and reduce tasks.

[0099] 2) MinCost: This method is used to minimize the execution cost of a single job. It transmits data along low-cost links and executes tasks in low-cost slots as much as possible.

[0100] 3) InPlace: This method is a widely accepted task placement method in the MapReduce framework. It executes map tasks at the site where the data is located and executes the same proportion of reduce tasks in each data center.

[0101] Experimental Metrics: The following two experimental metrics are used to measure the performance of different methods:

[0102] 1) Number of legal jobs: the total number of jobs that can meet the delay constraint. A job that can meet the delay constraint is considered a legal job.

[0103] 2) Average job completion cost: the total cost of the job divided by the number of jobs that can meet the delay constraint.

[0104] Performance Analysis:

[0105] Firstly, the MCMG method is compared with the traditional job scheduling method, and the number of legal jobs and the average job completion cost under different methods are compared. The compared job scheduling methods use Tetrium, MinCost or InPlace method as their task placement method. Figure 4 and Figure 5 The performance results are based on the Google and Alibaba datasets respectively. Figure 4 This is a performance comparison chart based on different job scheduling and task placement methods based on the Google dataset. Figure 4 (a) is a comparison chart of the number of legal jobs under different job scheduling and task placement methods based on the Google dataset. Figure 4 (b) is a comparison chart of the average job completion cost under different job scheduling and task placement methods based on the Google dataset. Figure 4 (c) is a comparison chart of the job completion cost of Tetrium under different job scheduling and task placement methods based on the Google dataset. Figure 4 (d) is a comparison chart of the job completion cost of MinCost under different job scheduling and task placement methods based on the Google dataset. Figure 5 This is a performance comparison chart of different job scheduling and task placement methods based on the Alibaba dataset. Figure 5 (a) is a comparison chart of the number of legal jobs under different job scheduling and task placement methods based on the Alibaba dataset. (b) is a comparison chart of the average job completion cost under different job scheduling and task placement methods based on the Alibaba dataset. Figure 5 (c) is a comparison chart of Tetrium's job completion costs under different job scheduling and task placement methods based on the Alibaba dataset. Figure 5 (d) is a comparison chart of the job completion cost of MinCost under different job scheduling and task placement methods based on the Alibaba dataset. The ratio U of the deadlines of Google and Alibaba is 0.2 and 0.9 respectively. Figure 4 (a) and Figure 5 (a) shows the number of jobs that meet the delay constraint under different methods and the ratio normalized to this method. Figure 4 (b) and Figure 5 (b) The average job completion cost of different methods and the ratio normalized to MCMG are recorded.

[0106] like Figure 4As shown in (a), when the compared job scheduling methods adopt MinCost algorithm as their task scheduling strategy, MCMG can ensure that more jobs meet the delay constraints compared with the compared methods. The reason is that MCMG schedules jobs according to their urgency, and the task placement strategy of each job takes the delay constraints of the job into consideration. Therefore, MCMG can ensure that more jobs meet their delay constraints. When the compared job scheduler adopts Tetrium as its task placement method, the number of jobs that can meet the delay constraints under the MCMG method is slightly less than that under Varys, Capacity, and Fair. However, the average job completion cost under the Varys, Capacity, and Fair methods is much higher than that under MCMG. This is because Tetrium can minimize the job completion time by optimizing the placement of map and reduce tasks, but it will result in a high job completion cost. When the InPlace method is adopted, the number of legal jobs under different job scheduling methods is less than the number of legal jobs under the MCMG method. In other words, compared with other methods that adopt the InPlace algorithm as their task placement strategy, MCMG can provide better performance guarantees.

[0107] like Figure 4 (b) and Figure 5 As shown in (b), the job completion costs of the comparison methods are all lower than the cost incurred by MCMG. In particular, based on the Alibaba dataset, MCMG can reduce the total cost by at least 16% compared to other comparison methods. The results show that the method proposed in this chapter can significantly reduce the cost of MapReduce jobs across data centers. In addition, Figure 5 (a), this section tests the number of legal jobs under different methods based on the Alibaba dataset. It can be found that due to the large amount of data that needs to be processed by the cluster per unit time, no matter what task placement method is used, MCMG can achieve better performance than other comparison methods. This is because MCMG can make appropriate task placement strategies based on the processing time of its remaining jobs before its deadline. However, traditional job schedulers do not consider the latency constraints of jobs, so many jobs are called but violate their deadlines. Therefore, based on the Alibaba dataset, the performance results under traditional job schedulers are lower than the performance that MCMG can achieve.

[0108] Figure 4(c), 4(d), 5(c), and 5(d) show the CDF of the job completion cost that satisfies the latency constraint compared to the comparison methods. When the job scheduler uses Tetrium as its task placement method, MCMG does not cause any job to incur higher costs compared to the other comparison methods, even though MCMG is a heuristic method. When the comparison method uses the MinCost job task placement strategy, the job completion cost caused by MCMG is very close to the job completion cost under the other comparison methods. This is because MCMG selects the resource allocation method that minimizes the cost, and the task placement strategy of each job can minimize the job completion cost before its deadline. Therefore, compared to the other comparison methods, the total cost caused by MCMG is low, and MCMG can enable more jobs to meet the latency constraint.

[0109] Next, Table 2 shows the running time and performance comparison of the optimal algorithm and MCMG under different sites and job numbers. The results are shown in Table 2:

[0110] Table 2 Performance comparison between the optimal algorithm and MCMG

[0111]

[0112] Among them, the optimal algorithm enumerates all possible solutions (job scheduling, resource allocation, and task placement) and obtains the optimal solution from them. Compared with the optimal solution, the difference between the number of legal job data and the average job completion cost under the MCMG method is 0.05 and 0.09 respectively. However, it can be found that under different site scales and job numbers, the optimal algorithm takes tens of seconds to hours to obtain the solution to the problem, while MCMG only takes a few hundred milliseconds to calculate the scheduling solution.

[0113] Job Urgency and Resources: We first evaluate the impact of job urgency and cluster resource ratio on short-latency and long-latency jobs. Figure 6 and Figure 7 Shows the deadline and resource ratio of the job (R s ,R l ) changes, the MCMG method is based on the average job completion time and the number of legal jobs in the Google and Alibaba datasets respectively. Figure 6 (a) and Figure 6 (b) shows the experimental results based on the Google dataset. Figure 7 (a) and 7(b) show the experimental results based on the Alibaba dataset.

[0114] Figure 6 This is a performance comparison chart of different methods under different latency constraints based on the Google dataset. Figure 6(a) is a comparison chart of the percentage of legitimate jobs under different delay constraints based on the Google dataset. Figure 6 (b) is a comparison chart of the average job completion cost under different delay constraints based on the Google dataset. Figure 7 This is a performance comparison chart of different methods under different delay constraints based on the Alibaba dataset. Figure 7 (a) is a comparison chart of the percentage of legal jobs under different delay constraints based on the Alibaba dataset. Figure 7 (b) is a comparison chart of the average job completion cost under different delay constraints based on the Alibaba dataset.

[0115] like Figure 6 As shown in (a), except when U = 2, the number of legal jobs increases with the increase of deadline. s When the range of R is [0.2, 0.5], MCMG can make 91% of the jobs meet the delay constraint. The reason is that when U = 2, the number of jobs in the short delay queue and the long delay queue is appropriate. The system can make a good balance between the job completion time and cost. However, when R s = 0.1 and U = 2, the number of long-delay queues is too large. Therefore, due to the lack of cluster resources, the number of jobs that the short-delay job module can process is limited. On the contrary, the cluster resources for long-delay jobs may be idle. Therefore, when R s =0.1, its U=2, and the number of jobs completed before the deadline is very small.

[0116] Figure 7 (a) records the number of legitimate jobs based on the Alibaba dataset. It can be found that the number of legitimate jobs increases with the increase of U or the proportion of short-delay queues. This phenomenon can be explained as follows: Alibaba's workload contains many short-delay jobs, that is, the deadline of the job is lower than T. win ×1.5. Therefore, when R s When it increases, MCMG can handle more short-latency jobs, and these short-latency jobs can be processed quickly. Generally, for different loads, setting appropriate loads for the two queues is very important to improve the performance of the system.

[0117] Figure 6 (b) and Figure 7(b) shows the average job completion cost when the deadline and the resource ratio of the job change. It can be seen that as U increases, the average job completion cost decreases. This is because the deadline of the job becomes longer and the system is able to choose cheaper resources to process these jobs. In addition, when U>0.2, based on the Google dataset, the difference in the average job completion cost under different resource allocation ratios is small. This phenomenon can be explained as follows: as R s As R increases, the number of short-delay jobs increases, and short-delay jobs that use Tetrium as their task placement method will result in high job completion costs. Therefore, the total job completion cost of short-delay jobs increases with R s In addition, when U becomes larger, MCMG-L is able to select cheaper resources to handle long-delay jobs, and the total cost of long-delay jobs will decrease. Therefore, based on the Google dataset, the average job completion cost of all jobs changes less under different resource allocation ratios and job urgency.

[0118] In addition, based on the Google dataset, when R s When U is equal to 0.4 or 0.5, the number of legal jobs and the average job completion cost are almost the same under different U values. Therefore, as U increases, the total cost of long-delay jobs and the number of legal jobs are increasing. However, the total cost of short-delay jobs and the number of legal jobs will decrease. Finally, the total cost of all legal jobs and the number of legal jobs can achieve an equilibrium.

[0119] Time Window: The average job completion cost and the number of legal jobs under different time windows were tested, such as Figure 8 As shown, Figure 8 The comparison chart of the number of legal jobs and the average job completion cost in different time windows is shown in Figure 2. Figure 8 (a) is a comparison chart of the number of legal jobs in different time windows. Figure 8 (b) is a comparison chart of the average job completion cost under different time windows. It can be found that as the time window increases, the number of legal jobs decreases, but the average job completion cost increases. The reason is that a longer time window will result in more waiting jobs, which will waste more time before they are scheduled. In each iteration, each scheduler can only process a certain number of jobs, and the scheduled jobs usually have less completion time. Therefore, they have to use more expensive resources. After the scheduled job is completed, the system resources may be idle for a long time. Therefore, a shorter time window can reduce the total cost while allowing more jobs to meet the delay constraints.

[0120] Job classification parameters: Based on the Google dataset, we then tested the effect of the job classification parameter θ on the average job completion cost and the number of legal jobs. The results are shown in the figure. Fig. 9 is a comparison chart of the number of legal jobs and the average job completion cost under different θ values, Fig. 9 (a) is a comparison chart of the number of legal jobs under different θ values. Fig. 9 (b) is a comparison chart of the average job completion cost under different θ values. It can be found that as θ increases, the average job completion cost increases, but the number of legal jobs increases at first and then decreases. When θ is set to 1.5, the number of legal jobs is high and the average job completion cost is low. Therefore, the default θ value is set to 1.5.

[0121] System Components: Fig.10 This is a comparison chart of the number of legal jobs and the average job completion cost under different components, where Fig.10 (a) is a comparison chart of the number of legal jobs under different components. Fig.10 (b) is a comparison chart of the average job completion cost of legal jobs under different components. The impact of the job scheduler (JS) in MCMG-L, the impact of the three components of resource allocation (RA) and task placement (TP), which are replaced by FIFO, minimum resources to meet delays, and Tetrium, respectively. In addition, the impact of not calling the module for short-delay job scheduling is tested. When the experiment was conducted based on the Google dataset, the percentage of jobs that met the delay data under different components changed little, but the average job completion cost under different components was higher than the cost under MCMG. Based on the Alibaba dataset, the number of jobs that met the delay constraints using the LS+JS mode was greater than the result under MCMG. However, the average job completion cost caused by LS+JS is higher than the cost using MCMG.

[0122] On the other hand, an embodiment provides a device for optimizing the cost of multi-data analysis operations across data centers based on latency constraints, including:

[0123] The first module is used to obtain a set of MapReduce jobs in a set of data centers;

[0124] The second module is used to classify jobs into short-latency jobs and long-latency jobs according to the deadline of each job in the MapReduce job set, and each data center allocates a fixed ratio of resources to short-latency jobs and long-latency jobs;

[0125] The third module is used to schedule short-latency jobs using the FIFO strategy and minimize the completion time of short-latency jobs by optimizing the task placement in the map and reduce phases of short-latency jobs.

[0126] The fourth module is used to first obtain the current waiting long-delay jobs and currently available resources for long-delay jobs. The current waiting long-delay jobs include newly arrived long-delay jobs and unprocessed long-delay jobs; calculate the priority of each job according to the urgency of the job and the minimum required resource ratio; select the current scheduling job from the current waiting long-delay jobs according to the priority of the long-delay job; generate the resource configuration of the current scheduling job and the task placement strategy that meets the delay constraints and minimizes the cost under the corresponding resource configuration to achieve the lowest total cost, and continuously update the current waiting long-delay job until it ends.

[0127] On the other hand, the present invention provides a computer device, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the multi-data analysis operation cost optimization method based on delay constraints across data centers provided in any of the above embodiments when executing the computer program. The computer device can be a server. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store sample data. The network interface of the computer device is used to communicate with an external terminal through a network connection.

[0128] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for optimizing the cost of multi-data analysis operations across data centers based on latency constraints provided in any of the above-mentioned embodiments.

[0129] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0130] Matters not covered by the present invention are known technologies.

[0131] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cost optimization method for multi-data analysis operations across data centers based on latency constraints, characterized in that: include: Get the MapReduce job set in the data center set; According to the deadline of each job in the MapReduce job set, the jobs are divided into short-latency jobs and long-latency jobs. Each data center allocates a fixed ratio of resources to short-latency jobs and long-latency jobs. For short-latency jobs, the FIFO strategy is used to schedule short-latency jobs and minimize the completion time of short-latency jobs by optimizing the task placement in the map and reduce phases of short-latency jobs. For long-delay jobs, first obtain the current waiting long-delay jobs and currently available resources. The current waiting long-delay jobs include newly arrived long-delay jobs and unprocessed long-delay jobs; calculate the priority of each job based on the urgency of the job and the minimum required resource ratio; select the current scheduling job from the current waiting long-delay jobs based on the priority of the long-delay job; Generate the resource configuration of the current scheduled job and the task placement strategy that satisfies the delay constraint and minimizes the cost under the corresponding resource configuration to achieve the lowest total cost, and continuously update the current waiting long-delay job until it ends.

2. The method for optimizing the cost of multi-data analysis operations across data centers based on latency constraints according to claim 1 is characterized in that: For the kth job in the MapReduce job set, let the kth job be at time was submitted and at the time Completion, duration of the kth job The corresponding delay constraint cannot be exceeded, that is, the kth job must be completed before its corresponding deadline T k Before completion, Set the time window T win , if the deadline of the kth job is less than T win ×θ, then the kth job is a short-delay job, otherwise, the kth job is a delay-constrained job, where θ is a constant.

3. The method for optimizing the cost of multi-data analysis operations across data centers based on latency constraints according to claim 2 is characterized in that: The currently scheduled job is selected from the currently waiting long-delay jobs according to the priority of the long-delay job, including: Determine the minimum resource requirement ratio required for long-delay jobs; Prioritize long-delay jobs based on their urgency and the ratio of the least resource requirements they require; Long-delay jobs are sorted in descending order from high to low priority; The current scheduled job is selected according to the priority and the minimum resource requirement ratio of the long-delay job. The current scheduled job is selected in descending order of priority until the currently available resources are saturated. All the current scheduled jobs constitute Q schedule gather.

4. The method for optimizing the cost of multi-data analysis operations across data centers based on delay constraints according to claim 3 is characterized in that: Also includes: Generate a corresponding job set P for each currently scheduled job match , in the task placement strategy set TP all In the included task placement strategy, when the i-th currently scheduled job performs a computation operation, the job set P match Each job in can effectively use the idle network resources of the i-th currently scheduled job for transmission; similarly, while one job is performing data transmission, another job uses the idle computing resources to perform computing operations; In Q schedule The first batch of jobs that can be executed simultaneously are selected from the set according to the priority and the minimum resource requirement ratio required by the long-delay jobs. At the same time, jobs that can share resources with the first batch of jobs when the first batch of jobs are executed are found, and the second batch of jobs with the highest idle resource utilization rate can use the idle resources during the execution of the first batch of jobs to perform related operations. The second batch of jobs constitutes the job set tempMatch, and the optimal job processing strategy match is calculated. The job processing strategy match records the processing strategies of all jobs included in the first and second batches of jobs, including the placement plan of job tasks and the resource ratio allocated to each job, as well as the number of jobs executed under the processing strategy tempNum and the cost tempCost consumed. The selection method of the first batch of jobs is: calculate Q schedule Each currently scheduled job in Q is added to the first batch of jobs in order of priority. The jobs with positive gains are added to the first batch of jobs. schedule Updated to the union of the first batch of jobs and the corresponding second batch of jobs.

5. The method for optimizing the cost of multi-data analysis operations across data centers based on delay constraints according to claim 3 or 4, characterized in that: The set of MapReduce jobs in the data center set, where D represents the data center set, D = {1, 2, ..., M}, and J represents the MapReduce job set, J = {1, 2, ..., n}; for the k-th job in the MapReduce job set, it includes the map phase and the reduce phase. The map phase consists of two processes: input data import and map calculation. The reduce phase consists of the shuffle process and the reduce calculation process. The total cost C of the k-th job is k for: in They respectively represent the cost of importing input data in the map phase of the k-th job, the map computation cost in the map phase of the k-th job, the cost caused by intermediate data transmission in the reduce phase of the k-th job, and the reduce computation cost in the reduce phase of the k-th job.

6. The method for optimizing the cost of multi-data analysis operations across data centers based on latency constraints according to claim 5 is characterized in that: represents the amount of data stored in the i-th data center by the k-th job. In the map phase, let the amount of data transferred from the i-th data center to the j-th data center by the k-th job be have Assume the price of transmitting a unit of data is P i,j , then the cost of importing input data in the map phase of the kth job is 7. The method for optimizing the cost of multi-data analysis operations across data centers based on latency constraints according to claim 5 is characterized in that: After the input data is imported, the data volume of the jth data center becomes The map task of the jth data center will generate intermediate data as the input data of the reduce task; represents the amount of intermediate data of the k-th job in the j-th data center, and qk represents the ratio of the amount of intermediate data to the amount of input data of the k-th job, and we get: In the reduce phase, represents the proportion of reduce tasks executed by the k-th job in the i-th data center, and has Ensure that the reduce task of the kth job will be completed by all data centers. In order to complete the reduce task, all data centers need to exchange some intermediate data with other data centers that have related reduce tasks. Let represents the cost caused by intermediate data transmission in the reduce phase of the k-th job, Calculated by the following formula: in, represents the proportion of reduce tasks executed by the k-th job in the j-th data center, It represents the amount of intermediate data of the k-th job in the o-th data center.

8. The method for optimizing the cost of multi-data analysis operations across data centers based on latency constraints according to claim 5 is characterized in that: The amount of input data for the map task of the kth job in the i-th data center is in represents the amount of data transferred from the jth data center to the ith data center by the kth job; after the shuffle process, the amount of data that the reduce task on the jth data center needs to process is make They represent the amount of data that each map and reduce task of the kth job can process, respectively. and They represent the number of map and reduce tasks on the i-th data center, respectively, as follows: Assume that each task requires a computation slot. The computation slots of the kth job in the map and reduce phases are and Let m i represents the price of a slot per unit time in the i-th data center, and Respectively represent the execution time of a map and a reduce task of the k-th job, and we get:

9. The method for optimizing the cost of multi-data analysis operations across data centers based on latency constraints according to any one of claims 6 to 8, characterized in that: Completion time of the kth job It is expressed as: in, They respectively represent the time used for importing input data in the map phase of the k-th job, the time used for map calculation in the map phase of the k-th job, the time used for intermediate data transmission in the reduce phase of the k-th job, the time used for reduce calculation in the reduce phase of the k-th job, and the waiting time of the k-th job.

10. The method for optimizing the cost of multi-data analysis operations across data centers based on latency constraints according to claim 9, characterized in that: Assume that the bandwidth and computing slot allocated to a job will not be preempted by other jobs before the job is completed; let represents the allocated bandwidth of the kth job from the i-th data center to the j-th data center at time t; therefore, in the map phase, the transmission time of the kth job from the i-th data center to the j-th data center is Represents the amount of data from the i-th data center to the j-th data center; in the reduce phase, the i-th data center needs to process reduce task, so the amount of data transferred from the i-th data center to the j-th data center is The transmission time from the i-th data center to the j-th data center is in and It is determined by the longest transmission time between any two data centers in the above process, and we get: And require The bandwidth allocated to all jobs from the i-th data center to the j-th data center cannot exceed the available bandwidth B i,j ; Since the number of tasks in a data center usually exceeds the number of available computing resources, they need to be executed in multiple batches. Suppose the computing slot allocated to the kth job in the i-th data center at time t is Then the i-th data center needs to spend and Complete map and reduce tasks in batches; make They represent the time required for each map or reduce task of the k-th job. The map and reduce computing times of the k-th job in the ith data center are and make and represents the computation time of the kth job in the map and reduce phases, and Determined by the site that is executed last in the corresponding process, there are: And require At time t, the computing resources of all jobs in the ith data center cannot exceed the total computing resources S of the ith data center. i .

Images