A method for cross-data center multi-data analysis job cost optimization based on delay constraint

Abstract

The application provides a cross-data-center multi-data analysis job cost optimization method based on a time delay constraint, which comprises the following steps: obtaining a MapReduce job set in a data center set; according to the deadlines of each job in the MapReduce job set, the jobs are divided into short-delay jobs and long-delay jobs; for the short-delay jobs, a FIFO strategy is used to schedule the short-delay jobs, and the completion time of the short-delay jobs is minimized by optimizing the task placement of the map and reduce stages of the short-delay jobs; for the long-delay jobs, the jobs are scheduled according to the urgency and cost of the jobs, and a suitable resource allocation scheme is found, which describes the amount of resources allocated to each scheduled job on each site, and then a task placement strategy is calculated, which can minimize the job completion cost and meet the time delay constraint of each job.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a multi-data analysis job cost optimization method based on delay constraint across data centers. BACKGROUND

[0002] With the rapid development of edge computing, Internet of Things and 5G technologies, many Internet companies, research institutions and government agencies and other institutions have deployed a large number of data centers around the world, and a large number of services are deployed on these data centers. The heterogeneity of resource capacity and price between different data centers brings great challenges to the scheduling and task placement of MapReduce jobs across data centers.

[0003] In cross-data-center data analysis, multiple sites are connected through a wide area network, and the bandwidth scarcity of the wide area network severely limits large-scale data transmission. General-purpose big data processing frameworks based on the MapReduce model (such as spark and Hadoop) are often used to analyze cross-regionally distributed data in parallel on multiple cross-regionally distributed sites. These cross-data-center data analysis jobs rely on the resources provided by the sites (such as clusters, data centers) that store the original input data. However, these sites may provide heterogeneous resources (bandwidth, computing, etc.) in terms of hardware capacity and resource price, and the data distribution of these sites is also heterogeneous. This heterogeneity makes it impossible for general-purpose big data processing frameworks to effectively analyze cross-regionally distributed data, and has a significant impact on the completion time and total cost of these cross-regionally distributed data analysis jobs. Existing scheduling strategies either only consider the total cost of the job or only optimize the completion time. Therefore, it is impossible to achieve a proper trade-off between the total completion time and the total cost. SUMMARY

[0004] In view of the problems existing in the prior art, the present application provides a multi-data analysis job cost optimization method based on delay constraint across data centers.

[0005] To achieve the above object, the technical scheme adopted by the present application is as follows:

[0006] On the one hand, the present application provides a multi-data analysis job cost optimization method based on delay constraint across data centers, comprising:

[0007] obtaining a set of MapReduce jobs in a set of data centers;

[0008] According to the deadlines of each job in the set of MapReduce jobs, the jobs are divided into short-delay jobs and long-delay jobs, and each data center allocates a fixed proportion of resources to short-delay jobs and long-delay jobs;

[0009] For short-delay jobs, a FIFO strategy is used to schedule them, and the completion time of short-delay jobs is minimized by optimizing the task placement in the map and reduce phases.

[0010] For long-delayed jobs, first obtain the currently waiting long-delayed jobs and the currently available resources. The currently waiting long-delayed jobs include newly arrived long-delayed jobs and unprocessed long-delayed jobs. Calculate the priority of each job based on its urgency and the minimum resource ratio required. Select the current scheduled job from the currently waiting long-delayed jobs according to the priority of the long-delayed jobs. Generate the resource configuration of the current scheduled job and the corresponding task placement strategy that satisfies the latency constraint and minimizes the cost under the resource configuration to achieve the lowest total cost. Continuously update the currently waiting long-delayed jobs until the process ends.

[0011] On one hand, the present invention provides a device for optimizing the cost of multi-data center analysis operations across data centers based on latency constraints, comprising:

[0012] The first module is used to obtain the MapReduce job set from the data center set;

[0013] The second module is used to divide the jobs into short-delay jobs and long-delay jobs according to the deadline of each job in the MapReduce job set, and each data center allocates a fixed proportion of resources to short-delay jobs and long-delay jobs.

[0014] The third module is used to schedule short-delay jobs using a FIFO strategy, minimizing the completion time of short-delay jobs by optimizing the task placement in the map and reduce phases.

[0015] The fourth module, for long-delay jobs, first obtains the currently waiting long-delay jobs and the currently available resources. The currently waiting long-delay jobs include newly arrived long-delay jobs and unprocessed long-delay jobs. It calculates the priority of each job based on its urgency and the minimum resource ratio required. Based on the priority of the long-delay jobs, it selects the current scheduled job from the currently waiting long-delay jobs. It generates the resource configuration of the current scheduled job and the corresponding task placement strategy that meets the latency constraints and minimizes the cost under the resource configuration to achieve the lowest total cost. It continuously updates the currently waiting long-delay jobs until the process ends.

[0016] On the other hand, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method for optimizing the cost of multi-data analysis jobs across data centers based on latency constraints.

[0017] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method for optimizing the cost of multi-data analysis jobs across data centers based on latency constraints.

[0018] On the other hand, the present invention provides a computer program product stored on a computer-readable storage medium and including computer instructions that, when executed by a processor, cause a computer device to implement the steps of the above-described method for optimizing the cost of multi-data analysis jobs across data centers based on latency constraints.

[0019] Compared with the prior art, the technical effects of the present invention are as follows:

[0020] This invention proposes an effective method to minimize the total cost of MapReduce jobs across data centers without violating their deadlines. The invention schedules jobs based on their urgency and cost, calculating appropriate resource allocation ratios and task placement strategies for each job. Experimental results based on real-world cluster runtime datasets demonstrate that, compared to other methods, the proposed method effectively reduces total cost while meeting latency constraints. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a method for optimizing the cost of multi-data center analysis operations based on latency constraints, provided in one embodiment.

[0023] Figure 2 This is an architecture diagram of a method for optimizing the cost of multi-data center analysis operations across data centers based on latency constraints, provided in one embodiment.

[0024] Figure 3 This is an example diagram of a resource sharing model;

[0025] Figure 4 This is a performance comparison chart based on different job scheduling and task placement methods using the Google dataset. Figure 4(a) is a comparison chart of the number of valid jobs under different job scheduling and task placement methods based on the Google dataset; 4(b) is a comparison chart of the average job completion cost under different job scheduling and task placement methods based on the Google dataset; 4(c) is a comparison chart of the job completion cost of Tetrium under different job scheduling and task placement methods based on the Google dataset; 4(d) is a comparison chart of the job completion cost of MinCost under different job scheduling and task placement methods based on the Google dataset.

[0026] Figure 5 This is a performance comparison chart based on different job scheduling and task placement methods using the Alibaba dataset. Figure 5 (a) is a comparison chart of the number of valid jobs under different job scheduling and task placement methods based on the Alibaba dataset; (b) is a comparison chart of the average job completion cost under different job scheduling and task placement methods based on the Alibaba dataset. Figure 5 (c) is a comparison chart of Tetrium's job completion costs under different job scheduling and task placement methods based on the Alibaba dataset. Figure 5 (d) is a comparison chart of job completion costs under different job scheduling and task placement methods based on the Alibaba dataset.

[0027] Figure 6 This is a performance comparison chart of different methods under different latency constraints based on the Google dataset. Figure 6 (a) is a comparison chart of the percentage of valid jobs under different latency constraints based on the Google dataset. Figure 6 (b) is a comparison chart of average job completion costs under different latency constraints based on the Google dataset;

[0028] Figure 7 This is a performance comparison chart of different methods under different latency constraints based on the Alibaba dataset. Figure 7 (a) is a comparison chart of the percentage of valid jobs under different latency constraints based on the Alibaba dataset. Figure 7 (b) is a comparison chart of average job completion costs under different time delay constraints based on the Alibaba dataset;

[0029] Figure 8 This chart compares the number of legal jobs and the average job completion cost under different time windows. Figure 8 (a) is a comparison chart of the number of legal jobs under different time windows. Figure 8 (b) A comparison chart of average task completion costs under different time windows;

[0030] Figure 9A comparison chart showing the number of legal jobs and average job completion cost under different θ values. Figure 9 (a) is a comparison chart of the number of valid jobs under different θ values. Figure 9 (b) is a comparison chart of average task completion costs under different θ values;

[0031] Figure 10 A comparison chart of the number of legal jobs and average job completion cost under different components, where Figure 10 (a) is a comparison chart of the number of valid jobs under different components. Figure 10 (b) is a comparison chart of the average job completion cost of legal jobs under different components. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0033] In one embodiment of the present invention, reference is made to... Figure 1 This paper provides a method for optimizing the cost of multi-datacenter data analytics jobs across data centers based on latency constraints, including:

[0034] Retrieve the collection of MapReduce jobs from the data center set;

[0035] Let D represent the set of data centers, D = {1, 2, ..., M}, and J represent the set of MapReduce jobs, J = {1, 2, ..., n}.

[0036] Based on the deadline of each job in the MapReduce job set, the jobs are divided into short-delay jobs and long-delay jobs, and each data center allocates a fixed proportion of resources to short-delay jobs and long-delay jobs.

[0037] For short-delay jobs, a FIFO strategy is used to schedule them, and the completion time of short-delay jobs is minimized by optimizing the task placement in the map and reduce phases.

[0038] For long-delayed jobs, first obtain the currently waiting long-delayed jobs and the currently available resources. The currently waiting long-delayed jobs include newly arrived long-delayed jobs and unprocessed long-delayed jobs. Calculate the priority of each job based on its urgency and the minimum resource ratio required. Select the current scheduled job from the currently waiting long-delayed jobs according to the priority of the long-delayed jobs. Generate the resource configuration of the current scheduled job and the corresponding task placement strategy that satisfies the latency constraint and minimizes the cost under the resource configuration to achieve the lowest total cost. Continuously update the currently waiting long-delayed jobs until the process ends.

[0039] For the k-th job in the MapReduce job set, let the k-th job be in time... Submitted, and in time Complete, duration of the k-th task It cannot exceed its corresponding time delay constraint, that is, the k-th job must be completed by its corresponding deadline T. k Completed before, with

[0040] Set time window T win If the deadline for the k-th job is less than T win If ×θ, then the k-th job is a short-delay job; otherwise, the k-th job is a delay-constrained job, where θ is a constant.

[0041] Figure 2 This is an architecture diagram of a method for optimizing the cost of multi-datacenter data analytics jobs across data centers based on latency constraints, provided in one embodiment. It includes job classification, cost optimization for short-latency jobs, and cost optimization for long-latency jobs. The deadlines for data analytics jobs across data centers vary significantly. If all jobs are placed in the same queue for scheduling, and a large job is already being scheduled, newly arriving jobs with shorter latency may violate their latency constraints. To avoid this and improve scheduling efficiency, jobs are divided into two queues based on their deadlines: short-latency jobs and long-latency jobs. A fixed proportion of cluster resources is allocated to short-latency jobs and long-latency jobs respectively, and each is managed by R... s and R lThis indicates that the resource allocation ratio for each queue can be obtained based on historical load. Short-latency jobs and long-latency jobs are processed by the MCMG-S and MCMG-L modules, respectively. For short-latency jobs, the completion cost is usually relatively small, so the number of jobs satisfying latency constraints is maximized. Since the latency of short-latency jobs is similar, a FIFO strategy is used to schedule them. For each short-latency job, it is optimized to complete as quickly as possible. Tetrium is an excellent task scheduling method in recent years. This invention utilizes the Tetrium task scheduling method as the task placement method for short-latency jobs, while considering the impact of computing and network resources on cross-datacenter data analysis. By optimizing the task placement in the map and reduce phases of short-latency jobs, the completion time of short-latency jobs is minimized.

[0042] For long-latency jobs, the MCMG-L module comprises three components: job scheduling, resource allocation, and task placement. Job scheduling determines the priority of jobs within each time window. The resource allocation strategy describes the amount of resources allocated to each job in each data center. Task placement generates a corresponding task placement strategy, determining the proportion of tasks allocated to each data center for the map and reduce phases of each job.

[0043] Regarding job scheduling, this invention selects the currently scheduled job from among the currently waiting long-delay jobs based on the priority of long-delay jobs, including:

[0044] Determine the minimum resource requirements for long-delayed tasks;

[0045] The priority of long-delayed tasks is determined based on the ratio of their urgency to the minimum resource requirements.

[0046] Long-delayed jobs are sorted in descending order of priority;

[0047] The currently scheduled job is selected based on priority and the minimum resource requirement ratio for long-latency jobs, in descending order of priority, until the currently available resources are saturated. All currently scheduled jobs constitute Q. schedule gather.

[0048] Next, generate Q. schedule The set of task placement strategies (TP) corresponding to the set all , of which TP all ={TP b ,b∈Q schedule}, TP b Q represents schedule The task placement strategy (TP) for the b-th currently scheduled job in the set. bIt records the resource configuration allocated to the b-th currently scheduled job and the task placement strategy that satisfies the latency constraint and minimizes the cost under the corresponding resource configuration.

[0049] Resource allocation determines Q schedule This invention proposes a resource-sharing scheme to fully utilize the resources of a data center (or site), allocating a certain proportion of available resources to each job within the set. A job can only complete when it is allocated sufficient resources (network resources and computing resources). MapReduce-based jobs cannot simultaneously occupy both types of resources. When network resources are used to transmit data, computing resources are idle, and vice versa. For each job currently being processed, consideration can be given to reallocating its idle resources to other jobs without interfering with the execution of other jobs. Figure 3 The diagram illustrates a resource-sharing model; the blue and white squares represent the alternating use of network and computing resources, respectively. When Job 1 is in the map phase and occupies computing resources while releasing network resources, Job 2 uses the idle network resources for its input data import process and completes this phase before Job 1's shuffle process begins. Similarly, when Job 1 temporarily releases computing resources, Job 2 reuses those resources; Job 2 is referred to as the matching job of Job 1. Figure 3 The dashed box in the figure represents resource sharing between assignment 1 and assignment 2.

[0050] Under a resource-sharing scheme, resources can be fully utilized, and matching jobs using idle resources will not affect the execution of original jobs. In other words, in... Figure 3 In this approach, Job 2 will not affect the execution of Job 1. Furthermore, the scheme also considers the synchronization overhead incurred by synchronizing and releasing load when the two jobs alternately use resources. For example, the total time for importing input data and synchronizing resources for Job 2 cannot exceed the time for the map calculation process of Job 1. Therefore, the execution of Job 1 will not be interrupted by Job 2.

[0051] Q schedule The set is selected based on priority and the minimum required resources. Allocating resources to more jobs will result in Q... schedule The tasks in the first batch cannot obtain the minimum required resources, thus timeouts and wasting the resources allocated to them. Therefore, this invention, when allocating resources, also tries to find tasks that can share resources with the first batch of tasks. These tasks are called the second batch of tasks. The second batch of tasks can perform corresponding data transmission or computation operations when the first batch of tasks has idle network or computing resources, making full use of system resources. It should be noted that not all tasks in Q... schedule All jobs in the set will receive resources at the beginning. Q scheduleThe assignments in the first batch may also be reassigned to the second batch. For example, suppose there are 10 assignments (Assignment 1, 2, ..., 10), sorted according to priority. Q schedule It initially consists of 5 tasks. If available resources are allocated to Q... schedule All jobs in the dataset may be able to complete before their deadlines using a resource-sharing mechanism. However, if resources are allocated only to jobs 1, 2, and 3, jobs 1, 2, and 3 will receive more resources than their minimum required resources, and jobs 4, 5, and 6 can also share resources with jobs 1, 2, and 3. Clearly, the second strategy can handle more jobs. This invention will... schedule The lowest-priority job in the set is moved to the second batch of job queues until the maximum number of jobs processed is reached. For jobs not in the first batch of job queues, their availability for idle resources is checked based on their priority.

[0052] Specifically, the present invention further includes: generating a corresponding job set P for each currently scheduled job. match In the task placement strategy set TP all In the included task placement strategy, when the i-th currently scheduled job performs a computation operation, the job set P match Each job in the process can effectively utilize the idle network resources of the i-th currently scheduled job for transmission; similarly, while one job is performing data transmission, another job can use idle computing resources to perform computational operations.

[0053] In Q schedule The first batch of jobs that can be executed simultaneously is selected from the set based on priority and the minimum resource requirement ratio for long-latency jobs. Simultaneously, jobs that can share resources with the first batch are identified, along with the second batch of jobs with the highest idle resource utilization. These second batch of jobs can perform data transfer or computation operations when network or computing resources are idle in the first batch, fully utilizing system resources. The second batch of jobs constitutes the job set `tempMatch`, and the optimal job processing strategy `match` is calculated. The job processing strategy `match` records the processing strategies for all jobs included in the first and second batches, including the job placement scheme, the resource ratio allocated to each job, the number of jobs executed under the processing strategy `tempNum`, and the cost `tempCost`. The method for selecting the first batch of jobs is: calculating Q... schedule Each currently scheduled job in the set is added sequentially according to its priority to the gain that the first batch of jobs can bring when executed. Jobs with positive gains are added to the set of the first batch of jobs, and Q is set accordingly. schedule Update to the union of the first batch of assignments and the corresponding second batch of assignments.

[0054] The task placement strategy for the current scheduled job determines the job's completion time and cost. The task placement strategy needs to determine the number of map and reduce tasks for the corresponding job placed in each data center, and which data center each task's input data should be obtained from. This invention proposes a task placement strategy that can generate a cost-minimizing strategy while considering deadlines and satisfying latency constraints.

[0055] For the k-th job in the MapReduce job set, regardless of whether it is a long-latency constraint job or a short-latency constraint job, it includes a map phase and a reduce phase. The map phase consists of two processes: input data import and map computation. The reduce phase consists of a shuffle process and a reduce computation process.

[0056] set up This represents the amount of data stored in the i-th data center for the k-th job. During the map phase, let the amount of data transferred by the k-th job from the i-th data center to the j-th data center be... have Let P be the price per unit of data transmitted. i,j The cost of importing input data in the map phase of the k-th job is...

[0057] After the input data is imported, the amount of data in the j-th data center becomes The map task in the j-th data center will generate intermediate data, which will then serve as input data for the reduce task; let q represents the amount of intermediate data for the k-th job in the j-th data center. k The ratio of intermediate data volume to input data volume for the k-th job is given by:

[0058]

[0059] During the reduce phase, let This represents the proportion of reduce tasks executed by the k-th job on the i-th data center, and has... To ensure that the reduce task of the k-th job is completed by all data centers, each data center needs to interact with other data centers that have related reduce tasks, sharing some intermediate data. This represents the cost incurred by intermediate data transfers during the reduce phase of the k-th job. It is calculated using the following formula: in, This represents the proportion of reduce tasks executed by the k-th job on the j-th data center. This represents the amount of intermediate data for the k-th job in the i-th data center.

[0060] The number of input data for the map task in the k-th job at the i-th data center is... in This represents the amount of data transferred by the k-th job from the j-th data center to the i-th data center; after the shuffle process, the amount of data that the reduce task on the j-th data center needs to process is... make These represent the amount of data that each map and reduce task in the k-th job can process, respectively. and Representing the number of map and reduce tasks on the i-th data center, respectively, as follows:

[0061]

[0062] Assuming each task requires one computation slot, then the computation slots for the k-th job in the map and reduce phases are respectively... and Let m i This represents the price of a slot per unit time in the i-th data center. and Let represent the execution time of a map task and a reduce task of the k-th job, respectively. Then we get:

[0063]

[0064] In summary, for the k-th job in the MapReduce job set, it includes a map phase and a reduce phase. The map phase consists of two processes: input data import and map computation. The reduce phase consists of a shuffle process and a reduce computation process. The total cost C of the k-th job is... k for:

[0065]

[0066] in These represent the costs of importing input data in the map phase of the k-th job, the map computation cost in the map phase of the k-th job, the intermediate data transfer cost in the reduce phase of the k-th job, and the reduce computation cost in the reduce phase of the k-th job, respectively.

[0067] For the k-th job in the MapReduce job set, regardless of whether it is a long-latency-constrained job or a short-latency-constrained job, the completion time of the k-th job is... Represented as:

[0068]

[0069] in, These represent the time taken to import input data in the map phase of the k-th job, the time taken to perform map calculations in the map phase of the k-th job, the time taken to transfer intermediate data in the reduce phase of the k-th job, the time taken to perform reduce calculations in the reduce phase of the k-th job, and the waiting time of the k-th job, respectively.

[0070] Assume that the bandwidth and computation slot allocated to a job will not be preempted by other jobs before the job finishes; let This represents the bandwidth allocated to the k-th job when it travels from the i-th data center to the j-th data center at time t; therefore, during the map phase, the transmission time for the k-th job from the i-th data center to the j-th data center is... This represents the amount of data from the i-th data center to the j-th data center; during the reduce phase, the i-th data center needs to process... The reduce task, therefore the amount of data transferred from the i-th data center to the j-th data center is The transmission time from the i-th data center to the j-th data center is in and This is determined by the longest transmission time between any two data centers in the above process, resulting in:

[0071]

[0072] And require The bandwidth allocated to all jobs from the i-th data center to the j-th data center cannot exceed the available bandwidth B. i,j .

[0073] Since the number of tasks in a data center often exceeds the available computing resources, they need to be executed in multiple batches. Let time t be the computing slot allocated to the k-th job in the i-th data center. So, what are the costs for the i-th data center? and Batch completion of map and reduce tasks;

[0074] make Let represent the time required for each map or reduce task of the k-th job, and let represent the map and reduce computation times of the k-th job on the i-th data center, respectively. and make and This represents the computation time of the k-th job in the map and reduce phases. and Determined by the site that executes the latest in the corresponding process, the following are included:

[0075]

[0076] And require At time t, the computing resources of all jobs in the i-th data center cannot exceed the total computing resources S of the i-th data center. i .

[0077] The task placement strategy problem of minimizing cost is transformed into two subproblems: minimizing the cost of the map phase and minimizing the cost of the reduce phase. Both problems are linear programming problems, solvable in multinomial time. They are combined into the minCost algorithm. The minCost algorithm is used to generate task placement strategies with optimal cost.

[0078] The task placement strategy that minimizes time aims to reduce job completion time, regardless of the impact of resource price heterogeneity. Much research has focused on optimizing job completion time. Tetrium is one of the most state-of-the-art methods in this field. This invention uses Tetrium to generate a task placement strategy that minimizes time for jobs.

[0079] Suppose that the k-th job is determined to be a long-delay-constrained job, let This represents the completion time of the k-th job while minimizing the cost (minCost). Let represent the minimum completion time of the k-th job obtained under Tetrium. Clearly, the completion time range of the k-th job is...

[0080] For long-latency-constrained jobs selected as the current scheduled jobs, two sets of task placement strategies are generated by executing the Tetrium and minCost algorithms, respectively. The decision variables for the task placement strategy problem of long-latency-constrained jobs selected as the current scheduled jobs include the amount of data transferred from each data center to all other data centers during the map phase, and the proportion of reduce tasks allocated to each data center for execution.

[0081] To further determine the existence of a feasible solution to the task allocation problem, we examine the time required for computation by the Tetrium algorithm (i.e., ...). Has the given deadline T been exceeded? kIf the time exceeds this limit, it means a feasible solution to the task placement strategy problem cannot be found. Then, determine the computation time of the minimum cost algorithm. Is it less than the given time T? k If it is less than, it will return directly. This shows that the job completion time of the feasible solution generated by the minimum cost algorithm satisfies the deadline T. k .

[0082] If a feasible solution cannot be found through the above steps, then the parameter settings must be heuristically adjusted to find a viable solution, thereby generating the optimal cost before the deadline. Specifically, the variables... Distributed to Then, in the while loop, adjust the variable. Until the task is completed in less than the deadline T k To quickly obtain a feasible solution, this invention employs a greedy strategy to update decision variables. For example, if the current value of the variable is... And the adjustment rate of the variable is β∈[0,1], then when Or when Variable values ​​adjusted to Through multiple rounds of such adjustments Ultimately, it will be less than T. k At the same time, a final feasible solution will be returned.

[0083] To verify the effectiveness of the method of the present invention, a large number of experiments were conducted based on real Google and Alibaba datasets to test the performance of the latency-constrained cross-datacenter multi-database analysis job cost optimization method (MCMG method) provided by the present invention.

[0084] The latency-constrained, cross-datacenter multi-datacenter analysis job cost optimization method (MCMG method) provided by this invention mainly includes the MCMG-S algorithm for short-latency jobs and the MCMG-L algorithm for long-latency jobs. The MCMG-L algorithm is more time-consuming than the MCMG-S algorithm. In the MCMG-L algorithm, the most time-consuming process is calculating the task placement strategy for each job to be scheduled under different resource quantities. The time complexity of the MCMG-L algorithm is O(n^2). 3.5 Therefore, the time complexity of MCMG-L is L). Where k is the number of jobs that need to be scheduled, n represents the number of variables, and L represents the problem size. Calculate the number of times the task scheduling strategy is applied for each job.

[0085] Cluster Setup: This experiment considers 10 cross-region data centers with |D| of different resource and pricing settings. The capacity and pricing of network and compute resources are set according to Amazon EC2. Specifically, the bandwidth range is [0.5, 2.75] GB / s, and the bandwidth price is [0.02, 0.15] $ / GB. The number of compute slots is [100, 1000], and the price per slot ranges from [4.5, 7.5] × 10⁻⁵ $ / s. The MCMG method proposed in this invention is triggered periodically, with a default window size (T). win The time limit is 10 seconds. The parameter θ for job classification is set to 1.5, and the default resource adjustment ratio τ is 0.1.

[0086] Job Load: The job load used was generated based on job sizes and their distribution within the clusters, obtained from job load datasets from Google's and Alibaba's clusters. The Google dataset collects machine, job, and task information for a cluster of 12,500 machines in a data center over approximately one month. For each job recorded in the dataset, its arrival time, task number, input data size and distribution, and required resources are recorded. Alibaba Group released the Alibaba dataset in 2018, which contains job execution records from approximately 4,000 machines over 8 days. Refer to Table 1 for the distribution of job input data sizes.

[0087] Table 1: Distribution of Input Data Size

[0088]

[0089] Both datasets contain a large number of jobs, with most jobs having relatively small input data volumes. Furthermore, the frequency of job arrivals in the Google and Alibaba datasets was analyzed, with the average interval between two consecutive jobs being approximately 157 seconds and 17 seconds, respectively. This means that the job load on the Alibaba cluster job load dataset is significantly higher than that on the Google dataset.

[0090] The minimum completion time T for each job when using half of the cluster resources is obtained using the Tetrium algorithm and the MinCost algorithm, respectively. time and the maximum completion time T cost And according to T time and T cost Set the deadline for the assignment. The deadline for the assignment is equal to T. time +((T cost -T time )×U), and U≥0. U is the ratio of the deadline.

[0091] Comparative job scheduling strategies:

[0092] 1) Varys: An advanced scheduler developed in recent years. It uses a minimum effective bottleneck priority strategy and a bottleneck completion time heuristic-based scheduling coflow to reduce job completion time.

[0093] 2) SJF: The job with the shortest completion time will be scheduled first, and the scheduled job will occupy all resources.

[0094] 3) EDF (Earliest Deadline First): The job with the shortest remaining time will be executed first. However, this job will consume all available resources.

[0095] 4) Capacity: A popular scheduling principle used in the YARN cluster resource manager. Each hierarchical queue is scheduled based on the FIFO principle. Based on the task data of each job, this section divides jobs into three queues (small job queue, medium-sized job queue, and large job queue), with each queue occupying a certain proportion of cluster resources.

[0096] 5) Fair: Idle resources are evenly distributed to the first few jobs in the waiting queue. Each job is allocated no less than one-tenth of the site's resources. Jobs are queued according to their arrival time.

[0097] Comparison of task placement strategies:

[0098] 1) Tetrium: An excellent task placement method in recent years. This method aims to optimize job completion time by optimizing the placement of map and reduce tasks.

[0099] 2) MinCost: This method is used to minimize the execution cost of a single job. It transfers data along low-cost links and executes tasks in low-cost slots whenever possible.

[0100] 3) InPlace: This method is a widely accepted task placement approach in the MapReduce framework. It executes map tasks at the site where the data resides and performs a proportional number of reduce tasks in each data center.

[0101] Experimental metrics: The following two experimental metrics were used to measure the performance of different methods:

[0102] 1) Number of valid jobs: The total number of jobs that can satisfy the delay constraint. A job that can satisfy the delay constraint is considered a valid job.

[0103] 2) Average job completion cost: The total cost of a job divided by the number of jobs that can meet the time delay constraint.

[0104] Performance Analysis:

[0105] First, the MCMG method is compared with traditional job scheduling methods, comparing the number of valid jobs and average job completion cost under different methods. The job scheduling methods compared use Tetrium, MinCost, or InPlace as their task placement methods. Figure 4 and Figure 5 The performance results are based on the Google and Alibaba datasets, respectively. Figure 4 This is a performance comparison chart based on different job scheduling and task placement methods using the Google dataset. Figure 4 (a) is a comparison chart of the number of valid jobs under different job scheduling and task placement methods based on the Google dataset. Figure 4 (b) is a comparison chart of average job completion costs under different job scheduling and task placement methods based on the Google dataset. Figure 4 (c) is a comparison chart of Tetrium's job completion costs under different job scheduling and task placement methods based on the Google dataset. Figure 4 (d) is a comparison chart of job completion costs based on different job scheduling and task placement methods using the Google dataset. Figure 5 This is a performance comparison chart based on different job scheduling and task placement methods using the Alibaba dataset. Figure 5 (a) is a comparison chart of the number of valid jobs under different job scheduling and task placement methods based on the Alibaba dataset; (b) is a comparison chart of the average job completion cost under different job scheduling and task placement methods based on the Alibaba dataset. Figure 5 (c) is a comparison chart of Tetrium's job completion costs under different job scheduling and task placement methods based on the Alibaba dataset. Figure 5 (d) is a comparison chart of the MinCost for job completion costs under different job scheduling and task placement methods based on the Alibaba dataset. The ratios U of the deadlines for Google and Alibaba are 0.2 and 0.9, respectively. Figure 4 (a) and Figure 5 (a) shows the number of jobs that meet the delay constraints under different methods and the ratio normalized to this method. Figure 4 (b) and Figure 5 (b) Records the average job completion cost and the ratio normalized to MCMG for different methods.

[0106] like Figure 4As shown in (a), when the compared job scheduling methods use the MinCost algorithm as their task scheduling strategy, MCMG ensures that more jobs meet latency constraints compared to the compared methods. This is because MCMG schedules jobs based on their urgency, and each job's task placement strategy considers its latency constraints. Therefore, MCMG guarantees that more jobs meet their latency constraints. When the compared job scheduler uses Tetrium as its task placement method, the number of jobs meeting latency constraints under the MCMG method is slightly less than that under Varys, Capacity, and Fair. However, the average job completion cost under Varys, Capacity, and Fair methods is much higher than that under MCMG. This is because Tetrium can minimize job completion time by optimizing the placement of map and reduce tasks, but this results in a high job completion cost. When the InPlace method is used, the number of valid jobs under all job scheduling methods is less than that under the MCMG method. In other words, compared to other methods using the InPlace algorithm as their task placement strategy, MCMG provides better performance guarantees.

[0107] like Figure 4 (b) and Figure 5 As shown in (b), the job completion costs of the comparative methods are all lower than those generated by MCMG. In particular, based on the Alibaba dataset, MCMG can reduce the total cost by at least 16% compared to other comparative methods. The results demonstrate that the method proposed in this chapter can significantly reduce the cost of MapReduce jobs across data centers. Furthermore, in... Figure 5 In section (a), this section tested the number of valid jobs under different methods based on the Alibaba dataset. It can be observed that, due to the large amount of data that the cluster needs to process per unit time, MCMG achieves better performance than other comparison methods regardless of the task placement method used. This is because MCMG can make appropriate task placement strategies based on the remaining processing time of its jobs before their deadlines. However, traditional job schedulers do not consider job latency constraints, resulting in many jobs being called but violating their deadlines. Therefore, based on the Alibaba dataset, the performance results of traditional job schedulers are lower than those achievable by MCMG.

[0108] Figure 4Tables (c), 4(d), 5(c), and 5(d) demonstrate the CDF of job completion costs that satisfy latency constraints compared to the comparison methods. When the job scheduler uses Tetrium as its task placement method, although MCMG is a heuristic approach, it does not increase the cost of any job compared to the other comparison methods. When the comparison methods use the MinCost job placement strategy, the job completion cost resulting from MCMG is very close to the job completion cost under the other comparison methods. This is because MCMG selects the resource allocation method that minimizes costs, and each job's task placement strategy minimizes the job completion cost before its deadline. Therefore, compared to the other comparison methods, MCMG results in a lower total cost, and MCMG enables more jobs to satisfy latency constraints.

[0109] Next, Table 2 shows the comparison of the running time and performance of the optimal algorithm and MCMG under different sites and number of jobs. The results are shown in Table 2:

[0110] Table 2 Performance Comparison of Optimal Algorithm and MCMG

[0111]

[0112] The optimal algorithm enumerates all possible solutions (job scheduling, resource allocation, and task placement) and obtains the optimal solution. Compared to the optimal solution, the difference in the number of valid job data and the average job completion cost under the MCMG method is 0.05 and 0.09, respectively. However, it can be observed that under different site sizes and job numbers, the optimal algorithm takes tens of seconds to several hours to obtain the solution, while MCMG only takes a few hundred milliseconds to calculate the scheduling scheme.

[0113] Job urgency and resources: First, the impact of job urgency and cluster resource ratio on short-latency and long-latency jobs was assessed. Figure 6 and Figure 7 This shows the job's deadline and resource ratio (R). s ,R l When the conditions change, the MCMG method is based on the average job completion time and the number of valid jobs on the Google and Alibaba datasets, respectively. Figure 6 (a) and Figure 6 (b) presents experimental results based on the Google dataset. Figure 7 (a) and 7(b) show the experimental results based on the Alibaba dataset.

[0114] Figure 6 This is a performance comparison chart of different methods under different latency constraints based on the Google dataset. Figure 6(a) is a comparison chart of the percentage of valid jobs under different latency constraints based on the Google dataset. Figure 6 (b) is a comparison chart of average job completion costs under different latency constraints based on the Google dataset. Figure 7 This is a performance comparison chart of different methods under different latency constraints based on the Alibaba dataset. Figure 7 (a) is a comparison chart of the percentage of valid jobs under different latency constraints based on the Alibaba dataset. Figure 7 (b) is a comparison chart of average job completion costs under different time delay constraints based on the Alibaba dataset.

[0115] like Figure 6 As shown in (a), except when U = 2, the number of valid jobs increases with the deadline. When U = 2, R s When R is in the range [0.2, 0.5], MCMG can ensure that 91% of jobs meet the latency constraints. This is because when U = 2, the number of jobs in the short-latency and long-latency queues is appropriate. The system can achieve a good balance between job completion time and cost. However, when R... s =0.1 and U=2, the number of long-latency queues is too large. Therefore, due to the lack of cluster resources, the number of jobs that the short-latency job module can process is limited. Conversely, the cluster resources for long-latency jobs may be idle. Therefore, when R... s =0.1, its U=2, the number of jobs completed before the deadline is very small.

[0116] Figure 7 (a) Records the number of valid jobs based on the Alibaba dataset. It can be observed that the number of valid jobs increases with the increase of U or the proportion of short-latency queues. This phenomenon can be explained as follows: Alibaba's job load includes many short-latency jobs, i.e., jobs with deadlines below T. win ×1.5. Therefore, when R s When added, MCMG can handle more short-latency jobs, and these jobs can be processed quickly. Generally, setting appropriate loads for the two queues for different workloads is crucial for improving system performance.

[0117] Figure 6 (b) and Figure 7(b) shows the average job completion cost as the deadline and resource allocation ratio for each job change. It can be seen that the average job completion cost decreases as U increases. This is because as the job deadlines become longer, the system can select cheaper resources to process these jobs. Furthermore, when U > 0.2, based on the Google dataset, the difference in average job completion cost is small under different resource allocation ratios. This phenomenon can be explained as follows: as R... s As R increases, the number of short-delay jobs increases, and short-delay jobs using Tetrium as their task placement method result in high job completion costs. Therefore, the total job completion cost of short-delay jobs increases with R. s The cost increases with the increase of U. Furthermore, as U increases, MCMG-L can select cheaper resources to handle long-latency jobs, reducing the overall cost of long-latency jobs. Therefore, based on the Google dataset, the average job completion cost of all jobs changes little under different resource allocation ratios and job urgency.

[0118] Furthermore, based on Google datasets, when R s When U is equal to 0.4 or 0.5, the number of valid jobs and the average job completion cost are almost the same for different U values. Therefore, as U increases, the total cost of long-delay jobs and the number of valid jobs both increase. However, the total cost of short-delay jobs and the number of valid jobs decrease. Finally, the total cost and number of valid jobs reach an equilibrium.

[0119] Time window: The average job completion cost and the number of valid jobs were tested under different time windows, such as Figure 8 As shown, Figure 8 This chart compares the number of legal jobs and the average job completion cost under different time windows. Figure 8 (a) is a comparison chart of the number of legal jobs under different time windows. Figure 8 (b) A comparison of average job completion costs under different time windows. It can be observed that as the time window lengthens, the number of valid jobs decreases, but the average job completion cost increases. This is because a longer time window leads to more waiting jobs, wasting more time before they are scheduled. In each iteration, each scheduler can only handle a certain number of jobs, and scheduled jobs typically have shorter completion times. Therefore, they have to use more expensive resources. After scheduled jobs are completed, the system's resources may be idle for a long time. Therefore, a shorter time window can reduce the total cost while allowing more jobs to meet latency constraints.

[0120] The parameters for job classification: Based on the Google dataset, the impact of the parameter θ on the average job completion cost and the number of valid jobs was tested. The results are shown in the figure. Figure 9 A comparison chart showing the number of legal jobs and average job completion cost under different θ values. Figure 9 (a) is a comparison chart of the number of valid jobs under different θ values. Figure 9 (b) is a comparison chart of average job completion costs under different θ values. It can be observed that as θ increases, the average job completion cost increases, but the number of valid jobs initially increases and then decreases. When θ is set to 1.5, the number of valid jobs is high, and the average job completion cost is low. Therefore, the default θ value is set to 1.5.

[0121] System components: Figure 10 A comparison chart of the number of legal jobs and average job completion cost under different components, where Figure 10 (a) is a comparison chart of the number of valid jobs under different components. Figure 10 (b) This section presents a comparison of the average job completion cost of valid jobs under different components. It analyzes the impact of the Job Scheduler (JS), Resource Allocation (RA), and Task Placement (TP) components in MCMG-L, which are replaced by FIFO, minimum resource requirement to meet latency, and Tetrium, respectively. Additionally, the impact of not calling the module for short-latency job scheduling was tested. When experiments were conducted on the Google dataset, the percentage of jobs meeting latency constraints changed little under different components, but the average job completion cost under different components was higher than that under MCMG. Based on the Alibaba dataset, the number of jobs meeting latency constraints using the LS+JS model was greater than that under MCMG. However, the average job completion cost resulting from LS+JS was higher than that using MCMG.

[0122] On the other hand, one embodiment provides a cost optimization apparatus for multi-datacenter data analysis operations based on latency constraints, comprising:

[0123] The first module is used to obtain the MapReduce job set from the data center set;

[0124] The second module is used to divide the jobs into short-delay jobs and long-delay jobs according to the deadline of each job in the MapReduce job set, and each data center allocates a fixed proportion of resources to short-delay jobs and long-delay jobs.

[0125] The third module is used to schedule short-delay jobs using a FIFO strategy, minimizing the completion time of short-delay jobs by optimizing the task placement in the map and reduce phases.

[0126] The fourth module, for long-delay jobs, first obtains the currently waiting long-delay jobs and the currently available resources. The currently waiting long-delay jobs include newly arrived long-delay jobs and unprocessed long-delay jobs. It calculates the priority of each job based on its urgency and the minimum resource ratio required. Based on the priority of the long-delay jobs, it selects the current scheduled job from the currently waiting long-delay jobs. It generates the resource configuration of the current scheduled job and the corresponding task placement strategy that meets the latency constraints and minimizes the cost under the resource configuration to achieve the lowest total cost. It continuously updates the currently waiting long-delay jobs until the process ends.

[0127] On the other hand, the present invention provides a computer device including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the latency-constrained multi-datacenter job cost optimization method provided in any of the above embodiments. The computer device may be a server. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store sample data. The network interface of the computer device is used for communication with external terminals via a network connection.

[0128] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the method for optimizing the cost of multi-data analysis jobs across data centers based on latency constraints provided in any of the above embodiments.

[0129] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0130] Matters not covered in this invention are common knowledge.

[0131] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

[0133] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for optimizing the cost of multi-data center analysis jobs across data centers based on latency constraints, characterized in that, include: Retrieve the collection of MapReduce jobs from the data center set; Based on the deadlines of each job in the MapReduce job set, jobs are divided into short-latency jobs and long-latency jobs. Each data center allocates a fixed proportion of resources to short-latency jobs and long-latency jobs. For the first job in the MapReduce job set... The assignment is set as follows: The assignment is in time Submitted, and in time Completed, number Duration of each task It cannot exceed its corresponding delay constraint, i.e., the first... Each assignment must be completed by its corresponding deadline. Completed before, with , Represents a collection of MapReduce jobs. Set time window If the first The deadline for each assignment is less than × θ Then the first The first task is a short-delay task; otherwise, the second task... One of the tasks is a long-delayed task, among which... θ It is a constant; For short-delay jobs, a FIFO strategy is used to schedule them, and the completion time of short-delay jobs is minimized by optimizing the task placement in the map and reduce phases. For long-delayed jobs, first, obtain the currently waiting long-delayed jobs and the currently available resources. The currently waiting long-delayed jobs include newly arrived long-delayed jobs and unprocessed long-delayed jobs. Calculate the priority of each job based on its urgency and the minimum resource requirement ratio. Select the current scheduled job from the currently waiting long-delayed jobs based on their priority, including: determining the minimum resource requirement ratio for each long-delayed job; determining the priority of each long-delayed job based on its urgency and the minimum resource requirement ratio; sorting the long-delayed jobs in descending order of priority; selecting the current scheduled job based on priority and the minimum resource requirement ratio for each long-delayed job, in descending order of priority, until the currently available resources are saturated. All currently scheduled jobs constitute the current scheduled job. Set; generates the resource configuration of the current scheduled job and the corresponding task placement strategy under the resource configuration that satisfies the latency constraint and minimizes the cost, so as to achieve the lowest total cost, and continuously updates the current waiting long-latency jobs until the end.

2. The method for optimizing the cost of multi-data center analysis jobs across data centers based on latency constraints according to claim 1, characterized in that, Also includes: Generate a corresponding set of jobs for each currently scheduled job. In the task placement strategy set In the included task placement strategy, when the i-th currently scheduled job performs a computation operation, the job set... Each job in the process can effectively utilize the idle network resources of the i-th currently scheduled job for transmission; similarly, while one job is performing data transmission, another job can use idle computing resources to perform computational operations. exist From the set, the first batch of jobs that can be executed simultaneously are selected based on priority and the minimum resource requirement ratio for long-delayed jobs. Simultaneously, jobs that can share resources with the first batch of jobs during execution are identified, along with the second batch of jobs with the highest idle resource utilization. These second batch of jobs can utilize the idle resources of the first batch of jobs to perform related operations. This second batch of jobs constitutes the job set. tempMatch And calculate the optimal job processing strategy. match Job processing strategies match The processing strategies for all jobs included in the first and second batches of jobs were recorded, including the job placement scheme, the resource allocation to each job, and the number of jobs executed under each processing strategy. tempNum and the cost of consumption tempCost The method for selecting the first batch of assignments is: calculation Each currently scheduled job is added to the first batch of jobs in order of priority based on the potential gain from the execution of those jobs. Jobs with positive gains are added to the first batch of jobs. Update to the union of the first batch of assignments and the corresponding second batch of assignments.

3. The method for optimizing the cost of multi-data center analysis operations based on latency constraints according to claim 1 or 2, characterized in that, The collection of MapReduce jobs in the data center set, where Represents a collection of data centers. , Represents a collection of MapReduce jobs. For the first MapReduce job in the set... The first job consists of a map phase and a reduce phase. The map phase comprises two processes: input data import and map computation. The reduce phase comprises two processes: shuffle and reduce computation. Total cost of each task for: in , , , They represent the first The cost of importing input data for the first job map stage, the cost of the first job map stage, the cost of the first job map stage, the cost of importing input data for ... The map computation cost of the first job's map phase, the first The cost of intermediate data transfers during the reduce phase of the first job and the cost of the second job The computational cost of the reduce phase of a job.

4. The method for optimizing the cost of multi-data center analysis operations based on latency constraints according to claim 3, characterized in that, Indicates the first The amount of data stored in the i-th data center for each job, during the map phase, is denoted as follows: The job moves from the i-th data center to the i-th data center. The amount of data transmitted by each data center is ,have , ; Let the price per unit of data transmission be Then the first The cost of importing input data for the map phase of a task .

5. The method for optimizing the cost of multi-data center analysis operations across data centers based on latency constraints according to claim 3, characterized in that, After the input data is imported, the first... The amount of data in each data center became ;No. Map tasks in each data center generate intermediate data that serves as input for reduce tasks; let Indicates the first The first assignment was in the... The amount of intermediate data on each data center Indicates the first The ratio of intermediate data volume to input data volume for each task is obtained as follows: During the reduce phase, let Indicates the first The proportion of reduce tasks executed by each job on the i-th data center, and have , , ensure the The reduce tasks of each job will be completed by all data centers. To complete the reduce tasks, each data center needs to interact with other data centers that have related reduce tasks, exchanging some intermediate data. Indicates the first The cost of intermediate data transfers during the reduce phase of a job. It is calculated using the following formula: ,in, Indicates the first The proportion of reduce tasks executed by a job on the j-th data center. Indicates the first The first assignment was in the... Intermediate data volume on a data center.

6. The method for optimizing the cost of multi-data center analysis operations across data centers based on latency constraints according to claim 3, characterized in that, No. The first assignment was in the... The number of input data for map tasks on each data center is ,in Indicates the first The amount of data transferred from the j-th data center to the i-th data center in the j-th job; after the shuffle process, the amount of data transferred from the j-th data center to the i-th data center. The amount of data that the reduce tasks need to process on each data center is: ; make , They represent the first The amount of data that each map and reduce task in the job can process. and Representing the number of map and reduce tasks on the i-th data center, respectively, as follows: Assuming each task requires a computation slot, then the... The computation slots for each job in the map and reduce phases are respectively... and ;make This represents the price of a slot per unit time in the i-th data center. and They represent the first The execution time of one map and one reduce task for each job is obtained as follows: 。 7. The method for optimizing the cost of multi-data center analysis jobs across data centers based on latency constraints according to any one of claims 4 to 6, characterized in that, No. Completion time of each assignment Represented as: in, , , , ， They represent the first The time taken to import input data for the first job map phase, the first The time taken for map calculation in the map phase of the first job, the time taken for map calculation in the first job, The time taken for intermediate data transfer during the reduce phase of the first job, the time taken for the second job, the time taken for intermediate data transfer during the reduce phase of the first ... The time taken for the reduce calculation in the reduce phase of the first job and the time taken for the second job. The waiting time for each task.

8. The method for optimizing the cost of multi-data center analysis jobs across data centers based on latency constraints according to claim 7, characterized in that, Assume that the bandwidth and computation slot allocated to a job will not be preempted by other jobs before the job finishes; let Indicates the first The assignment is in time t The allocated bandwidth from the i-th data center to the j-th data center; therefore, in the map phase, the... The transmission time for a task from the i-th data center to the j-th data center is , This represents the amount of data from the i-th data center to the j-th data center; during the reduce phase, the i-th data center needs to process... The reduce task, therefore the amount of data transferred from the i-th data center to the j-th data center is The transmission time from the i-th data center to the j-th data center is ,in and satisfy: And require The bandwidth allocated to all jobs from the i-th data center to the j-th data center cannot exceed the available bandwidth. ; Since the number of tasks in a data center often exceeds the available computing resources, they need to be executed in multiple batches, with time constraints. t Assigned to the i-th data center The calculation slot for each job is Then the i-th data center needs to spend respectively and Batch completion of map and reduce tasks; make , They represent the first The time required for each map or reduce task in a given job, the first The map and reduce computation times for job i on the i-th data center are respectively... and ;make and Indicates the first The computation time of each job in the map and reduce phases. and Determined by the site that executes the latest in the corresponding process, the following are included: And require In time t The computing resources of all jobs on the i-th data center cannot exceed the total computing resources of the i-th data center. .

Images