Batch job resource configuration optimization method and device

Abstract

A batch job resource configuration optimization method and device, the batch job resource configuration optimization method comprises: determining a cloud resource configuration optimization target; the cloud resource configuration optimization target comprises a maximum job completion running time and a minimum cloud service cost; determining a target cloud resource configuration corresponding to the cloud resource configuration optimization target. By using the above scheme, the target cloud resource configuration with the minimum cloud service cost can be obtained within the maximum job completion running time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing, and particularly relates to a batch processing job resource configuration optimization method and device. BACKGROUND

[0002] With the development of cloud computing technology, the number of cloud big data jobs is growing exponentially, and the business scenarios of big data jobs are becoming increasingly complex and diverse. Therefore, public cloud vendors provide a large number of virtual machine instances and clusters, allowing users to flexibly select specific cloud resource configurations for each specific big data job to meet its computing needs. At the same time, the selection of cloud resource configurations has a significant impact on the completion time and running cost of big data jobs. Inappropriate cloud resource configurations can greatly increase the completion time and running cost of big data jobs.

[0003] For batch processing jobs (such as Spark) that are repeatedly run at a fixed time, it is particularly important to select appropriate cloud resource configurations. However, the optimization problem of resource configuration is difficult, and it is difficult to obtain cloud resource configurations that meet multiple requirements. SUMMARY

[0004] One of the purposes of the present application is to provide a batch processing job resource configuration optimization method that can find a cloud resource configuration for a batch processing job that meets the job completion time requirement and has the minimum completion cost.

[0005] In a first aspect, an embodiment of the present application provides a batch processing job resource configuration optimization method, comprising: obtaining a cloud resource configuration optimization target; the cloud resource configuration optimization target includes a maximum job completion running time and a minimum cloud service cost; determining a target cloud resource configuration corresponding to the cloud resource configuration optimization target, comprising: using candidate cloud resource configurations in a preset search space, sequentially running a target batch processing job, and obtaining cloud service costs and completion times obtained by running the target batch processing job using the first t-1 candidate cloud resource configurations; t is a positive integer; based on the cloud service costs and completion times obtained by running the target batch processing job using the first t-1 candidate cloud resource configurations, performing a cost prediction operation to obtain a predicted cloud service cost, and performing a time prediction operation to obtain a predicted completion time; based on the predicted cloud service cost and the predicted completion time, selecting a tth candidate cloud resource configuration from the search space, and running the target batch processing operation using the tth candidate cloud resource configuration to obtain a cloud service cost and a completion time corresponding to the tth candidate cloud resource configuration running the target batch processing job; detecting an instruction to stop running the target batch processing operation, selecting a candidate cloud resource configuration corresponding to the minimum cloud service cost as the target cloud resource configuration.

[0006] Optionally, the cloud resource configuration optimization target is: s.t. g(x)≤D; wherein, configuring, for the target cloud resource, for the search space, x is the candidate cloud resource configuration, g(x) is the completion time corresponding to x, p t (x) is the cloud service cost corresponding to x, D is the maximum job completion running time, g(x)·p t (x) represents the cloud service cost of running the target batch job by using the tth candidate cloud resource, s.t. g(x)≤D represents that g(x) is not greater than the maximum job completion running time, represents selecting the candidate cloud resource configuration corresponding to the minimum cloud service cost as

[0007] Optionally, the batch job resource configuration optimization method further comprises: converting, based on a Bayesian optimization model, the cloud resource configuration optimization target, and the cloud resource configuration optimization target after conversion is: wherein f(x) is the completion speed corresponding to x, and f(x)=1 / g(x); represents obtaining the candidate cloud resource configuration that makes f(x) / p t (x) maximum.

[0008] Optionally, the obtaining, by using the candidate cloud resource configurations in the preset search space, the completion time of running the target batch job by using the first t-1 candidate cloud resource configurations comprises: obtaining the average value and the covariance of the f(x); and obtaining the Gaussian process corresponding to the f(x) based on the average value and the covariance of the f(x).

[0009] Optionally, the Gaussian process corresponding to the f(x) is GP(μ(x),k(x,x′)), wherein μ(x) is the average value of the f(x), and k(x,x′) is the covariance of the f(x); after running the target batch job by using the t candidate cloud resource configurations, μ(x) and k(x,x′) are estimated in the following manner: wherein k t (x)=[k(x1,x),k(x2,x),···,k(x t ,x)] T , I is a unit matrix, and K t is a positive definite kernel matrix A t is a set of the first t candidate cloud resource configurations, and A t ={x1,x2,…,x t}, x1 is the first candidate cloud resource configuration, x2 is the second candidate cloud resource configuration, and x nx' is the candidate cloud resource configuration for the n th candidate cloud resource configuration.

[0010] Optionally, the time prediction operation is performed based on the completion times of the target batch processing job running on the first t-1 candidate cloud resource configurations to obtain a predicted completion time, including: performing a prediction operation based on the Gaussian process corresponding to the f(x) and the completion times of the target batch processing job running on the first t-1 candidate cloud resource configurations to obtain the predicted completion time.

[0011] Optionally, the cost prediction operation is performed on the cloud service costs of the target batch processing job running on the first t-1 candidate cloud resource configurations to obtain a predicted cloud service cost, including: using the following formula the cost prediction operation is performed to obtain the predicted cloud service cost; and the h is a prediction function.

[0012] Optionally, the predicted cloud service cost is: is the cloud service cost of the target batch processing job running on the t-1 th candidate cloud resource configuration, p t-2 (x t ) is the cloud service cost of the target batch processing job running on the t-2 th candidate cloud resource configuration, μ t-1 (x t ) is the posterior distribution of the mean corresponding to the f(x).

[0013] Optionally, the t th candidate cloud resource configuration is selected from the search space based on the predicted cloud service cost and the predicted completion time, including: using the following formula to select the t th candidate cloud resource configuration from the search space: wherein β t is a confidence upper limit weight, and δ∈(1,∞); σ t-1 (x t ) is the posterior distribution of the variance corresponding to the f(x).

[0014] The embodiment of the present application also provides a batch processing job resource configuration optimization device, comprising: an acquisition unit configured to acquire a cloud resource configuration optimization target; the cloud resource configuration optimization target comprises maximum job completion running time and minimum cloud service cost; a determination unit configured to determine a target cloud resource configuration corresponding to the cloud resource configuration optimization target, comprising: using candidate cloud resource configurations in a preset search space, sequentially running a target batch processing job, and acquiring cloud service costs and completion times of running the target batch processing job using the first t-1 candidate cloud resource configurations; t is a positive integer; based on the cloud service costs and the completion times of running the target batch processing job using the first t-1 candidate cloud resource configurations, performing a cost prediction operation to obtain a predicted cloud service cost, and performing a time prediction operation to obtain a predicted completion time; based on the predicted cloud service cost and the predicted completion time, selecting a tth candidate cloud resource configuration from the search space, and running the target batch processing operation using the tth candidate cloud resource configuration, and acquiring a cloud service cost and a completion time corresponding to running the target batch processing job using the tth candidate cloud resource configuration; detecting an instruction to stop running the target batch processing operation, selecting a candidate cloud resource configuration corresponding to the minimum cloud service cost as the target cloud resource configuration.

[0015] Compared with the prior art, the technical scheme of the embodiment of the present application has the following beneficial effects:

[0016] The cloud resource configuration optimization target is acquired, and then the maximum job completion running time and the minimum cloud service cost are obtained. Based on the cloud resource configuration optimization target, the target cloud resource configuration corresponding to the cloud resource configuration optimization target is determined. By taking the maximum job completion running time and the minimum cloud service cost as constraint conditions when the target cloud resource configuration is determined, the target cloud resource configuration finally obtained can only cost the minimum cloud service cost on the premise that the completion time is less than the maximum job completion running time. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 It is a flowchart of a batch processing job resource optimization method in the embodiment of the present application;

[0018] Figure 2 It is a system framework schematic diagram of a batch processing job resource optimization device in the embodiment of the present application. DETAILED DESCRIPTION

[0019] In the prior art, for a batch job (such as Spark) running repeatedly in time, how to select a suitable cloud resource configuration is related to the completion time and the completion cost of the batch job. However, the optimization problem of the cloud resource configuration is difficult. On the one hand, the completion time of the batch job is determined by the cloud resource configuration, and the mapping relationship is unpredictable. On the other hand, the cloud resource can be shared by other users / jobs, and the running time of the same batch job can be quite different under the same cloud resource configuration. In addition, the cost needs to be considered and predicted when selecting the cloud resource configuration.

[0020] In the embodiment of the present application, when determining the target cloud resource configuration, the maximum job completion running time and the minimum cloud service cost are taken as constraint conditions, and finally the target cloud resource configuration can be obtained, which can only cost the minimum cloud service cost under the premise that the completion time is less than the maximum job completion running time.

[0021] In order to make the above-mentioned purposes, characteristics and beneficial effects of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings.

[0022] The embodiment of the present application provides a batch job resource configuration optimization method, referring to Figure 1 The specific steps are described in detail as follows.

[0023] In step 101, a cloud resource configuration optimization target is obtained.

[0024] In the specific implementation, the cloud resource configuration optimization target can include the maximum job completion running time and the minimum cloud service cost.

[0025] In step 102, a target cloud resource configuration corresponding to the cloud resource configuration optimization target is determined.

[0026] In the specific implementation, candidate cloud resource configurations in a preset search space can be used to run the target batch job in sequence, to obtain the cloud service cost and the completion time of running the target batch job by using the first t-1 candidate cloud resource configurations; t is a positive integer; based on the cloud service cost and the completion time of running the target batch job by using the first t-1 candidate cloud resource configurations, a cost prediction operation is performed to obtain a predicted cloud service cost, and a time prediction operation is performed to obtain a predicted completion time; based on the predicted cloud service cost and the predicted completion time, the tth candidate cloud resource configuration is selected from the search space, and the target batch job is run by using the tth candidate cloud resource configuration to obtain the cloud service cost and the completion time corresponding to the tth candidate cloud resource configuration; and when a command to stop running the target batch job is detected, the candidate cloud resource configuration with the minimum cloud service cost is selected as the target cloud resource configuration.

[0027] The principle and process of the batch job resource configuration optimization method provided in the embodiments of the application are described in detail below.

[0028] In a specific implementation, the cloud resource configuration optimization problem can be modeled first to obtain a cloud resource configuration optimization target.

[0029] Specifically, the following function can be used to represent the mapping relationship between the batch job resource configuration and the completion time, where the completion time function g(x) depends on the candidate cloud resource configuration x, and the candidate cloud resource configuration x includes the number of instances in each stage of the batch job, the number of CPU cores corresponding to the server, the memory capacity size corresponding to the server, the network bandwidth, and the like. All candidate cloud resource configurations x can form a search space . The search space may include a plurality of candidate cloud resource configurations x.

[0030] The following function can be used to represent the cloud service cost of the candidate cloud resource configuration x for running the job for the t th time.

[0031] Therefore, the cloud resource configuration optimization target in the embodiments of the application is to find the cloud resource configuration that minimizes the cloud service cost under the condition that the job completion time requirement is met (that is, the completion time is less than the maximum job completion running time).

[0032] The cloud resource configuration optimization target in the embodiments of the application can be represented as the following formula (1):

[0033]

[0034] s.t.g(x)≤D, which represents a constraint condition, that is, in formula (1), g(x)≤D needs to be met.

[0035] For the optimization target in the above formula (1), the target cloud resource configuration can be represented as the following formula (2):

[0036]

[0037] wherein, the target cloud resource configuration is x*, the search space is X, x is the candidate cloud resource configuration, g(x) is the completion time corresponding to x, p t (x) is the cloud service cost corresponding to x, and D is the maximum job completion running time. t g(x)·p characterize the candidate cloud resource configuration corresponding to the minimum cloud service cost as

[0038] Since the job completion time function g(x) and the cloud service cost function p t (x) are unknown, multiple runs and sample collection are required to learn the characterization and search for the target cloud resource configuration

[0039] However, the search space may contain a large number of candidate cloud resource configurations x, making it difficult to achieve detailed characterization and exhaustive search. In some special application scenarios, the cloud service cost function p t (x) may change over time, making it more difficult to obtain the target cloud resource configuration .

[0040] Therefore, in the embodiments of the present application, a Bayesian optimization model is used to characterize the cloud resource configuration to completion cost using a small number of samples.

[0041] In the embodiments of the present application, a function is introduced to represent the job completion speed. The job completion speed function f(x) is the inverse of the job completion time. Thus, the cloud resource configuration optimization objective in the above formula (1) can be transformed into:

[0042]

[0043] where f(x) is the completion speed corresponding to x, f(x) = 1 / g(x); is represented as obtaining the candidate cloud resource configuration that maximizes f(x) / p t (x).

[0044] By using the Bayesian optimization model, the cloud resource configuration is continuously selected, the posterior distribution of the objective function is updated, and the optimal resource configuration is found to maximize the cloud resource configuration optimization objective in the above (3).

[0045] The job completion speed f(x) is set to follow a Gaussian process (GP), and the Gaussian process GP(μ(x),k(x,x')) is determined by its mean μ(x) and covariance (i.e., kernel function). The mean μ(x) = E[f(x)].

[0046] In specific implementations, μ(x) and k(x,x') are estimated as follows: The kernel function where k t (x) = [k(x1,x), k(x2,x),..., k(x t,x)] T I is the identity matrix, K t It is a positive definite kernel matrix. A t Let A be the set of the first t candidate cloud resource configurations, and A t ={x1,x2,…,x t}, x1 is the first candidate cloud resource configuration, x2 is the second candidate cloud resource configuration, x n Let x′ be the configuration of the nth candidate cloud resource and x′ be the configuration of the candidate cloud resource that has been completed.

[0047] For point A t ={x1,x2,…,x t Noisy sample y at} t ={y1,y2,…,y t The posterior of} is a GP distribution. After running the target batch job with t candidate cloud resource configurations, the mean μ(x) and kernel function k(x,x′) can be estimated posteriorly using the following formula:

[0048]

[0049] Where, k t (x)=[k(x1,x),k(x2,x),···,k(x t ,x)] T ,K t It is a positive definite kernel matrix. A t Let A be the set of the first t candidate cloud resource configurations, and A t ={x1,x2,…,x t}, x1 is the first candidate cloud resource configuration, x2 is the second candidate cloud resource configuration, x n Let x′ be the configuration of the nth candidate cloud resource and x′ be the configuration of the candidate cloud resource that has been completed.

[0050] In this embodiment of the invention, a squared exponential kernel is used to characterize the nonlinear relationship between cloud resource allocation and job completion speed. During the t-th run, the obtained y... t-1 ={y1,y2,…,y t-1} and resource allocation A t-1 ={x1,x2,…,x t-1}, update the posterior distribution of job completion speeds for all candidate cloud resource configurations.

[0051] In this embodiment of the invention, before the t-th run, the cloud service cost for the t-th run can be predicted, and the predicted cloud service cost for the t-th run can be set. Represented as:

[0052]

[0053] wherein h is a prediction function.

[0054] In the embodiments of the present application, a first-order incremental estimation method can be used for prediction to obtain more accurate prediction results, and specifically:

[0055]

[0056] In the above formula (6), p t-1 (x t ) is a cloud service cost obtained by running the target batch processing job using the t-1th candidate cloud resource configuration, p t-2 (x t ) is a cloud service cost obtained by running the target batch processing job using the t-2th candidate cloud resource configuration, μ t-1 (x t ) is a posterior distribution of the mean corresponding to f(x).

[0057] In combination with the above cloud service estimation cost and the completion speed posterior distribution GP(μ(x),k(x,x′)), an extended Gaussian process upper confidence bound (UCB) algorithm is used to select a cloud resource configuration for running in the next round.

[0058] In the tth running, the optimal solution in the following formula (7) is selected as the target cloud resource configuration:

[0059]

[0060] wherein β is a UCB weight, and δ∈(1,∞). Specifically, μ t-1 (x t ) is a posterior distribution of the mean corresponding to f(x), σ t-1 (x t ) is a posterior distribution of the variance corresponding to f(x). The UCB weight β t is a weighted sum of the mean μ t-1 (x t ) and the variance σ t-1 (x t ).

[0061] The specific process of determining the target cloud resource configuration in the above embodiments of the present application will be described below through a specific example.

[0062] In a specific implementation, the initial candidate cloud resource configuration of the batch processing job is set as x1, the maximum job completion running time of the batch processing job is set as D, and all candidate cloud resource configurations x can constitute a search space The three parameters are taken as inputs and initialized.

[0063] After obtaining the user-submitted batch job, the batch job is assigned a default initial candidate cloud resource configuration x1 and run, while recording the cloud service cost p1(x) of the first run of the batch job and the job completion speed y1(x) of the first run of the batch job.

[0064] According to the candidate cloud resource configurations x2, x3, … x n , the second, third, …, nth run of the batch job.

[0065] Before the tth run of the batch job, the completion speed history information y t-1 ={y1, y2, …, y t-1} and the configuration information A t-1 ={x1, x2, …, x t-1} of the first t-1 runs of the batch job can be obtained. Then, the completion speed history information y t-1 and the above formula (4) are used to update the completion time posterior distribution μ t-1 (x), .

[0066] And before the tth run of the batch job, the cloud service estimated cost p

[0067] Combined with the above predicted cloud service cost and the predicted completion time, the candidate cloud resource configuration x t of the tth run of the batch job is selected using formula (7).

[0068] The cloud service cost p t (x) of the tth run of the batch job and the job completion speed y t (x) of the tth run of the batch job are obtained.

[0069] If it is detected that the user has stopped running the batch job, the optimal resource configuration under the current state is output; if it is not detected that the user has stopped running the batch job, the batch job can continue to be run using the candidate cloud resource configuration.

[0070] Referring to Figure 2 , a same framework diagram of a batch job resource optimization device in an embodiment of the application is given. Specifically, the batch job resource optimization method provided in the above embodiment can be executed by a batch job resource optimization device (hereinafter referred to as an optimization device).

[0071] In a specific implementation, the optimization device can include an optimization engine, a database, a price predictor, a job monitor, etc.

[0072] Specifically, after receiving a user-submitted Spark batch job, the optimization engine assigns the batch job a default initial resource configuration and notifies the job monitor to monitor its runtime information. Then, according to the runtime information and the next-time instance price predicted by the price predictor, the optimization engine selects a next candidate cloud configuration resource;

[0073] Database: used to save a search space and store historical data of batch jobs, including job name, resource configuration, completion time, completion cost, etc. The database is responsible for maintaining and sending job history data to the optimization engine, and has a sliding window N for limiting the size of data sent to the optimization engine. The sliding window N allows the optimization engine to focus on the last N runs, reducing the impact of dynamic cloud noise caused by resource overprovisioning;

[0074] Price predictor: used to collect real-time unit time prices of all candidate cloud resource configurations, and use a first-order increasing method to predict the price of the next candidate cloud resource configuration, and send the predicted price to the optimization engine;

[0075] Job monitor: when a job is submitted to the Kubernetes platform, the job monitor starts communicating with the SparkContext to monitor its runtime information. When the optimization engine predicts that the cloud service cost of the current job is 30% higher than the best cloud service cost currently observed, the job monitor will abort the running job and record the aborted job information in the database.

[0076] Running the batch job resource configuration optimization method provided in the above embodiments of the application on the Kubernetes container management platform optimizes the resource configuration for Spark batch jobs. Through a large number of real system experiments, the algorithm is verified. The experimental results show that, by using the batch job resource configuration optimization method provided in the embodiments of the application, the best resource configuration can be learned from more than 7000 candidate resource configurations after 20 iterations. Compared with the most advanced algorithm CherryPick, the speed is improved by 2 times, and the cost is saved by 17.9%.

[0077] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by programs instructing related hardware, which can be stored in a computer readable storage medium, including ROM, RAM, magnetic disk or optical disk, etc.

[0078] Although the present application has been disclosed with reference to the above embodiments, the application is not limited to the above embodiments. It will be apparent to those skilled in the art that various modifications and changes can be made thereto without departing from the spirit and scope of the application. The scope of the application should be limited only by the appended claims.

Claims

1. A method for optimizing resource allocation in batch processing jobs, characterized in that, The method comprises the following steps: obtaining a cloud resource configuration optimization target; the cloud resource configuration optimization target comprises maximum job completion running time and minimum cloud service cost; The cloud resource configuration optimization target is: ; wherein, is a target cloud resource configuration, is a search space, is a candidate cloud resource configuration, is a corresponding completion time, is a corresponding cloud service cost, D is the maximum job completion running time, characterizes a cloud service cost of running a target batch processing job by using the tth candidate cloud resource, characterizes is not greater than the maximum job completion running time, characterizes selecting the candidate cloud resource configuration corresponding to the minimum cloud service cost as ; determining a target cloud resource configuration corresponding to the cloud resource configuration optimization target, comprising: using candidate cloud resource configurations in a preset search space, sequentially running a target batch processing job, and obtaining cloud service costs and completion times of running the target batch processing job using the first t-1 candidate cloud resource configurations; t is a positive integer; based on the cloud service costs and the completion times of running the target batch processing job using the first t-1 candidate cloud resource configurations, performing a cost prediction operation to obtain a predicted cloud service cost and performing a time prediction operation to obtain a predicted completion time; based on the predicted cloud service cost and the predicted completion time, selecting a tth candidate cloud resource configuration from the search space and using the tth candidate cloud resource configuration to run the target batch processing operation to obtain a cloud service cost and a completion time corresponding to running the target batch processing job using the tth candidate cloud resource configuration; detecting an instruction to stop running the target batch processing operation, selecting a candidate cloud resource configuration corresponding to the minimum cloud service cost as the target cloud resource configuration.

2. The method of claim 1, wherein, Further comprising: based on a Bayesian optimization model, converting the cloud resource configuration optimization target, and the converted cloud resource configuration optimization target is: ； wherein, is corresponding completion speed, =1 / ; characterized by obtaining a candidate cloud resource configuration that maximizes the maximum.

3. The method of claim 2, wherein, the using candidate cloud resource configurations in a preset search space, sequentially running a target batch processing job, and obtaining completion times of running the target batch processing job using the first t-1 candidate cloud resource configurations, comprising: acquiring the average value and covariance; Based on the average and covariance, obtain the corresponding Gaussian process.

4. The batch job resource configuration optimization method of claim 3, wherein, The The corresponding Gaussian process is where: is the average value of the is the covariance of the ; after running the target batch job with t candidate cloud resource configurations, and The posterior estimation is performed in the following way: ; where ; I is the identity matrix, is a positive definite kernel matrix A t is the set of the first t candidate cloud resource configurations, and , is the first candidate cloud resource configuration, is the second candidate cloud resource configuration, is the nth candidate cloud resource configuration, is the candidate cloud resource configuration that completes the running.​ 5. The batch job resource configuration optimization method of claim 4, wherein, based on the completion times of running the target batch processing job using the first t-1 candidate cloud resource configurations, performing a time prediction operation to obtain a predicted completion time, comprising: based on the corresponding Gaussian processes, and a completion time resulting from running the target batch job on the first t-1 candidate cloud resource configurations, to obtain the predicted completion time.

6. The batch job resource configuration optimization method of claim 4, wherein, based on the cloud service costs of running the target batch processing job using the first t-1 candidate cloud resource configurations, performing a cost prediction operation to obtain a predicted cloud service cost, comprising: Cost prediction operations are performed using the following equation to obtain the predicted cloud service cost; h is a prediction function, is the cloud service cost for the first run of the target batch job, is the cloud service cost for the second run of the target batch job, is the cloud service cost for the nth run of the target batch job.

7. The batch job resource configuration optimization method of claim 6, wherein, The predicted cloud service cost is: , is a cloud service cost obtained by running the target batch processing job using a t-1th candidate cloud resource configuration, is a cloud service cost obtained by running the target batch processing job using a t-2th candidate cloud resource configuration, is the target batch processing job, corresponding mean posterior distribution.

8. The batch job resource configuration optimization method of claim 7, wherein, the based on the predicted cloud service cost and the predicted completion time, selecting a tth candidate cloud resource configuration from the search space, comprising: selecting the tth candidate cloud resource configuration from the search space using the following equation: , ; wherein, is a confidence upper bound weight, and , ; is the corresponding variance of the posterior distribution.

9. A batch job resource configuration optimization apparatus, characterized by, comprising: an obtaining unit, configured to obtain a cloud resource configuration optimization target; the cloud resource configuration optimization target comprises maximum job completion running time and minimum cloud service cost; The cloud resource configuration optimization target is: ; wherein, is a target cloud resource configuration, is a search space, is a candidate cloud resource configuration, is a corresponding completion time, is a corresponding cloud service cost, D is the maximum job completion running time, characterizes a cloud service cost of running a target batch job using the tth candidate cloud resource, characterizes is not greater than the maximum job completion running time, characterizes selecting a candidate cloud resource configuration corresponding to the minimum cloud service cost as ; The determining unit is used for determining a target cloud resource configuration corresponding to the cloud resource configuration optimization target, comprising: using candidate cloud resource configurations in a preset search space, sequentially running a target batch processing job, and obtaining cloud service costs and completion times obtained by running the target batch processing job using the first t-1 candidate cloud resource configurations; t is a positive integer; based on the cloud service costs and the completion times obtained by running the target batch processing job using the first t-1 candidate cloud resource configurations, performing a cost prediction operation to obtain a predicted cloud service cost, and performing a time prediction operation to obtain a predicted completion time; based on the predicted cloud service cost and the predicted completion time, selecting a tth candidate cloud resource configuration from the search space, and running the target batch processing operation using the tth candidate cloud resource configuration to obtain a cloud service cost and a completion time corresponding to the tth candidate cloud resource configuration running the target batch processing job; detecting an instruction to stop running the target batch processing operation, selecting a candidate cloud resource configuration corresponding to a minimum cloud service cost as the target cloud resource configuration.

Images