A running time prediction system and method for graph-oriented iterative jobs in Gaia system

Abstract

The present invention discloses a runtime prediction system and method for graph iteration jobs in a Gaia system. Before the job is executed, the offline features of the current graph iteration algorithm are quickly captured through sampling execution, including convergence features and key input features of each iteration; during the execution of the job, runtime features are continuously captured, including job parameters, resource utilization, and detailed statistical data; the similarity between jobs is used as the basis for job matching and final prediction value calculation, mainly including static similarity captured by sampling execution and dynamic similarity captured by real execution. The matching algorithm can train specific parameters of the algorithm through the established similarity evaluation criteria so that the iterative job can automatically adapt to various similarities. The present invention is an end-to-end runtime prediction method that integrates the offline features and runtime features of graph iteration jobs, and can accurately predict the runtime of distributed graph iteration jobs with low training overhead.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed big data computing, and in particular to a runtime prediction system for graph iteration jobs in a Gaia system. Background Art

[0002] The Gaia system is a highly efficient and scalable new generation big data computing system based on the coexistence of multiple computing models. It solves a series of key technical problems in several core aspects of big data analysis systems, such as adaptive and scalable big data storage, batch-stream fusion big data computing, high-dimensional large-scale machine learning, and high-efficiency big data intelligent interactive wizards, and builds an autonomous and controllable high-efficiency and scalable new generation big data analysis system, mastering the core technologies of the internationally leading big data analysis system.

[0003] Iterative computing is usually the core part of machine learning and graph processing algorithms, that is, one or more steps are repeatedly executed until the convergence criteria are met. It is common in various applications such as recommendation systems and data mining. Moreover, with the rapid development of Internet technology, the size of data sets in various scenarios is increasing, making it very common to use distributed computing frameworks for iterative computing in industry and science. Distributed computing systems such as Gaia allow users to perform iterative computing on computer clusters to analyze large-scale graph data sets. The Gaia system supports iterative computing by defining a step function and embedding it into a special iteration operator, and further supports incremental iteration. In order to maintain a DAG-based runtime and scheduler, Gaia builds an iteration channel by establishing iteration "head" and "tail" tasks and interconnected feedback edges. Figure 1 Shows the structure of a Gaia iterative program.

[0004] Most graph iteration jobs running on a distributed computing framework need to comply with service-level objectives, because external applications usually require timely feedback of analysis results. For example, Twitter explicitly requires that searches and queries on TB-level log data indexes be completed within ten minutes. Too high a delay will violate the service level agreement signed with users. In most cases, users tend to over-provision resources for graph iteration jobs, which leads to low overall cluster resource utilization and huge cluster operation and maintenance costs. In particular, in the worst case, even if all remaining resources are used, it is not possible to guarantee that the job will be completed within the required time limit, and these graph iteration jobs are often executed repeatedly.

[0005] Although the Gaia system has extremely high iterative computing efficiency compared to other distributed frameworks, it does not use the time limit requirements of graph iteration jobs as the basis for resource scheduling. Therefore, in order to meet the time limit requirements of graph iteration jobs and optimize their resource supply in a timely manner, predicting the running time of graph iteration jobs on the distributed computing framework is currently an urgent problem to be solved.

[0006] However, it is difficult to estimate the running time or resource requirements of distributed graph iteration jobs in the early stage of execution, because the running time of distributed graph iteration jobs depends on many factors, including the logic of user-defined functions, program parameters, input data sets, and system configuration. In addition, each iteration of incremental iterative calculations supported by the Gaia system is dependent on the results of the previous iteration, making it even more difficult to obtain the above information. Traditional methods simply use the execution data of job history to predict the running time. Some methods train performance models to predict the running time of jobs under given resource configurations by sampling data sets and running jobs on a small number of servers to collect data. Other methods directly perform sampling execution to extract input features of each iteration for running time prediction. Although these methods are quite effective, they are limited to predictions under fixed configuration clusters, and because the data obtained are all offline data or rough statistical data, the prediction accuracy is not high. Summary of the invention

[0007] In view of this, the present invention provides a runtime prediction system for graph iteration jobs in a Gaia system, which can predict the runtime of graph iteration jobs without being limited to clusters with fixed configurations and has high prediction accuracy.

[0008] To achieve the above-mentioned purpose, the technical solution of the present invention is: a runtime prediction system for graph iterative jobs in a Gaia system, comprising a similarity management module, a job matching module and a runtime prediction module.

[0009] The similarity management module includes a sampling execution unit, a static similarity calculation unit and a dynamic similarity calculation unit.

[0010] The sampling execution unit samples the graph data set by a biased random jump method to obtain sample graph data, and executes an iterative algorithm on the sample graph data set to capture offline features of the sample graph iterative job.

[0011] The static similarity calculation unit calculates the static similarity between the current graph iteration job and the historical graph iteration jobs by sampling the offline features captured by the execution unit.

[0012] The dynamic similarity calculation unit calculates the dynamic similarity between the current graph iteration job and the historical graph iteration jobs by capturing the statistical information during the actual execution of the graph iteration job.

[0013] The similarity management module sends the calculated static similarity and dynamic similarity to the job matching module.

[0014] The job matching module includes a similarity evaluation unit, a training point collection unit and a parameter training unit.

[0015] The similarity evaluation unit constructs a similarity evaluation function, performs validity evaluation on the static similarity and dynamic similarity sent by the similarity management module, and selects matching historical graph iteration jobs based on the results of the validity evaluation and sends them to the runtime prediction module.

[0016] The training point collection unit is used to collect training data points for parameters in the training similarity evaluation function.

[0017] The parameter training unit uses the training data points to train the parameters of the similarity evaluation function and sends them to the similarity evaluation unit.

[0018] The runtime prediction module predicts the remaining runtime of the current graph iteration job through the historical graph iteration jobs matched by the job matching module.

[0019] Furthermore, the static similarity calculation unit uses the following method to calculate the current graph iteration job Job run and historical graph iteration jobs comp The static similarity between them includes static computational similarity and static message passing similarity.

[0020] Furthermore, the static similarity is calculated as:

[0021]

[0022] Among them, S1 is the current graph iteration job Job run and historical graph iteration jobs comp The static calculation similarity between the two, the active node ratio of the graph iteration job at the i-th iteration ActVert i (Job) represents the number of active nodes in the graph iteration job Job at the i-th iteration, TotVert i (Job) indicates the summary points of the graph iteration job at the i-th iteration, and N indicates the total number of iterations during the sampling execution; the graph iteration job includes the current graph iteration job run and historical graph iteration jobs comp ;

[0023] The static message passing similarity is

[0024]

[0025] Among them, S2 represents the current graph iteration job Job run and historical graph iteration jobs comp The static message transmission similarity between them; the total amount of messages transmitted by the graph iteration job Job at the i-th iteration is RemMsg i (Job) represents the number of messages delivered by the graph iteration job Job at the i-th iteration. It represents the average message size of the graph iteration job at the i-th iteration, and N represents the total number of iterations during the sampling execution.

[0026] Furthermore, the dynamic similarity calculation of the current graph iteration job run and historical graph iteration jobs comp The static similarity between them includes runtime similarity S3, resource allocation similarity S4, convergence similarity S5 and resource utilization similarity S6.

[0027] The running time similarity S3 is

[0028]

[0029] Among them, Step represents the current iteration number of the job, runtime(Job,i) represents the running time of the graph iteration job Job at the i-th iteration; the graph iteration job Job includes the current graph iteration job Job run and historical graph iteration jobs comp ;

[0030] The resource allocation similarity is:

[0031]

[0032] Among them, Step represents the current iteration number of the job, cons avg (Job, Step) represents the average number of containers used by all iterations of the graph iteration job from the beginning to the current execution; the graph iteration job includes the current graph iteration job. run and historical graph iteration jobs comp .

[0033] The convergence similarity is:

[0034]

[0035] Among them, workSet(Job,i) represents the number of data actively processed by the graph iteration job Job at the i-th iteration;

[0036] The resource utilization similarity is

[0037]

[0038] Among them, cons is the number of containers used by the graph iteration job; RsUtil job [hw,Step,i] represents the average utilization of the hardware device hw in container i when the graph iteration job executes to the Step-th iteration.

[0039] Furthermore, the similarity evaluation function constructed by the similarity evaluation unit is specifically:

[0040] Average optimal precision function h(t) and optimal similarity ratio function n(t);

[0041] h(t):=avg{acc|(DoS,acc)∈P,DoS≥t};

[0042] n(t):=|{acc|(DoS,acc)∈P,DoS≥t}| / |P|;

[0043] Where acc is the prediction accuracy of the running time prediction system; DoS is the overall similarity of the two graph iteration jobs, which is the weighted average of all individual similarities, and the weight values ​​of all individual similarities constitute the weight vector; t is the similarity threshold; P is a simulation point set consisting of a set of simulation points (DoS, acc); the weight vector and the similarity threshold are the parameters of the similarity evaluation function.

[0044] Furthermore, the training point collection unit first extracts n times of history graph iteration job set in an orderly and non-repetitive manner, and extracts two graph iteration jobs from it each time, which are the first graph iteration job J and i and the second graph iteration job K i , i=1,2,3,...,n.

[0045] Extracted first graph iteration job J i Running and just finished executing the M∈[1,m i ] iterations, m i It is a graph iteration job J i The maximum number of iterations.

[0046] Extracted second graph iterative job K i Yes and J i Based on the similarity between the two extracted graph iteration jobs, the running graph iteration job J is predicted by the runtime prediction module. i The remaining running time of the first graph iteration job J iThe actual remaining running time is known. By using different current graph iteration jobs and matching historical graph iteration jobs and assuming that the current graph iteration job is in a different iteration order, a set P(w) of the form of (predict(w), actual) is calculated as a training data point; where predict(w) is the predicted remaining execution time with w as the weight vector, and actual is the actual remaining execution time of the current graph iteration job.

[0047] The parameter training unit finds a suitable similarity threshold t and a similarity weight vector w = (w1, ..., w6) for the currently running graph iteration job, so that the values ​​of the similarity evaluation functions h(t) and n(t) in the similarity evaluation unit are both greater than the set thresholds, to ensure that the number of matching results of the current graph iteration job is greater than the set threshold.

[0048] Furthermore, the runtime prediction module includes a single prediction unit and a final prediction unit; the single prediction unit predicts the runtime of the current graph iteration job based on the matching single historical job, obtains a series of single prediction estimation values, constitutes a prediction time estimation set and outputs it to the final prediction unit.

[0049] The final prediction unit is based on the prediction estimation set, and all the estimated values ​​in the prediction estimation set are weighted averaged to merge into a final prediction value; the weight uses the overall similarity value corresponding to each historical graph iteration job during matching execution.

[0050] The present invention also provides a method for predicting the running time of a graph iteration job in a Gaia system, and the prediction process is as follows:

[0051] Step 1: Obtain the historical graph iteration job set, the initial input data set, the to-be-predicted graph iteration job and its job parameters and iteration termination conditions;

[0052] Step 2: The sampling execution unit samples the input data set with a sampling ratio of 10%, and then scales the convergence parameters of the graph iteration job to be predicted according to the sampling ratio, and initializes the parameters after scaling;

[0053] Step 3: Set the data source of the iterative job of the graph to be predicted to the sample set after sampling, set the initial number of iterations to 0, and then execute the iterative job of the prediction graph, and store the statistical information captured during each iteration in the memory database;

[0054] Step 4: Calculate the static similarity between the to-be-predicted graph iteration job and each job in the historical graph iteration job set based on the statistical information captured in step 3;

[0055] Step 5: Re-obtain the job parameters and iteration termination conditions of the iteration job of the graph to be predicted, and initialize the parameters, static similarity, and dynamic similarity; set the data source of the iteration job of the graph to be predicted to the initial input data set, and set the number of iterations to 0;

[0056] Step 6: When the number of iterations is 0, the historical graph iteration job matching the to-be-predicted graph iteration job is output according to the static similarity calculated in step 4, and then the running time prediction module calculates the static prediction value of the running time of the to-be-predicted graph iteration job according to the matched historical graph iteration job;

[0057] Step 7: Run the iteration job of the graph to be predicted, perform iterative calculations on the input data, and obtain the dynamic similarity between the iteration job of the graph to be predicted and each graph iteration job in the historical graph iteration job set;

[0058] Step 8: The job matching module outputs the historical graph iteration job that matches the graph iteration job to be predicted based on the static similarity calculated in step 4 and the dynamic similarity calculated in step 7, and then calculates the dynamic prediction value of the running time of the graph iteration job to be predicted based on the matched historical graph iteration job;

[0059] Step 9: Determine whether the number of iterations or the iteration output result meets the iteration termination condition. If not, repeat step 7. If so, execute step 10.

[0060] Step 10: Calculate the prediction deviation of the running time of the iterative job of the graph to be predicted under different iterations. The dynamic prediction value with the smallest prediction deviation is the final prediction value of the running time of the iterative job of the graph to be predicted.

[0061] Further, step 7 is performed using the following steps 7.1 to 7.4:

[0062] Step 7.1: Increase the number of iterations by 1.

[0063] Step 7.2: Execute the iterative step function in parallel on each node of the Gaia cluster. After each node is executed, the data on each node is synchronized by broadcasting to ensure the consistency of data on different nodes.

[0064] Step 7.3: After data synchronization is completed, append the statistical information captured during this iteration to the in-memory database for update.

[0065] Step 7.4: Calculate the dynamic similarity between the to-be-predicted graph iteration job and each graph iteration job in the historical graph iteration job set based on the statistical information updated in step 7.3.

[0066] Beneficial effects:

[0067] The present invention provides a runtime prediction method for graph iteration jobs in a Gaia system. Before job execution, sampling execution is used to quickly capture offline features of the current graph iteration algorithm, including convergence features and key input features of each iteration; runtime features are continuously captured during job execution, including job parameters, resource utilization, and detailed statistical data; and similarities between jobs are defined as the basis for job matching and final prediction value calculation, mainly including static similarities captured by sampling execution and dynamic similarities captured by real execution. The core matching algorithm can train specific parameters of the algorithm through established similarity evaluation criteria so that iterative jobs can automatically adapt to various similarities. This is an end-to-end runtime prediction method that integrates offline features and runtime features of graph iteration jobs, and can accurately predict the runtime of distributed graph iteration jobs with low training overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 Iterative model architecture diagram supported in the Gaia system provided as background technology of the present invention;

[0069] Figure 2 A prediction flow chart of the running time prediction method provided by an embodiment of the present invention;

[0070] Figure 3 A system architecture diagram of the running time prediction system provided by an embodiment of the present invention on the Gaia system;

[0071] Figure 4 An organizational chart of the run time prediction method provided by an embodiment of the present invention;

[0072] Figure 5 A distribution diagram of relative errors of predictions of various iterative jobs by the runtime prediction system provided by an embodiment of the present invention;

[0073] Figure 6 The average relative prediction error of the runtime prediction system provided by the embodiment of the present invention for multiple iterative jobs. DETAILED DESCRIPTION

[0074] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0075] The present invention provides a running time prediction system for graph iteration jobs in a Gaia system, comprising: a similarity management module, a job matching module and a running time prediction module.

[0076] The similarity management module includes a sampling execution unit, a static similarity calculation unit and a dynamic similarity calculation unit.

[0077] The sampling execution unit samples the graph data set by a biased random jump method to obtain sample graph data. Then, for iterative algorithms such as PageRank whose convergence threshold changes according to the size of the data set, a scaling function needs to be used to adjust its parameters. The form of the scaling function T can be described as follows:

[0078]

[0079] Among them, Conf S =>Conf G Represents the mapping of configuration parameters, conf configuration parameters, s original image, G sample image after sampling, Conv S = > Conv G represents the mapping of convergence parameters, and sr is the sampling ratio. Then, the iterative algorithm after parameter scaling is quickly executed on the sampled data set to capture the offline features of the iterative job of the sample graph.

[0080] The static similarity calculation unit calculates the static similarity we define based on comprehensive consideration of the execution process of the iterative algorithm by sampling the feature information captured by the execution unit. The static similarity is the current graph iteration job Job run and historical graph iteration jobs comp There are two types of similarity:

[0081] (1) Calculate similarity. In the calculation phase, each vertex of the input graph data is used to execute the iterative algorithm logic defined by the user. For large-scale iterative algorithms, the cost of performing a calculation on each vertex is fixed. The time spent in the calculation phase is proportional to the number of messages to be processed, that is, proportional to the number of active neighbor nodes. The calculation similarity is then defined as

[0082]

[0083] in, ActVert i (Job) represents the number of active nodes of the job at the i-th iteration, TotVert i (Job) represents the summary points of the job at the i-th iteration, and N represents the total number of iterations during the sampling execution.

[0084] (2) Message passing similarity. Since messages from different processors need to be sent through the network and added to the memory of the destination node during the message passing phase, the running time of this phase is proportional to the number and size of remote messages. In particular, the memory sharing of the Gaia system makes the communication cost of local messages negligible. The message passing similarity is then defined as

[0085]

[0086] in, RemMsg i (Job) represents the number of messages delivered by the job at the i-th iteration, represents the average message size of the job at the i-th iteration, and N represents the total number of iterations during the sampled execution.

[0087] The dynamic similarity calculation unit calculates the dynamic similarity we define based on comprehensive consideration of domain knowledge and experiments by capturing statistical information during the actual execution of the graph iteration job. Similarly, the dynamic similarity is also the similarity between the current graph iteration job and the historical graph iteration job. These dynamic similarities cover the entire process of the graph iteration job at the runtime layer. There are four types:

[0088] (1) Runtime similarity. Runtime similarity compares the runtime of each iteration of a graph iteration job from the beginning to the current execution. For the currently executing graph iteration job, assuming that the iterations that have been executed have a great similarity in runtime with the historical graph iteration jobs, then we have reason to believe that the runtime of the remaining iterations is also very likely to be similar. Runtime similarity is obtained by calculating the average runtime deviation of each iteration corresponding to the two graph iteration jobs. Runtime similarity is defined as

[0089]

[0090] Among them, Step indicates the current iteration number of the job;

[0091]

[0092] runtime(Job,i) represents the running time of the graph iteration job at the i-th iteration.

[0093] (2) Resource allocation similarity. Resource allocation similarity compares the number of containers used by a graph iteration job from the beginning to the current iteration. As introduced in Section 3.2, the container here is an abstract representation of basic computing resources such as the number of CPU cores, memory, disk, and network bandwidth by the resource management system. Resource allocation similarity can make full use of the resource allocation strategy of the resource management system to feedback the performance consumption of the graph iteration job during execution. Resource allocation similarity is defined as:

[0094]

[0095] Step represents the current iteration number of the job, cons(Job,i) represents the number of containers used by the graph iteration job in the i-th iteration, and consavg (Job, Step) represents the average number of containers used by the graph iteration job from the beginning to the current execution of all iterations.

[0096] (3) Convergence similarity. Without considering the parallel capability of the distributed cluster and the impact of data partitioning, the convergence behavior of the graph iteration job mainly depends on the input data set and system parameters. We use the number of data actively processed by the graph iteration job to intuitively measure its iteration behavior, and obtain the convergence similarity by calculating the average deviation of the number of data actively processed by the two graph iteration jobs from the beginning of execution to each iteration of the current state. Convergence similarity is defined as:

[0097]

[0098]

[0099] Among them, Step represents the current iteration number of the job, and workSet(Job,i) represents the number of data actively processed by the graph iteration job in the i-th iteration.

[0100] (4) Resource utilization similarity. The resource management system monitors the iterative application and supervises the lifecycle management of the container, and returns the utilization of these resources in each container through heartbeats. Then, we calculate the arithmetic mean of resource utilization from the perspectives of container and iteration number during the execution of the graph iteration job based on the returned information, and maintain these averages as the status of the graph iteration job in a three-dimensional array RsUtil job [hw, step, containerId], the three dimensions of the array represent the system hardware device, the number of iterations, and the container number from low to high, that is, RsUtil job [i,j,k] represents the average utilization of hardware device i in container k when the graph iteration job executes to the jth iteration. Resource utilization similarity is defined as:

[0101]

[0102]

[0103] Where Step represents the current iteration number of the job, and cons is the number of containers used by the graph iteration job. Formula 8 is only applicable to iterations using the same number of containers, because it is performed by calculating the relative average deviation of resource utilization under paired containers at each iteration. For iterations using different numbers of containers, their average deviation under all containers is directly calculated.

[0104] The job matching module includes a similarity evaluation unit, a training point collection unit, and a parameter training unit.

[0105] The similarity evaluation unit ensures the effectiveness of the above similarity management module by establishing an evaluation function. Specifically, for the historical graph iteration job set H, we assume that the historical job J (J∈H) is currently running and has just completed the Ith iteration. Assuming that the historical job K (K∈HandK≠J) has been completed before job J starts running, then we can calculate the job similarity S={S1,...,S7} of the two jobs when the Ith iteration is completed. S1 to S5 correspond to the seven individual similarities of the jobs respectively. Finally, we define the overall similarity DoS of job J and job K as the weighted average of each individual similarity:

[0106]

[0107] Among them, the parameter w i (i=1,...,6) respectively correspond to the similarity S of each item of the task i Then assume that the runtime prediction system of this paper predicts the remaining runtime of job J based on the similarity between them. Then we can calculate the prediction accuracy acc. Therefore, by repeating the above simulation operation many times, we can finally get a set P of points (DoS, acc).

[0108] Obviously, the points in P express the relationship between the similarity of graph iteration jobs and the final prediction accuracy. We define the similarity evaluation criteria based on the set P obtained from the above simulation:

[0109] h(t):=avg{acc|(DoS,acc)∈P,DoS≥t} (11)

[0110] n(t):=|{acc|(DoS,acc)∈P,DoS≥t}| / |P| (12)

[0111] The t in the evaluation criteria is the threshold of similarity. The overall similarity between the current graph iteration job and the matched historical job should be greater than the threshold t. Specifically, the average optimal precision function h(t) is used to calculate the average precision of all similarities in the set P that are greater than a specific threshold t. Then, the optimal similarity share function n(t) represents the share of all points in the set P whose similarity is greater than a specific threshold t.

[0112] The training point collection unit is used to collect data points for training similarity weight vectors. First, the historical graph iteration job set is extracted n times in an orderly and non-repetitive manner, and two graph iteration jobs J are extracted from it each time. i and K i (i=1,2,3,...,n), and assume that job Ji Running and just finished executing (M∈[1,m i ]) iterations, m i It is a graph iteration job J i The maximum number of iterations. Then assume that job K i Yes and J i The matching historical jobs can be used to predict the running graph iteration job J based on the similarity between the two jobs through the runtime prediction module. i The remaining running time of job J i The actual remaining running time is known, so by using different current jobs and matching historical jobs and assuming that the current jobs are in different iteration orders, a set P(w) of elements in the form of (predict(w), actual) can be calculated as training points. Among them, predict(w) is the predicted remaining execution time after using w as the weight vector, and actual is the actual remaining execution time of the current graph iteration job.

[0113] The parameter training unit finds a suitable similarity threshold t and a similarity weight vector w = (w1, ..., w6) for the currently running graph iteration job, so that the values ​​of the functions h(t) and n(t) in the similarity evaluation unit are at a high level to ensure that the current graph iteration job has enough matching results to predict the running time.

[0114] Training similarity threshold. The similarity threshold is used to determine which jobs in the historical graph iteration job set will be matched. In addition to using the global threshold t to remove historical graph iteration jobs with insufficient overall similarity, for each individual similarity S of the graph iteration job i (i=1,...,6) all set the corresponding threshold t i , which is used to remove historical jobs whose similarity is not high enough in a certain dimension. The parameter training unit uses the similarity evaluation standard to evaluate the global threshold t and each single threshold t i Specifically, it sets the minimum similarity ratio n min and the minimum average precision h min This is to avoid the inability to filter out enough historical jobs due to too high a threshold during the parameter search process. The specific conditions are as follows.

[0115] t=max{t∈[0,1]|h(t)≥h min ∧n(t)≥n min} (13)

[0116] t i =max{t i ∈[0,1]|h(t i)≥h min ∧n(t i )≥n min} (14)

[0117] Train the similarity weight vector. In order to quickly obtain effective hyperparameters, we use the Hyperopt tool (Python, a tool for training parameter vectors) to perform distributed asynchronous optimization on the weight vector w. The data set uses the set P(w) collected by the data point collection unit, and the average relative prediction error in the training set is used as the loss function o:

[0118]

[0119] Here, predict(w) is the predicted remaining execution time with w as the weight vector, and actual is the actual remaining execution time of the current graph iteration job.

[0120] The runtime prediction module mainly includes a single prediction unit and a final prediction unit. It predicts the remaining runtime of the current graph iteration job through the historical graph iteration jobs matched by the job matching module. The specific prediction process is as follows: Figure 2 shown.

[0121] A single prediction unit mainly predicts the running time of the current graph iteration job based on the matching single historical job. First, it is assumed that the current job will have the same number of iterations as the matching historical job. Since cluster configuration and program logic are usually not exactly the same, the current job may have a slight difference in running time even though it is very similar to the historical job. To this end, a single prediction unit can calculate the relative average deviation coefficient δ between the current graph iteration job and each matching historical job. The current job Job run The i-th historical job that matches The calculation process of the relative average deviation coefficient executed to the Sth iteration is as follows:

[0122]

[0123] in, and Represent the actual running time of the current job and the historical job in the kth iteration respectively.

[0124] Assuming that the coefficient of variance of the running time up to the current iteration will be maintained until the end of the job execution, the running time prediction based on a single matching historical job is calculated as follows:

[0125]

[0126] The output of a single prediction unit is an estimate set runtimerun , where the elements are a series of estimates of the running time of the current job obtained through a single prediction. end The maximum number of iterations.

[0127] The final prediction unit is based on the estimated set we calculated in the previous step. It combines the single estimated values ​​based on the matched historical jobs into a final prediction value by weighted averaging. The weights are based on the overall similarity value corresponding to each historical job when the matching execution is executed. In summary, the historical jobs with high similarity to the current job also contribute more to the final estimate. Therefore, the runtime prediction value of the current graph iteration job when executing to the Sth iteration is runtime S The calculation process is as follows:

[0128]

[0129] Among them, DoS i Represents the overall similarity between the i-th historical job and the current job, end is the total number of iterations, and N is the number of matched historical jobs.

[0130] The specific prediction process of the running time prediction method of the graph iteration job is as follows:

[0131] Step 1: Obtain the historical graph iteration job set, the initial input data set, the graph iteration job to be predicted, and the job parameters and iteration termination conditions.

[0132] Step 2: The sampling execution unit samples the input data set with a sampling ratio of 10%, and then scales the convergence parameters of the graph iteration job to be predicted according to the sampling ratio, and initializes the parameters after scaling.

[0133] Step 3: Set the data source of the iterative job of the graph to be predicted to the sample set after sampling, set the initial number of iterations to 0, and then execute the job, and store the statistical information captured during each iteration in the memory database.

[0134] Step 4: The static similarity calculation unit calculates the computational similarity and message transmission similarity between each job in the to-be-predicted graph iteration job and the historical graph iteration job set according to the statistical information captured in step 3.

[0135] Step 5: Re-obtain the parameters and iteration termination conditions, and initialize the parameters and the similarities of the six jobs. Set the data source of the iterative job of the graph to be predicted to the initial input data set, and set the number of iterations to 0.

[0136] Step 6: If the number of iterations is 0, the job matching module outputs the historical graph iteration job that matches the graph iteration job to be predicted based on the static similarity calculated in step 4, and then the running time prediction module calculates the static prediction value of the running time of the graph iteration job to be predicted based on the matched historical jobs, which is used to quickly grasp the running time of the job in a short time.

[0137] Step 7: Run the graph iteration job to be predicted and perform iterative calculations on the input data.

[0138] Step 7.1: Increase the number of iterations by 1.

[0139] Step 7.2: Execute the iterative step function in parallel on each node of the cluster. After each node completes the execution, the data on each node is synchronized by broadcasting to ensure the consistency of data on different nodes.

[0140] Step 7.3: After data synchronization is completed, append the statistical information captured during this iteration to the in-memory database for update.

[0141] Step 7.4: The dynamic similarity calculation unit calculates four types of dynamic similarities between each job in the to-be-predicted graph iteration job and the historical graph iteration job set according to the statistical information updated in step 7.4.

[0142] Step 8: The job matching module outputs the historical graph iteration job that matches the graph iteration job to be predicted based on the static similarity calculated in step 4 and the dynamic similarity calculated in step 7.4, and then the running time prediction module calculates the dynamic prediction value of the running time of the graph iteration job to be predicted based on the matched historical jobs.

[0143] Step 9: Determine whether the number of iterations or the iteration output result meets the iteration termination condition. If not, repeat step 7. If yes, execute step 10.

[0144] Step 10: Calculate the prediction deviation of the running time of the graph iteration job under different iterations. The dynamic prediction value with the smallest deviation is the final prediction value.

[0145] Embodiment 1:

[0146] This embodiment is as follows Figure 3 The running time prediction system integrated into the Gaia system shown in the figure is used to process four iterative algorithms: PageRank, Connected Components, SSSP, and Adsorption as actual application scenarios. The data sets used by the above iterative algorithms are shown in Table 1.

[0147] The Single Source Shortest Path algorithm (SSSP) calculates the shortest distance from a source node to all other nodes in the graph. PageRank is a well-known web page ranking algorithm that updates the importance score of each page node through iterative recursive calculations. The Adsorption algorithm diffuses labels in the graph according to the Random Walk model until the distribution of labels in each node in the graph reaches a stable state. The Connected Components algorithm uses iterative searches to find the interconnected parts of a large graph. The data sets in Table 1 are all from the real world and downloaded from the Stanford Large Network Dataset website.

[0148] Table 1 Dataset

[0149]

[0150] In this embodiment, a method for predicting the running time of a graph-oriented iterative job in a Gaia system is provided. Figure 4 As shown, it includes: a similarity management module, a job matching module and a running time prediction module;

[0151] The similarity management module includes a sampling execution unit, a static similarity calculation unit and a dynamic similarity calculation unit.

[0152] The sampling execution unit samples the data set by a biased random jump method. Then, for iterative algorithms such as PageRank whose convergence threshold changes according to the size of the data set, a scaling function needs to be used to adjust its parameters. The form of the scaling function T can be described as follows:

[0153]

[0154] Among them, Conf S =>Conf G Represents the mapping of configuration parameters, Conv S = > Conv G represents the mapping of convergence parameters, and sr is the sampling ratio. Then, the iterative algorithm after parameter scaling is quickly executed on the sampled dataset to capture the offline characteristics of the graph iteration job.

[0155] The static similarity calculation unit calculates the static similarity defined by comprehensively considering the execution process of the iterative algorithm by sampling the feature information captured by the execution unit. The static similarity is the similarity between the current graph iteration job and the historical graph iteration job, including the following two types:

[0156] 1. Calculate similarity. In the calculation phase, each vertex of the input graph data is used to execute the iterative algorithm logic defined by the user. For large-scale iterative algorithms, the cost of calculating each vertex once is fixed, so the time spent in the calculation phase is proportional to the number of messages to be processed, that is, proportional to the number of active neighbor nodes. The calculation similarity is defined as

[0157]

[0158] in, ActVert i (Job) represents the number of active nodes of the job at the i-th iteration, TotVert i (Job) represents the summary points of the job at the i-th iteration, and N represents the total number of iterations during the sampling execution.

[0159] 2. Message passing similarity. Since messages from different processors need to be sent through the network and added to the memory of the destination node during the message passing phase, the running time of this phase is proportional to the number and size of remote messages. In particular, the memory sharing of the Gaia system makes the communication cost of local messages negligible. Then the message passing similarity is defined as

[0160]

[0161] in, RemMsg i (Job) represents the number of messages delivered by the job at the i-th iteration, represents the average message size of the job at the i-th iteration, and N represents the total number of iterations during the sampled execution.

[0162] The dynamic similarity calculation unit calculates the dynamic similarity we define based on comprehensive consideration of domain knowledge and experiments by capturing statistical information during the actual execution of the graph iteration job. Similarly, the dynamic similarity is also the similarity between the current graph iteration job and the historical graph iteration job. These dynamic similarities cover the entire process of the graph iteration job at the runtime layer. There are four types:

[0163] (1) Runtime similarity. Runtime similarity compares the runtime of each iteration of a graph iteration job from the beginning to the current execution. For the currently executing graph iteration job, assuming that the iterations that have been executed have a great similarity in runtime with the historical graph iteration jobs, then we have reason to believe that the runtime of the remaining iterations is also very likely to be similar. Runtime similarity is obtained by calculating the average runtime deviation of each iteration corresponding to the two graph iteration jobs. Runtime similarity is defined as

[0164]

[0165] Among them, Step represents the current iteration number of the job, and runtime(Job,i) represents the running time of the graph iteration job at the i-th iteration.

[0166] (2) Resource allocation similarity. Resource allocation similarity compares the number of containers used by a graph iteration job from the beginning to the current iteration. As introduced in Section 3.2, the container here is an abstract representation of basic computing resources such as the number of CPU cores, memory, disk, and network bandwidth by the resource management system. Resource allocation similarity can make full use of the resource allocation strategy of the resource management system to feedback the performance consumption of the graph iteration job during execution. Resource allocation similarity is defined as:

[0167]

[0168] Step represents the current iteration number of the job, cons(Job,i) represents the number of containers used by the graph iteration job in the i-th iteration, and cons avg (Job, Step) represents the average number of containers used by the graph iteration job from the beginning to the current execution of all iterations.

[0169] (3) Convergence similarity. Without considering the parallel capability of the distributed cluster and the impact of data partitioning, the convergence behavior of the graph iteration job mainly depends on the input data set and system parameters. We use the number of data actively processed by the graph iteration job to intuitively measure its iteration behavior, and obtain the convergence similarity by calculating the average deviation of the number of data actively processed by the two graph iteration jobs from the beginning of execution to each iteration of the current state. Convergence similarity is defined as:

[0170]

[0171]

[0172] Among them, Step represents the current iteration number of the job, and workSet(Job,i) represents the number of data actively processed by the graph iteration job in the i-th iteration.

[0173] (4) Resource utilization similarity. The resource management system monitors the iterative application and supervises the lifecycle management of the container, and returns the utilization of these resources in each container through heartbeats. Then, we calculate the arithmetic mean of resource utilization from the perspectives of container and iteration number during the execution of the graph iteration job based on the returned information, and maintain these averages as the status of the graph iteration job in a three-dimensional array RsUtil job[hw, step, containerId], the three dimensions of the array represent the system hardware device, the number of iterations, and the container number from low to high, that is, RsUtil job [i,j,k] represents the average utilization of hardware device i in container k when the graph iteration job executes to the jth iteration. Resource utilization similarity is defined as:

[0174]

[0175]

[0176] Where Step represents the current iteration number of the job, and cons is the number of containers used by the graph iteration job. Formula 26 is only applicable to iterations using the same number of containers, because it is performed by calculating the relative average deviation of resource utilization under paired containers at each iteration. For iterations using different numbers of containers, their average deviation under all containers is directly calculated.

[0177] The job matching module includes a similarity evaluation unit, a training point collection unit, and a parameter training unit.

[0178] The similarity evaluation unit ensures the effectiveness of the above similarity management module by establishing an evaluation function. Specifically, for the historical graph iteration job set H, we assume that the historical job J (J∈H) is currently running and has just completed the Ith iteration. Assuming that the historical job K (K∈HandK≠J) has been completed before job J starts running, then we can calculate the job similarity S={S1,...,S7} of the two jobs when the Ith iteration is completed. S1 to S5 correspond to the seven individual similarities of the jobs respectively. Finally, we define the overall similarity CoS of job J and job K as the weighted average of each individual similarity:

[0179]

[0180] Among them, the parameter w i (i=1,...,6) respectively correspond to the similarity S of each item of the task i Then assume that the runtime prediction system of this paper predicts the remaining runtime of job J based on the similarity between them. Then we can calculate the prediction accuracy acc. Therefore, by repeating the above simulation operation many times, we can finally get a set P of points (DoS, acc).

[0181] Obviously, the points in P express the relationship between the similarity of graph iteration jobs and the final prediction accuracy. We define the similarity evaluation criteria based on the set P obtained from the above simulation:

[0182] h(t):=avg{acc|(DoS,acc)∈P,DoS≥t} (29)

[0183] n(t):=|{acc|(DoS,acc)∈P,DoS≥t}| / |P| (30)

[0184] The t in the evaluation criteria is the threshold of similarity. The overall similarity between the current graph iteration job and the matched historical job should be greater than the threshold t. Specifically, the average optimal precision function h(x) is used to calculate the average precision of all similarities in the set P that are greater than a specific threshold t. Then, the optimal similarity share function n(t) represents the share of all points in the set P whose similarity is greater than a specific threshold t.

[0185] The training point collection unit is used to collect data points for training similarity weight vectors. First, the historical graph iteration job set is extracted n times in an orderly and non-repetitive manner, and two graph iteration jobs J are extracted from it each time. i and K i (i=1,2,3,...,n), and assume that job J i Running and just finished executing (M∈[1,m i ]) iterations, m i It is a graph iteration job J i The maximum number of iterations. Then assume that job K i Yes and J i The matching historical jobs can be used to predict the running graph iteration job J based on the similarity between the two jobs through the job prediction module. i The remaining running time of job J i The actual remaining running time is known. By using different current jobs and matching historical jobs and assuming that the current jobs are in different iteration orders, a set P(w) of the form of (predict(w), actual) can be calculated as training points.

[0186] The parameter training unit finds a suitable similarity threshold t and a similarity weight vector w = (w1, ..., w6) for the currently running graph iteration job, so that the values ​​of the functions h(t) and n(t) in the similarity evaluation unit are at a high level to ensure that the current graph iteration job has enough matching results to predict the running time.

[0187] Training similarity threshold. The similarity threshold is used to determine which jobs in the historical graph iteration job set will be matched. In addition to using the global threshold t to remove historical graph iteration jobs with insufficient overall similarity, for each individual similarity S of the graph iteration job i(i=1,...,6) all set the corresponding threshold t i , which is used to remove historical jobs whose similarity is not high enough in a certain dimension. The parameter training unit uses the similarity evaluation standard to evaluate the global threshold t and each single threshold t i Specifically, it sets the minimum similarity ratio n min and the minimum average precision h min This is to avoid the inability to filter out enough historical jobs due to too high a threshold during the parameter search process. The specific conditions are as follows.

[0188] t=max{t∈[0,1]|h(t)≥h min ∧n(t)≥n min} (31)

[0189] t i =max{t i ∈[0,1]|h(t i )≥h min ∧n(t i )≥n min} (32)

[0190] Train the similarity weight vector. In order to quickly obtain effective hyperparameters, we use the Hyperopt tool to perform distributed asynchronous optimization on the weight vector w. The data set uses the set P(w) collected by the data point collection unit, and the average relative prediction error in the training set is used as the loss function o:

[0191]

[0192] Here, predict(w) is the predicted remaining execution time with w as the weight vector, and actual is the actual remaining execution time of the current graph iteration job.

[0193] The runtime prediction module mainly includes a single prediction unit and a final prediction unit. It predicts the remaining runtime of the current graph iteration job through the historical graph iteration jobs matched by the job matching module. The specific prediction process is as follows: Figure 2 shown.

[0194] A single prediction unit mainly predicts the running time of the current graph iteration job based on the matching single historical job. First, it is assumed that the current job will have the same number of iterations as the matching historical job. Since cluster configuration and program logic are usually not exactly the same, the current job may have a slight difference in running time even though it is very similar to the historical job. To this end, a single prediction unit can calculate the relative average deviation coefficient δ between the current graph iteration job and each matching historical job. The current job Job runThe i-th historical job that matches The calculation process of the relative average deviation coefficient executed to the Sth iteration is as follows:

[0195]

[0196] in, and Represent the actual running time of the current job and the historical job in the kth iteration respectively.

[0197] Assuming that the coefficient of variance of the running time up to the current iteration will be maintained until the end of the job execution, the running time prediction based on a single matching historical job is calculated as follows:

[0198]

[0199] The output of a single prediction unit is an estimate set runtime run , where the elements are a series of estimates of the running time of the current job obtained through a single prediction.

[0200] The final prediction unit is based on the estimated set we calculated in the previous step. It combines the single estimated values ​​based on the matched historical jobs into a final prediction value by weighted averaging. The weights are based on the overall similarity value corresponding to each historical job when the matching execution is executed. In summary, the historical jobs with high similarity to the current job also contribute more to the final estimate. Therefore, the runtime prediction value of the current graph iteration job when executing to the Sth iteration is runtime S The calculation process is as follows:

[0201]

[0202] Among them, DoS i Represents the overall similarity between the i-th historical job and the current job, end is the total number of iterations, and N is the number of matched historical jobs.

[0203] This embodiment uses the runtime prediction method of the present invention to predict the runtime of various graph iteration jobs composed of the above four iteration algorithms under different data sets, and obtains the prediction relative error that changes with the iteration process, such as Figure 5As shown in the figure. It can be seen that the prediction error of the running time of each graph iteration job basically remains at about 10% after stabilization, but there is a short-term trend of relative error from low to high at the beginning of the graph iteration job. This is because at the beginning of the job running, the proportion of static similarity is relatively large. However, as the iteration begins, the proportion of dynamic similarity gradually increases, but only a small amount of statistical information can be captured, so the accuracy of dynamic similarity is low, resulting in an increase in relative error. As the iteration proceeds, more and more statistical information used to construct dynamic similarity is captured, and the relative error of the prediction is getting lower and lower, reaching the lowest at about 25%-30% of the total number of iterations. At the end of the graph iteration job running, since the predictable remaining running time is small, even a small absolute error will result in a high relative error.

[0204] For Figure 5 For each graph iteration job in the present invention, the average relative error of the job is calculated by selecting representative sample points. The selection of sample points follows the following rule: starting from 5% of the total iteration times, taking 20% ​​as a step, and taking 5 sample points. Figure 6 The figure shows the average relative error of each graph iteration job, that is, the average relative error of the above four graph iteration algorithms under each data set. It can be seen that the relative average error of prediction for all jobs is around 8%.

[0205] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A running time prediction system for graph iterative jobs in a Gaia system, characterized in that: It includes similarity management module, job matching module and running time prediction module; The similarity management module includes a sampling execution unit, a static similarity calculation unit and a dynamic similarity calculation unit; The sampling execution unit samples the graph data set by a biased random jump method to obtain sample graph data, and executes an iterative algorithm on the sample graph data set to capture offline features of the sample graph iterative job; The static similarity calculation unit calculates the static similarity between the current graph iteration job and the historical graph iteration job by sampling the offline features captured by the execution unit; The dynamic similarity calculation unit calculates the dynamic similarity between the current graph iteration job and the historical graph iteration job by using the statistical information captured during the actual execution of the graph iteration job; The similarity management module sends the calculated static similarity and dynamic similarity to the job matching module; The job matching module includes a similarity evaluation unit, a training point collection unit and a parameter training unit; The similarity evaluation unit constructs a similarity evaluation function, performs validity evaluation on the static similarity and dynamic similarity sent by the similarity management module, and selects matching historical graph iteration jobs according to the validity evaluation results and sends them to the running time prediction module; The training point collection unit is used to collect training data points of parameters in the training similarity evaluation function; The parameter training unit uses the training data points to train the parameters of the similarity evaluation function and sends them to the similarity evaluation unit; The runtime prediction module predicts the remaining runtime of the current graph iteration job through the historical graph iteration jobs matched by the job matching module; The static similarity calculation unit calculates the current graph iteration job using the following method: run and historical graph iteration jobs comp The static similarity between them includes static computational similarity and static message passing similarity; The static calculation similarity is: Among them, S1 is the current graph iteration job Job run and historical graph iteration jobs comp The static calculation similarity between the two, the active node ratio of the graph iteration job at the i-th iteration ActVert i (Job) represents the number of active nodes in the graph iteration job Job at the i-th iteration, TotVert i (Job) indicates the summary points of the graph iteration job at the i-th iteration, and N indicates the total number of iterations during the sampling execution; the graph iteration job includes the current graph iteration job run and historical graph iteration jobs comp ; The static message delivery similarity is Among them, S2 represents the current graph iteration job Job run and historical graph iteration jobs comp The static message transmission similarity between them; the total amount of messages transmitted by the graph iteration job Job at the i-th iteration is RemMsg i (Job) represents the number of messages delivered by the graph iteration job Job at the i-th iteration. represents the average message size of the graph iteration job Job at the i-th iteration, and N represents the total number of iterations during the sampling execution; The dynamic similarity calculation job of the current graph iteration run and historical graph iteration jobs comp The dynamic similarity between them includes runtime similarity S3, resource allocation similarity S4, convergence similarity S5, and resource utilization similarity S6; The running time similarity S3 is Among them, Step represents the current iteration number of the job, runtime(Job,i) represents the running time of the graph iteration job Job at the i-th iteration; the graph iteration job Job includes the current graph iteration job Job run and historical graph iteration jobs comp ; The resource allocation similarity is: Among them, Step represents the current iteration number of the job, cons avg (Job, Step) represents the average number of containers used by all iterations of the graph iteration job from the beginning to the current execution; the graph iteration job includes the current graph iteration job. run and historical graph iteration jobs comp ; The convergence similarity is: Among them, workSet(Job,i) represents the number of data actively processed by the graph iteration job Job at the i-th iteration; The resource utilization similarity is Among them, cons is the number of containers used by the graph iteration job; RsUtil job [hw,Step,i] represents the average utilization of the hardware device hw in container i when the graph iteration job executes to the Step-th iteration.

2. The system according to claim 1, characterized in that The similarity evaluation function constructed by the similarity evaluation unit is specifically: Average optimal precision function h(t) and optimal similarity ratio function n(t); h(t):=avg{acc|(DoS,acc)∈P,DoS≥t}; n(t):=|{acc|(DoS,acc)∈P,DoS≥t}| / |P|; Where acc is the prediction accuracy of the runtime prediction system; DoS is the overall similarity of the two graph iteration jobs, which is the weighted average of all individual similarities, and the weight values ​​of all individual similarities form a weight vector; t is the similarity threshold; P is a simulation point set consisting of a set of simulation points (DoS, acc); The weight vector and similarity threshold are the parameters of the similarity evaluation function.

3. The system according to claim 2, characterized in that The training point collection unit first extracts n times of history graph iteration job set in an orderly and non-repetitive manner, and extracts two graph iteration jobs from it each time, which are the first graph iteration job J i and the second graph iteration job K i , i=1,2,3,...,n; Extracted first graph iteration job J i Running and just finished executing the M∈[1,m i ] iterations, m i It is a graph iteration job J i The maximum number of iterations; Extracted second graph iterative job K i Yes and J i Based on the similarity between the two extracted graph iteration jobs, the running graph iteration job J is predicted by the runtime prediction module. i The remaining running time of the first graph iteration job J i The actual remaining running time is known. By using different current graph iteration jobs and matching historical graph iteration jobs and assuming that the current graph iteration jobs are in different iteration orders, a set P(w) of the form of (predict(w), actual) is calculated as training data points; where predict(w) is the predicted remaining execution time with w as the weight vector, and actual is the actual remaining execution time of the current graph iteration job. The parameter training unit finds a suitable similarity threshold t and a similarity weight vector w = (w1, ..., w6) for the currently running graph iteration job, so that the values ​​of the similarity evaluation functions h(t) and n(t) in the similarity evaluation unit are both greater than the set thresholds, to ensure that the number of matching results of the current graph iteration job is greater than the set threshold.

4. The system according to claim 3, characterized in that The runtime prediction module includes a single prediction unit and a final prediction unit; The single prediction unit predicts the running time of the current graph iteration job based on the matched single historical job, obtains a series of single prediction estimation values, constitutes a prediction time estimation set and outputs it to the final prediction unit; The final prediction unit is based on the prediction estimation set, and combines all the estimation values ​​in the prediction estimation set into a final prediction value by weighted average; The weight uses the overall similarity value corresponding to each history graph iteration job during matching execution.

5. A method for predicting the running time of a graph-oriented iterative job in a Gaia system based on the system of claim 1, characterized in that: The prediction process is as follows: Step 1: Obtain the historical graph iteration job set, the initial input data set, the to-be-predicted graph iteration job and its job parameters and iteration termination conditions; Step 2: The sampling execution unit samples the input data set with a sampling ratio of 10%, and then scales the convergence parameters of the graph iteration job to be predicted according to the sampling ratio, and initializes the parameters after scaling; Step 3: Set the data source of the iterative job of the graph to be predicted to the sample set after sampling, set the initial number of iterations to 0, and then execute the iterative job of the prediction graph, and store the statistical information captured during each iteration in the memory database; Step 4: Calculate the static similarity between the to-be-predicted graph iteration job and each job in the historical graph iteration job set based on the statistical information captured in step 3; Step 5: Re-obtain the job parameters and iteration termination conditions of the iteration job of the graph to be predicted, and initialize the parameters, static similarity, and dynamic similarity; set the data source of the iteration job of the graph to be predicted to the initial input data set, and set the number of iterations to 0; Step 6: When the number of iterations is 0, the historical graph iteration job matching the to-be-predicted graph iteration job is output according to the static similarity calculated in step 4, and then the running time prediction module calculates the static prediction value of the running time of the to-be-predicted graph iteration job according to the matched historical graph iteration job; Step 7: Run the iteration job of the graph to be predicted, perform iterative calculations on the input data, and obtain the dynamic similarity between the iteration job of the graph to be predicted and each graph iteration job in the historical graph iteration job set; Step 8: The job matching module outputs the historical graph iteration job that matches the graph iteration job to be predicted based on the static similarity calculated in step 4 and the dynamic similarity calculated in step 7, and then calculates the dynamic prediction value of the running time of the graph iteration job to be predicted based on the matched historical graph iteration job; Step 9: Determine whether the number of iterations or the iteration output result meets the iteration termination condition. If not, repeat step 7. If yes, execute step 10. Step 10: Calculate the prediction deviation of the running time of the iterative job of the graph to be predicted under different iterations. The dynamic prediction value with the smallest prediction deviation is the final prediction value of the running time of the iterative job of the graph to be predicted.

6. The method according to claim 5, characterized in that The step 7 is performed by the following steps 7.1 to 7.4: Step 7.1: The number of iterations increases by 1; Step 7.2: Execute the iterative step function in parallel on each node of the Gaia cluster. After each node completes the execution, the data on each node is synchronized by broadcasting to ensure the consistency of data on different nodes. Step 7.3: After data synchronization is completed, append the statistical information captured during this iteration to the in-memory database for update; Step 7.4: Calculate the dynamic similarity between the to-be-predicted graph iteration job and each graph iteration job in the historical graph iteration job set based on the statistical information updated in step 7.3.

Images