Spark automatic energy efficiency optimization method

Abstract

The invention discloses a Spark automatic energy efficiency optimization method, relates to the technical field of computers, and aims to solve the problem of low energy efficiency optimization efficiency of an existing Spark application, and the technical scheme of the invention can significantly improve the energy efficiency optimization efficiency of the Spark application. In the application, the user only needs to submit the command, and the system can automatically analyze the parameters and the application requirements and call the optimization scheme to carry out parameter tuning, so that the workload of manual tuning is greatly reduced, and the deployment and execution process of the Spark application is accelerated. According to the technical scheme provided by the invention, the advantages of heuristic optimization and machine learning methods are combined, preliminary adjustment and optimization (cold start) can be carried out by depending on hardware conditions under the condition of insufficient data, and training data are provided for a subsequent machine learning model. According to the scheme, the adaptability and robustness of the system in different scenes are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a Spark automatic energy efficiency optimization method. Background Art

[0002] With the advent of the information age, the demand for data processing has surged, driving the development of distributed computing platforms. As one of the best, Apache Spark has attracted widespread attention for its excellent hardware compatibility and performance. Spark uses the Resilient Distributed Memory Abstraction Model (RDD) to efficiently process data in the cluster memory. Compared with Hadoop, its computing speed is significantly improved, thereby effectively reducing energy consumption. The Spark ecosystem is comprehensive and supports SQL queries, stream data processing, machine learning, and graph processing. It also provides Java, Python, Scala, and R programming interfaces, making it highly user-friendly. The Spark cluster consists of master and worker nodes. It implements parallel computing by decomposing tasks into several Stages and Tasks, forming a directed acyclic graph (DAG) to represent application features. As a data representation method, RDD supports lazy computing and Lineage-based fault tolerance mechanisms.

[0003] However, Spark's energy efficiency is highly dependent on configuration parameters. Improper parameter settings can significantly reduce efficiency and increase energy consumption. Spark has more than 180 adjustable parameters, and the number increases with version upgrades. Dozens of these parameters have a significant impact on system energy consumption. Parameter tuning is complex because energy efficiency bottlenecks may involve computing, memory, or network overhead. Studies have shown that parameter adjustment has a significant impact on application runtime, memory usage, system parallelism, I / O, and network bandwidth. Therefore, improper optimization may lead to increased energy consumption. Due to differences in cluster hardware configuration and applications, parameter optimization needs to be adjusted according to different environments, and manual optimization is difficult to meet the needs of all scenarios.

[0004] In recent years, research on energy efficiency optimization for Apache Spark has been widely carried out, covering a variety of scenarios such as machine learning, matrix computing, equi-joins, and OLAP. Although these studies provide effective optimization solutions for specific applications, they generally have the limitations of narrow coverage and focus only on optimization at the algorithm or logical operation level, making it difficult to reduce the energy consumption of written applications. Some work attempts to optimize performance from the perspective of hardware configuration and heterogeneous cluster scheduling, but due to hardware cost and environmental stability constraints, its application is limited.

[0005] Currently, a more common optimization method is to improve energy efficiency by automatically adjusting Spark system parameters. Although Spark provides default parameter configurations, parameter optimization can still significantly reduce energy efficiency for specific applications. However, existing automated parameter optimization methods tend to establish application performance prediction models and search for optimization parameters based on them, but ignore the running time of parameter optimization algorithms and the running efficiency of application performance prediction models. This leads to the problem of low efficiency in energy efficiency optimization of Spark applications. Summary of the invention

[0006] The purpose of the present invention is to provide a Spark automatic energy efficiency optimization method to address the problem of low energy efficiency optimization efficiency of existing Spark applications.

[0007] The technical solution adopted by the present invention to solve the above technical problems is:

[0008] A Spark automatic energy efficiency optimization method, the method comprising the following steps:

[0009] Step 1: Based on the Spark application submitted by the user, the system obtains the parameters for running the Spark application and the default values ​​of the parameters, and uses the heuristic parameter optimization method to tune the default values ​​of the parameters to obtain the heuristic optimization parameters;

[0010] Step 2: Based on the default parameter values, determine whether the energy consumption model of the Spark application exists in the model library. If so, proceed to step 3. Otherwise, proceed to step 4 based on the heuristic optimization parameters:

[0011] Step 3: Based on the energy consumption model, the default values ​​of the parameters are tuned using a random recursive search algorithm to obtain the machine learning optimization parameters, and it is determined whether the amount of available historical data exceeds the threshold set by the user. If the amount of available historical data exceeds the threshold, the energy consumption model is used to obtain the actual operating energy consumption corresponding to the heuristic optimization parameters and the predicted operating energy consumption corresponding to the machine learning optimization parameters, and the final energy efficiency optimization parameters are determined based on the actual operating energy consumption and the predicted operating energy consumption. Finally, step 4 is performed based on the final energy efficiency optimization parameters.

[0012] Otherwise, execute step 4 based on the heuristic optimization parameters:

[0013] In step 3, based on the energy consumption model, the steps of tuning the default values ​​of the parameters using a random recursive search algorithm to obtain the machine learning optimization parameters are as follows:

[0014] First, a random recursive search algorithm is used to generate candidate parameters. Then, based on the energy consumption model, the predicted value of the operating energy consumption corresponding to each candidate parameter is obtained. Then, the candidate parameter with the lowest predicted value of operating energy consumption is selected as the machine learning optimization parameter.

[0015] Step 4: Based on the optimization parameters, call the shell command to run the Spark application submitted by the user, and save the running information as newly added available historical data. The available historical data includes parameters and running energy consumption. After that, based on the optimization parameters, determine whether there is a corresponding energy consumption model in the model library. If so, use the newly added available historical data to update the energy consumption model. If not, determine whether the amount of available historical data at this time exceeds the threshold. If so, use the available historical data to build a new model and save the model in the model library. Otherwise, end directly.

[0016] Furthermore, in step 1, the heuristic parameter optimization method is used to optimize the default values ​​of the parameters, and the specific steps are:

[0017] Step 11: Read the basic hardware configuration of the Spark cluster, which includes the number of available CPU cores and memory size of each physical node in the Spark cluster;

[0018] Step 1 and 2: Based on the number of available CPU cores and memory size, for the number of CPU cores occupied by each executor in Spark, an exhaustive method is used to obtain the number of CPU cores and memory size corresponding to the maximum task parallelism. The remaining parameters take the default values ​​to obtain the heuristic optimization parameters.

[0019] Furthermore, the task parallelism is expressed as:

[0020]

[0021] Among them, Parallelism represents the degree of task parallelism, C t Indicates the number of cores required for each task, M workeri and C workeri Respectively represent the available memory and number of CPU cores of the i-th worker, M e and C e Indicates the memory size and number of CPU cores of each executor.

[0022] Furthermore, the specific steps of using the machine learning method to tune the parameters of the default values ​​in step 3 are:

[0023] Step 31: Read the operating energy consumption and parameters from the available historical data, and train the neural network using the parameters as input and the operating energy consumption as output;

[0024] Step 32: Generate candidate parameters using a random recursive search algorithm, input the candidate parameters into the trained neural network, obtain the operating energy consumption corresponding to the candidate parameters, and add the current candidate parameters and their corresponding operating energy consumption to the access record list;

[0025] Step 33: Repeat step 32 until the number of iterations is reached;

[0026] Step 34: Select the parameters with the lowest energy consumption in the access record list, which are the machine learning optimization parameters.

[0027] Furthermore, the random recursive search algorithm is an improved random recursive search algorithm, and the specific steps of the improved random recursive search algorithm are:

[0028] Step 1: Calculate the number of sample searches n, where n is expressed as:

[0029]

[0030] Among them, p and r represent the requirement for the performance quantile of the subspace center point and the confidence level that the performance of the subspace center point reaches the quantile during the search process, respectively;

[0031] Step 2: Determine whether the number of available historical data is less than n. If so, execute step 3. Otherwise, directly select the parameter corresponding to the lowest operating energy consumption from the available historical data.

[0032] Step 3: Randomly select n parameters from the parameter space, and use the energy consumption model to obtain the predicted value of the operating energy consumption of each parameter. Select the parameter with the lowest operating energy consumption as the center point of the next round of sampling, and shrink the size of the current search subspace to r times the current search space, where r represents the shrinkage coefficient, and r is less than 1;

[0033] Step 4: Repeat step 3 until the search condition is met or the parameter space is less than the preset threshold, then stop searching and select the current parameter as the search result.

[0034] Furthermore, the hybrid tuning specifically comprises the following steps:

[0035] Step A: Get the generalization error err during neural network training i , and calculate the mean of the generalization error E(Err), and then get the standard deviation σ based on E(Err);

[0036] Step B: According to the set confidence probability p, calculate the confidence interval factor k, expressed as:

[0037]

[0038] Step C: Based on the heuristic optimization parameters and the machine learning optimization parameters, the difference t in the operating energy consumption between the two is obtained;

[0039] If t<-kσ, the parameter is a machine learning optimization parameter;

[0040] If -kσ≤t≤kσ, then with probability Returns mix(x opt ,x h ), with probability Select x opt As a parameter;

[0041] If t>kσ, return mix(x opt ,x h );

[0042] Among them, return mix(x opt ,x h ) indicates that among both the heuristic optimization parameters and the machine learning optimization parameters, if the parameters have been optimized, the parameters among the heuristic optimization parameters are selected; and among only the machine learning optimization parameters, if the parameters have been optimized, the parameters among the machine learning optimization parameters are selected, and the remaining parameters are the default values.

[0043] Furthermore, the standard deviation σ is expressed as:

[0044]

[0045]

[0046] Furthermore, the difference t in the operating energy consumption is expressed as:

[0047] t=f(x opt )-E(Err)-Energy(x h )

[0048] Among them, x h represents the heuristic optimization parameter, x opt represents the machine learning optimization parameter, f(x opt ) indicates that the energy consumption model is used to predict the application with parameter x opt The predicted power consumption under the condition of h ) means that the application is set to x h The actual power consumption under the condition.

[0049] Furthermore, the operating energy consumption is monitored by a power consumption meter.

[0050] Furthermore, the energy consumption model is expressed as:

[0051] f(c,d)=t

[0052] Among them, c represents the operating parameters, d represents the amount of input data, and t represents the predicted operating energy consumption.

[0053] The beneficial effects of the present invention are:

[0054] The technical solution of this application can significantly improve the efficiency of energy efficiency optimization of Spark applications. In this application, users only need to submit commands, and the system can automatically parse parameters and application requirements, and call optimization solutions for parameter tuning, which greatly reduces the workload of manual tuning and accelerates the deployment and execution of Spark applications. The technical solution proposed in this application combines the advantages of heuristic optimization and machine learning methods, and can rely on hardware conditions for preliminary tuning ("cold start") when data is insufficient, and provide training data for subsequent machine learning models. This solution enhances the adaptability and robustness of the system in different scenarios. It establishes a Spark parameter optimization algorithm for machine learning, which can use the parameter search range and task history data set by the user to output finely tuned parameter configurations. This not only improves the energy efficiency of Spark applications, but also optimizes resource utilization and reduces operating costs. This application further proposes a machine learning model selection algorithm with speed-precision trade-off, which improves the prediction speed while ensuring the accuracy of the application energy efficiency prediction model, thereby improving the final parameter optimization effect. This algorithm balances model complexity and operating efficiency, making the tuning process more efficient and accurate. The above work has significantly improved the energy efficiency optimization efficiency, accuracy and running speed of Spark applications through strategies such as automation, intelligence, hybrid optimization and speed-accuracy trade-off, providing users with a more convenient, efficient and accurate Spark parameter tuning solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Automatically optimize the system flow chart for Spark;

[0056] Figure 2 Automatically optimize the system structure diagram for Spark;

[0057] Figure 3 Flowchart for obtaining machine learning optimization parameters;

[0058] Figure 4 Schematic diagram of hybrid optimization algorithm;

[0059] Figure 5 Diagram of selecting an algorithm for a machine learning model. DETAILED DESCRIPTION

[0060] It should be particularly noted that, in the absence of conflict, the various embodiments disclosed in this application can be combined with each other.

[0061] Specific implementation method 1: A Spark automatic energy efficiency optimization method described in this implementation implements a Spark automatic tuning system by parsing the parameters and applications specified by user commands, "cold start" parameter tuning using heuristic optimization methods, searching parameter space using machine learning methods, and balancing model accuracy and speed.

[0062] The main contents of this application are: (1) The project implements a Spark automatic tuning system. When the user submits a command, the system parses the specified parameters and applications, and automatically tunes the parameters based on the application by calling a hybrid optimization scheme. (2) A Spark parameter tuning method based on a hybrid optimization scheme is proposed. In the absence of data, a heuristic optimization method is called to tune the parameters based on hardware conditions to achieve a "cold start". In this process, training data is provided for the application energy efficiency prediction model based on machine learning methods. (3) A Spark parameter optimization algorithm based on machine learning is proposed. It inputs the optimization parameter search range set by the user and the task history data, and outputs the parameters tuned by the system. (4) A machine learning model selection algorithm based on speed-accuracy trade-off is proposed. While ensuring the accuracy of the application energy efficiency prediction model, the prediction speed is improved to improve the final parameter optimization effect.

[0063] Specifically, the following steps are included:

[0064] Step 1: Parameter tuning based on machine learning methods:

[0065] The system establishes an energy efficiency prediction model:

[0066] f(c,d)=t

[0067] Where c is the runtime parameter, d is the input data volume, and t is the predicted energy efficiency. The system trains the model through regression analysis methods (such as linear regression, decision tree regression, etc.) to predict the operating energy efficiency under different parameter configurations. The trained model can output the predicted energy efficiency t based on the input parameters c and d.

[0068] Based on this model, the parameter space search algorithm is used to find the optimal parameters in the parameter space given by the user through a heuristic search method. The algorithm will explore different parameter combinations to find the parameter configuration that optimizes energy efficiency.

[0069] The machine learning model is used to predict the energy efficiency of the application and output the optimized parameter configuration. The output parameters of this step will be used for subsequent tuning and provide a basis for optimization decisions.

[0070] Step 2: Parameter tuning based on hybrid optimization scheme:

[0071] A heuristic optimization method is proposed to find a balance between hardware conditions and task requirements by dynamically adjusting the core parameters of the Spark cluster (such as the number of executor cores, memory size, RDD storage mode, etc.) to improve the parallel capability and resource utilization of the cluster. This method is particularly suitable for scenarios with insufficient historical data, and can quickly provide optimization solutions, providing a basis for subsequent hybrid optimization.

[0072] Based on the heuristic optimization method, a parameter tuning method based on a hybrid optimization scheme is proposed.

[0073] Leverage heuristics to quickly provide optimized parameters when historical data is lacking.

[0074] Train machine learning models based on historical data and predict parameter performance, providing more potential optimization solutions.

[0075] The weights between optimization methods are dynamically adjusted through error analysis and confidence intervals to ensure the credibility of the results.

[0076] The hybrid strategy ensures that even in uncertain situations, the optimization parameters can still effectively balance the advantages of the two methods. The output of this step is the preliminary parameter configuration and optimization effect obtained through heuristic optimization, and at the same time, new historical data is generated for training machine learning models. These data will help improve the prediction accuracy of the machine learning model in the next step.

[0077] Step 3: Machine learning model selection algorithm based on speed-accuracy trade-off:

[0078] According to the speed-accuracy trade-off, a suitable machine learning model is selected to obtain the optimal parameter configuration within a limited time. In this way, the efficiency and accuracy of model selection in the optimization process can be balanced, ensuring that the tuning process is both fast and efficient.

[0079] Specifically, consider two candidate machine learning models A and B and evaluate their search efficiency in a given time:

[0080] Assume that model A can perform n random searches and model B can perform m random searches (m <n)。

[0081] The expected values ​​of the performance indicators of the optimization parameters obtained through random search are EA and EB respectively.

[0082] In the model selection process, the algorithm calculates the prediction error of each model on the sample data through cross-validation, and estimates the generalization error of the model on the overall data. The generalization error reflects the prediction ability of the model on unknown data. According to the prediction error and time budget, the selection priority of different models is dynamically adjusted. The algorithm selects the best model from the list of alternative machine learning models (such as linear regression, decision tree, support vector machine, etc.) based on the following two points:

[0083] Prediction accuracy: The smaller the generalization error of the model, the higher the prediction accuracy and the higher the priority.

[0084] Operation efficiency: The faster the model runs per unit time, the more random searches can be performed and the higher the priority.

[0085] Finally, the algorithm outputs the final selected machine learning model mopt, which will be applied to the entire tuning process to provide fast and accurate support for energy efficiency prediction of Spark applications, thereby guiding subsequent tuning decisions.

[0086] Step 4: Implement Spark automatic optimization system

[0087] Users submit applications through the Spark automatic optimization system. The system automatically analyzes the commands submitted by users, parses out the specified parameters and application types, and matches them according to the historical database.

[0088] If there is an application energy efficiency prediction model, the system will guide the energy efficiency optimization process based on the model.

[0089] If there is no predictive model, the system will use heuristic methods to adjust parameters.

[0090] Users can force the system to use a certain optimization method through the configuration file, or decide whether to perform automatic tuning.

[0091] When the system determines that it can provide optimized parameters for the application, it will ask the user whether to perform the tuning. The user can avoid interacting with the system through configuration file settings.

[0092] If the user refuses tuning, the system directly submits the application to the cluster according to the parameters specified by the user; if the user accepts tuning, the system uses the hybrid optimization algorithm in the subsequent steps to generate optimization parameters and submits them to the cluster.

[0093] According to the configuration file set by the user, the default value of the parameter is obtained and the

[0094] Current candidate parameters

[0095] Step 1: Parameter tuning based on machine learning methods

[0096] Spark parameter optimization algorithm based on machine learning method

[0097] Read the running energy efficiency t and each set of parameters c of the user's historical running tasks, and use t and each parameter to obtain the optimal prediction model (multiple models, select the best one), select the optimal model BestModel from the historical data, and use it to predict the subsequent task running time: BestModel = SelectBestModel(c,t), and use the historical task data to train BestModel: BestModel.Fit(c,t).

[0098] Parameter List and the candidate optimal parameter copt is empty

[0099] When the stop condition is not met, perform the following steps:

[0100] Use the trained model BestModel to predict the operating energy consumption of the current candidate parameter p (set by the user):

[0101] If the current energy efficiency BestModel.Predict(p) is better than Energy, then Energy = BestModel.Predict(p), and the current candidate parameter is used as the candidate optimal parameter copt, otherwise it is not used as the candidate optimal parameter copt

[0102] Add the current candidate parameters and their predicted operating energy efficiency to the access record list:

[0103] VisitedParaList=VisitedParaList∪{(p,Energy)}

[0104] Based on Energy, a random recursive search algorithm is used to generate candidate parameters, and the candidate parameters are used as current candidate parameters to repeat the above steps for iteration. When the stopping condition is met (such as reaching the maximum number of iterations or Energy convergence), the optimization ends.

[0105] In VisitedParaList, find the parameter configuration that predicts the optimal operating energy efficiency:

[0106]

[0107] Parameter space search algorithm (improved random recursive search algorithm---random recursive search algorithm based on confidence probability)

[0108] Given a parameter space D

[0109] Sample quantity initialization: According to the confidence probability p of the sample, calculate the required number of sample searches:

[0110] (Parameters p and r represent the user's requirement for the performance quantile of the subspace center point and the confidence that the subspace center point performance reaches that quantile during the search process. These two parameters together determine the number of explorations in a subspace during the parameter search process.)

[0111] Historical data screening: If the number of historical data is ≥ n, directly select the best parameters in the historical sample Otherwise, a new round of sampling starts from the parameter space D (n parameters are randomly selected from the parameter space, and the energy consumption of each parameter is predicted, and the one with the best energy consumption is selected as the center point of the next round of sampling).

[0112] Initialize optimization parameters xopt =x0, as the starting point of the optimization process.

[0113] Iterative search: When the search times do not reach the upper limit, perform the following steps:

[0114] Randomly select n sample points x1, x2, ..., xn from the current candidate parameter space D

[0115] Calculate the optimal parameters in the current sample according to the objective function value f(c) If f(x')<f(x opt ) Update the optimal parameter x opt =x', and set the optimal parameter x' among the sample points x1, x2, ..., xn as the subspace center for the next search.

[0116] If the size of the subspace currently being searched is shrunk to r times the size of the current space (r shrinkage factor, r is less than 1), the subspace is updated to gradually narrow the parameter search range.

[0117] For the reduced parameter space D′, continue random sampling and repeat the search until the search condition is met or the parameter space is exhausted.

[0118] If the parameter search in the current subspace is completed or the maximum number of searches n is reached, the search stops.

[0119] Output the final optimal parameter xopt as the result of searching the entire parameter space.

[0120] Step 2: Parameter tuning based on hybrid optimization scheme

[0121] 1. Heuristic parameter optimization

[0122] Initialization parameters: Read the basic hardware configuration of the Spark cluster, including:

[0123] The number of cores and memory available for each Worker;

[0124] The total number of Workers in the cluster, K;

[0125] Storage mode of RDD.

[0126] Initialize Spark default configuration parameters:

[0127] spark.executor.cores: Exhaustive list of the number of cores per executor (1 to min{C worker1 ,C worker2 ,…,C workerk});

[0128] spark.executor.memory: set to

[0129] Calculate parallelism: In Spark onYarn mode, the parallelism of a task can be calculated using the following formula:

[0130]

[0131] Exhaustively list the above parameters, and the value of spark.executor.cores and spark.executor.memory is the largest value of Parallelism.

[0132] spark.default.parallelism: The default number of partitions in the RDD generated by Spark join, reduceByKey and other transformation operations. Set to the value of Parallelism.

[0133] in:

[0134] C t Indicates the number of cores required for each task, determined by spark.task.cpus.

[0135] Mworkeri and Cworkeri represent the available memory and number of CPU cores of the i-th worker, respectively.

[0136] M e and C e Indicates the memory size and core number of each executor, which is determined by spark.exector.memory and spark.exector.cores.

[0137] Adjust the number of cores and memory allocation: For Spark's core and memory allocation, use the formula to ensure that each executor maximizes the use of Worker resources while avoiding resource waste. By default, each executor in Yarn mode is allocated only one core. You need to adjust spark.executor.cores according to the task computing load to increase parallelism.

[0138] Optimize RDD storage mode:

[0139] If RDD is called frequently, use MEMORY_ONLY mode to reduce repeated calculations.

[0140] If memory is insufficient, use MEMORY_AND_DISK mode to balance performance and storage requirements.

[0141] For data that is no longer needed after a one-time calculation, use DISK_ONLY to save memory.

[0142] Task distribution optimization:

[0143] During the task running process, tasks are reasonably allocated to executors to maximize the number of parallel tasks while ensuring task load balancing.

[0144] For computing-intensive tasks, reasonably limit the upper limit of concurrency to avoid CPU saturation.

[0145] Adjust the number of data partitions (spark.default.parallelism) based on the I / O characteristics of the task, ensuring that the number of partitions can fully utilize the available CPU cores.

[0146] RDD caching strategy:

[0147] If some RDDs are calculated multiple times or called frequently, it is recommended to use a caching mechanism (such as MEMORY_ONLY mode) to avoid repeated calculations.

[0148] Adjust cache priority or clearing mechanism according to data size and frequency requirements to avoid excessive memory usage.

[0149] Generate optimized configuration:

[0150] The optimized configuration is summarized as follows:

[0151] spark.executor.cores

[0152] spark.executor.memory

[0153] spark.default.parallelism

[0154] RDD storage mode and caching strategy

[0155] Returns the optimized parameter configuration for cluster operation.

[0156] Hybrid Optimization Algorithm and Error Analysis

[0157] Use the heuristic optimization algorithm to obtain the optimization parameter x according to the cluster hardware configuration information h , and record x h Energy efficiencyEnergy(x h ).

[0158] If the amount of available historical data n does not exceed the threshold set by the user, the heuristic optimization parameter x is returned immediately. h , as the final optimization result.

[0159] If the amount of historical data meets the requirements, use a machine learning-based method to train the energy efficiency prediction model f:x->R on the historical data. Calculate the model error erri=f(xi)-Energy(xi) on the cross-validation set, where xi is the parameter sample in the validation set. Calculate the mean error E(Err) and the standard deviation σ of the error:

[0160] x h ←Use heuristic optimization method to obtain optimization parameters based on cluster hardware information

[0161] E(x h )←The average energy consumption of pure heuristic method based on historical records

[0162] IF the amount of available historical data ≤ the user-set threshold, return the parameters obtained by the heuristic method

[0163] The model is trained on historical data using machine learning methods to obtain the generalization error err1.err2...errn of the model on the training data set. i )-Energy(x i ).

[0164] Calculate the mean error E(Err) and the standard deviation σ of the error:

[0165]

[0166]

[0167] A random recursive search algorithm is used to find the optimal parameter xopt in the parameter search space C, and the energy efficiency f(xopt) of xopt is predicted and compared with the energy efficiency obtained by the heuristic algorithm.

[0168] Calculate the confidence interval factor k based on the user-specified confidence probability p:

[0169]

[0170] Based on the heuristic parameters xh and the predicted optimal parameters xopt, the energy efficiency difference t is calculated:

[0171] t=f(x opt )-E(Err)-Energy(x h )

[0172] If the error difference t<-kσ: return xopt, which means that the machine learning prediction parameters are considered to have better performance.

[0173] If kσ≤t≤kσ: Mix the parameters of xopt and xh to generate the mixed parameters mix(xopt,xh), where the mixing probability is given by the formula and Decide.

[0174] If t>-kσ: Return the mixing parameters mix(xopt,xh) to further adjust the parameters of the heuristic method.

[0175] According to the above judgment and mixing steps, the optimized parameter Copt is finally output

[0176] Step 3: Machine Learning Model Selection Algorithm Based on Speed-Accuracy Tradeoff

[0177] Calculate the error index: Use the cross-validation method to calculate the error err of all models on the training set i =f(x i )-Energy(x i ).

[0178] The following indicators of error are calculated:

[0179] Mean Absolute Error (MAE):

[0180]

[0181] Mean Squared Error (MSE):

[0182]

[0183] Standard deviation of error (σ):

[0184]

[0185] Sort the models: Sort all the models in ModelList according to the MAE index from small to large, and get the sorted model sequence M1, M2, …, Mn, where:

[0186] MAE(M1)≤MAE(M2)≤...≤MAE(M n )

[0187] Select the initial model mopt=M1.

[0188] Model-by-model evaluation: traverse the sorted model list M1, M2, …, Mn:

[0189] Calculate the average energy efficiency of the current model Mi: PredictEnergy(Cost)i.

[0190] Calculate the standard deviation of the prediction error of the current model

[0191] Calculate the evaluation round n of the current model:

[0192] Calculate the expected error E(n) of the current model based on the distribution density function fs(x):

[0193] If E(n)+2kσ<E(m), the current model Mi is replaced with the optimal model mopt, and PredictEnergy(Cost)mopt=PredictEnergy(Cost)i is updated.

[0194] Output the optimal model: After the traversal is completed, the current optimal machine learning model mopt is output to complete model selection and optimization.

[0195] Step 4: Implement Spark automatic optimization system

[0196] Users submit Spark application tasks through the system's application submission interface and specify necessary operating parameters (such as application type, data scale, runtime configuration, etc.).

[0197] The system parses the tasks submitted by the user, including: extracting the operating parameters and task type specified by the user and checking whether the submitted tasks contain sufficient optimization information.

[0198] Retrieve historical running records similar to the tasks submitted by the user from the historical information database, including: retrieving the parameter configuration of the historical tasks and retrieving the running performance of the historical tasks (such as running time and memory usage).

[0199] If the historical data is sufficient, the system directly uses the historical information and the trained model to optimize the task. If the historical data is insufficient, the system selects a heuristic method for preliminary optimization and records the optimized task operation data.

[0200] Based on historical operation records, the task performance prediction model is trained using machine learning methods. The training process includes:

[0201] Data preprocessing: Extract task parameters and operation performance data from the historical database.

[0202] Model selection: Select the optimal machine learning model for training.

[0203] Model training: Generates a performance model that can predict run time and resource usage.

[0204] The system optimizes parameters by combining the operating parameters of user tasks, cluster hardware information, and performance prediction models. When historical data is insufficient, the system adjusts parameters (such as the number of executor cores, memory size, etc.) based on cluster hardware information. When historical data is sufficient, the system uses the trained performance prediction model to search for the optimal configuration in the parameter space.

[0205] The system converts the optimization results into a parameter configuration file that Spark can execute. The optimized parameters include spark.executor.cores, spark.executor.memory, spark.default.parallelism, etc.

[0206] The system submits the optimized task parameters to the Spark cluster server for execution.

[0207] The system writes the actual running results of the task (such as running time, resource utilization) into the historical information database to update historical data and provide a reference for subsequent task optimization.

[0208] Existing research on Spark performance optimization has shortcomings in terms of versatility, historical data dependence, and optimization algorithm efficiency. New methods and frameworks are urgently needed to overcome these limitations. In general, there are three main problems: a complete automatic tuning tool has not yet been implemented from an engineering perspective. The tool should be able to take over the user's Spark command submission, automatically capture historical data, and optimize according to the user's wishes. At the same time, the method of adjusting Spark system parameters has not been fully utilized to improve application operation efficiency, although this method does not require code intervention. At the same time, the potential of system administrators in improving system performance and stability has not been fully explored, especially when faced with the immutable limitations of compiled programs.

[0209] The design of automatic parameter optimization methods based on the characteristics of Spark applications has not been completed, including the establishment of application energy efficiency prediction models and the implementation of heuristic parameter space search strategies. Methods for parameter optimization in the absence of historical data have not been developed, including heuristic settings using parameter configuration and cluster hardware resource estimation, and strategies for solving other parameter adjustment problems in combination with black box methods. In the parameter optimization process, the impact of the trade-off between model accuracy and speed on the optimization effect has not been considered, and there is a lack of reasonable system energy efficiency model selection solutions.

[0210] A complete automatic tuning tool has not yet been implemented from an engineering perspective. This tool should be able to take over the user's Spark command submission, automatically capture historical data, and optimize according to the user's wishes.

[0211] This application uses the energy efficiency optimization method described in this chapter on a cluster with three nodes to optimize the applications in the test benchmark program set SparkBench. After parameter optimization based on the parameter tuning method of machine learning methods, the running time of the four classic applications PageRank, Sort, KMeans, and LogisticRegression decreased by 50.14%, 34.56%, 91.65%, and 78.63% respectively compared with the default parameters. For the heuristic optimization algorithm, it has a good optimization effect on the cluster in Spark onyarn mode. Compared with the default parameters of the Spark system, the running time of PageRank, Sort, KMeans, and LogisticRegression has been greatly improved. This application implements a Spark automatic tuning system. The system can automatically adjust the parameters of Spark and effectively improve the energy efficiency of the system.

[0212] This application builds an efficient and intelligent Spark automatic tuning system, which aims to simplify the parameter configuration process of Spark applications and improve the accuracy and efficiency of energy efficiency optimization. By combining heuristic optimization and machine learning methods, the system can flexibly respond to different data conditions and realize automatic optimization of parameters, thereby improving the energy consumption of Spark applications, improving resource utilization, reducing the user's manual tuning costs, and accelerating the deployment and execution speed of applications. At the same time, through the speed-accuracy trade-off model selection algorithm, the efficiency and accuracy of the tuning process are further ensured to meet the needs of actual application scenarios.

[0213] It should be noted that the specific implementation is only an explanation and description of the technical solution of the present invention, and cannot be used to limit the scope of protection of the rights. Any partial changes made according to the claims and description of the present invention should still fall within the scope of protection of the present invention.

Claims

1. A Spark automatic energy efficiency optimization method, characterized in that The method comprises the following steps: Step 1: Based on the Spark application submitted by the user, the system obtains the parameters for running the Spark application and the default values ​​of the parameters, and uses the heuristic parameter optimization method to tune the default values ​​of the parameters to obtain the heuristic optimization parameters; Step 2: Based on the default parameter values, determine whether the energy consumption model of the Spark application exists in the model library. If so, proceed to step 3. Otherwise, proceed to step 4 based on the heuristic optimization parameters: Step 3: Based on the energy consumption model, the default values ​​of the parameters are tuned using a random recursive search algorithm to obtain the machine learning optimization parameters, and it is determined whether the amount of available historical data exceeds the threshold set by the user. If the amount of available historical data exceeds the threshold, the energy consumption model is used to obtain the actual operating energy consumption corresponding to the heuristic optimization parameters and the predicted operating energy consumption corresponding to the machine learning optimization parameters, and the final energy efficiency optimization parameters are determined based on the actual operating energy consumption and the predicted operating energy consumption. Finally, step 4 is performed based on the final energy efficiency optimization parameters. Otherwise, execute step 4 based on the heuristic optimization parameters: In step 3, based on the energy consumption model, the steps of tuning the default values ​​of the parameters using a random recursive search algorithm to obtain the machine learning optimization parameters are as follows: First, a random recursive search algorithm is used to generate candidate parameters. Then, based on the energy consumption model, the predicted value of the operating energy consumption corresponding to each candidate parameter is obtained. Then, the candidate parameter with the lowest predicted value of operating energy consumption is selected as the machine learning optimization parameter. Step 4: Based on the optimization parameters, call the shell command to run the Spark application submitted by the user, and save the running information as newly added available historical data. The available historical data includes parameters and running energy consumption. After that, based on the optimization parameters, determine whether there is a corresponding energy consumption model in the model library. If so, use the newly added available historical data to update the energy consumption model. If not, determine whether the amount of available historical data at this time exceeds the threshold. If so, use the available historical data to build a new model and save the model in the model library. Otherwise, end directly.

2. A Spark automatic energy efficiency optimization method according to claim 1, characterized in that In step 1, the heuristic parameter optimization method is used to optimize the default values ​​of the parameters. The specific steps are: Step 11: Read the basic hardware configuration of the Spark cluster, which includes the number of available CPU cores and memory size of each physical node in the Spark cluster; Step 1 and 2: Based on the number of available CPU cores and memory size, for the number of CPU cores occupied by each executor in Spark, an exhaustive method is used to obtain the number of CPU cores and memory size corresponding to the maximum task parallelism. The remaining parameters take the default values ​​to obtain the heuristic optimization parameters.

3. A Spark automatic energy efficiency optimization method according to claim 2, characterized in that The task parallelism is expressed as: Among them, Parallelism represents the degree of task parallelism, C t Indicates the number of cores required for each task, M workeri and C workeri Respectively represent the available memory and number of CPU cores of the i-th worker, M e and C e Indicates the memory size and number of CPU cores of each executor.

4. A Spark automatic energy efficiency optimization method according to claim 3, characterized in that The specific steps of using the machine learning method to tune the parameters of the default values ​​in step 3 are: Step 31: Read the operating energy consumption and parameters from the available historical data, and train the neural network using the parameters as input and the operating energy consumption as output; Step 32: Generate candidate parameters using a random recursive search algorithm, input the candidate parameters into the trained neural network, obtain the operating energy consumption corresponding to the candidate parameters, and add the current candidate parameters and their corresponding operating energy consumption to the access record list; Step 33: Repeat step 32 until the number of iterations is reached; Step 34: Select the parameters with the lowest energy consumption in the access record list, which are the machine learning optimization parameters.

5. A Spark automatic energy efficiency optimization method according to claim 4, characterized in that The random recursive search algorithm is an improved random recursive search algorithm, and the specific steps of the improved random recursive search algorithm are: Step 1: Calculate the number of sample searches n, where n is expressed as: Among them, p and r represent the requirement for the performance quantile of the subspace center point and the confidence level that the performance of the subspace center point reaches the quantile during the search process, respectively; Step 2: Determine whether the number of available historical data is less than n. If so, execute step 3. Otherwise, directly select the parameter corresponding to the lowest operating energy consumption from the available historical data. Step 3: Randomly select n parameters from the parameter space, and use the energy consumption model to obtain the predicted value of the operating energy consumption of each parameter. Select the parameter with the lowest operating energy consumption as the center point of the next round of sampling, and shrink the size of the current search subspace to r times the current search space, where r represents the shrinkage coefficient, and r is less than 1; Step 4: Repeat step 3 until the search condition is met or the parameter space is less than the preset threshold, then stop searching and select the current parameter as the search result.

6. A Spark automatic energy efficiency optimization method according to claim 5, characterized in that The specific steps of hybrid tuning are: Step A: Get the generalization error err during neural network training i , and calculate the mean of the generalization error E(Err), and then get the standard deviation σ based on E(Err); Step B: According to the set confidence probability p, calculate the confidence interval factor k, expressed as: Step C: Based on the heuristic optimization parameters and the machine learning optimization parameters, the difference t in the operating energy consumption between the two is obtained; If t<-kσ, the parameter is a machine learning optimization parameter; If -kσ≤t≤kσ, then with probability Returns mix(x opt ,x h ), with probability Select x opt As a parameter; If t>kσ, return mix(x opt ,x h ); Among them, return mix(x opt ,x h ) indicates that among both the heuristic optimization parameters and the machine learning optimization parameters, if the parameters have been optimized, the parameters among the heuristic optimization parameters are selected; and among only the machine learning optimization parameters, if the parameters have been optimized, the parameters among the machine learning optimization parameters are selected, and the remaining parameters are the default values.

7. A Spark automatic energy efficiency optimization method according to claim 6, characterized in that The standard deviation σ is expressed as:

8. A Spark automatic energy efficiency optimization method according to claim 7, characterized in that The difference t in the operating energy consumption is expressed as: t=f(x opt )-E(Err)-Energy(x h ) Among them, x h represents the heuristic optimization parameter, x opt represents the machine learning optimization parameter, f(x opt ) indicates that the energy consumption model is used to predict the power consumption of the application when the parameter is xopt. h ) means that the application is set to x h The actual power consumption under the condition.

9. A Spark automatic energy efficiency optimization method according to claim 1, characterized in that The operating energy consumption is monitored by a power consumption meter.

10. A Spark automatic energy efficiency optimization method according to claim 1, characterized in that The energy consumption model is expressed as: f(c,d)=t Among them, c represents the operating parameters, d represents the amount of input data, and t represents the predicted operating energy consumption.

Images