An automatic tuning method of spark parameters based on cost model

Abstract

The present invention proposes a cost model-based Spark parameter automatic tuning method, which includes the following steps. Step 1: By acquiring task execution configuration and corresponding cost information, a cost-based performance model is constructed, and a cost-based performance model is obtained in a given parameter space. Optimize configuration; Step 2: For unknown types of tasks, use default parameters to run once, and provide reference values ​​for optimized configuration by judging the similarity of tasks. The invention proposes a cost-based performance model in view of the problems that may exist in the current configuration parameter tuning. The performance model is generated by analyzing Spark historical tasks, and the optimization parameters are obtained through the parameter space search algorithm. Upgrade and adjust to increase the accuracy of the model. For unknown types of tasks, after running it once, provide parameter reference values.

Description

technical field [0001] The invention relates to Spark task performance model establishment, big data system configuration parameter space search, and task similarity judgment. Background technique [0002] With the continuous development of science and technology, everything from a mobile phone and a tablet to an astronomical telescope and the Large Hadron Collider are all generators of data. massive data. How to store, process, and analyze these data has also become a practical issue in front of everyone. Since Google published the Google File System paper in 2003, a number of distributed computing frameworks have emerged, such as hadoop and sparkSpark—cluster computing on working sets. The emergence of these frameworks provides a basis for the storage and processing of big data, and also It plays an important role in various application scenarios. In response to the inefficiency of hadoop in iterative processing, Spark came into being. Although spark has a good perform...

AI Technical Summary

Problems solved by technology

[0009] For experienced operation and maintenance personnel, when there are few parameters to be adjusted, the task performance can be greatly increased by adjusting the configuration parameters, but this requires very familiarity with the system to achieve, and it is impossible to determine whether there is a better Configuration

But with the increase of parameters, even experienced operation and maintenance personnel, it is difficult to tune by manually adjusting the configuration parameters

[0010] To sum up, the current method is difficult to dynamically upgrade the performance model in the configuration parameter tuning; the generated performance model is greatly affected by the system operating state, and the model cannot be accurately established; manual tuning cannot adapt to the large parameter space

Images