Data sensing-based Spark configuration parameter automatic optimization method

Abstract

The invention belongs to the technical field of electronic information, big data, cloud computing and the like, and particularly relates to a data sensing-based Spark configuration parameter automatic optimization method. The method comprises the steps of predetermining a Spark application and parameters influencing Spark performance; randomly configuring the parameters to obtain a training set; building a performance model by the training set through a random forest algorithm; and searching out optimal configuration parameters through a genetic algorithm. According to the method, under the condition that a user is not required to understand a Spark running mechanism, a parameter meaning effect, a value range, application characteristics and an input set, the optimal configuration parameters of a specific application running in a specific cluster environment can be found for the user; compared with a conventional parameter configuration method, the automatic optimization method is simpler and quicker; and the used random forest algorithm combines the advantages of machine learning and statistical reasoning, so that relatively high precision can be achieved by using relatively few training sets.

Description

technical field [0001] The invention belongs to the technical fields of electronic information, big data, cloud computing, etc., and in particular relates to a data-aware Spark configuration parameter automatic optimization method. Background technique [0002] Spark is a Hadoop-like MapReduce general-purpose parallel framework open sourced by UC Berkeley AMP lab (AMP Lab of the University of California, Berkeley). It has developed rapidly, and in just five years, it has become the top project of the Apache Foundation. Because Spark has the characteristics of storing intermediate results in memory, Spark runs iterative and interactive programs 10 times faster than the traditional disk computing framework Hadoop. Due to the important position of Spark in the field of big data analysis, according to the survey of Typesafe, more than 500 enterprises have used Spark in 2015. [0003] Configuration parameter optimization has always been one of the research hotspots in big data ...

AI Technical Summary

Problems solved by technology

Due to the different optimal configuration parameters for different cluster environments, different applications, and different input data sets, the manual configuration parameter method is a time-consuming and boring work

[0010] The disadvantages of the existing automatic parameter configuration methods are the low precision of the performance model and the high cost of modeling

Some methods use artificial neural network (Artificial Neural Network) and support vector machine (Support Vector Machine) to model, but to achieve higher accuracy (within 10%), a very large training set is required

Images