Big data system configuration parameter adjusting and optimizing method and system based on deep learning

Abstract

The invention provides a big data system configuration parameter adjusting and optimizing method and system based on deep learning. The a big data system configuration parameter adjusting and optimizing method comprises a nerve network training step of preliminarily constructing a deep neural network, taking at least one MapReduce parameter as an input parameter, taking an optimal configuration parameter to be predicted as an output parameter, and taking historical data of a big data system as a training sample set; taking the MapReduce time as the measuring standard of the deep neural network, and adjusting the weight of each layer of nerve cells based on the parameter learning rule of the back propagation idea until the MapReduce time satisfies the time cost demand; and a configuration parameter prediction step of setting the initial value of the at least one MapReduce parameter, reading the current test data, inputting the current test data into the deep neural network obtained in the nerve network training step, and obtaining the configuration parameter. According to the invention, the configuration parameter in the MapReduce framework is adjusted and optimized by means of the deep neural network, manual adjustment is avoided, and the predicted parameter is good in application effect.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a method and system for optimizing configuration parameters of a big data system based on deep learning. Background technique [0002] In recent years, big data exploration and analysis has flourished in various fields. The big data system can be divided into three levels: (1) Basic layer: the basic data processing layer, which allocates hardware resources to the execution platform layer that supports computing tasks; (2) Platform layer: the core business layer, which is the application layer It provides an interface that is easy to process data sets and can manage resources allocated by the infrastructure layer; (3) Application layer: the output layer of prediction results, which predicts expert decisions and gives big data analysis results. [0003] The platform layer plays a connecting role in the big data system and is also the core part of a big data system. MapR...

AI Technical Summary

Problems solved by technology

[0006] The technical problem to be solved by the present invention is to provide a method and system for optimizing the configuration parameters of a big data system based on deep learning, aiming at the defects of low efficiency and poor effect of manual methods or using regression to automatically adjust configuration parameters in the prior art

Images