A method for dynamically configuring parameters of a batch streaming computing system

Abstract

The invention discloses a method for dynamically configuring parameters of a batch flow computing system, including: sample collection, model construction, optimal solution generation, online matching and parameter optimization; wherein, sample collection, model construction and optimal solution generation are in the background stage Complete; online matching and parameter optimization are completed in the online phase. According to the significance of the Pearson correlation between each parameter and the data processing response delay, this method selects the core parameter set that has a greater impact on application performance; on the premise of parameter reduction, the support vector regression method is selected to construct the application performance model. The model can quantitatively evaluate the response delay performance of data processing under a given data arrival rate and related parameter configuration; on the basis of performance modeling, a heuristic genetic algorithm is selected to quickly search for an optimal configuration scheme in a large solution space.

Description

technical field [0001] The invention belongs to the field of distributed computing, and in particular relates to a parameter configuration and optimization method of a batch streaming computing system. Background technique [0002] The development of the Internet and the Internet of Things drives the arrival of the era of big data. Streaming data is an important data form of big data, and with the development of applications such as social networks, it occupies a higher and higher proportion. Streaming data is defined as a series of massive, high-speed, and continuous arrival data sequences, which is a dynamic data collection that grows infinitely over time. Volatility is the core feature of streaming data: the generation of data is completely determined by the data source, and the dynamic changes of different data sources cause the data generation rate to fluctuate, that is, the data rate at the previous moment and the next moment may be very different. [0003] Big data ...

AI Technical Summary

Problems solved by technology

[0006] Static configuration is the default mode provided by the current batch streaming computing system, and its advantages are simplicity and low cost; the disadvantage is that it cannot adapt to the natural volatility characteristics of streaming data

In response to this shortcoming, dynamic configuration was proposed, but the current dynamic configuration only focuses on the data partition stage

In fact, the parameter configuration of the other two stages also has an important impact on performance: for example, as the parameter value of the data calculation stage of the batch stream computing system decreases, the response delay will increase

Obviously, the existing work is limited to the dynamic configuration of parameters in the data partitioning stage, resulting in a reduction in the flexibility of performance tuning; in addition, there is a dependency relationship between the parameters of each stage: for example, with the batch stream computing system As the parameter value increases, the optimal value of the data batch division interval will decrease

It can be seen that the robustness of the existing optimization work is also poor, that is, the existing optimization scheme for the data division stage will fail due to parameter changes in the data receiving or data calculation stage

Images