Big data cluster self-adaptive resource scheduling method based on cloud platform

Abstract

The invention belongs to the technical field of computing, and relates to a big data cluster adaptive resource scheduling method based on a cloud platform. The method comprises the steps that in the big data analysis task classification analysis stage, CPU and I / O characteristics of big data analysis tasks are preliminarily analyzed through a neural network classifier; in the initial stage of configuration of a small number of sample clusters, the optimal configuration is rapidly obtained by means of a Bayesian optimization algorithm; in the cluster configuration online optimization stage, iterative optimization of a selection strategy is configured; and in a configuration selection stage in which sufficient samples have time limitation, execution time of the big data analysis task is predicted under different configurations based on a non-negative least square method, and optimal configuration is selected under the condition of time limitation. The method can solve the problem of reasonable selection of cluster configuration for running big data analysis tasks on the cloud platform, and guarantees the resource utilization rate of the cloud platform while guaranteeing the task execution efficiency.

Description

technical field [0001] The invention belongs to the technical field of cloud computing, and in particular relates to a method for adaptive resource scheduling of large data clusters based on a cloud platform. Background technique [0002] With the improvement of computer storage capacity and the development of complex algorithms, the amount of data on the Internet has grown exponentially in recent years. These trends have led to the rapid development of science and technology. By 2020, the total amount of newly added and copied data worldwide is expected to grow to 44ZB. The industry believes that the large amount of data accumulation and growth is due to the data-related practices of various companies and individuals; taking Facebook as an example, the total number of pictures uploaded by users every day can reach up to 300 million, and the daily content delivery can reach up to 2.5 billion. The total amount of data can be added up to 500TB per day; similarly, Google's mon...

AI Technical Summary

Problems solved by technology

It is increasingly difficult for traditional resource scheduling schemes to meet the requirements of complex and changeable environments and rapid response

With the popularity of data-driven methods in recent years, some research programs based on data-driven cloud platform resource scheduling have emerged in academia, including research based on traditional machine learning (Machine Learning, ML) and research based on reinforcement learning (Reinforcement Learning, RL). ) research, etc., but when these studies are applied to the cloud platform, there are still shortcomings, such as Christina Delimitrou et al [CC2013, Christina Delimitrou and Christos Kozyrakis. QoS-Awarescheduling indigenous datacenters with paragon. 2013.] Grasp the characteristics of the task by modeling the workload, associate it with tasks with similar characteristics, and use historical experience to guide the resource allocation of upcoming tasks. This method relies on a large amount of workload data for Accurate modeling requires sufficient historical operating data to support resource allocation decisions; Shivaram Venkataraman et al [SZM2016, Shivaram Venkataraman, Zongheng Yang, Michael Franklin, Benjamin Recht, and Ion Stoica. Ernest: Efficient performance prediction for large-scale advanced analytics. NSDI'16, 363 -378.] Aiming at the problem of big data cluster configuration selection on the cloud platform, a method based on machine learning modeling is proposed. For a specific task and a specific type of virtual machine, given the number of virtual machines and data In the case of a large amount of data, the prediction model obtained through historical data training can predict the execution time of the task in the case of the current data size and the number of virtual machines. Although this method can achieve accurate time prediction, and based on the predicted time A reasonable configuration is assigned to it, but its training cost is too high, a large amount of historical data is required to ensure the accuracy of the model, and it needs to be retrained for different tasks and virtual machine types, and the flexibility is poor; Omid Alipourfard et al. OHJ2017, Omid Alipourfard, Hongqiang Harry Liu, Jianshu Chen, Shivaram Venkataraman, Minlan Yu, and Ming Zhang. Cherrypick: Adaptively unearthing the best cloud configurations for b ig dataanalytics.NSDI'17, 469-482.] In the case of only a small number of data samples, a data-driven method can also be used to obtain a near-optimal cluster configuration through a small number of samples, but this solution can only be used for Daily repetitive tasks, and re-sampling training is required for different tasks; etc.

Images