Storage scheduling method and system, job scheduling method and system and management node

Abstract

The invention provides a storage scheduling method and a job scheduling method for scheduling according to a path available bandwidth, and further provides a management node, a storage scheduling system and a job scheduling system corresponding to the storage scheduling method and the job scheduling method. According to the storage scheduling algorithm, a data block is stored in a data node list with the maximum weighted mean of the path available bandwidth, network blocking is avoided, and the network transmission time of storage of the data block is shortened. Distribution optimization operation of data nodes is processed according to the path available bandwidth information, the localization probability of operation is improved, and the total completion time of operation is shortened.

Description

technical field [0001] The present invention relates to the technical field of cloud computing, in particular to a storage scheduling method and system in cloud computing task scheduling, a job scheduling method and system, and a management node. Background technique [0002] With the rapid development of data-intensive Internet applications such as search engines, social networking sites, and e-commerce, massive data processing platforms based on distributed infrastructure (such as Hadoop, MapReduce, distributed file systems, and data definition storage) have attracted widespread attention. On these platforms, it is not only necessary to support large-capacity file (TB-level, PB-level) storage / download and fast retrieval, but also to perform large-scale data processing operations such as machine learning, data mining, and data analysis, and to respond quickly to interactive users. Therefore, the cloud computing platform needs to distribute and execute multiple data-intensiv...

AI Technical Summary

Problems solved by technology

[0003] This type of task scheduling algorithm has the following disadvantages: 1. In the process of remote data storage and reading, the distributed file system does not perceive the available bandwidth information of the network path in the platform, and the data node selected by the task scheduling algorithm for the job may be affected by the network. Poor quality leads to extended data access time and poor user experience; 2. Environmental factors such as data center workload and network load may change at any time, and the waiting time threshold of the existing delay scheduling algorithm is set when the user submits the job , can not adapt to the change of data center load, although there are improved algorithms, such as the dynamic change of time threshold, but the calculation formula is complex, the calculation load is large, it is not suitable for large-capacity distributed systems, resulting in reduced practicability

Images