Storage scheduling method and device

Abstract

The embodiment of this application provides a storage scheduling method and device, where the storage scheduling method includes: the performance evaluation model is established according to the dynamic factor of the data node;The configuration attributes of the data node are scheduled to perform the data nodes that implement the storage function, thereby providing an effective scheduling mechanism, partly or all to avoid or alleviate the above technical problems in existing technologies.

Description

technical field [0001] The embodiments of the present application relate to the field of storage technologies, and in particular, to a storage scheduling method and device. Background technique [0002] HDFS is the Hadoop distributed file system, which has the advantages of high fault tolerance, high reliability, and high scalability. It simplifies the consistency model of the file, improves the throughput of data access through the streaming method, and is especially suitable for the storage and access of massive data. [0003] The fault tolerance of HDFS allows cheap hosts in the cluster to serve as data nodes for storing data. Therefore, many HDFS clusters contain a large number of server nodes with heterogeneous performance, including newly purchased high-performance servers and legacy low-cost servers. low-performance, low-reliability servers. Moreover, differences in CPU and memory performance, network card and disk I / O speed, and network location of different server...

AI Technical Summary

Problems solved by technology

Although this ensures the reliability of data storage to a certain extent, it may cause cluster performance problems caused by node load imbalance in heterogeneous clusters with large performance differences.

On the one hand, all copies of frequently accessed data may be stored on data nodes with poor performance, causing the low performance data nodes to be overloaded, thereby reducing the speed of cluster operation; on the other hand, data that is not frequently accessed may be stored All copies of are stored on high-performance data nodes, resulting in underutilization of high-performance nodes

And when data with low access frequency occupies too many high-performance data nodes, it will force the data nodes with high access frequency to be stored on low-performance data nodes. When the access pressure increases, the response speed of the cluster will be further reduced

Images