Method of HDFS Distributed and Centralized Hybrid Data Storage System Based on Hierarchical Governance

Abstract

The invention provides a HDFS distributed and centralized hybrid data storage system based on hierarchical governance. A distributed file system HDFS is combined with a centralized storage network file system (NFS), the high-activity data (or called as hot data) is stored by the HDFS, and the low-activity data (or called as warm-cold data) is stored by the NFS; the low-activity file based on the user policy definition can be slow-released to the NFS system from the HDFS by providing the user customizable file storage positioning policy, and the corresponding space on the HDFS is released. Through the method provided by the invention, the storage source can be effectively managed and scheduled by the application manufacturer using the HDFS-based big data, thereby ensuring that the high-activity data is managed in a distributed storage way, and the high-concurrence correspondence is provided; and the low-activity data can be stored in the NFS way, and the resource waste caused by addingunnecessary computing resource by expanding the HDFS capacity (horizontal expansion) is avoided. The warm-cold data staying ahead in overall share of the data can be managed in a relatively cheap andsafe way, and the effective governance on the hierarchical data is realized.

Description

technical field [0001] The invention belongs to the field of information technology, and in particular relates to a HDFS distributed and centralized hybrid data storage method based on data hierarchical management. Background technique [0002] In recent years, with the extensive development of big data applications, Hadoop-based computing framework has become one of the industry standard parallel computing environments. The distributed file system HDFS corresponding to Hadoop map-reduce has also become an industry-standard distributed storage system. Its multi-frame and erasure code mechanism can protect data in a more convenient way and provide high concurrency. . HDFS mainly relies on the disks of the computing nodes (computing server nodes) of the Hadoop cluster for storage, so it has the ability to expand horizontally, but it also restricts the expansion of storage capacity to a certain extent. Especially for scenarios that require vertical expansion capabilities, tha...

AI Technical Summary

Problems solved by technology

[0004] Aiming at the problems of vertical expansion capability and data governance existing in the existing HDFS distributed file system, the purpose of the present invention is to provide an efficient automatic data governance method to automatically move down the inactive data of HDFS to the NFS storage system

Images