Distributed management method and system for metadata of large-scale storage system

Abstract

The invention discloses a distributed management method and system for metadata of a large-scale storage system. According to the method, metadata, stored in a NameNode memory, of an HDFS is abstracted into a structure of a two-dimensional table, and the structure of the two-dimensional table is stored in a distributed database in the form of the two-dimensional table; and the abstracted two-dimensional tables are associated with each other through an inode _ id. The Namenode becomes a bridge for the client to access the metadata, the client is firstly connected with the Namenode, the Namenode is used for operating the metadata in the distributed database, and the metadata is returned to the client. The single-point fault problem of the HDFS is solved.

Description

technical field [0001] The invention belongs to the technical field of distributed data storage, and in particular relates to a metadata distributed management and organization method of a large-scale storage system. Background technique [0002] With the rapid development of big data technology and applications, the Internet of Things, and cloud computing, the amount of data stored in a centralized manner can usually reach PB or even EB levels. Distributed file systems are a common solution for storing and managing large-scale data files. The distributed file system uses multiple machines to build a storage cluster, and the data storage capacity also increases linearly with the number of machines. To support the storage of large-scale data, in addition to hardware support, metadata management technology is also one of the essential key technologies. HDFS (Hadoop Distributed File System) is the most common distributed file system, but HDFS stores metadata in the memory of ...

AI Technical Summary

Problems solved by technology

HDFS (Hadoop Distributed File System) is the most common distributed file system, but HDFS stores metadata in the memory of a single Namenode machine, which limits the size of metadata storage space and becomes a bottleneck of system performance

Ceph is a high-performance, high-availability, and high-expansion distributed file system. It proposes a dynamic subtree partition method that dynamically divides metadata according to load conditions, and dynamically migrates metadata on nodes with higher loads to nodes with lower loads. On the lower nodes, dynamic load balancing is realized, but when the number of small files is large, Ceph's expansion of multiple metadata service nodes is still unstable, and it cannot provide online smooth expansion services

GlusterFS uses the elastic hash algorithm to replace the metadata management service, and uses the file path and file name to calculate the storage location of the file, which fundamentally solves the system bottleneck problem caused by the metadata service, and the traversal efficiency is low. When a directory When there are many files under download, it will be very slow to run file scanning, statistics, traversal and other operations

Images