Intelligent hash data layout method, cluster storage system and cluster storage method

Abstract

The invention discloses an intelligent hash data layout method, a cluster storage system and a cluster storage method. The intelligent hash data layout method is used for laying out storage nodes in a data volume. Data is distributed through a hash distribution method by taking a directory as a basic unit. The parent directory of a file utilizes extended attributes to record the mapping information of the storage nodes. Sub-files are distributed in the storage nodes to which the parent directory belongs. The distribution method of the sub-files in the storage nodes to which the parent directory belongs specifically comprises one of the following situations that: the sub-files are distributed in the storage nodes to which the parent directory belongs through the hash distribution method; the sub-files are distributed in the storage nodes to which the parent directory belongs through a zonal two-level distribution method; the sub-files are distributed in the storage nodes to which the parent directory belongs through a duplicate two-level distribution method; and the sub-files are firstly distributed in the storage nodes to which the parent directory belongs through the zonal two-level distribution method and then are distributed through a duplicate three-level distribution method. The system and the methods provided by the invention have the advantages that the extensibility, the performance, the availability and the applicability of the cluster storage system can be remarkably improved, and the load pressure of a storage server can be greatly decreased.

Description

technical field [0001] The invention relates to the technical field of data storage, in particular to an intelligent hash data layout method, a cluster storage system and a method thereof. Background technique [0002] Under the background of cloud storage and big data, data presents an explosive growth trend. According to research, the digital universe will reach 35.2ZB in 2020, a 44-fold increase from 0.8ZB in 2009, of which more than 80% are unstructured data. A large number of data-intensive applications such as high-performance computing, medical imaging, oil and gas exploration, digital media and social WEB lead to a blowout of data, which constantly poses new and severe challenges to storage methods. Clustered storage is a scale-out storage architecture that has the advantages of linear expansion in capacity and performance, and has been widely recognized by the global market. Cluster storage technology involves two key issues, namely metadata management and data la...

AI Technical Summary

Problems solved by technology

Centralized or distributed metadata management has a series of related issues such as performance bottlenecks, single point of failure, and data consistency, which directly affect system performance and scalability

The data layout mainly adopts stripe (Stripe), mirror (Mirror), hash (Hash), consistent hash (DHT) and other methods. The common problem with them is that the scalability is not high. When the cluster scale expands, a large number of Data needs to be migrated and redistributed

This method relies on the metadata server, and there are a series of related problems such as performance bottlenecks, single point failures, and data consistency in centralized or distributed metadata management, which directly affect system performance and scalability.

Images