Distributed file system and data distribution method thereof

Abstract

The invention discloses a distributed file system and a data distribution method thereof. The data distribution method includes the following steps of driving the distributed file system into a plurality of storage layers, wherein each storage layer comprises a plurality of storage devices; arranging constitution information and replica placement strategy information of the storage layers and the storage devices of the storage layers; distributing object replicas to the storage devices in the storage layers according to the constitution information and the replica placement strategy information. According to the distributed file system and the data distribution method thereof, the pseudo-random data distribution method is used, the object replicas can be effectively and robustly distributed in a structured and hierarchical storage cluster, data distribution can be optimized to fully utilize available resources, and data security can be guaranteed to the greatest possible under the condition of consistent failure. The distributed file system and the data distribution method thereof have two advantages that data are completely distributed, and data in any part of a large-scale system can independently calculate object storage positions; a small number of requested metadata are basically static, and only when the devices perform addition or deletion do the metadata change.

Description

technical field [0001] The invention relates to data management of a file system, in particular to a distributed file system and a data distribution method thereof. Background technique [0002] In the era of rapid development of information technology, the massive expansion of data has brought the distributed file system into a golden age of development. From data sharing to Internet applications, it has been applied to all walks of life. For most distributed file systems, cluster file systems, and parallel file systems, metadata and data are usually separated, that is, control flow and data flow are separated, so as to obtain higher system scalability and I / O O Concurrency, which shows the importance and necessity of metadata. [0003] However, with the rapid expansion of data volume, the file system will face PB (million GB) level data and tens of millions of storage nodes, whether it is a centralized metadata service file system, a distributed metadata service file syst...

AI Technical Summary

Problems solved by technology

The main problem with this approach is that data is rarely moved after being written

However, a simple hash-based distribution strategy cannot adapt to changes in storage devices, resulting in re-migration and flushing of a large amount of data

Also, existing random distribution strategies approach replication by assigning a copy of each disk to other devices, thus losing data if multiple devices fail together.

Images