Distributed file system and file sharding method thereof

Abstract

The invention discloses a distributed file system and a file sharding method thereof. The system and the method aim to solve the problems of supporting a flexible data sharding method and meeting different application requirements. The file sharding method for the distributed file system comprises the steps of: determining a file sharding scheme according to the application requirements; and carrying out sharding processing according to the sharding scheme to acquire at least two data pieces. The distributed file system comprises a decision unit and a sharding unit, wherein the decision unit is used for determining the file sharding scheme according to the application requirements; and the sharding unit is used for carrying out the sharding processing according to the sharding scheme to acquire the at least two data pieces. The method is applicable to file sharding storage and data cache of the distributed file system.

Description

technical field [0001] The invention relates to the field of computer distributed storage, in particular to file slice storage and data cache of a distributed file system in computer cluster technology. Background technique [0002] In the current storage based on computer cluster technology, in order to meet the requirements of various file system operations and the efficiency of data storage and caching, various file fragmentation methods are adopted, such as splitting a large file according to a fixed fragmentation size Divide into consecutive data pieces of equal size, store each data piece on a different computer in a cluster computer system, or slice a large file in an exponentially increasing form, and the size of each piece increases exponentially The method changes and puts different shards on corresponding computers. [0003] In the process of realizing the above-mentioned data fragmentation storage method and system, the inventor found that there are at least the...

AI Technical Summary

Problems solved by technology

[0003] In the process of realizing the above-mentioned data fragmentation storage method and system, the inventor found that there are at least the following problems in the prior art: different file fragmentation storage technologies have different applicable services, and the fixed-size fragmentation method is suitable for equal fragmentation sizes For systems that require load balancing and small bandwidth, the fragmentation method in the form of exponential increase is a storage method based on the assumption that the access rate of part of the content behind the file is very small

Both of these methods are based on a fixed allocation strategy, which cannot meet various specific business needs.

Images