A data equalization method, apparatus and computer readable storage medium

Abstract

The embodiment of the invention discloses a data equalization method, device and computer-readable storage medium. When a newly created directory is smaller than the fragmentation requirement, data corresponding to the directory is grouped according to fragmentation rules to obtain multiple fragmented data. According to the principle of load balancing, configure each piece of data to the corresponding metadata server. When the directory is smaller than the fragmentation requirement, it means that the directory contains a large number of subdirectories or files. According to the prior art, it takes a lot of time to allocate data with the subdirectory as the minimum allocation unit. By packaging multiple directories into a fragmented data, and using the fragmented data as the smallest processing unit for data allocation, the amount of data allocation is effectively reduced, and the processing performance of the distributed file system is improved. And the data is grouped according to the sharding rules, which effectively solves the problem in the prior art that the processing performance of the distributed file system is degraded when a single subdirectory contains a large number of files.

Description

technical field [0001] The present invention relates to the technical field of distributed file systems, in particular to a data equalization method, device and computer-readable storage medium. Background technique [0002] In the process of using the distributed file system, in order to achieve load balancing, it is necessary to balance the file data corresponding to all directories to different metadata servers (Meta Data Server, MDS). [0003] The directory structure of the file system constitutes the directory tree of the entire file system. The distributed file system consists of multiple MDSs, and each MDS is responsible for a part of the directory tree. These part of the directory tree are called subtrees of the file system. In a scenario where there are a large number of subdirectories, tens of thousands to hundreds of thousands of subtrees will be generated if hundreds of thousands or even more subdirectories exist, and the distributed file system will call more sy...

AI Technical Summary

Problems solved by technology

In a scenario where there are a large number of subdirectories, tens of thousands to hundreds of thousands of subtrees will be generated if hundreds of thousands or even more subdirectories exist, and the distributed file system will call more system resources to achieve load balancing , leading to a decline in the overall performance of the distributed file system

At the same time, the subtree is only for subdirectories. When a single subdirectory contains a large number of files, the processing performance of the distributed file system will be worse.

Images