Method and system for dynamic load balancing of metadata cluster directory

Abstract

The invention discloses a metadata-cluster-directory dynamic load balancing method and system. The metadata-cluster-directory dynamic load balancing method includes the steps that whether the heat degree of a root directory of a directory tree of a primary metadata server is larger than a preset threshold value or not is judged; if the heat degree of the root directory of the directory tree of the primary metadata server is larger than the preset threshold value, whether a current subdirectory only contains files or not is judged, wherein the current subdirectory is a subdirectory of a root directory meeting the heat degree larger than a threshold value; if the current subdirectory only contains the files, the current subdirectory is distributed to a secondary metadata server in a polling mode; if the current subdirectory does not only contain the files, the subdirectory of the current subdirectory serves as a new current subdirectory. According to the metadata-cluster-directory dynamic load balancing method and system, the threshold value of the heat degree is set, whether the heat degree of the root directory of the directory tree of the primary metadata server of a metadata service is larger than the preset threshold value or not is judged, and if the heat degree of the root directory of the directory tree of the primary metadata server of the metadata service is larger than the preset threshold value, the subdirectory serving as a heat degree source, only containing the files and meeting the heat degree larger than the threshold value is sought, the subdirectory is distributed to the secondary metadata server corresponding to the primary metadata server in a polling mode, and metadata service load balancing is achieved.

Description

technical field [0001] The invention relates to the field of distributed files, in particular to a method and system for dynamic load balancing of metadata cluster directories. Background technique [0002] In distributed file system applications, there will be a large number of small file usage scenarios. For example, in video surveillance, the camera port stores bayonet pictures (500KB), the material files in high-performance computing examples (several KB to hundreds of KB), and the animation rendering materials in radio and television media scenes (several KB to hundreds of KB), etc. The scene generates millions of files every day, and the file operation requirement per second is as high as 5000 / s. The metadata performance of the distributed file system is a bottleneck, and metadata clusters need to be used to improve metadata concurrency. The key problem facing the metadata cluster is how to realize the dynamic load balancing of the directory. At present, there are m...

AI Technical Summary

Problems solved by technology

Due to the different IO models of different application scenarios, it is difficult to have a dynamic directory load balancing mechanism applicable to all scenarios

[0003] In the existing technology, the metadata service in the metadata cluster counts the popularity of directories in units of directories, and the popularity of directories represents the load of metadata services. Part of the load is divided from the mds process with high load to the mds with light load, and through multiple directories Metadata concurrency improves metadata performance, but requires frequent statistics of directory popularity and metadata load, increasing metadata processing delays, and existing algorithms will frequently switch between directories and metadata services, which will degrade performance and be difficult to apply to production environments

Images