Method and apparatus for balancing metadata cluster load

Abstract

The embodiment of the invention discloses a method for balancing the metadata cluster load. The method comprises the following steps: receiving a new catalogue creation request sent by a client; obtaining heat values of all metadata nodes in a cluster; determining the metadata node having the minimum heat value as the allocated metadata node; and creating a new catalogue under the allocated metadata node. Therefore, according to the method disclosed by the invention, the heat values of all metadata nodes are obtained, the metadata node having the minimum heat value is selected to create the new catalogue, thereby preventing the metadata node from becoming the hotspot metadata node in advance, balancing the metadata cluster load and improving the cluster performance, and then the performance of the whole file system is improved. In addition, the invention further discloses a system for balancing the metadata cluster load.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a method and device for balancing metadata cluster loads. Background technique [0002] The file system refers to the software mechanism responsible for managing and storing file information in the operating system, which is called the file management system. The existing file system can be mainly divided into two parts: metadata management and data management. Metadata is the data describing the data, mainly the information describing the attributes of the data. Metadata is an electronic catalog. In order to achieve the purpose of cataloging, The content or characteristics of the data must be described and collected to achieve the purpose of assisting data retrieval. According to statistics, it is found that more than 50% of all requests in the file system are requests for metadata. Therefore, the performance of the entire file system can be improved by improving the pr...

AI Technical Summary

Problems solved by technology

At this time, if multiple users access the files under this metadata node at the same time, the popularity of this metadata node will continue to increase. If you create a new subdirectory under the hot directory under this metadata node, the load on this metadata node will be caused. The overall metadata cluster performance degrades when the load is extremely unbalanced with other metadata nodes, for example, as in figure 1 In the directory structure diagram of the existing mechanism shown, there are three metadata nodes in total, which are metadata node No. 0, metadata node No. 1, and metadata node No. 2. When creating subdirectories / subfiles of A, B, and C When , it is still created under the metadata nodes where A, B, and C are located. If the number of accesses to directory a increases at a certain time, metadata node 0 will become a hot metadata node

[0004] After research, the inventor found that in the prior art, the method to solve the extremely unbalanced load of metadata nodes is to separate the hot directories in the hot metadata nodes by manually executing commands, for example, as figure 1 As shown, the existing technical method is to distribute the directory aa to the No. 1 metadata node or the No. 2 metadata node with a lower heat value, so as to reduce the pressure on the No. 0 metadata node, but the existing technology is in After the metadata node becomes a hotspot metadata node, the solution is implemented. The load imbalance of the metadata cluster and the overall performance of the file system have already appeared.

Images