An efficient distributed metadata management method combining memory and prefix tree

Abstract

The application discloses a kind of memory and prefix tree combined efficient distributed metadata management method.The method involves the management mode of metadata in distributed system, contain using improved consistent Hash algorithm to map metadata, and utilize the way of prefix tree to store metadata to reduce memory space.The application is based on the form of memory, and search and add all use binary search, greatly improve in the performance of search and add.The uniformity of metadata distribution is solved by the balance factor of MDS node, so that metadata can be as evenly distributed as possible in each MDS node, improve the performance of metadata management.In addition, the application stores actual metadata index by efficient prefix tree, obtains the metadata directly from disk by index, and uses small amount of space to exchange efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to a method for managing metadata in a distributed system, comprising using an improved consistent Hash algorithm to map metadata, and using a prefix tree to store metadata to reduce memory space. BACKGROUND

[0002] Metadata is a special data used to describe data. In a file system, metadata is data describing file attributes, including file directory content, file size, and file pointers. Metadata contains the location of data from the file name to the record.

[0003] With the development of the Internet and the digital transformation in various aspects, a large number of applications have generated massive data, such as image data and system log data. These massive data require a large amount of storage resources to store and manage data. In order to better manage these data, these image resources and log data are compressed and stored in a file system, and the operating system organizes these files, and finally the resource metadata is stored in a structured database, which serves as a file-to-resource mapping tool. Therefore, the management of massive metadata has become a major difficulty for systems using file systems as storage media.

[0004] Common methods for distributed metadata management include: static subtree partitioning method, suitable for scenarios where metadata lookup is frequent, which can cause uneven load between distributed metadata servers (MDS) when metadata is dynamically increased. Dynamic subtree partitioning method: the dynamic adjustment strategy of the subtree can greatly increase the communication between MDSs, and the system performance will be affected to a certain extent. Hash mapping method: when the data characteristics are similar, there is a data "skew" phenomenon, which leads to uneven load between MDSs.

[0005] In order to cope with the access pressure of massive data on single database, a multi-database cluster method is adopted to reduce the access of single database. However, this approach reduces the access pressure of each database, but also leads to data redundancy, and does not fundamentally solve the impact of massive data on database performance. Therefore, in order to fundamentally solve the impact of massive data on the database, the data amount of a single database needs to be reduced to improve performance, so data sharding storage is adopted. Massive data shards are stored in a database cluster to reduce the data amount of a single database. Data shards are stored for different metadata rather than sharding between metadata. This approach relieves the pressure on the database and improves the metadata query efficiency, but also greatly increases the complexity of the system, which needs to solve the problem of distributed transactions caused by sharded data, the sharding strategy of data, and the complex positioning of data shards. SUMMARY

[0006] The present application is based on the deficiencies of the prior art, and proposes a high-efficiency distributed metadata management method based on the combination of memory and prefix tree for the massive metadata generated by application, to provide an efficient storage and query scheme for massive metadata.

[0007] The present application comprises the following two stages:

[0008] The first stage: locate the distributed metadata server MDS where the metadata is located through the improved consistent Hash; the improved consistent Hash comprises: a MDS object set, a Hash ring and a metadata identification bucket set Key Bucket pointed to by each MDS.

[0009] The second stage: obtain the corresponding metadata from the located distributed metadata server MDS, share the part with the same metadata Key through the prefix tree, and store the address where the metadata is located to the node of the corresponding prefix tree.

[0010] Compared with the prior art, the present application has the following advantages:

[0011] 1. Through the form based on memory, the operation efficiency will be much higher than the traditional IO mode, and the binary search is used for searching and adding, which greatly improves the performance of searching and adding.

[0012] 2. Through the balance factor of the MDS node, the uniformity of metadata distribution is solved, so that the metadata can be as evenly distributed as possible in each MDS node, and the tilt problem caused by excessive load of a certain MDS node is avoided, and the performance of metadata management is improved.

[0013] 3. The actual metadata index is stored in the efficient prefix tree, and the metadata is directly obtained from the disk through the index, which uses a small amount of space to exchange efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 : compressed Bloom Filter schematic diagram;

[0015] Figure 2 : improved consistent Hash schematic diagram;

[0016] Figure 3 : prefix tree storage schematic diagram. DETAILED DESCRIPTION

[0017] The present application will be further described below in combination with the drawings and specific implementation steps:

[0018] The application adopts the idea of secondary cache, and manages metadata by dividing it into two stages:

[0019] The first stage: locate the MDS where the metadata is located by improved consistent Hash. The original consistent Hash adopts single ring structure. When a large number of data have similar characteristics, a large number of data will be mapped to the Hash ring by the hash function, and the problem of uneven distribution will occur. The improved consistent Hash has good data uniformity and high data query efficiency. The improved consistent Hash includes: MDS object set, Hash ring (32bit, data range (0~2 31 -1)) and metadata identification bucket set (Key Bucket) pointed by each MDS. The MDS object has the IP address of the target MDS, MDS serial number, Hash ring position and balance factor attribute, see Figure 2 .

[0020] The second stage: obtain the corresponding metadata from the located MDS. In order to improve the efficiency and reduce the memory occupation, the prefix tree is used to share the same part of all metadata Key (metadata identification request), and the address where the metadata is stored is stored in the node of the corresponding prefix tree.

[0021] Based on the above idea, the application adopts the following technical means:

[0022] When initialized, the initial MDS is uniformly distributed on the Hash ring, and then the Hash ring position corresponding to the MDS is written into the MDS object. All MDS objects are loaded into a set to form an MDS set. The structure of Key Bucket adopts the structure of array, such as Figure 2 which includes 8 bytes of metadata Key Hash Code, 1 byte of check code (when hash conflict occurs, the check code is used to judge whether it is the same Key), 1 byte of MDS serial number (record the MDS to which the metadata identification belongs), and Key Bucket adopts the way of binary search to locate data.

[0023] For example Figure 1As shown, the compressed Bloom Filter groups Bloom Filters BFa and BFb, and hashes and inserts the original data into BFa and BFb. The compressed Bloom Filter is denoted as BFc, and the insertion process of BFa and BFb is the same as that of a conventional BF. The compressed Bloom Filter can be expressed as: assuming that the required compressed Bloom Filter (CBF) length is m, BFa of length m / 2 is first created, data is filled into BFa, when the data filling amount exceeds a threshold (set according to the false positive rate requirement), BFb of length m / 2 is created, and data is filled into BFb. When BFa and BFb both reach the threshold under the respective false positive rates, BFa and BFb are combined into BFc through a table rule, so as to achieve the purpose of compressing memory.

[0024] Embodiments:

[0025] The embodiment includes the following steps:

[0026] Step (1): A consistent Hash server is created through an IO multiplexing technology to receive external requests. The operating system uses NIO select to listen to the file descriptor, and the overall Reactor model is used to distribute events, and a work thread is used to receive requests and forward them to a work thread, and the work thread uniformly processes them.

[0027] When a read request arrives, the metadata Key (the identification of the request metadata) is first judged through a consistent Hash compressed Bloom Filter (Compress Bloom Filter), for a BFc of length m, the number of hash mapping functions is k, the number of stored data is n, and the false positive rate of the Compress Bloom Filter is as follows:

[0028]

[0029] The non-existent data is directly returned to the corresponding requester, and the connection is closed.

[0030] It is preliminarily estimated that 1000W of metadata occupies about 1000W*10B≈95.37M, and such memory occupation is considerable. In the Key Bucket lookup process, binary search is used, and the time complexity of a single MDS lookup is O(logn), and the overall time complexity is O(Klogn).

[0031] When the Compress Bloom Filter judges that the metadata Key does not exist, it directly returns that the metadata does not exist. If the Compress Bloom Filter judges that the metadata Key exists, it preliminarily identifies that the metadata corresponding to the Key exists, and continues step (2).

[0032] The use of the Compress Bloom Filter in the embodiment is to efficiently screen out the non-existing metadata and provide a basis for uniform hashing, and only occupies a small amount of memory, thereby reducing the pressure on the server.

[0033] Step (2): When the Compress Bloom Filter judges that the metadata Key may exist, the request is retrieved to the MDS where the metadata Key is located through the improved consistent Hash. In order to solve the problem of uneven distribution of metadata Key caused by the similar characteristic values of the metadata Key through the consistent Hash, the improved consistent Hash contains two Hashes. First, the metadata Key is mapped to one of the N Hash functions through a first Hash function, and the first Hash function is as follows:

[0034] H i =a i *W+H i-1

[0035] Where H i represents the hash value of the metadata Key, a i represents the Ascii value of a single character contained in the metadata Key, and W represents a perturbation parameter.

[0036] The metadata Key is mapped to one of the N Hash functions in the Compress Bloom Filter through the above Hash function, and the metadata Key is mapped to the corresponding position of the Hash ring through the selected Hash function.

[0037] The above two Hash methods effectively solve the problem of uneven distribution caused by the similar characteristic values of the metadata Key through the consistent Hash, and then the metadata Key is retrieved through binary search.

[0038] For a write request, the requested metadata Key is directly added to the BFc, the metadata Key is recorded, and then the corresponding MDS set is found through the improved consistent Hash, the MDS set is traversed through binary search to find the metadata Key, and then the metadata Key is overwritten. For the case of Hash conflict, the conflict list needs to be traversed, and whether the metadata exists is determined by comparing the check code. If the metadata does not exist, if the load factor of the MDS node is too high, the MDS set is searched to find an MDS with relatively low load, and then the metadata Key is stored in the located MDS, and the check value and the adjusted MDS sequence number are written into the node and the located MDS.

[0039] Step (3): When the metadata Key is located through the improved consistent Hash, the target MDS is found, and the request is forwarded to the actual metadata storage node, which manages and stores the metadata. Each storage node uses a prefix tree structure as an index of the metadata storage location. The node of the prefix tree stores a data structure of a specific index, such as Figure 3 The structure shown in the figure contains 1B index number, 4B offset and 2B file sequence number. The prefix tree adopts a four-layer structure, each node contains 26 lowercase letters, and the entire prefix tree occupies about 26 4 *8B≈3.48M, wherein 8B represents the metadata index address (64-bit machine). 1000W data occupies about 1000W*7B≈66.76M. Therefore, 1000W data occupies about 70.24M of memory, and the disk memory can occupy: 2 32 *2 16 / 2 40 = 256T; the data is shared in a cyclic manner to save storage.

[0040] When the read and write metadata Key reaches the MDS storage node, the address of the prefix tree is first obtained from the memory, the address of the next node is found by searching the root node, and then the subscript of the metadata Key is also increased. When the Key is traversed, the metadata index contained in the character of the tree node is returned, otherwise the recursion from the root node is continued until the metadata index represented by the Key is found. The index is used to find the file sequence number, and then the metadata is found through the offset.

[0041] Other matters not covered by the present application are known techniques.

[0042] The above examples are only for illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made according to the spirit and essence of the present application shall be covered within the protection scope of the present application.

Claims

1. A highly efficient distributed metadata management method combining memory and prefix trees, characterized in that: The first stage: Locating the distributed metadata server (MDS) where the metadata resides through improved consistent hashing; the improved consistent hashing includes: a set of MDS objects, a hash ring, and a set of metadata identifier buckets (Key Buckets) pointed to by each MDS. The MDS object contains the IP address of the target MDS, the MDS sequence number, the position of the hash ring, and the balance factor attribute. The second stage involves retrieving the corresponding metadata from the located distributed metadata server (MDS), sharing all metadata keys with the same value through a prefix tree, and storing the address of the metadata in the corresponding prefix tree node. The aforementioned metadata identifier bucket set, Key Bucket, contains an 8-byte hash code for the metadata key, a 1-byte checksum, and a 1-byte MDS sequence number. The aforementioned metadata identifier bucket set, Key Bucket, uses a binary search method to locate data; The nodes of the prefix tree store a data structure for a specific index, which includes a 1B index count, a 4B offset, and a 2B file sequence number; the prefix tree adopts a four-level structure, and each node contains 26 lowercase letters.

2. The efficient distributed metadata management method combining memory and prefix trees according to claim 1, characterized in that: Before the first stage, a preliminary judgment of the metadata key is made using a compressed Bloom filter after a read request arrives. If the compressed Bloom filter determines that the above metadata key does not exist, it will directly return that the metadata does not exist. When the compressed Bloom filter determines that the above metadata key exists, it preliminarily concludes that the metadata corresponding to the above metadata key exists.

3. The efficient distributed metadata management method combining memory and prefix trees according to claim 2, characterized in that: The compressed Bloom filter is implemented as follows: Let the required length of the compressed Bloom filter be m. Create a first Bloom filter BFa with a length of m / 2 and fill it with data. When the amount of data filled exceeds the threshold, create a second Bloom filter BFb with a length of m / 2 and fill it with data. When both the first Bloom filter BFa and the second Bloom filter BFb reach their respective false positive rates, the first Bloom filter BFa and the second Bloom filter BFb are merged into a compressed Bloom filter using table rules, thereby achieving the purpose of memory compression.

4. The efficient distributed metadata management method combining memory and prefix trees according to claim 2, characterized in that: After initially confirming the existence of metadata corresponding to the aforementioned metadata key, the distributed metadata server (MDS) where the aforementioned metadata key resides is retrieved using an improved consistent hashing method.

5. The efficient distributed metadata management method combining memory and prefix trees according to claim 2, characterized in that: For write requests, the request's metadata key is added to the compressed Bloom filter, the metadata key is recorded, and then the corresponding MDS set is found through improved consistent hashing. The metadata key is found by traversing the MDS set through binary search; then it is overwritten.

6. The efficient distributed metadata management method combining memory and prefix trees according to claim 1, characterized in that: When the read / write metadata key reaches the distributed metadata server (MDS) storage node, the address of the prefix tree is first obtained from memory. The address of the next node is found by searching the root node, and then the index of the metadata key is also incremented. When the metadata key has been traversed, the metadata index contained in the characters in the tree node is returned. Otherwise, the recursion continues from the root node until the metadata index represented by the above metadata key is found. Find the index of the metadata, use the index to find the file number, and after finding the file, find the metadata using the offset.

Images