Directory distribution method, electronic device, and computer-readable storage medium

Abstract

The application belongs to the field of data storage, and discloses a directory distribution method, an electronic device and a computer readable storage medium. The method comprises the following steps: inserting a to-be-created directory item into a virtual directory of a virtual set of a distributed file system based on a skip consistent hashing algorithm; if the storage amount of the virtual directory is greater than a preset storage threshold, determining whether the number of sets of the virtual set in the distributed file system is less than a preset set threshold; if yes, splitting a new directory set from the virtual set and allocating a metadata node to the new directory set; if no, determining whether the number of directories of the virtual directory is less than a preset directory threshold; if the number of directories is less than the preset directory threshold, performing directory splitting based on a global bitmap of the virtual set where the virtual directory is located; and if the number of directories is greater than or equal to the preset directory threshold, controlling the directory expansion and the directory splitting to be alternately split. The application aims to solve the technical problem that the creation of a directory in a distributed file system is not efficient enough.

Description

Technical Field

[0001] The present application belongs to the technical field of data storage, and relates to a directory distribution method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Currently, with the rapid development of mobile Internet applications and scientific computing applications, data is growing explosively. The increasing amount of data requires the underlying distributed file system to support a large number of files and large directories containing massive files. Therefore, how to achieve efficient directory distribution among multiple data nodes is an urgent problem that needs to be solved in the design of distributed file systems. Currently, in a single directory partitioning mode, a directory is divided into fixed-size directory blocks using GIGA+ (Generalized Infinitesimal Gradient Ascent, gradient ascent algorithm), each directory block is extended to a multi-data node, and the directory blocks in each metadata node are stored through a cross-node bitmap, thereby achieving directory storage on the multi-data node and enabling directory block search through the cross-node bitmap. However, since fixed-size directory blocks are stored, it is difficult to balance the size of the directory blocks. If the divided directory blocks are too small, the number of directory blocks will continue to increase as the directory continues to grow, causing the cross-node bitmap to be too large, thereby affecting the query efficiency of the directory blocks. If the divided directory blocks are too large, the efficiency of querying data within the directory blocks is low. Moreover, when the directory blocks are stored in blocks, the data that needs to be migrated and stored will also increase due to the large directory blocks, which will also affect the efficiency of directory splitting and storage. Therefore, when storing massive data, the technical problem of inefficient directory creation in the distributed file system is that it is difficult to balance the size of the stored directory blocks when creating directories on each metadata node of the distributed file system.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a directory distribution method, electronic device and computer-readable storage medium, aiming to solve the technical problem of inefficient directory creation in a distributed file system.

[0005] To achieve the above objectives, the present application provides a directory distribution method, which is applied to a distributed file system. The directory distribution method includes:

[0006] In response to a directory entry creation instruction, starting a directory creation process in the distributed file system;

[0007] The directory creation process is as follows: inserting a directory entry to be created into a virtual directory of a virtual set of the distributed file system based on a jump consistent hashing algorithm; if the storage capacity of the virtual directory is greater than a preset storage threshold, determining whether the number of sets of the virtual set in the distributed file system is less than a preset set threshold;

[0008] If so, splitting a new directory set from the virtual set, and allocating metadata nodes to the new directory set for directory storage in the new directory set;

[0009] If not, determining whether the number of directories in the virtual directory is less than a preset directory threshold;

[0010] If the number of directories is less than the preset directory threshold, performing directory splitting based on the global bitmap of the virtual set where the virtual directory is located, wherein the directory splitting is splitting the virtual directory;

[0011] If the number of directories is greater than or equal to the preset directory threshold, directory expansion and directory splitting are controlled to be performed alternately, wherein the directory expansion is to expand the storage capacity of the virtual directory.

[0012] To achieve the above objectives, the present application provides a directory creation device, which includes:

[0013] A response module, configured to initiate a directory creation process in the distributed file system in response to a directory entry creation instruction;

[0014] A first judgment module is used in the directory creation process of: inserting a directory entry to be created into a virtual directory of a virtual set of the distributed file system based on a jump consistent hashing algorithm, and if the storage capacity of the virtual directory is greater than a preset storage threshold, then judging whether the number of sets of the virtual set in the distributed file system is less than a preset set threshold;

[0015] a set splitting module, configured to, if yes, split a new directory set from the virtual set, and allocate metadata nodes to the new directory set for directory storage in the new directory set;

[0016] A second judgment module is used to judge whether the number of directories in the virtual directory is less than a preset directory threshold if no;

[0017] a directory splitting module, configured to perform directory splitting based on a global bitmap of the virtual set where the virtual directory is located if the number of the directories is less than the preset directory threshold, wherein the directory splitting is to split the virtual directory;

[0018] The alternating splitting module is used to control directory expansion and directory splitting to be performed alternately if the number of directories is greater than or equal to the preset directory threshold, wherein the directory expansion is to expand the storage capacity of the virtual directory.

[0019] The present application also provides an electronic device, which includes: a memory, a processor, and a program of the directory distribution method stored in the memory and runnable on the processor. When the program of the directory distribution method is executed by the processor, the steps of the directory distribution method described above can be implemented.

[0020] The present application also provides a computer-readable storage medium, on which is stored a program for implementing the directory distribution method. When the program of the directory distribution method is executed by a processor, the steps of the directory distribution method as described above are implemented.

[0021] The present application also provides a computer program product, comprising a computer program, which implements the steps of the above-mentioned directory distribution method when executed by a processor.

[0022] The present application provides a directory distribution method, an electronic device, and a computer-readable storage medium. The present application determines the storage capacity of a virtual directory of a virtual set in a distributed file system when creating a directory item to be created in the distributed file system. When the storage capacity of the virtual directory is greater than a preset storage threshold and the number of virtual sets is less than the preset set threshold, the virtual set is split, and metadata nodes are allocated to the new directory set generated by the split in the distributed file system to store the directory item to be created in the new directory set. When a new directory item needs to be inserted, the storage capacity of the virtual directory reaches the preset storage threshold and the number of virtual sets in the distributed file system does not reach the preset set threshold, the virtual set can be split, and metadata nodes are allocated to the new directory set generated by the split to store the directory in the new directory set, thereby avoiding always inserting the directory item to be created into the same virtual set, and realizing that the directory item to be created can be inserted in each virtual set, thereby realizing inserting the directory item to be created between each metadata node, and realizing directory storage on each metadata node.

[0023] Furthermore, when the storage capacity of the virtual directory reaches a preset storage threshold and the number of virtual sets is greater than or equal to the preset set threshold, and the number of directories in the virtual directory is less than the preset directory threshold, the virtual directories in the virtual set can be split based on the global bitmap of the virtual set, thereby expanding the storage space of the virtual set to store a large amount of data. Furthermore, when the number of directories in the virtual set is greater than or equal to the preset directory threshold, the directory expansion and directory splitting are controlled to be performed alternately, so that the size expansion of the virtual directory and the increase in the number of virtual directories are achieved alternately, thereby avoiding the problem that the global bitmap of the virtual set is too large due to the excessive number of virtual directories in the virtual set, and the low efficiency of virtual directory splitting caused by the large global bitmap, which in turn leads to the problem of low directory creation efficiency. Therefore, the present application realizes data storage in each metadata node by splitting the virtual set, and splits the virtual directories in the virtual set when the number of sets of the virtual set reaches the preset set threshold, and then controls the directory expansion and directory splitting to be performed alternately when the number of directories of the virtual directory reaches the preset directory threshold, thereby realizing adaptive growth of the number of virtual directories and the size of virtual directories in the distributed file system, and solving the technical problem of inefficient directory creation in the distributed file system due to the difficulty in balancing the number of directories and the size of directories. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 This is a flowchart of the first embodiment of the directory distribution method of this application;

[0027] Figure 2 This is a schematic diagram of the splitting of the virtual collection among the metadata nodes in the directory distribution method of this application;

[0028] Figure 3 A schematic diagram of a global bitmap of a virtual set of directory distribution methods of this application;

[0029] Figure 4 This is a flow chart of the second embodiment of the directory distribution method of this application;

[0030] Figure 5 This is a flowchart of the third embodiment of the directory distribution method of this application;

[0031] Figure 6 A schematic diagram of a distributed file system for the directory distribution method of this application;

[0032] Figure 7 A schematic diagram of an apparatus according to an embodiment of the directory distribution method of the present application;

[0033] Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the directory distribution method in the embodiment of the present application.

[0034] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0035] To make the above-mentioned purposes, features, and advantages of the present application more clearly understood, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making any creative work are within the scope of protection of this application.

[0036] Current directory distribution methods in distributed file systems can be broadly categorized into the following modes: unit data node mode, subtree partitioning mode, and single-directory partitioning mode. The unit data node mode stores all metadata on a single node, storing the entire directory. This mode, such as GFS, HDFS, and earlier versions of Lustre, avoids cross-node operations and data migration overhead during metadata operations. However, storing massive files on a single node makes it difficult to partition them, resulting in a single bottleneck. The subtree partitioning mode divides stored data into multiple subtrees (the smallest subtree is a single directory) and distributes them across multiple metadata nodes. There are two types of subtree partitioning: static subtree partitioning and dynamic subtree partitioning. Static subtree partitioning requires preconfiguration, assigning subtrees to designated metadata nodes. For example, NFS (Network File System) and Lustre (Parallel Distributed File System) versions 2.3 and later use static subtree partitioning to quickly utilize multiple metadata nodes. Dynamic subtree partitioning, based on the load of each metadata node, dynamically migrates subtrees and adjusts load, effectively ensuring load balancing. Ceph (Distributed File System) is a typical example. However, regardless of dynamic subtree partitioning or static subtree partitioning, the smallest unit of data storage is still a single directory, which makes it difficult to handle a single large directory.

[0037] The single directory partitioning method is mainly aimed at distributing a single directory on multiple metadata nodes. Through the GIGA+ directory partitioning method, efficient dynamic distribution of a single directory on multiple metadata nodes is achieved. GIGA+ divides a single directory into fixed-size directory blocks. When the directory is small, only one directory block is used. As the directory grows, the directory blocks will continue to split and expand to multiple metadata nodes, realizing the dynamic distribution of the directory on multiple data nodes. GIGA+ uses a full binary tree structure and utilizes the deterministic structure of the full binary tree. The next split block index is directly calculated through the current directory block index and the tree depth information, and a cross-node bitmap is used to store the directory blocks on each metadata node, realizing fast storage and search of directory blocks through the cross-node bitmap. However, as more and more data is stored on each metadata node, the cross-node bitmap will also become larger. The storage status of each metadata node's directory blocks is recorded in the cross-node bitmap in the form of a node bitmap. When a directory block in a metadata node is updated, the client needs to update the node bitmap of the metadata node to update the cross-node bitmap. When there are a large number of directory blocks to be stored, the cross-node bitmap will be updated frequently. Frequent updates of the cross-node bitmap can easily cause the client's cached cross-node bitmap to become invalid, thus affecting the performance of the metadata node. GIGA+ also divides the directory into fixed-size directory blocks. Since it stores fixed-size directory blocks, it is difficult to balance the size of the directory blocks. If the directory blocks are too small, the number of directory blocks will continue to increase as the directory grows, causing the cross-node bitmap to become too large, affecting the efficiency of directory block queries. If the directory blocks are too large, data queries within the directory blocks will be inefficient. When directory blocks are stored in blocks, the data that needs to be migrated due to the large directory blocks will also increase, which will also affect the efficiency of directory split storage. Therefore, the current directory distribution method in the distributed file system is difficult to partition and store a single large directory, and it is difficult to achieve efficient distribution of directories among multiple data nodes.

[0038] Example 1

[0039] Reference Figure 1 The present application provides a directory distribution method, which is applied to a distributed file system. In a first embodiment of the directory distribution method of the present application, the directory distribution method includes:

[0040] Step S10, in response to the directory entry creation instruction, starting a directory creation process in the distributed file system;

[0041] Step S20, wherein the directory creation process comprises: inserting a directory entry to be created into a virtual directory of a virtual set of the distributed file system based on a jump consistent hashing algorithm; if the storage capacity of the virtual directory is greater than a preset storage threshold, determining whether the number of sets of the virtual set in the distributed file system is less than a preset set threshold;

[0042] It should be noted that the embodiment of the present application introduces the concepts of virtual sets and virtual directories in a distributed file system, and then inserts directory entries into the virtual directories of the virtual sets to store data. In a distributed file system, it can be divided into preset large directories, virtual sets, virtual directories, and sub-directory entries. Virtual sets include virtual directories. Virtual sets are set on metadata nodes. Metadata nodes include multiple virtual sets. The distributed file system includes multiple metadata nodes. The directory entries to be created are data that need to be stored in the distributed file system. The directory entries to be created can be files. The preset storage threshold is the maximum storage capacity of the virtual directory that triggers the splitting of the virtual set. The number of sets is the number of current virtual sets when the preset large directory is allocated. The preset set threshold is the maximum number of virtual sets set when the preset large directory is allocated. Among them, the preset large directory is characterized as a large directory stored in the distributed system. The large directory is a directory containing a large number of files. The preset large directory can be determined based on actual conditions. The preset large directory can be allocated in each virtual set.

[0043] Step S30: If yes, split the virtual set, and allocate metadata nodes to the new directory set in the distributed file system to migrate the directory entries to be created to the new directory set, wherein the new directory set is a new virtual set generated by the split;

[0044] It should be noted that when a virtual set is split, the original virtual set and the new directory set are obtained. The original virtual set is characterized as a virtual set whose set subscript remains unchanged before and after the split, and the metadata node where the virtual set resides remains unchanged before and after the split. The new directory set is characterized as a new virtual set split from the virtual set, and the set subscript of the new directory set is different from the set subscript of the original virtual directory. Because a metadata node includes multiple virtual sets, the metadata node where the new directory set resides may be the same as or different from the metadata node where the original virtual set resides.

[0045] Step S40: If not, determine whether the number of directories in the virtual directory is less than a preset directory threshold;

[0046] Step S50: If the number of directories is less than the preset directory threshold, performing directory splitting based on the global bitmap of the virtual set where the virtual directory is located, wherein the directory splitting is splitting the virtual directory;

[0047] Step S60: If the number of directories is greater than or equal to the preset directory threshold, directory expansion and directory splitting are controlled to be performed alternately, wherein the directory expansion is to expand the storage capacity of the virtual directory.

[0048] It should be noted that the preset directory threshold represents the maximum number of virtual directories when the storage capacity of virtual directories is expanded. The number of directories represents the number of virtual directories in a virtual set. The global bitmap includes all virtual directories in the virtual set and is derived based on the GIGA+ algorithm. When the number of virtual directories in a virtual set is greater than or equal to the preset directory threshold, the size of the virtual directory is expanded. This prevents the global bitmap of the virtual set from becoming too large due to the continuous increase in the number of virtual directories, which in turn leads to inefficient storage of directory entries and inefficient searches based on the global bitmap due to an overly large global bitmap.

[0049] As an example, steps S10 to S60 include: in response to a directory entry creation instruction, starting a directory creation process in the distributed file system, wherein the directory creation process is: based on the directory entry to be created, determining the virtual directory of the virtual set of the directory entry to be created in the distributed file system, inserting the directory entry to be created in the virtual directory, judging whether the storage capacity of the virtual directory is greater than a preset storage threshold, and if so, judging whether the number of sets of the virtual sets in the distributed file system is less than the preset set threshold; if so, splitting the virtual set in the distributed file system to obtain a new virtual set in the distributed file system; if not, judging whether the number of directories of the virtual directory is less than the preset directory threshold, and if the number of directories is less than the preset directory threshold, performing directory splitting based on the global bitmap of the virtual set where the virtual directory is located, wherein the directory splitting is splitting the virtual directory; if the number of directories is greater than or equal to the preset directory threshold, controlling directory expansion and directory splitting to be performed alternately, wherein the directory expansion is expanding the storage capacity of the virtual directory.

[0050] When directory storage is first started in a distributed file system, there is only one virtual set and one virtual directory in the distributed file system. When the storage capacity of the virtual directory exceeds a preset storage threshold, the virtual set is split and configured on each metadata node in the distributed file system. When the new directory set is obtained by splitting the virtual set and metadata nodes are allocated for the new directory set, the virtual sets are allocated sequentially among the metadata nodes to ensure that the virtual sets are evenly distributed among the metadata nodes. When directory storage is first started, the metadata node corresponding to the directory entry to be created can be randomly selected, and the subscript of the metadata node can be defined based on actual conditions.

[0051] For example, refer to Figure 2 , Figure 2Figure 1 is a diagram of the splitting of virtual sets among metadata nodes. There are three metadata nodes in the distributed file system, namely Meta0, Meta1 and Meta2. Meta0 is the metadata node with node subscript 0, Meta1 is the metadata node with node subscript 1 and Meta2 is the metadata node with node subscript 2. vset is a virtual set, where Meta0 includes virtual set vset0 and virtual set vset3, Meta1 includes virtual set vset1 and virtual set vset4, and Meta2 includes virtual set vset3 and virtual set vset5. The initial state of the distributed file system is Me ta0 includes vset0. When the virtual set is split, the new directory set subscript is determined to be 1, that is, vset1. According to the preset loop algorithm, vset1 is assigned to Meta1. When the virtual set is split again, the new directory set subscript can be determined to be 2, that is, vset2. According to the preset loop algorithm, vset2 is assigned to Meta2. New virtual sets are configured on metadata nodes in sequence according to the subscripts of metadata nodes. The embodiment of the present application determines the metadata node where the split virtual set is located through the preset loop algorithm, so that the data stored on each metadata node in the distributed file system is more uniform, thereby improving the performance of metadata nodes in the distributed file system.

[0052] In an embodiment of the present application, when creating a directory item to be created in a distributed file system, the storage capacity of the virtual directory of the virtual collection in the distributed file system is judged, and when the storage capacity of the virtual directory is greater than a preset storage threshold and the number of sets of the virtual collection is less than the preset collection threshold, the virtual collection is split, and metadata nodes are allocated to the new virtual collection generated by the split in the distributed file system to store the directory item to be created in the new virtual collection. Therefore, when a new directory item needs to be inserted, the storage capacity of the virtual directory reaches the preset storage threshold and the number of virtual collections in the distributed file system does not reach the preset collection threshold, the virtual collection can be split, and metadata nodes are allocated to the new virtual collection generated by the split to migrate the directory item to be created to the new directory collection to store the new directory item in the new virtual collection, thereby avoiding constantly inserting the new directory item into the same virtual collection, so that the new directory item can be inserted into each virtual collection, thereby realizing the insertion of a new directory between each metadata node, and then realizing the storage of data on each metadata node, thereby improving the uniformity of data distribution on each metadata node.

[0053] Furthermore, when the storage capacity of the virtual directory reaches a preset storage threshold and the number of virtual sets is greater than or equal to the preset set threshold, and the number of directories in the virtual directory is less than the preset directory threshold, the virtual directories in the virtual set can be split based on the global bitmap of the virtual set, thereby expanding the storage space of the virtual set to store a large amount of data. Furthermore, when the number of directories in the virtual set is greater than or equal to the preset directory threshold, the directory expansion and directory splitting are controlled to be performed alternately, so as to achieve the alternating expansion of the size of the virtual directory and the increase in the number of virtual directories, thereby avoiding the global bitmap of the virtual set caused by the excessive number of virtual directories in the virtual set. Too large, to avoid the low efficiency of virtual directory splitting caused by the large global bitmap, which in turn leads to the low efficiency of directory item creation. Therefore, the present application splits the virtual collection to store data in each metadata node, and splits the virtual directory in the virtual collection when the number of collections of the virtual collection reaches the preset collection threshold, and then controls the directory expansion and directory splitting to be performed alternately when the number of directories of the virtual directory reaches the preset directory threshold, thereby realizing the adaptive growth of the number of virtual directories and the size of virtual directories in the distributed file system, and solving the technical problem of inefficient directory creation in the distributed file system due to the difficulty in balancing the number of directories and the size of directories.

[0054] The step of creating a directory entry to be created in the virtual directory of the virtual set of the distributed file system includes:

[0055] Step S21, determining the virtual directory where the directory item to be created is located according to the creation name of the directory item to be created and the number of sets;

[0056] It should be noted that the number of collections refers to the number of all virtual collections included in the current distributed file system. The directory entry to be created is created under a virtual directory. Before determining the virtual directory, the virtual collection in which the virtual directory is located and the metadata node in which the virtual collection is located must also be determined. Exemplarily, the virtual directory in which the directory entry to be created is located is determined based on the creation name, the number of collections, a jump consistent hashing algorithm, a preset round-robin algorithm, and a preset gradient algorithm.

[0057] In one possible embodiment, step S21 includes:

[0058] Step S211: Perform jump consistent hashing on the creation name according to the jump consistent hashing algorithm and the number of sets to obtain the original hash value of the directory entry to be created, and use the original hash value as the set subscript of the virtual set where the directory entry to be created is located;

[0059] Step S212, calculating the node subscript of the metadata node where the virtual set is located according to the set subscript, the number of metadata nodes in the distributed file system, and a preset loop algorithm;

[0060] Step S213: searching the metadata node for the virtual set that matches the set subscript according to the node subscript, so as to determine the virtual directory where the directory entry to be created is located in the virtual set based on a preset gradient algorithm.

[0061] It should be noted that the original hash value refers to the hash value calculated before the contemporary directory entry is inserted into the virtual directory. The set subscript refers to the subscript of the virtual set, and the set subscript is used to distinguish between virtual sets. Jump Consistent Hash is a minimalist and fast consistent hash algorithm released by Google in 2014. When the number of sets in the virtual set changes, the jump consistent hash algorithm will remap the number of sets. The jump consistent hash algorithm uses a uniformly distributed random function to achieve a high mapping uniformity and a better uniform distribution effect. Among them, the preset loop algorithm is the round-robin method. Through the preset loop algorithm, the virtual set is controlled to be evenly distributed among the metadata nodes. After the metadata node of the virtual set is determined, the virtual set is found in the metadata node, and the virtual directory where the directory entry to be created is located is determined based on the preset gradient algorithm. Among them, the preset gradient algorithm is the GIGA+ algorithm. Generally, the GIGA+ algorithm is used between metadata nodes, so there will be a cross-node bitmap, and the cross-node bitmap stores directories corresponding to multiple metadata nodes. In the embodiment of the present application, the GIGA+ algorithm is only used within the metadata node, and each virtual set within the metadata node corresponds to a global bitmap. Therefore, when multiple directory items are stored on the metadata node, the storage efficiency of the directory items and the query efficiency of the directory items will not be too low due to the large cross-node bitmap. The embodiment of the present application uses a global bitmap within the virtual set of the metadata node to implement directory storage and directory search within the virtual set through the global bitmap, without the need for cross-node directory search. Since the global bitmap is not a cross-node bitmap, but a global bitmap corresponding to the virtual set, when the virtual directory in the virtual set is split, the global bitmap does not need to be updated on the client, and the global bitmap corresponding to the virtual set can be directly updated, thereby improving the storage performance of the directory in the distributed file system.

[0062] As an example, steps S211 to S212 include: performing a hash conversion on the creation name based on a jump consistent hashing algorithm to obtain a hash name value of the creation name, performing a modulo operation on the hash name value with respect to the number of sets to obtain an original hash value, and the original hash value being the set index of the original virtual set where the directory entry to be created is located; calculating the node index of the metadata node where the virtual set is located based on the number of nodes of the metadata node, the set index, and a preset loop algorithm, and searching the metadata node for the virtual set that matches the set index based on the node index to determine the virtual directory where the directory entry to be created is located in the virtual set based on a preset gradient algorithm. The step of calculating the node index of the metadata node where the virtual set is located based on the number of nodes of the metadata node, the set index, and the preset loop algorithm also includes: performing a modulo operation on the set index with respect to the number of nodes to obtain the node index of the metadata node. For example, when the set index is a and the number of nodes is b, the node index of the metadata node is a%b.

[0063] The embodiment of the present application calculates the set index of the virtual set where the directory entry to be created is located through a jump consistent hash algorithm, and allocates corresponding metadata nodes to the virtual set through a preset loop algorithm, thereby achieving uniform distribution of directories in a distributed file system. Since the storage capacity on each metadata node is comparable, the performance of each metadata node of the distributed file system can be maximized.

[0064] Step S22, determining whether there is a subdirectory entry with the same name as the created one in the virtual directory;

[0065] Step S23: If yes, output an error message to prompt the user that the created name is a duplicate;

[0066] Step S24: If not, insert the directory item to be created into the virtual directory where the directory item to be created is located.

[0067] It should be noted that, when determining the virtual directory into which the directory item to be created is currently inserted, it is necessary to determine whether there is a sub-directory item with the same name in the virtual directory. The sub-directory item is the data stored in the virtual directory. When the directory item to be created is inserted into the virtual directory, the directory item to be created becomes a sub-directory item in the virtual directory. An error message is used to inform the user if the creation name of the directory item to be created is the same as the name of the directory item to be created, so that the user can modify the creation name of the directory item to be created. If there is no sub-directory item with the same name as the directory item to be created in the virtual directory, the directory item to be created will be inserted into the virtual directory.

[0068] As an example, steps S21 to S24 include: determining the virtual directory where the directory item to be created is located based on the creation name, the number of sets, the jump consistency hash algorithm, the preset loop algorithm and the preset gradient algorithm; obtaining the creation name and the number of sets of the directory item to be created, performing jump consistency hashing on the creation name based on the jump consistency hashing algorithm to obtain the original hash value of the directory item to be created; calculating the metadata node where the virtual set of the directory item to be created is located based on the original hash value, the number of nodes and the preset loop algorithm; locating the metadata node, determining the virtual set of the directory item to be created in the metadata node based on the set subscript, and determining the virtual directory where the directory item to be created is located based on the preset gradient algorithm; judging whether there is a sub-directory item with the same creation name in the virtual directory; if so, outputting an error message to prompt the user that the creation name is duplicated; if not, inserting the directory item to be created into the virtual directory where the directory item to be created is located.

[0069] Among them, the remainder of the number of sets in the distributed file system is taken according to the hash name value of the creation name of the directory item to be created to obtain the original hash value, and the original hash value is used as the set subscript of the virtual set where the directory item to be created is located. According to the preset loop algorithm, the metadata node where the set subscript is located is determined. The set subscripts corresponding to different hash name values ​​may be the same, and the directory items to be created with the same set subscript exist in the same virtual set.

[0070] Step S23 also includes: if the number of virtual directories in the virtual set is 1, the directory to be created is inserted into the number of virtual directories; if the number of virtual directories in the virtual set is greater than 1, the virtual directory into which the directory to be created is inserted is determined according to the GIGA+ algorithm (preset gradient algorithm); specifically, the virtual directory has a directory subscript, and the virtual directory into which the directory item to be created is inserted can be determined according to the directory subscript, and the directory subscript can be determined based on the GIGA+ algorithm (preset gradient algorithm).

[0071] The step of allocating metadata nodes for the new directory set in the distributed file system includes:

[0072] Step S31, determining the set index of the virtual set where the directory entry to be created is located and the node index of the metadata node where the directory entry to be created is located;

[0073] Step S32: Calculate the split subscript of the new directory set and the new node where the new directory set is located according to the set subscript, the node subscript, and the number of metadata nodes in the distributed file system according to a preset loop algorithm, wherein the new node is the metadata node where the new directory set is located;

[0074] Step S33, traversing all the virtual sets in the distributed file system, obtaining the traversal number of the virtual sets in the distributed file system, and performing hash processing on the sub-directory entries in the virtual sets according to the traversal number and a jump consistent hashing algorithm to obtain new hash values ​​of the sub-directory entries;

[0075] Step S34: taking the sub-directory entry whose new hash value is different from the original hash value as the directory entry to be migrated;

[0076] Step S35: storing the directory entries to be migrated in all the virtual sets into the cache area of ​​the distributed file system;

[0077] Step S36: Migrate the to-be-migrated directory entries in the cache area to the new directory set to perform directory storage in the new directory set.

[0078] It should be noted that after the directory item to be created is inserted into a virtual directory in the distributed file system, a virtual set split or virtual directory split is triggered only when it is detected that the storage capacity of the virtual directory exceeds a preset storage threshold. After the virtual set split or virtual directory split, the virtual set or virtual directory stored in the distributed file system of the directory item to be created is changed. In the embodiment of the present application, when triggering a virtual set split, it is necessary to determine the set index of the virtual set where the directory item to be created is located and the node index of the metadata node.

[0079] The new directory set is a new virtual set obtained after the virtual directory is split. The set subscript of the new virtual set is different from that of the virtual set before the split. The number of sets is the number of all virtual sets existing in the distributed file system. The new hash value is the hash value of the directory item to be created modulo the number of traversed sets. The number of traversed sets is the number of new virtual sets obtained after traversing all the virtual sets in the distributed file system. The new node is the metadata node where the new directory set is located. The directory item to be migrated is the subdirectory item that needs to be migrated to the new directory set. The directory item to be migrated is the subdirectory item whose original hash value is different from the new hash value. The new hash value of the directory item to be created may be the same as or different from the original hash value. Among them, when traversing all virtual sets in the distributed file system, a split request will be initiated for all virtual sets to determine the directory item to be migrated in the virtual set. The split subscript is the set subscript of the new directory set.

[0080] As an example, steps S31 to S36 include: determining the set index of the virtual set where the directory entry to be created is located and the node index of the metadata node where the directory entry to be created is located; calculating the new node of the metadata node where the new directory set is located based on the set index, the node index, the number of nodes and a preset loop algorithm; determining the split index of the new directory set in the new node, the split index being determined based on the set index of the split virtual set. Traversing all virtual sets in the distributed file system, obtaining the number of sets after traversal of the virtual set, and re-performing jump consistency hashing on the sub-directory items based on the number of sets after traversal to obtain a new hash value; using the sub-directory items whose new hash values ​​are different from the original hash values ​​as the directory entries to be migrated, storing the directory entries to be migrated in all the virtual sets in the cache area of ​​the distributed file system, and migrating the directory entries to be migrated in the cache area to the new directory set, wherein when the directory entries to be migrated are stored in the cache area, the pre-migration position of the directory entries to be migrated is stored in the distributed file system, and then after the migration of the directory entries to be migrated, the directory entries to be migrated are deleted from the original virtual set where the directory entries to be migrated are located based on the stored pre-migration position. For example, when the set index is 2, the split index can be 1 plus the set index, that is, the split index is 3. It can be considered that the order of the storage directory items of the new directory set where the split index is located is after the virtual directory where the set index is located.

[0081] The embodiment of the present application determines the split subscript of the new directory set by a preset loop algorithm, node subscript, set subscript and the number of metadata nodes, thereby generating a new virtual directory, expanding the storage capacity of data stored in the distributed file system, and by first splitting the virtual set at each metadata node when the storage capacity of the virtual directory exceeds the preset storage threshold, thereby achieving uniform distribution of the virtual set on each metadata node, and when the virtual set is split, the amount of data stored in the distributed file system is small, so when traversing all virtual sets in the distributed file system and recalculating the jump consistency hash value of all subdirectory items in the virtual set, it does not generate a large amount of calculation, and the present application ensures the uniformity of the distribution of virtual sets among each metadata node through the jump consistency hash algorithm and the preset loop algorithm. Because the virtual set is distributed among multiple nodes, the virtual set can be quickly distributed to all metadata nodes of the distributed file system at the beginning, so that all metadata nodes can be quickly used to maximize the concurrency performance of the entire system.

[0082] The step of performing directory splitting based on the global bitmap of the virtual set where the virtual directory is located includes:

[0083] Based on a global bitmap of the virtual set where the virtual directory is located, the virtual directory is split into half in the global bitmap.

[0084] It should be noted that the global bitmap is generated based on the GIGA+ algorithm. Each virtual set in the distributed file system corresponds to a global bitmap. When the number of virtual sets does not reach the preset set threshold, each virtual set contains one virtual directory, and the depth of the global bitmap for that virtual set is 1. The depth of the global bitmap represents the depth to which the virtual directories in the virtual set are split downward. When the number of virtual sets reaches the preset set threshold, the virtual directories in the virtual set begin to split. Exemplarily, the virtual directory is split in half in the global bitmap through the global bitmap in the virtual set. Specifically, the virtual directory has a corresponding hash space in the global bitmap, wherein the hash space can be (0,1] (the hash space is between 0 and 1, including 1 but not including 0). When the virtual directory is split in half in the global bitmap, the split is performed based on the hash space of the virtual directory. Exemplarily, based on the file name of any sub-directory item in the virtual directory, the file name of the sub-directory item is hashed to obtain a file hash value of the file name of the directory item. The file hash value is divided by the maximum value of the unsigned integer to obtain a space hash value of the hash space of the virtual directory. Then, the split can be performed in half based on the space hash value of the hash space. For example, when the number of bits of the file hash value is 32 bits, the maximum value of the unsigned integer is the maximum value of a 32-bit integer.

[0085] In the virtual set, the virtual directory also has a corresponding directory subscript. The directory subscript of the virtual directory where the directory item to be created is located can be determined based on a preset gradient algorithm (GIGA+). For example, the creation name of the directory item to be created is hashed to obtain the creation name, and the split depth of the global bitmap of the virtual set is obtained. According to the split depth and the hash name value of the directory item to be created, the directory subscript of the virtual directory is determined, and the virtual directory to be inserted into the directory to be created is determined according to the directory subscript. Specifically, the directory subscript of the virtual directory to be inserted into the directory to be created is obtained by taking the remainder of the split depth according to the original hash value. The split depth refers to the depth level of the global bitmap. The global bitmap is a binary tree structure, so the split depth can be the depth of the binary tree in the global bitmap. Among them, the formula for taking the remainder of the original hash value according to the split depth is:

[0086] hash(filename)mod2 x =y

[0087] Among them, "filename" is the creation name of the directory entry to be created, mod is the remainder, 2 is a binary tree, x is the split depth of the split virtual directory, hash(filename) is the hash name value of the creation name of the directory entry to be created, and y is the directory index of the virtual directory where the directory entry to be created is located.

[0088] For a better understanding of this embodiment, refer to Figure 3 , Figure 3 Schematic diagram of the global bitmap of the virtual collection. Figure 3 In the figure, P0 is the initial virtual directory, and the 0 in P0 represents the directory subscript of the initial virtual directory. When virtual directory P0 is split for the first time, a new virtual directory P1 is generated from virtual directory P0. Therefore, the first split of virtual directory P0 results in virtual directory P0 and virtual directory P1, and the split depth is 1. When another split is required, the virtual directory can be split based on P0 with a split depth of 2, generating a new virtual directory P2 from P0 with a split depth of 2, or a new virtual directory P3 from virtual directory P1, in which case the split depth of the global bitmap is 3. During the split, the directory subscript of the new virtual directory is determined by the subscript of the split virtual directory and the split depth of the split virtual directory. For example, when virtual directory P0 with a split depth of 1 is split, the directory subscript of the new virtual directory generated from virtual directory P0 is: virtual directory P0 subscript 0 + 2 ^ virtual directory P0's split depth 1, that is, the directory subscript of the new virtual directory is 2.

[0089] Example 2

[0090] Further, refer to Figure 4 Based on the above embodiment of the present application, in another embodiment of the present application, the same or similar contents as the above embodiment can be referred to the above introduction and will not be repeated hereafter. On this basis, the steps of controlling the alternating directory splitting and directory expansion include:

[0091] Step A10, doubling the directory storage capacity of the virtual directory to determine the doubled storage capacity of the virtual directory;

[0092] Step A20: After doubling the directory storage capacity, if a new directory entry is created in the virtual directory and the directory storage capacity of the virtual directory is greater than or equal to the doubled storage capacity, splitting the virtual directory in the virtual set;

[0093] Step A30: If the number of directories in the virtual directory of the virtual set is greater than or equal to the preset balanced directory number value, return to the execution step of doubling the directory storage capacity of the virtual directory to determine the doubled storage capacity of the virtual directory.

[0094] It should be noted that when the number of virtual sets in the distributed file system reaches the preset set threshold, the virtual directories in the virtual set will be split, and when the number of directories in the virtual directory in the virtual set reaches the preset directory threshold, the storage capacity of the virtual directory will begin to expand. Since the insertion of the directory item to be created in the distributed file system in the embodiment of the present application is determined based on the jump consistency algorithm and the preset loop algorithm, when the number of directories in the virtual directory of a certain virtual set reaches the preset directory threshold, it can be considered that the number of directories in the virtual directory of other virtual sets in the distributed file system has also reached the preset directory threshold. The directory storage capacity refers to the current storage capacity of the virtual directory, and the doubled storage capacity is obtained by doubling the directory storage capacity of the virtual directory; the preset balanced directory quantity value can be obtained by doubling the number of directories.

[0095] As an example, steps A10 to A30 include: doubling the directory storage capacity of the virtual directory to determine the doubled storage capacity of the virtual directory; when the amount of data stored in the virtual directory reaches the doubled storage capacity, splitting the virtual directory in the virtual set, and when the number of directories of the virtual directory in the virtual set is greater than or equal to the preset balance directory number value, returning to the execution step: doubling the directory storage capacity of the virtual directory to determine the doubled storage capacity of the virtual directory.

[0096] The embodiment of the present application realizes the adaptive growth of the virtual set in the distributed file system by controlling the alternating expansion between the number of virtual directories and the storage capacity of the virtual directory in the virtual set. When the number of directories in the virtual directory in the virtual set grows to a preset directory threshold, the virtual directory is stopped from being split, and the size of the virtual directory is then increased. In this way, while ensuring the storage of massive data, the global bitmap caused by the excessive number of directories in the virtual directory is avoided from being too large, thereby affecting the search efficiency of sub-directory items in the virtual set. Therefore, the storage capacity of the virtual directory is expanded when the number of directories reaches the preset directory threshold. However, the storage capacity of the virtual directory cannot grow indefinitely. Therefore, when the storage capacity of the virtual directory reaches double the storage capacity, the virtual directory is split again to increase the number of directories in the virtual directory again, thereby realizing the adaptive growth of the virtual directory, and maximizing the performance of the metadata node in the distributed file system.

[0097] Example 3

[0098] Further, refer to Figure 5 Based on the above embodiment of the present application, in another embodiment of the present application, the same or similar contents as the above embodiment can be referred to the above introduction and will not be repeated hereafter. On this basis, after the step of splitting the virtual set in the distributed file system, the directory distribution method further includes:

[0099] Step B10: If a virtual set is detected to be split, the preset large directory corresponding to the virtual set is locked;

[0100] In step B20, if it is detected that the virtual directory in the virtual set is being split, the virtual set is locked when the global bitmap of the virtual set is updated;

[0101] Step B30: If it is detected that the directory item to be migrated is being migrated, the virtual directory where the directory item to be migrated is located before the migration and the virtual directory where the directory item to be migrated is located after the migration are locked;

[0102] Step B40: If it is detected that a sub-directory item in the virtual directory is updated, the updated sub-directory item is locked.

[0103] It should be noted that when performing operations on metadata nodes in a distributed file system, the virtual collections, virtual directories, or sub-directory items on the metadata nodes are locked, thereby enabling concurrent data operations. Specifically, a distributed file system can be divided into preset large directories, virtual collections, virtual directories, and sub-directory items. The preset large directory includes each virtual collection, so in the embodiments of the present application, virtual collections, virtual directories, and sub-directory items can be locked. Exemplarily, when it is detected that a virtual directory in a virtual set is splitting, the virtual set is locked when the global bitmap of the virtual set is updated so that the virtual set is operated on by acquiring the lock of the virtual set on one thread. When it is detected that a directory item to be migrated is migrating between virtual directories, the virtual directory before the migration of the directory item to be migrated and the virtual directory after the migration of the directory item to be migrated are locked so that the virtual directory is operated on by acquiring the lock of the virtual directory on one thread. If it is detected that a sub-directory item in the virtual directory is being updated, the updated sub-directory item is locked so that the sub-directory item is operated on by acquiring the lock of the sub-directory item on one thread. If it is detected that the virtual set is splitting, all metadata nodes of the distributed file system are locked. Wherein, the sub-directory item can be a file, and the sub-directory item update can be to delete the file or update the file content. The sub-directory item can also be understood as creating a sub-directory item in the distributed file system. Exemplarily, when it is detected that the virtual set is splitting, a preset large directory for directory storage in the virtual set can be determined and the preset large directory can be locked.

[0104] The embodiment of the present application can lock virtual collections, virtual directories, and sub-directory items, that is, the present embodiment implements the setting of locks at multiple levels, thereby enabling concurrent control at multiple levels. The embodiment of the present application not only locks the directory composed of the entire virtual collection in the distributed file system, but is refined to lock the virtual collection, lock the virtual directory, and lock the sub-directory items, thereby refining the locking granularity, so that the sub-directory items can also be processed concurrently, thereby improving the concurrent processing performance of metadata and being able to cope with high-concurrency metadata requests.

[0105] Wherein, after the step of starting a directory creation process in the distributed file system in response to the directory entry creation instruction, the directory distribution method further includes:

[0106] Step C10, in response to the directory entry search instruction, obtaining the directory name of the directory entry to be searched;

[0107] Step C20, performing hash processing on the directory name according to a jump consistent hashing algorithm and the number of sets in the distributed file system to obtain a set index of the virtual set where the directory entry to be searched is located;

[0108] Step C30, determining the metadata node of the virtual set according to the set index;

[0109] Step C40, searching the metadata node for a virtual set that matches the set index to obtain a global bitmap of the virtual set;

[0110] Step C50: Match the directory name in the global bitmap to find the directory entry to be found.

[0111] It should be noted that the directory item search instruction may be triggered by the user, the directory item to be searched is guaranteed to be the sub-directory item that the user wants to search, and the directory name is the file name of the directory item to be searched. Exemplarily, the directory name is hashed by a jump consistency hashing algorithm to obtain a hash name value of the directory item to be searched. According to the number of sets in the distributed file system, the set index of the virtual set where the name to be searched is located is obtained by taking the modulus of the number of sets according to the hash name value. Then, according to the set index and the number of nodes, the node index of the metadata node where the virtual set is located is obtained by a preset loop algorithm. Then, according to the node index, the metadata node is located. The virtual set matching the set index is searched in the metadata node. Then, in the virtual set, according to the hash name value of the directory item to be searched and the split depth of the global bitmap of the virtual set, the directory index of the virtual directory where the directory item to be searched is located is determined according to a preset gradient algorithm. Then, according to the directory index, the virtual directory is located. The sub-directory item matching the directory name is searched in the virtual directory. If there is a sub-directory item matching the directory name in the virtual directory, it is determined that the search for the directory item to be searched is successful. If there is no sub-directory item matching the directory name in the virtual directory, it is determined that the search for the directory item to be searched fails. Specifically, the step of obtaining the node subscript of the metadata node where the virtual set is located through a preset loop algorithm based on the set subscript and the number of nodes includes: obtaining the node subscript of the metadata node by taking the modulus of the set subscript and the number of nodes.

[0112] The embodiment of the present application determines the metadata node where the directory entry is located by using a jump consistent hash algorithm between metadata nodes, and then determines a virtual set within the metadata node to search for the corresponding directory entry in the global bitmap of the virtual set, thereby improving the search efficiency of the directory entry. Since the present application uses the GIGA+ algorithm to create a global bitmap within the metadata node, and does not use the GIGA+ algorithm between metadata nodes, there will be no cross-node bitmap (the bitmap of the directory entry stored by each metadata node) in the present application, and thus it will not be difficult to find the corresponding directory entry in the distributed file system due to the excessive size of the cross-node bitmap. Nor will it be difficult to find the corresponding directory entry in the distributed file system due to the failure of the cross-node bitmap stored by the client due to the frequent updates of the cross-node bitmap. In the embodiment of the present application, the global bitmap includes the directory entries stored in the virtual set. Therefore, when searching for directory entries based on the global bitmap, there will not be a large amount of data compared to the cross-node bitmap, so the query efficiency of the directory entry can be improved.

[0113] Further, for a better understanding of this application, refer to Figure 6 , Figure 6 This is a schematic diagram of a distributed file system. Figure 6In the distributed file system, there are three metadata nodes Meta0, Meta1 and Meta2. Each metadata node includes two virtual sets. Figure 6 The maximum number of virtual collections in the distributed file system is 6. Figure 6 The vset in the table is a virtual set, and the vdir is a virtual directory. The value after vset is the set subscript of the virtual set, and the value after vdir is the directory subscript of the virtual directory. The virtual directory in the virtual set vset0 is determined by the GIGA+ algorithm. a, b, and c can be considered as the subdirectory items corresponding to the virtual directory vdir2. GIGA+ means that the GIGA+ algorithm is used to split the virtual directory in the virtual set.

[0114] Example 4

[0115] Reference Figure 7 , an embodiment of the present application further provides a directory creation device, the directory creation device comprising:

[0116] A response module 10 is configured to initiate a directory creation process in the distributed file system in response to a directory entry creation instruction;

[0117] A first judgment module 20 is used in the directory creation process as follows: inserting a directory entry to be created into a virtual directory of a virtual set of the distributed file system based on a jump consistent hashing algorithm, and if the storage capacity of the virtual directory is greater than a preset storage threshold, determining whether the number of sets of the virtual set in the distributed file system is less than a preset set threshold;

[0118] a set splitting module 30 for, if yes, splitting a new directory set from the virtual set, and allocating metadata nodes to the new directory set for directory storage in the new directory set;

[0119] The second judgment module 40 is used to judge whether the number of directories in the virtual directory is less than a preset directory threshold if no;

[0120] a directory splitting module 50, configured to perform directory splitting based on a global bitmap of the virtual set where the virtual directory is located if the number of directories is less than the preset directory threshold, wherein the directory splitting is to split the virtual directory;

[0121] The alternating splitting module 60 is configured to control directory expansion and directory splitting to be performed alternately if the number of directories is greater than or equal to the preset directory threshold, wherein the directory expansion is to expand the storage capacity of the virtual directory.

[0122] Optionally, the first determining module 20 is further configured to:

[0123] Determine the virtual directory where the directory item to be created is located according to the creation name of the directory item to be created and the number of sets;

[0124] Determine whether there is a subdirectory entry with the same name as the created one in the virtual directory;

[0125] If so, an error message is output to prompt the user that the created name is duplicated;

[0126] If not, insert the directory item to be created into the virtual directory where the directory item to be created is located.

[0127] Optionally, the first determining module 20 is further configured to:

[0128] Performing jump consistent hashing on the creation name according to the jump consistent hashing algorithm and the number of sets to obtain an original hash value of the directory entry to be created, and using the original hash value as the set subscript of the virtual set where the directory entry to be created is located;

[0129] Calculate the node subscript of the metadata node where the virtual set is located according to the set subscript, the number of metadata nodes in the distributed file system, and a preset loop algorithm;

[0130] The virtual set matching the set subscript is searched in the metadata node according to the node subscript, so as to determine the virtual directory where the directory entry to be created is located in the virtual set based on a preset gradient algorithm.

[0131] Optionally, the set splitting module 30 is further configured to:

[0132] Determine the set index of the virtual set where the directory entry to be created is located and the node index of the metadata node where the directory entry to be created is located;

[0133] Calculate, according to the set index, the node index, and the number of metadata nodes in the distributed file system, the split subscript of the new directory set and the new node where the new directory set is located according to a preset loop algorithm, wherein the new node is the metadata node where the new directory set is located;

[0134] Traversing all the virtual sets in the distributed file system to obtain a traversal number of the virtual sets in the distributed file system, and performing hash processing on sub-directory entries in the virtual sets according to the traversal number and a jump consistent hashing algorithm to obtain a new hash value of the sub-directory entry;

[0135] The sub-directory entry with a new hash value different from the original hash value is used as the directory entry to be migrated;

[0136] Storing the directory entries to be migrated in all the virtual sets in a cache area of ​​the distributed file system;

[0137] Based on the new node, the to-be-migrated directory entries in the cache area are migrated to the new directory set to perform directory storage in the new directory set.

[0138] Optionally, the directory splitting module 50 is further configured to:

[0139] Based on a global bitmap of the virtual set where the virtual directory is located, the virtual directory is split into half in the global bitmap.

[0140] Optionally, the alternating splitting module 60 is further configured to:

[0141] doubling the directory storage capacity of the virtual directory to determine the doubled storage capacity of the virtual directory;

[0142] After doubling the directory storage capacity, if a new directory entry is created in the virtual directory and the directory storage capacity of the virtual directory is greater than or equal to the doubled storage capacity, splitting the virtual directory in the virtual set;

[0143] If the directory quantity of the virtual directory in the virtual set is greater than or equal to the preset balanced directory quantity value, return to the execution step: doubling the directory storage capacity of the virtual directory to determine the doubled storage capacity of the virtual directory.

[0144] Optionally, the alternating splitting module 60 is further configured to:

[0145] If a virtual set is detected to be split, the preset large directory corresponding to the virtual set is locked;

[0146] If a virtual directory in a virtual set is detected to be split, the virtual set is locked when the global bitmap of the virtual set is updated;

[0147] If it is detected that the directory item to be migrated is being migrated, the virtual directory where the directory item to be migrated is located before the migration and the virtual directory where the directory item to be migrated is located after the migration are locked;

[0148] If it is detected that a sub-directory item in the virtual directory is updated, the updated sub-directory item is locked.

[0149] Optionally, the response module 10 is further configured to:

[0150] In response to a directory entry search instruction, obtaining a directory name of a directory entry to be searched;

[0151] Performing hash processing on the directory name according to a jump consistent hashing algorithm and the number of sets in the distributed file system to obtain a set index of the virtual set where the directory entry to be found is located;

[0152] Determine the metadata node of the virtual set according to the set subscript;

[0153] Searching the metadata node for a virtual set that matches the set index to obtain a global bitmap of the virtual set;

[0154] The directory name is matched in the global bitmap to find the directory entry to be found.

[0155] The directory creation device provided in this application utilizes the directory distribution method described in the aforementioned embodiments, aiming to address the technical issue of inefficient directory creation in distributed file systems. Compared to the prior art, the directory distribution method provided in this embodiment achieves the same beneficial effects as those provided in the aforementioned embodiments. Other technical features of this directory creation device are the same as those disclosed in the aforementioned embodiments and are not further detailed here.

[0156] Example 5

[0157] An embodiment of the present application provides an electronic device, which may be a playback device, and the electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the directory distribution method in the above embodiment.

[0158] Reference below Figure 8 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (portable Android devices), PMPs (portable media players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0159] like Figure 8As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a ROM (Read-Only Memory) 1002 or a program loaded from a storage device 1003 into a RAM (Random Access Memory) 1004. Various programs and data required for the operation of the electronic device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus.

[0160] Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, tachometer, gyroscope, etc.; output devices 1008 including, for example, an LCD (Liquid Crystal Display), speaker, vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, hard disk, etc.; and communication devices 1009. The communication devices can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented or have instead.

[0161] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication system, or installed from a storage system, or installed from a ROM. When the computer program is executed by a processing system, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0162] The electronic device provided in this application utilizes the directory distribution method described in the first embodiment above to address the technical issue of inefficient directory creation in distributed file systems. Compared to the prior art, the beneficial effects of product flow data distribution provided in this embodiment are the same as those of the directory distribution method described in the first embodiment above. The other technical features of this directory creation device are the same as those disclosed in the first embodiment above and are not further elaborated here.

[0163] It should be understood that various parts of the present disclosure can be implemented with hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in an appropriate manner.

[0164] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0165] Example 6

[0166] This embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, and the computer-readable program instructions are used to execute the directory distribution method in the above-mentioned embodiment 1.

[0167] The computer-readable storage medium provided in the embodiment of the present application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, equipment or devices, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable EPROM (Electrical Programmable Read Only Memory, read-only memory) or flash memory, an optical fiber, a portable compact disk CD-ROM (compact discread-only memory, read-only memory), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution device, device or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency, radio frequency) and the like, or any suitable combination thereof.

[0168] The computer-readable storage medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0169] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device: responds to a directory entry creation instruction and starts a directory creation process in a distributed file system; wherein, the directory creation process is: based on a jump consistency hash algorithm, inserting a directory entry to be created in a virtual directory of a virtual set of the distributed file system; if the storage capacity of the virtual directory is greater than a preset storage threshold, then determining whether the number of sets of virtual sets in the distributed file system is less than a preset set threshold; if so, splitting a new directory set from the virtual set, and allocating metadata nodes to the new directory set for directory storage in the new directory set; if not, determining whether the number of directories of the virtual directory is less than a preset directory threshold; if the number of directories is less than the preset directory threshold, performing directory splitting based on the global bitmap of the virtual set where the virtual directory is located, wherein directory splitting is splitting the virtual directory; if the number of directories is greater than or equal to the preset directory threshold, then controlling directory extension and directory splitting to be performed alternately, wherein directory extension is expanding the storage capacity of the virtual directory.

[0170] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a LAN (local area network) or WAN (wide area network), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0171] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based device that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0172] The modules involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0173] The computer-readable storage medium provided in this application stores computer-readable program instructions for executing the aforementioned directory distribution method, aiming to address the technical issue of inefficient directory creation in distributed file systems. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this embodiment of the application are similar to those of the directory distribution method provided in the aforementioned embodiment, and are not further elaborated here.

[0174] Example 7

[0175] The present application also provides a computer program product, comprising a computer program, which implements the steps of the above-mentioned directory distribution method when executed by a processor.

[0176] The computer program product provided in this application is intended to address the technical problem of inefficient directory creation in a distributed file system. Compared to the prior art, the beneficial effects of the computer program product provided in the embodiments of this application are the same as those of the directory distribution method provided in the above embodiments, and will not be elaborated on here.

[0177] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A directory distribution method, characterized in that: Applied to a distributed file system, the directory distribution method includes: In response to a directory entry creation instruction, starting a directory creation process in the distributed file system; The directory creation process is as follows: inserting a directory entry to be created into a virtual directory of a virtual set of the distributed file system based on a jump consistent hashing algorithm; if the storage capacity of the virtual directory is greater than a preset storage threshold, determining whether the number of sets of the virtual set in the distributed file system is less than a preset set threshold; If so, splitting a new directory set from the virtual set, and allocating metadata nodes to the new directory set for directory storage in the new directory set; If not, determining whether the number of directories in the virtual directory is less than a preset directory threshold; If the number of directories is less than the preset directory threshold, performing directory splitting based on the global bitmap of the virtual set where the virtual directory is located, wherein the directory splitting is splitting the virtual directory; If the number of directories is greater than or equal to the preset directory threshold, controlling directory expansion and directory splitting to be performed alternately, wherein the directory expansion is to expand the storage capacity of the virtual directory; The step of inserting a directory entry to be created into a virtual directory of a virtual set of the distributed file system comprises: Determine the virtual directory where the directory item to be created is located according to the creation name of the directory item to be created and the number of sets; Determine whether there is a subdirectory entry with the same name as the created one in the virtual directory; If so, an error message is output to prompt the user that the created name is duplicated; If not, inserting the directory item to be created into the virtual directory where the directory item to be created is located; The step of determining the virtual directory where the directory item to be created is located according to the creation name of the directory item to be created and the number of sets includes: Performing jump consistent hashing on the creation name according to the jump consistent hashing algorithm and the number of sets to obtain an original hash value of the directory entry to be created, and using the original hash value as the set subscript of the virtual set where the directory entry to be created is located; Calculate the node subscript of the metadata node where the virtual set is located according to the set subscript, the number of metadata nodes in the distributed file system, and a preset loop algorithm; The virtual set matching the set subscript is searched in the metadata node according to the node subscript, so as to determine the virtual directory where the directory entry to be created is located in the virtual set based on a preset gradient algorithm.

2. The directory distribution method according to claim 1, wherein: The step of splitting a new directory set from the virtual set and allocating metadata nodes to the new directory set for directory storage in the new directory set includes: Determine the set index of the virtual set where the directory entry to be created is located and the node index of the metadata node where the directory entry to be created is located; Calculate, according to the set index, the node index, and the number of metadata nodes in the distributed file system, the split subscript of the new directory set and the new node where the new directory set is located according to a preset loop algorithm, wherein the new node is the metadata node where the new directory set is located; Traversing all the virtual sets in the distributed file system to obtain a traversal number of the virtual sets in the distributed file system, and performing hash processing on sub-directory entries in the virtual sets according to the traversal number and a jump consistent hashing algorithm to obtain a new hash value of the sub-directory entry; The sub-directory entry with a new hash value different from the original hash value is used as the directory entry to be migrated; Storing the directory entries to be migrated in all the virtual sets in a cache area of ​​the distributed file system; Based on the new node, the to-be-migrated directory entries in the cache area are migrated to the new directory set to perform directory storage in the new directory set.

3. The directory distribution method according to claim 1, wherein: The step of performing directory splitting based on the global bitmap of the virtual set where the virtual directory is located comprises: Based on a global bitmap of the virtual set where the virtual directory is located, the virtual directory is split into half in the global bitmap.

4. The directory distribution method according to claim 1, wherein: The steps of controlling directory expansion and directory splitting alternately include: doubling the directory storage capacity of the virtual directory to determine the doubled storage capacity of the virtual directory; After doubling the directory storage capacity, if a new directory entry is created in the virtual directory and the directory storage capacity of the virtual directory is greater than or equal to the doubled storage capacity, splitting the virtual directory in the virtual set; If the directory quantity of the virtual directory in the virtual set is greater than or equal to the preset balanced directory quantity value, return to the execution step: doubling the directory storage capacity of the virtual directory to determine the doubled storage capacity of the virtual directory.

5. The directory distribution method according to any one of claims 1 to 4, characterized in that: After the step of splitting the virtual set in the distributed file system, the directory distribution method further includes: If a virtual set is detected to be split, the preset large directory corresponding to the virtual set is locked; If a virtual directory in a virtual set is detected to be split, the virtual set is locked when the global bitmap of the virtual set is updated; If it is detected that the directory item to be migrated is being migrated, the virtual directory where the directory item to be migrated is located before the migration and the virtual directory where the directory item to be migrated is located after the migration are locked; If it is detected that a sub-directory item in the virtual directory is updated, the updated sub-directory item is locked.

6. The directory distribution method according to claim 1, wherein: After the step of starting a directory creation process in the distributed file system in response to the directory entry creation instruction, the directory distribution method further includes: In response to a directory entry search instruction, obtaining a directory name of a directory entry to be searched; Performing hash processing on the directory name according to a jump consistent hashing algorithm and the number of sets in the distributed file system to obtain a set index of the virtual set where the directory entry to be found is located; Determine the metadata node of the virtual set according to the set subscript; Searching the metadata node for a virtual set that matches the set index to obtain a global bitmap of the virtual set; The directory name is matched in the global bitmap to find the directory entry to be found.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the directory distribution method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program for implementing the directory distribution method, and the program for implementing the directory distribution method is executed by a processor to implement the steps of the directory distribution method according to any one of claims 1 to 6.

Images