Metadata index system for distributed file system, construction method and metadata access method

Abstract

The invention discloses a metadata indexing system for a distributed file system, a metadata indexing construction method and a metadata access method, and belongs to the technical field of data storage. The metadata indexing system comprises a metadata indexing structure used for indexing storage paths of all storage files in the distributed file system, and specifically comprises a root node, an intermediate node and a leaf node; the node type of the root node is a common model index node; the node type of the intermediate node comprises a common model index node and a small directory cluster compaction node; the node type of the leaf node is a directory entry node; the node types of the next layer of nodes pointed by the table entries in the common model index nodes comprise common model index nodes, small directory cluster compaction nodes and directory entry nodes; the node type of the next layer node pointed by the table item in the small directory cluster compaction node is a directory entry node; by means of the design, the technical problem that in the prior art, delay is high when files of a large scale are oriented can be solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data storage, and more specifically, relates to a metadata indexing system, a construction method, and a metadata access method for a distributed file system. Background Art

[0002] In recent years, large model and container technologies have been continuously developed for edge intelligent computing scenarios. Model training and container image files are small and numerous, reaching hundreds of millions or even tens of billions of files. Distributed file systems, as basic software for highly scalable data storage and management, have become a key technology for solving the distributed storage of massive data at the edge. Metadata services, as one of the core components of distributed file systems, are responsible for managing and providing file metadata to ensure efficient operation and reliability of the system. Massive small files place higher demands on metadata management. First, high scalability is required. The storage overhead and high throughput of metadata for massive small files require metadata services to be highly scalable. Second, low latency is required. Due to metadata request bottlenecks, existing systems cannot fully meet the low latency requirements of scenarios such as large model training and container startup acceleration.

[0003] Traditional metadata indexing systems for distributed file systems create corresponding inodes for each directory and file in their directory index tree, storing metadata for that directory or file. Each directory's inode stores inode index information for all subdirectories and subfiles within it. Ultimately, a tree-like directory index tree is constructed, with the root directory as the root node, other directories as intermediate nodes, and files as leaf nodes. After receiving a metadata request, a path resolution process must be performed. Starting from the root directory, the inodes of the directories along the path are resolved layer by layer according to the file path. The inodes of the next-level directories or files are then retrieved, ultimately obtaining the file or directory pointed to by the path pointer before the corresponding metadata processing operations can be performed. When dealing with massive amounts of files, the large and deep nature of the directory tree exacerbates path resolution latency, leading to high latency in metadata operations. Summary of the Invention

[0004] In response to the above-mentioned defects or improvement needs of the existing technology, the present invention provides a metadata indexing system, construction method and metadata access method for a distributed file system, so as to solve the technical problem of high latency in the existing technology when dealing with massive files.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a metadata indexing system for a distributed file system, comprising: a metadata indexing structure for indexing storage paths of various stored files in the distributed file system; the metadata indexing structure is a tree structure, comprising: a root node, intermediate nodes, and leaf nodes;

[0006] The node type of the root node is a common model index node;

[0007] The node types of intermediate nodes include: common model index nodes and small directory cluster compaction nodes;

[0008] The node type of a leaf node is a directory entry node;

[0009] The information stored in the common model index node includes: a first table header and a first table item array; the information stored in the first table header includes: a corresponding file path prefix and a local index model; the local index model is used to establish a mapping relationship between the path segments of each storage file in the distributed file system and the table item position pointing to the corresponding storage file in the first table item array stored at the node where the local index model is located; the path segment is a path prefix of a preset length intercepted from the sub-path after eliminating the file path prefix stored at the node where the corresponding local index model is located from the file path of the storage file; the information stored in each table item in the first table item array includes: the table item type, the next level node to which the table item refers, and the next level node to which the table item refers. The length of the file path prefix in the point and the pointer to the next layer node pointed to by the table entry; the table entry types include: empty table entry, ordinary model index node table entry, small directory cluster compaction node table entry and file directory entry table entry; among them, the node information pointed to by the pointer in the empty table entry is invalid; the pointer in the ordinary model index node table entry points to the node address of the ordinary model index node; the pointer in the small directory cluster compaction node table entry points to the node address of the small directory cluster compaction node; the pointer in the file directory entry table entry points to the storage address of the file directory entry metadata; the node types of the next layer node pointed to by the table entry in the ordinary model index node include ordinary model index node, small directory cluster compaction node and directory entry node;

[0010] The small directory cluster compaction node includes: a second header and a second entry array; information stored in the second header includes: a corresponding file path prefix and a total number of entries in the second entry array; information stored in each entry in the second entry array includes: a path suffix corresponding to the entry and a pointer to a next-level node pointed to by the entry; the node type of the next-level node pointed to by the entry in the small directory cluster compaction node is a directory entry node;

[0011] The information stored in the directory entry node includes: metadata and path suffix of the corresponding file.

[0012] Further preferably, the information stored in each entry in the second entry array further includes: a hash value corresponding to the path suffix corresponding to the entry.

[0013] In a second aspect, the present invention provides a method for constructing a metadata index structure of a distributed file system, comprising: upon receiving an insert request for a file to be processed, performing the following operations:

[0014] B100, create a target node of the type of directory entry node for the file to be processed; the information stored in the directory entry node includes: metadata and path suffix of the corresponding file; the metadata in the target node is the metadata of the file to be processed, and the path suffix is ​​the path of the file to be processed; target node; when the root node of the metadata index structure in the distributed file system is empty, create a node node0 of the type of ordinary model index node in the metadata index system as the root node of the metadata index structure; the information stored in the ordinary model index node includes: the corresponding file path prefix P c , local index model M and fixed-length first table array A1; the information stored in the table in A1 includes: table type, length L of the file path prefix in the next layer node pointed to by the table np And the pointer to the next layer node pointed to by the table entry P in node0 c Empty; M in node0 is used to reflect the correspondence between the path prefix of the preset length intercepted from the file path of the file to be processed and the position of the entry pointing to the target node in the first entry array of node0; A1 in node0 stores the entry T; the entry type of T is the file directory entry; L of T np is the file path length of the file to be processed; T Point to the target node; otherwise, use the root node of the metadata index structure as the current node;

[0015] B200. Get the file path prefix P stored in the current node c and eliminate it from the file path of the file to be processed to obtain the current subpath; determine the node type of the current node. If the current node is a common model index node, execute B300; if the current node is a small directory cluster compaction node, execute B400-B600;

[0016] B300: Extract a path prefix of a preset length from the current subpath and input it into the local index model M stored in the current node. Obtain the position of the entry in the first entry array of the current node that points to the file to be processed, and use the entry at that position as the current target entry.

[0017] When the entry type of the current target entry is an empty entry, the entry type of the current target entry is updated to a file directory entry, and the length of the current subpath is used as L np , the address of the target node is Insert the data into the current target table entry, thereby constructing a metadata index structure that responds to the current insert request, and the operation ends;

[0018] When the entry type of the current target entry is a common model index node entry or a small directory cluster compaction node entry, the next layer node pointed to by the current target entry is obtained and used as the new current node, and the process goes to B200;

[0019] When the entry type of the current target entry is a file directory entry, it is determined whether the file path stored in the directory entry node pointed to by the current target entry is consistent with the current sub-path. If so, it returns a message indicating that the metadata index structure creation in response to the current insert request has failed, and the creation operation is terminated. Otherwise, a node node1 of the type small directory cluster compaction node is created in the metadata index system. The information stored in the small directory cluster compaction node includes: the corresponding file path prefix P c , the second entry array A2 and the total number of entries N2 in A2; the information stored in the entry in A2 includes: the path suffix S corresponding to the entry s And the pointer to the next layer node pointed to by the table entry Node1 stores table entries P and Q; N2 in node1 is 2; P in node1 c P in the node where the current target entry is located c The common prefix with the current subpath; With the current target table entry Same; S in entry P s Eliminate the P in node1 from the path suffix stored in the directory entry node pointed to by the current target table entry. c The remaining part after the entry Q Points to the target node; S in table entry Q s Eliminate P in node1 for the current subpath c The remaining part after the current target table entry; update the type of the current target table entry to the small directory cluster compaction node table entry and point it to node1, thereby constructing a metadata index structure that responds to the current insert request, and the operation ends;

[0020] B400, querying the current subpath in the second entry array of the current node. If the current subpath exists, returning a message indicating that the metadata index structure creation in response to the current insert request has failed, and ending the creation operation. If the current subpath does not exist, going to B500;

[0021] B500, inserting the entry R pointing to the target node into the second entry array of the current node, and updating the total number of entries N2 stored in the current node; the path suffix S stored in the entry R s is the current subpath; determine whether the total number of entries of the current node exceeds the preset threshold. If so, create a node node2 of the type of ordinary model index node in the metadata index system; P in node2 cThe current subpath; A1 in node2 stores the entries in the second entry array of the current node; the local index model M in node2 is used to reflect the correspondence between the path prefix of the preset length intercepted from the current subpath and the entry position pointing to the target node in the first entry array of node2; delete the current node, store node2 at the original position of the current node, and go to B600; otherwise, go directly to B600;

[0022] B600. Use the current metadata index structure as the metadata index structure responding to the current insert request. The operation ends.

[0023] Further preferably, the corresponding path suffix S is also stored in the above table item P and table item Q. s The hash value of

[0024] The above querying the current subpath in the second entry array of the current node includes:

[0025] S1. Check whether the hash value corresponding to the current subpath exists in the second entry array of the current node. If so, take the entry in the second entry array containing the hash value corresponding to the current subpath as a candidate entry and go to S2; otherwise, go to S3.

[0026] S2, determine whether there is a stored S s The candidate table entry is the same as the current subpath. If so, determine whether the current subpath exists in the second table entry array of the current node; otherwise, go to S3;

[0027] S3. Determine whether the current subpath does not exist in the second entry array of the current node.

[0028] Further preferably, the above B100 further includes: after creating a common model index node node2 in the metadata index system, using a path prefix of a preset length intercepted from the current subpath as an input sample and the position of the entry pointing to the target node in the first entry array of node2 as the corresponding label, and performing incremental training on the local index model in node2;

[0029] The B600 also includes: after creating a normal model index node node0 in the metadata index system, the path prefix of a preset length intercepted from the current sub-path is used as an input sample, and the table entry position pointing to the target node in the first table entry array of node0 is used as the corresponding label to perform incremental training on the local index model in node0.

[0030] In a third aspect, the present invention provides a metadata access method, which is applied to the metadata index structure constructed by the construction method provided in the second aspect of the present invention; the metadata access method includes: when receiving a file query request, performing a query operation on the file to be processed, including:

[0031] A100, determining whether the root node of the metadata index structure is empty, if so, proceeding to A600; otherwise, taking the root node of the metadata index structure as the current node;

[0032] A200, obtaining the file path prefix stored in the current node, and eliminating it from the file path of the file to be processed to obtain the current subpath; determining the node type of the current node, if the node type of the current node is a common model index node, proceeding to A300; if the node type of the current node is a small directory cluster compaction node, proceeding to A400;

[0033] A300, intercepting a path prefix of a preset length from the current subpath, and inputting it into the local index model stored in the current node, obtaining the table entry position pointing to the file to be processed in the first table entry array of the current node, and using the table entry under this position as the current target table entry; when the table entry type of the current target table entry is an empty table entry, go to A600; when the table entry type of the current target table entry is a common model index node table entry or a small directory cluster compaction node table entry, obtain the next layer node pointed to by the current target table entry, and use it as the new current node, and go to A200; when the table entry type of the current target table entry is a file directory item, determine whether the length of the file path prefix in the next layer node pointed to by the current target table entry is consistent with the length of the current subpath, if consistent, go to A700; otherwise, go to A600;

[0034] A400, query the current subpath in the second entry array of the current node. If the current subpath exists, go to A600; if the current subpath does not exist, go to A500;

[0035] A500 determines that the query has failed, returns a message indicating that no files to be processed were found, and ends the query;

[0036] A600: Determine that the query is successful, return the pointer of the next layer node pointed to by the current target table entry, and end the query.

[0037] Further preferably, when the hash value of the corresponding path suffix is ​​further stored in the entry of the second entry array, querying the current subpath in the second entry array of the current node includes:

[0038] S1. Check whether the hash value corresponding to the current subpath exists in the second entry array of the current node. If so, take the entry in the second entry array containing the hash value corresponding to the current subpath as a candidate entry and go to S2; otherwise, go to S3.

[0039] S2, determine whether there is a stored S s The candidate table entry is the same as the current subpath. If so, determine whether the current subpath exists in the second table entry array of the current node; otherwise, go to S3;

[0040] S3. Determine whether the current subpath does not exist in the second entry array of the current node.

[0041] In a fourth aspect, the present invention provides an electronic device comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the method provided in the second aspect or the third aspect of the present invention when executing the computer program.

[0042] In a fifth aspect, the present invention provides a distributed file system, comprising:

[0043] An index structure construction module, configured to execute the construction method provided by the second aspect of the present invention;

[0044] The metadata access module is used to execute the metadata access method provided by the third aspect of the present invention.

[0045] In a sixth aspect, the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the method provided in the second aspect or the third aspect of the present invention.

[0046] In a seventh aspect, the invention further provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the method provided in the second or third aspect of the invention.

[0047] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0048] 1. The first aspect of the present invention provides a metadata indexing system for a distributed file system, comprising a metadata indexing structure for indexing the storage path of each storage file in the distributed file system; the metadata indexing structure comprises: a root node, an intermediate node and a leaf node; the node type of the root node is a common model index node; the node types of the intermediate nodes include: a common model index node and a small directory cluster compaction node; the node type of the leaf node is a directory entry node; the node types of the next layer node pointed to by the table entry in the common model index node include: a common model index node, a small directory cluster compaction node and a directory entry node; the node types of the next layer node pointed to by the table entry in the small directory cluster compaction node include: a common model index node, a small directory cluster compaction node and a directory entry node; The node type of the next layer node is a directory entry node; by designing corresponding node types for the root node, intermediate node and leaf node in the metadata index structure to store different key information, a better hierarchical design is achieved, which is flatter and provides a basis for metadata operations; on this basis, a local index model is stored in the ordinary index model node, which can index the directory entry nodes of the multi-layer directory within the node, effectively shortening the index path length in the process of indexing the target file; based on this, the present invention avoids the layer-by-layer directory parsing in the traditional path parsing process, and can solve the technical problem of high latency in the existing technology when facing massive files.

[0049] 2. The second aspect of the present invention provides a method for constructing a metadata index structure of a distributed file system. When an insert request for a file to be processed is received, a metadata index structure responsive to the current insert request is created or updated. The present invention designs a node expansion mechanism driven by hash collision of a common index model node and a small directory cluster compaction node design mechanism. The common index model node expansion mechanism driven by hash collision automatically expands the node when the total number of entries in the second entry array of the current node reaches a certain scale, expands it into a common model index node, and uses a local index model for indexing, forming a multi-layer index node to index the interior of a super-large directory, thereby avoiding the search degradation phenomenon caused by too many entries in the small directory cluster compaction node. The small directory cluster compaction node design mechanism solves the problem that in the case of a large number of small directory clusters, a large number of small nodes containing only a small number of entries are caused by the common index model node hash collision-driven node expansion mechanism, thereby affecting the indexing efficiency, and shortens the index path length. In the metadata index structure formed by the node expansion mechanism driven by the hash collision of the above-mentioned ordinary index model nodes and the small directory cluster compaction node design mechanism, the node type of the root node is the ordinary model index node; the node types of the intermediate nodes include: ordinary model index node and small directory cluster compaction node; the node type of the leaf node is the directory entry node; by designing corresponding node types for the root node, intermediate node and leaf node in the metadata index structure to store different key information, a better hierarchical design is achieved, which is flatter and can ensure the low-tree-high feature of the directory index tree when continuously creating new file directory entries, thereby reducing the operation delay of subsequent metadata access.

[0050] 3. Furthermore, the metadata access method provided in the second aspect of the present invention further includes: performing incremental training on the local index model, thereby further improving the accuracy of the model.

[0051] 4. The third aspect of the present invention provides a metadata access method. When a file query request is received, a query operation is performed on the file to be processed in the metadata index structure constructed by the construction method provided in the second aspect of the present invention. Based on the design of the metadata index structure, the metadata of the file to be processed can be accessed with a shorter index path length, thereby reducing the delay in metadata access. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of the structure of a metadata indexing system for a distributed file system provided by an embodiment of the present invention;

[0053] Figure 2 A schematic diagram of the overall architecture design of a distributed file system metadata service based on a learning-based index provided by an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of the data structure of a common model index node provided by an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of the data structure of a small directory cluster compaction node provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0057] In order to achieve the above objectives, in a first aspect, the present invention provides a metadata indexing system for a distributed file system, such as Figure 1 As shown, it includes: a metadata index structure for indexing the storage path of each storage file in the distributed file system; the metadata index structure is a tree structure, including: a root node, an intermediate node and a leaf node;

[0058] The node type of the root node is a common model index node;

[0059] The node types of intermediate nodes include: common model index nodes and small directory cluster compaction nodes;

[0060] The node type of a leaf node is a directory entry node;

[0061] The information stored in the common model index node includes: a first table header and a first table item array; the information stored in the first table header includes: a corresponding file path prefix and a local index model; the local index model is used to establish a mapping relationship between the path segments of each storage file in the distributed file system and the table item position pointing to the corresponding storage file in the first table item array stored at the node where the local index model is located; the path segment is a path prefix of a preset length intercepted from the sub-path after eliminating the file path prefix stored at the node where the corresponding local index model is located from the file path of the storage file; the information stored in each table item in the first table item array includes: the table item type, the next level node to which the table item refers, and the next level node to which the table item refers. The length of the file path prefix in the point and the pointer to the next layer node pointed to by the table entry; the table entry types include: empty table entry, ordinary model index node table entry, small directory cluster compaction node table entry and file directory entry table entry; among them, the node information pointed to by the pointer in the empty table entry is invalid; the pointer in the ordinary model index node table entry points to the node address of the ordinary model index node; the pointer in the small directory cluster compaction node table entry points to the node address of the small directory cluster compaction node; the pointer in the file directory entry table entry points to the storage address of the file directory entry metadata; the node types of the next layer node pointed to by the table entry in the ordinary model index node include ordinary model index node, small directory cluster compaction node and directory entry node;

[0062] The small directory cluster compaction node includes: a second header and a second entry array; information stored in the second header includes: a corresponding file path prefix and a total number of entries in the second entry array; information stored in each entry in the second entry array includes: a path suffix corresponding to the entry and a pointer to a next-level node pointed to by the entry; the node type of the next-level node pointed to by the entry in the small directory cluster compaction node is a directory entry node;

[0063] The information stored in the directory entry node includes: metadata and path suffix of the corresponding file.

[0064] The complete file path of the stored file is the concatenation result of the path sub-segments in each node on the path from the root node to the directory entry node corresponding to the stored file in the metadata index structure; among them, the path sub-segment in the root node is the file path prefix field on its header; the path sub-segment in the intermediate node whose node type is the ordinary model index node is the file path prefix field on its header; the path sub-segment in the intermediate node whose node type is the small directory cluster compaction node is the file path prefix field on its header and the path suffix field on the corresponding table item; the path sub-segment in the leaf node is the path suffix stored on it.

[0065] It should be noted that the above-mentioned local index model is a pre-trained local index model, which can be a CDF model, a neural network model, etc., and is not limited here.

[0066] In an optional implementation manner, the information stored in the first header further includes: the number of valid entries.

[0067] In an optional implementation manner, the information stored in each entry in the second entry array further includes: a hash value corresponding to the path suffix corresponding to the entry.

[0068] In a second aspect, the present invention provides a method for constructing a metadata index structure of a distributed file system, comprising: upon receiving an insert request for a file to be processed, performing the following operations:

[0069] B100, create a target node of the type of directory entry node for the file to be processed; the information stored in the directory entry node includes: metadata and path suffix of the corresponding file; the metadata in the target node is the metadata of the file to be processed, and the path suffix is ​​the path of the file to be processed; target node; when the root node of the metadata index structure in the distributed file system is empty, create a node node0 of the type of ordinary model index node in the metadata index system as the root node of the metadata index structure; the information stored in the ordinary model index node includes: the corresponding file path prefix P c , local index model M and fixed-length first table array A1; the information stored in the table in A1 includes: table type, length L of the file path prefix in the next layer node pointed to by the table np And the pointer to the next layer node pointed to by the table entry P in node0 c Empty; M in node0 is used to reflect the correspondence between the path prefix of the preset length intercepted from the file path of the file to be processed and the position of the entry pointing to the target node in the first entry array of node0; A1 in node0 stores the entry T; the entry type of T is the file directory entry; L of T np is the file path length of the file to be processed; T Point to the target node; otherwise, the root node of the metadata index structure is used as the current node; it should be noted that the above fixed length is an empirical value, and in an optional implementation method, the value is 64, indicating that a maximum of 64 table entries can be stored.

[0070] B200. Get the file path prefix P stored in the current node c and eliminate it from the file path of the file to be processed to obtain the current subpath; determine the node type of the current node. If the current node is a common model index node, execute B300; if the current node is a small directory cluster compaction node, execute B400-B600;

[0071] B300: Extract a path prefix of a preset length from the current subpath and input it into the local index model M stored in the current node. Obtain the position of the entry in the first entry array of the current node that points to the file to be processed, and use the entry at that position as the current target entry.

[0072] When the entry type of the current target entry is an empty entry, the entry type of the current target entry is updated to a file directory entry, and the length of the current subpath is used as L np , the address of the target node is Insert the data into the current target table entry, thereby constructing a metadata index structure that responds to the current insert request, and the operation ends;

[0073] When the entry type of the current target entry is a common model index node entry or a small directory cluster compaction node entry, the next layer node pointed to by the current target entry is obtained and used as the new current node, and the process goes to B200;

[0074] When the entry type of the current target entry is a file directory entry, it is determined whether the file path stored in the directory entry node pointed to by the current target entry is consistent with the current sub-path. If so, it returns a message indicating that the metadata index structure creation in response to the current insert request has failed, and the creation operation is terminated. Otherwise, a node node1 of the type small directory cluster compaction node is created in the metadata index system. The information stored in the small directory cluster compaction node includes: the corresponding file path prefix P c , the second entry array A2 and the total number of entries N2 in A2; the information stored in the entry in A2 includes: the path suffix S corresponding to the entry s And the pointer to the next layer node pointed to by the table entry Node1 stores table entries P and Q; N2 in node1 is 2; P in node1 c P in the node where the current target entry is located c The common prefix with the current subpath; With the current target table entry Same; S in entry P s Eliminate the P in node1 from the path suffix stored in the directory entry node pointed to by the current target table entry. c The remaining part after the entry Q Points to the target node; S in table entry Q s Eliminate P in node1 for the current subpath c The remaining part after the current target table entry; update the type of the current target table entry to the small directory cluster compaction node table entry and point it to node1, thereby constructing a metadata index structure that responds to the current insert request, and the operation ends;

[0075] B400, querying the current subpath in the second entry array of the current node. If the current subpath exists, returning a message indicating that the metadata index structure creation in response to the current insert request has failed, and ending the creation operation. If the current subpath does not exist, going to B500;

[0076] B500, inserting the entry R pointing to the target node into the second entry array of the current node, and updating the total number of entries N2 stored in the current node; the path suffix S stored in the entry R s is the current subpath; determine whether the total number of entries of the current node exceeds the preset threshold. If so, create a node node2 of the type of ordinary model index node in the metadata index system; P in node2 c is the current subpath; A1 in node2 stores the entries in the second entry array of the current node; the local index model M in node2 is used to reflect the correspondence between the path prefix of a preset length intercepted from the current subpath and the entry position pointing to the target node in the first entry array of node2; delete the current node, store node2 at the original position of the current node, and go to B600; otherwise, go directly to B600; wherein the above-mentioned preset threshold is an empirical value. In an optional implementation manner, the above-mentioned preset threshold is 256.

[0077] B600. Use the current metadata index structure as the metadata index structure responding to the current insert request. The operation ends.

[0078] It should be noted that in the presence of extremely large directories, the traditional directory tree index structure will cause the search for directory entries pointing to the target file path within the node to degenerate into a binary search, thereby reducing search efficiency. The expansion mechanism driven by node hash collision in the ordinary index model can automatically expand the node when the directory table entries reach a certain size, forming a multi-layer index node to index the interior of the extremely large directory, avoiding the problem of indexing efficiency degradation caused by the degeneration of the extremely large directory;

[0079] At the same time, the design mechanism of compacting nodes in small directory clusters solves the problem that in the case of a large number of small directory clusters, the node expansion mechanism driven by node hash collision in the ordinary index model causes a large number of small nodes containing only a small number of table entries, thereby affecting the indexing efficiency, and further shortens the index path length.

[0080] It should be noted that the local index model in node0 and node2 can be an existing pre-trained model, a model incrementally trained during the file creation process based on an initial model, or a model incrementally trained during the file creation process based on a pre-trained model. There are no restrictions here. The local index model can be a CDF model, a neural network model, or other models.

[0081] Preferably, in an optional embodiment, the local index model is further incrementally trained during the process of creating the file. In this case, the above B100 also includes: after creating the common model index node node2 in the metadata index system, the path prefix of the preset length intercepted from the current sub-path is used as the input sample, and the table position of the first table item array of node2 pointing to the target node is used as the corresponding label, and the local index model in node2 is incrementally trained. The above B600 also includes: after creating the common model index node node0 in the metadata index system, the path prefix of the preset length intercepted from the current sub-path is used as the input sample, and the table position of the first table item array of node0 pointing to the target node is used as the corresponding label, and the local index model in node0 is incrementally trained.

[0082] It should be noted that the metadata index structure in the metadata index system provided in the first aspect of the present invention can be constructed using the metadata index structure construction method provided in the second aspect of the present invention, wherein the preset length in the second aspect is the same as the preset length in the first aspect. The preset length is an empirical value. In one optional embodiment, the preset length is 8 to 64, preferably 8.

[0083] It should be noted that there are multiple ways to query whether the current subpath exists in the second table item array of the current node. For example, in the optional implementation method, the current subpath is directly compared with the path suffix stored in each table item of the second table item array for query; in order to further improve the query efficiency, in the optional implementation method, the hash value corresponding to the current subpath is compared with the hash value corresponding to the path suffix stored in each table item of the second table item array for query (query whether the hash value corresponding to the current subpath exists in the second table item array of the current node), which is not limited here. In the above optional implementation method, when the hash value of the corresponding path suffix is ​​also stored in the table item of the second table item array, the corresponding path suffix S is also stored in the above table item P and table item Q. s The hash value of ; at this time, querying the current subpath in the second table array of the current node includes:

[0084] S1. Check whether the hash value corresponding to the current subpath exists in the second entry array of the current node. If so, take the entry in the second entry array containing the hash value corresponding to the current subpath as a candidate entry and go to S2; otherwise, go to S3.

[0085] S2, determine whether there is a stored S s The candidate table entry is the same as the current subpath. If so, determine whether the current subpath exists in the second table entry array of the current node; otherwise, go to S3;

[0086] S3. Determine whether the current subpath does not exist in the second entry array of the current node.

[0087] It should be noted that the query can be performed using methods such as sequential search method and binary search method, which are not limited here. It is preferred to use binary search method to improve query efficiency.

[0088] The present invention is applicable to distributed file systems, and is particularly applicable to distributed file systems for massive-scale files. Large model and container technologies are continuously developing towards edge intelligent computing scenarios. The files for model training and container images are small and numerous, and can reach hundreds of millions or even tens of billions. Massive small files place high scalability and low latency requirements on metadata management. Due to the metadata request bottleneck, existing systems cannot fully meet the low latency requirements of scenarios such as large model training and container startup acceleration. The present invention proposes to introduce low-latency, low-space-overhead learning indexes into the path resolution process, replacing the inefficient directory tree index structure and index mechanism with a more efficient learning index to achieve a low-latency, low-space-overhead path resolution mechanism to meet the storage needs of massive small files. In summary, the present invention meets the high scalability requirements for metadata access of massive small files in edge scenarios, reduces the storage overhead of metadata for massive small files, improves the throughput of metadata operations, and has high scalability; at the same time, the present invention meets the low latency requirements for metadata access of massive small files in edge scenarios, breaks through the metadata request latency bottleneck for massive small files, and fully meets the low latency requirements of scenarios such as large model training and container startup acceleration.

[0089] In a third aspect, the present invention provides a metadata access method, which is applied to the metadata index structure constructed by the construction method provided in the second aspect of the present invention; the metadata access method includes: when receiving a file query request, performing a query operation on the file to be processed, including:

[0090] A100, determining whether the root node of the metadata index structure is empty, if so, proceeding to A600; otherwise, taking the root node of the metadata index structure as the current node;

[0091] A200, obtaining the file path prefix stored in the current node, and eliminating it from the file path of the file to be processed to obtain the current subpath; determining the node type of the current node, if the node type of the current node is a common model index node, proceeding to A300; if the node type of the current node is a small directory cluster compaction node, proceeding to A400;

[0092] A300, intercepting a path prefix of a preset length from the current subpath, and inputting it into the local index model stored in the current node, obtaining the table entry position pointing to the file to be processed in the first table entry array of the current node, and using the table entry under this position as the current target table entry; when the table entry type of the current target table entry is an empty table entry, go to A600; when the table entry type of the current target table entry is a common model index node table entry or a small directory cluster compaction node table entry, obtain the next layer node pointed to by the current target table entry, and use it as the new current node, and go to A200; when the table entry type of the current target table entry is a file directory item table entry, determine whether the length of the file path prefix in the next layer node pointed to by the current target table entry is consistent with the length of the current subpath, if consistent, go to A700; otherwise, go to A600; wherein, the preset length is the same as the preset length in the second aspect;

[0093] A400, query the current subpath in the second entry array of the current node. If the current subpath exists, go to A600; if the current subpath does not exist, go to A500;

[0094] A500 determines that the query has failed, returns a message indicating that no files to be processed were found, and ends the query;

[0095] A600: Determine that the query is successful, return the pointer of the next layer node pointed to by the current target table entry, and end the query.

[0096] It should be noted that there are multiple ways to query whether the current subpath exists in the second table item array of the current node. For example, in the optional implementation method, the current subpath is directly compared with the path suffix stored in each table item of the second table item array for query; in order to further improve the query efficiency, in the optional implementation method, the hash value corresponding to the current subpath is compared with the hash value corresponding to the path suffix stored in each table item of the second table item array for query (query whether the hash value corresponding to the current subpath exists in the second table item array of the current node), which is not limited here. Under the above optional implementation method two, when the hash value of the corresponding path suffix is ​​also stored in the table item of the second table item array, at this time, querying the current subpath in the second table item array of the current node includes:

[0097] S1. Check whether the hash value corresponding to the current subpath exists in the second entry array of the current node. If so, take the entry in the second entry array containing the hash value corresponding to the current subpath as a candidate entry and go to S2; otherwise, go to S3.

[0098] S2, determine whether there is a stored S sThe candidate table entry is the same as the current subpath. If so, determine whether the current subpath exists in the second table entry array of the current node; otherwise, go to S3;

[0099] S3. Determine whether the current subpath does not exist in the second entry array of the current node.

[0100] It should be noted that the query can be performed using methods such as sequential search method and binary search method, which are not limited here. It is preferred to use binary search method to improve query efficiency.

[0101] The relevant technical solution is the same as the construction method provided in the second aspect of the present invention, and will not be described in detail here.

[0102] To further illustrate the metadata indexing system for a distributed file system and the corresponding metadata operation method (index structure construction method (corresponding file creation method) and metadata access method) provided by the present invention, an application example thereof in a specific scenario is described in detail below:

[0103] like Figure 2 The figure shows a schematic diagram of the overall architecture design of the distributed file system metadata service based on learning indexing provided by this embodiment, the core of which includes a metadata indexing system.

[0104] The metadata indexing system is similar to the recursive model index (RMI) of the learned index. It consists of a hierarchical model structure consisting of multiple models. The first layer, the root node, has a single model node, and each subsequent layer contains multiple model nodes. Each model in the RMI index tree takes query data as input, performs model calculations, returns the address of the next layer's index node, and then enters the next layer's model nodes for indexing. The output of the last layer of models serves as the final index result, i.e., the metadata storage location. Compared to traditional index structures, the learned index provides better search performance and lower space requirements.

[0105] Furthermore, the path resolution module relies on the learning index model to build an index structure for path resolution, which is composed of multiple index nodes. There are different types of index nodes: ordinary model index nodes, small directory cluster compaction nodes, and directory entry nodes. In this embodiment, the ordinary index node is composed of a header containing a local index model and an entry array. The file path obtains its corresponding index position in the array after calculation through the local index model. The small directory cluster compaction node is composed of a header and an entry array. The hash value is calculated within the node to perform a binary search on the entry array to determine the entry position corresponding to the file path. The directory entry node is the metadata of the file directory entry corresponding to the final file path, which is stored in the metadata storage module.

[0106] like Figure 3 As shown in Figure 1, a common model index node consists of a header and an entry array. The header consists of the node path prefix (Prefix), the total size of the entry array (Cout), the number of valid entries (Array size), and the local index model. Each entry contains the entry type, prefix length, and node pointer.

[0107] There are four types of entries: empty entries, normal model index node entries, small directory cluster compaction node entries, and file directory entry entries. Empty entries are reserved space for nodes; the node information pointed to by the node pointer in an empty entry is invalid. Normal model index node entries and small directory cluster compaction node entries point to the node addresses of normal model index nodes and small directory cluster compaction nodes, respectively. The node pointer in a file directory entry points to the storage address of the file directory entry metadata.

[0108] For each table entry, there are three cases: ① The table entry is empty, and the table entry type is an empty table entry. ② A directory entry is mapped to the table entry, and the node points to the storage location of the directory entry metadata. The table entry type is a file directory entry. ③ If multiple directory entries are mapped to the table entry, a new node is created to store these directory entries, which is called a node expansion operation. This avoids error searches after local index model prediction failures. The table entry type is a normal model index node table entry or a small directory cluster compaction node table entry.

[0109] In the common model index node, after eliminating the redundant prefix part of the file path through the node path prefix, the node table entry only intercepts the sub-path that can guarantee the unique distinction of each table entry, that is, the redundant file path suffix is ​​eliminated at the same time, based on the local index model and the model reasoning in the file path parsing process.

[0110] like Figure 4 As shown in Figure 1, the small directory cluster compaction node contains a header and an entry array. The header consists of the node path prefix and the number of entries. Each entry contains the path suffix hash value and a node pointer.

[0111] Due to the node expansion mechanism of collision-driven design, small directory clusters are prone to causing a large number of small nodes containing only a small number of valid entries. Replacing the compaction operation of these small nodes with the compaction node of the small directory cluster can shorten the index tree height and reduce index overhead.

[0112] In the small directory cluster compaction node, after eliminating the redundant prefix part of the file path through the node path prefix, the hash value is calculated using the file path suffix to query the directory entry table, and finally the file directory entry metadata storage address corresponding to the file path is obtained.

[0113] The metadata operation process in this embodiment is as follows:

[0114] On the client side, the user calls the LearnFS library to send a metadata request to the metadata server (MDS) to obtain relevant metadata information and perform the corresponding file metadata operation. The metadata request mainly includes the type of metadata request and the file path address of the file directory item to which the metadata operation belongs.

[0115] On the server side, it is mainly composed of a path resolution module (which performs path resolution based on the metadata index system) and a metadata storage module. After receiving a metadata request from a client, the server first uses the file path resolution module to resolve the path of the file to be processed in the metadata request to obtain the storage address of the relevant directory item involved in the metadata request, obtains the directory item metadata from the metadata storage module, and returns the metadata information to the client after performing the metadata operation.

[0116] Take the file query operation as an example:

[0117] For a stat file query, the client performs a metadata operation, calling the LearnFS library to send a metadata request containing the file path to the remote metadata server (MDS). Upon receiving the request, the metadata server determines the metadata request type and, if it is a file query, passes the file path included in the request to the path resolution module for resolution. The path resolution module begins its resolution process by starting with the root directory index node, which is a standard model index node.

[0118] The internal parsing process of the common model index node is as follows: according to the path prefix information of the node header, the redundant prefixes in the file path to be parsed are eliminated, and then the remaining file path is inferred through the local index model. During the inference process, only the path of the partial length required by the model is used. After model inference, the index position of the file path in the node table array is obtained, and the information of the table item is used to determine whether the parsing is completed. If the table item type is empty, it means that the path parsing failed, the file path does not have a corresponding file directory item, and the parsing failure information is returned. If the table item type is a common model index node table item or a small directory cluster compaction node table item, it means that the path parsing process is not completed, and the node pointer jumps to the next level node for subsequent parsing operations. If the table item type is a file directory item table item, it means that the path parsing is complete, and the node pointer is returned, that is, the storage address of the directory item metadata corresponding to the file path.

[0119] The internal parsing process of the small directory cluster compaction node is as follows: based on the path prefix information in the node header, eliminate the redundant prefixes in the file path to be parsed to obtain the unprocessed file path suffix, obtain the corresponding path suffix hash value through hash calculation, query and read the corresponding table entry in the node table array through binary search, compare the file path with the file path of the directory item metadata pointed to by the table entry pointer, and if the file paths are consistent, it means that the parsing is complete, and return the node pointer, that is, the storage address of the directory item metadata corresponding to the file path. The metadata server performs metadata operations based on the return information of the path parsing module. If the file path parsing fails, a path parsing failure message is sent to the client, ending the metadata request service. If the file path parsing is successful, the metadata server reads the directory item metadata information in the metadata storage module through the directory item metadata storage address returned by the path parsing module, and finally sends the metadata information to the client to complete the metadata request service.

[0120] After the client receives the information returned by the metadata server, it parses the returned information through the LearnFS function library and finally feeds it back to the user.

[0121] Take the file creation operation (i.e., the method of building the metadata index structure in response to the current insert request) as an example:

[0122] For a create file operation, the client performs a create metadata operation, calling the LearnFS library to send a metadata request containing the desired file name and path to the remote metadata server (MDS). Upon receiving the request, the metadata server determines the metadata request type. If it indicates a file creation request, the path resolution module resolves the file path in the request. The path resolution module operates in the same manner as the stat operation.

[0123] The metadata server processes the information returned by the path resolution module. If path resolution fails, indicating that the path to the file being created does not exist, the server sends a message to the client stating that file creation failed, terminating the metadata request. If path resolution succeeds, the server inserts the directory entry and allocates space for the new file in the metadata storage module, obtaining the corresponding metadata storage address.

[0124] The directory entry insertion operation process is as follows: according to the previous path resolution module parsing process, first determine the type of the index node. If it is a normal model index node, then determine the type of the corresponding table item in the array. If the type is empty, insert it directly into the table item, and change the type to a file directory item table item. The node pointer points to the corresponding metadata storage address. If the type is a file directory item table item, it means that a directory item already exists at this location, and a new small directory cluster compaction node needs to be created. The directory item node pointer pointed to by the original array table item and the directory item node pointer to be created are inserted into the small directory cluster compaction node, and then the node pointer is inserted into the original array table item, and the table item type is updated to a small directory cluster compaction node table item. If it is a small directory cluster compaction node, it is necessary to first determine the filling status of the index node to which it belongs. If the number of valid entries (i.e. the total number of entries in the second entry array) is lower than the threshold, the directory entry node pointer is inserted into the entry array in order by calculating the hash value of the path suffix; if the number of valid entries is higher than the threshold, a new normal model index node is created to replace the entry pointer of the original small directory cluster compaction node in the upper layer node, and the entry type is modified to a normal model index node entry, and the node pointer points to the newly created normal model index node. For the newly created index node, the node header needs to be filled, and the path prefix and local index model are filled in through the training model.

[0125] After completing the directory entry insertion operation, the metadata server sends a creation success message to the client.

[0126] After the client receives the information returned by the metadata server, it parses the returned information through the LearnFS function library and finally feeds it back to the user.

[0127] This embodiment optimizes the path resolution process to improve metadata service performance. Observing that the entire path resolution process can be abstracted as an indexing operation based on the file path, using the directory tree as the index structure, and the inode as the query result, this embodiment optimizes the path resolution process by introducing a learned index mechanism to accelerate metadata services. Learned indexing is a machine learning-based approach that replaces traditional database index structures, such as B+ trees and hash tables, to improve query performance and efficiency.

[0128] This embodiment proposes introducing a low-latency, low-space-cost learning index into the path resolution process, replacing the inefficient directory tree index structure and indexing mechanism with a more efficient learning index to implement a low-latency, low-space-cost path resolution mechanism to meet the storage needs of massive small files. To achieve the above objectives, the efficient metadata management mechanism implemented in this embodiment addresses several challenges: the low efficiency of the file path learning index model, the high last-mile search overhead of the learning index, and the high additional overhead caused by file path skew.

[0129] Specifically, file path is a string type key, and there is currently little research on learning indexing of edge length strings. The present invention designs a learning index model optimized for file paths.

[0130] This embodiment designs a learning index model for file path optimization. The existing learning index mainly studies integer or floating-point type keys, while the key used in the path resolution process is a file path of variable-length string type. If an integer or floating-point type learning index model is used, the model training overhead is large, resulting in high delay and low efficiency in the path resolution, i.e., the model inference process. The attribute of the variable-length string of the file path causes the training overhead of the learning index model to be too large, and also reduces the accuracy of the model training. This embodiment establishes a learning index model that reflects the correspondence between the path prefix of a preset length intercepted from the file path of the file to be processed and the position of the table entry pointing to the target node in the first table entry array of the learning index model. It should be noted that the preset length generally adopts an empirical value. Preferably, under an optional implementation method, the common prefix of the file name is optimized, and the common path prefix is ​​used as the distinguishing attribute of the model node. For the node, only the shortest string that can uniquely distinguish each file path is used for training the local index model.

[0131] This embodiment designs an index node driven by hash collision: the storage position pos predicted by the key according to the learning index model is not completely correct, and it is usually necessary to perform a binary search within the error range of the predicted position to determine the final data location. The complete file path is usually in the range of tens of B to several KB in length, and the long string key leads to a large string comparison overhead in the search process. This embodiment constructs a node expansion mechanism driven by hash collision, so that paths in the same directory or with the same file name prefix are mapped to the same hash bucket. If the number of collisions is too many, a new index node is established, avoiding the problem of overly bloated index nodes, and solving the last mile search problem of general learning indexes, reducing the frequent file path string comparison overhead caused by range searches of array table entries in nodes in the final stage of indexing.

[0132] This embodiment designs a compaction mechanism for small directory nodes: the distribution of files in the file system directory tree is not uniform, and there are both small directories (containing only a few directory entries) and super directories (containing thousands to tens of thousands of directory entries), which is generally referred to as a skew phenomenon. This skew phenomenon causes the height of the learning index tree to be uncontrollable, and a large number of small directories lead to a large number of small index nodes, causing additional IO overhead to reduce path resolution performance. Super directories cause index nodes to be too bloated, resulting in a decrease in the accuracy of local models within the index and increased training overhead. This embodiment performs a compaction process on multiple small index nodes formed by continuously distributed small directories, reduces the tree height of the learning index, avoids redundant IO operations caused by long index paths, and reduces the delay in path resolution.

[0133] In a fourth aspect, the present invention provides an electronic device comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the method provided in the second aspect or the third aspect of the present invention when executing the computer program.

[0134] The relevant technical solutions are the same as the methods provided in the second or third aspects of the present invention and will not be described in detail here.

[0135] In a fifth aspect, the present invention provides a distributed file system, comprising:

[0136] An index structure construction module, configured to execute the construction method provided by the second aspect of the present invention;

[0137] The metadata access module is used to execute the metadata access method provided by the third aspect of the present invention.

[0138] The relevant technical solutions are the same as the methods provided in the second or third aspects of the present invention and will not be described in detail here.

[0139] In a sixth aspect, the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the method provided in the second aspect or the third aspect of the present invention.

[0140] The relevant technical solutions are the same as the methods provided in the second or third aspects of the present invention and will not be described in detail here.

[0141] In a seventh aspect, the invention further provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the method provided in the second or third aspect of the invention.

[0142] The relevant technical solutions are the same as the methods provided in the second or third aspects of the present invention and will not be described in detail here.

Claims

1. A metadata indexing system for a distributed file system, characterized in that: include: A metadata index structure for indexing the storage path of each storage file in the distributed file system; The metadata index structure is a tree structure, including: root node, intermediate nodes and leaf nodes; The node type of the root node is a common model index node; The node types of intermediate nodes include: common model index nodes and small directory cluster compaction nodes; The node type of a leaf node is a directory entry node; The information stored in the common model index node includes: a first table header and a first table item array; the information stored in the first table header includes: a corresponding file path prefix and a local index model; the local index model is used to establish a mapping relationship between the path fragments of each storage file in the distributed file system and the table item position pointing to the corresponding storage file in the first table item array stored at the node where the local index model is located; the path fragment is a path prefix of a preset length intercepted from the sub-path after eliminating the file path prefix stored at the node where the corresponding local index model is located from the file path of the storage file; the information stored in each table item in the first table item array includes: table item type, The length of the file path prefix in the next-level node pointed to by the table entry and the pointer to the next-level node pointed to by the table entry; the table entry types include: empty table entry, normal model index node table entry, small directory cluster compaction node table entry and file directory entry table entry; among them, the pointer in the normal model index node table entry points to the node address of the normal model index node; the pointer in the small directory cluster compaction node table entry points to the node address of the small directory cluster compaction node; the pointer in the file directory entry table entry points to the storage address of the file directory entry metadata; the node types of the next-level node pointed to by the table entry in the normal model index node include normal model index node, small directory cluster compaction node and directory entry node; The small directory cluster compaction node includes: a second header and a second entry array; information stored in the second header includes: a corresponding file path prefix and a total number of entries in the second entry array; information stored in each entry in the second entry array includes: a path suffix corresponding to the entry and a pointer to a next-level node pointed to by the entry; the node type of the next-level node pointed to by the entry in the small directory cluster compaction node is a directory entry node; The information stored in the directory entry node includes: metadata and path suffix of the corresponding file.

2. The metadata indexing system according to claim 1, characterized in that: The information stored in each entry in the second entry array further includes: a hash value corresponding to the path suffix corresponding to the entry.

3. A method for constructing a metadata index structure of a distributed file system, characterized in that: include: When receiving an insert request for a pending file, perform the following operations: B100, creating a target node of directory entry type for the file to be processed; The information stored in the directory entry node includes: metadata and path suffix of the corresponding file; the metadata in the target node is the metadata of the file to be processed, and the path suffix is ​​the path of the file to be processed; the target node; when the root node of the metadata index structure in the distributed file system is empty, a node node0 of the type of ordinary model index node is created in the metadata index system as the root node of the metadata index structure; the information stored in the ordinary model index node includes: the corresponding file path prefix P c , local index model M and fixed-length first table array A1; the information stored in the table in A1 includes: table type, length L of the file path prefix in the next layer node pointed to by the table np And the pointer to the next layer node pointed to by the table entry P in node0 c Empty; M in node0 is used to reflect the correspondence between the path prefix of the preset length intercepted from the file path of the file to be processed and the position of the entry pointing to the target node in the first entry array of node0; A1 in node0 stores the entry T; the entry type of T is the file directory entry; L of T np is the file path length of the file to be processed; T Point to the target node; otherwise, use the root node of the metadata index structure as the current node; B200. Get the file path prefix P stored in the current node c and eliminate it from the file path of the file to be processed to obtain the current subpath; determine the node type of the current node. If the current node is a common model index node, execute B300; if the current node is a small directory cluster compaction node, execute B400-B600; B300: Extract a path prefix of a preset length from the current subpath and input it into the local index model M stored in the current node. Obtain the position of the entry in the first entry array of the current node that points to the file to be processed, and use the entry at that position as the current target entry. When the entry type of the current target entry is an empty entry, the entry type of the current target entry is updated to a file directory entry, and the length of the current subpath is used as L np , the address of the target node is Insert the data into the current target table entry, thereby constructing a metadata index structure that responds to the current insert request, and the operation ends; When the entry type of the current target entry is a common model index node entry or a small directory cluster compaction node entry, the next layer node pointed to by the current target entry is obtained and used as the new current node, and the process goes to B200; When the entry type of the current target entry is a file directory entry, it is determined whether the file path stored in the directory entry node pointed to by the current target entry is consistent with the current sub-path. If so, it returns a message indicating that the metadata index structure creation in response to the current insert request has failed, and the creation operation is terminated. Otherwise, a node node1 of the type small directory cluster compaction node is created in the metadata index system. The information stored in the small directory cluster compaction node includes: the corresponding file path prefix P c , the second entry array A2 and the total number of entries N2 in A2; the information stored in the entry in A2 includes: the path suffix S corresponding to the entry s And the pointer to the next layer node pointed to by the table entry Node1 stores table entries P and Q; N2 in node1 is 2; P in node1 c P in the node where the current target entry is located c The common prefix with the current subpath; With the current target table entry Same; S in entry P s Eliminate the P in node1 from the path suffix stored in the directory entry node pointed to by the current target table entry. c The remaining part after the entry Q Points to the target node; S in table entry Q s Eliminate P in node1 for the current subpath c The remaining part after the current target table entry; update the type of the current target table entry to the small directory cluster compaction node table entry and point it to node1, thereby constructing a metadata index structure that responds to the current insert request, and the operation ends; B400, querying the current subpath in the second entry array of the current node. If the current subpath exists, returning a message indicating that the metadata index structure creation in response to the current insert request has failed, and ending the creation operation. If the current subpath does not exist, going to B500; B500, inserting the entry R pointing to the target node into the second entry array of the current node, and updating the total number of entries N2 stored in the current node; the path suffix S stored in the entry R s is the current subpath; determine whether the total number of entries of the current node exceeds the preset threshold. If so, create a node node2 of the type of ordinary model index node in the metadata index system; P in node2 c The current subpath; A1 in node2 stores the entries in the second entry array of the current node; the local index model M in node2 is used to reflect the correspondence between the path prefix of the preset length intercepted from the current subpath and the entry position pointing to the target node in the first entry array of node2; delete the current node, store node2 at the original position of the current node, and go to B600; otherwise, go directly to B600; B600. Use the current metadata index structure as the metadata index structure responding to the current insert request. The operation ends.

4. The construction method according to claim 1, characterized in that The corresponding path suffix S is also stored in the table item P and the table item Q. s The hash value of The querying of the current subpath in the second entry array of the current node includes: S1. Check whether the hash value corresponding to the current subpath exists in the second entry array of the current node. If so, take the entry in the second entry array containing the hash value corresponding to the current subpath as a candidate entry and go to S2; otherwise, go to S3. S2, determine whether there is a stored S s The candidate table entry is the same as the current subpath. If so, determine whether the current subpath exists in the second table entry array of the current node; otherwise, go to S3; S3. Determine whether the current subpath does not exist in the second entry array of the current node.

5. The construction method according to claim 3 or 4, characterized in that: The B100 further includes: after creating a common model index node node2 in the metadata index system, using a path prefix of a preset length intercepted from the current subpath as an input sample and the position of the entry pointing to the target node in the first entry array of node2 as the corresponding label, and performing incremental training on the local index model in node2; The B600 also includes: after creating a normal model index node node0 in the metadata index system, the path prefix of a preset length intercepted from the current sub-path is used as an input sample, and the table entry position pointing to the target node in the first table entry array of node0 is used as the corresponding label to perform incremental training on the local index model in node0.

6. A metadata access method, characterized in that: A metadata index structure constructed by the construction method according to any one of claims 3 to 5; The metadata access method includes: when receiving a file query request, performing a query operation on the file to be processed, including: A100, determining whether the root node of the metadata index structure is empty, if so, proceeding to A600; otherwise, taking the root node of the metadata index structure as the current node; A200, obtaining the file path prefix stored in the current node, and eliminating it from the file path of the file to be processed to obtain the current subpath; determining the node type of the current node, if the node type of the current node is a common model index node, proceeding to A300; if the node type of the current node is a small directory cluster compaction node, proceeding to A400; A300, intercepting a path prefix of a preset length from the current subpath, and inputting it into the local index model stored in the current node, obtaining the table entry position pointing to the file to be processed in the first table entry array of the current node, and using the table entry under this position as the current target table entry; when the table entry type of the current target table entry is an empty table entry, go to A600; when the table entry type of the current target table entry is a common model index node table entry or a small directory cluster compaction node table entry, obtain the next layer node pointed to by the current target table entry, and use it as the new current node, and go to A200; when the table entry type of the current target table entry is a file directory item, determine whether the length of the file path prefix in the next layer node pointed to by the current target table entry is consistent with the length of the current subpath, if consistent, go to A700; otherwise, go to A600; A400, query the current subpath in the second entry array of the current node. If the current subpath exists, go to A600; if the current subpath does not exist, go to A500; A500 determines that the query has failed, returns a message indicating that no files to be processed were found, and ends the query; A600: Determine that the query is successful, return the pointer of the next layer node pointed to by the current target table entry, and end the query.

7. The metadata access method according to claim 6, characterized in that: When the hash value of the corresponding path suffix is ​​also stored in the entry of the second entry array, querying the current subpath in the second entry array of the current node includes: S1. Check whether the hash value corresponding to the current subpath exists in the second entry array of the current node. If so, take the entry in the second entry array containing the hash value corresponding to the current subpath as a candidate entry and go to S2; otherwise, go to S3. S2, determine whether there is a stored S s The candidate table entry is the same as the current subpath. If so, determine whether the current subpath exists in the second table entry array of the current node; otherwise, go to S3; S3. Determine whether the current subpath does not exist in the second entry array of the current node.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the method according to any one of claims 3 to 7 when executing the computer program.

9. A distributed file system, characterized in that: include: An index structure construction module, configured to execute the construction method according to any one of claims 3 to 5; A metadata access module, configured to execute the method of claim 6 or 7.

10. A computer program product, characterized in that The method comprises a computer program / instruction, which implements the method according to any one of claims 3 to 7 when executed by a processor.

Images