Catalog metadata operation method and device, electronic equipment and readable storage medium
By pre-building prefix and full directory data tables and combining them with path matching, the problem of low efficiency in directory metadata operations is solved, query efficiency and change operation efficiency are improved, and system performance is enhanced.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING BAIDU NETCOM SCI & TECH CO LTD
- Filing Date
- 2023-12-20
- Publication Date
- 2026-08-04
AI Technical Summary
In existing technologies, directory metadata operations are inefficient in distributed file systems, especially during query operations where path resolution is performed many times and during renaming operations where path modification is costly.
By pre-constructing a prefix directory data table and a full directory data table, the key of the prefix directory data table is the prefix path of the directory metadata, and the key of the full directory data table is the identification information of the directory metadata. The path is determined by combining path matching and query results, reducing the number of path parsing operations.
It improves the efficiency of directory query operations, reduces the overhead of directory change operations, and enhances system performance.
Smart Images
Figure CN117931740B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of data processing technology, and more particularly to the fields of cloud computing, metadata management and distributed file system technology. Specifically, this disclosure relates to a directory metadata operation method, apparatus, electronic device and readable storage medium. Background Technology
[0002] File management in distributed file systems is mostly based on directory metadata. Directory metadata is stored in a specific storage format.
[0003] In the use of distributed file systems, operations on directory metadata constitute a significant proportion of all operations on the distributed file system. Currently, directory metadata operations suffer from inefficiency. Summary of the Invention
[0004] To address at least one of the aforementioned deficiencies, this disclosure provides a method, apparatus, electronic device, and readable storage medium for manipulating directory metadata.
[0005] According to a first aspect of this disclosure, a method for manipulating directory metadata is provided, the method comprising:
[0006] In response to receiving a directory query request, obtain the query path carried in the directory query request;
[0007] Obtain the pre-built prefix directory data table and the pre-built full directory data table. The key of the prefix directory data table is the prefix path of the directory metadata. The prefix path is the sub-path corresponding to the level of the path corresponding to the directory metadata, except for the last preset level. The value of the prefix directory data table is the directory metadata. The key of the full directory data table is the identification information of the directory metadata. The value of the full directory data table is the directory metadata.
[0008] Determine the matching results between the query path and the keys in the prefix directory data table;
[0009] Based on the matching results and the full catalog data table, the query results are determined.
[0010] According to a second aspect of this disclosure, a directory metadata manipulation apparatus is provided, the apparatus comprising:
[0011] The directory query request receiving module is used to respond to a received directory query request and obtain the query path carried in the directory query request;
[0012] The data table acquisition module is used to acquire a pre-built prefix directory data table and a pre-built full directory data table. The key of the prefix directory data table is the prefix path of the directory metadata, and the prefix path is the sub-path corresponding to the level of the path corresponding to the directory metadata except for the last preset level. The value of the prefix directory data table is the directory metadata. The key of the full directory data table is the identification information of the directory metadata, and the value of the full directory data table is the directory metadata.
[0013] The prefix matching module is used to determine the matching results between the query path and the keys of the prefix directory data table;
[0014] The query result determination module is used to determine the query results based on the matching results and the full catalog data table.
[0015] According to a third aspect of this disclosure, an electronic device is provided, the electronic device comprising:
[0016] At least one processor; and
[0017] A memory communicatively connected to at least one of the aforementioned processors; wherein,
[0018] The memory stores instructions that can be executed by at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the directory metadata operation method.
[0019] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described directory metadata operation method.
[0020] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described directory metadata manipulation method.
[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0023] Figure 1 This is a flowchart illustrating a directory metadata manipulation method provided in an embodiment of this disclosure;
[0024] Figure 2 This is a schematic diagram of the data structure involved in the solution provided in the embodiments of this disclosure;
[0025] Figure 3 This is a flowchart illustrating a specific implementation of the method provided in this disclosure.
[0026] Figure 4 This is a flowchart illustrating a specific implementation of the method provided in this disclosure.
[0027] Figure 5 This is a flowchart illustrating the process of deleting invalid entries in the prefix directory data table in the method provided in this embodiment of the disclosure;
[0028] Figure 6 This is a schematic diagram of the structure of a directory metadata manipulation device provided in an embodiment of this disclosure;
[0029] Figure 7 This is a block diagram of an electronic device used to implement the directory metadata operation method of the embodiments of this disclosure. Detailed Implementation
[0030] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0031] The use of distributed file systems often requires operations on directory metadata, but current operations on directory metadata are inefficient.
[0032] Operations on directory metadata include look-up, rename, and delete operations. Look-up is the most basic operation, and its efficiency must be guaranteed to improve the overall efficiency of directory metadata operations. Look-up operations involve frequent path resolution, and the efficiency of this step directly impacts the efficiency of the look-up operation.
[0033] In related technologies, there are generally two schemes for storing directory metadata. Scheme 1: Use the full path of the directory metadata as the key and the directory metadata as the value. Scheme 2: Use the identification information of the directory metadata as the key and the directory metadata as the value. The identification information of the directory metadata can be the node identifier of the inode, the directory, or the filename.
[0034] In the first scheme described above, the full path of the directory metadata is used as the key, allowing direct location of the queried directory metadata based on the full path during query operations. This results in fewer path resolutions and higher query efficiency. However, when directory changes are involved, such as renaming across directories, the keys corresponding to all affected paths need to be modified, which is extremely costly.
[0035] In Scheme 2, which uses the directory metadata's identifier information as the key, renaming operations only require modifying the keys of the target node and its parent node, resulting in high efficiency. However, Scheme 2 requires path parsing layer by layer according to the query path hierarchy, leading to higher path parsing overhead.
[0036] It is evident that both Scheme 1 and Scheme 2 described above suffer from inefficiency in directory metadata operations.
[0037] The directory metadata manipulation method, apparatus, electronic device, and readable storage medium provided in this disclosure are intended to solve at least one of the above-mentioned technical problems of the prior art.
[0038] Figure 1 A flowchart illustrating a directory metadata manipulation method provided in an embodiment of this disclosure is shown, such as... Figure 1 As shown, the method can mainly include:
[0039] Step S110: In response to receiving a directory query request, obtain the query path carried in the directory query request;
[0040] Step S120: Obtain the pre-built prefix directory data table and the pre-built full directory data table. The key of the prefix directory data table is the prefix path of the directory metadata, the prefix path is the sub-path of the path corresponding to the directory metadata at a preset number of levels, and the value of the prefix directory data table is the directory metadata. The key of the full directory data table is the identification information of the directory metadata, and the value of the full directory data table is the directory metadata.
[0041] Step S130: Determine the matching results between the query path and the keys of the prefix directory data table;
[0042] Step S140: Determine the query results based on the matching results and the full catalog data table.
[0043] In this embodiment of the disclosure, a prefix directory data table and a full directory data table can be pre-built. The full directory data table is a data table containing all directory metadata, constructed using the identification information of the directory metadata as the key and the directory metadata as the value. As an example, the identification information of the directory metadata may include node identifiers, directory names, or file names.
[0044] The prefix path is the sub-path corresponding to the level of the full path corresponding to the directory metadata, excluding the last preset level. As an example, the last preset level can be the last three levels. For example, if the full path corresponding to the directory metadata is " / a / b / c / d / e / f / g / h / i / j / k", its last three levels sub-path is "i / j / k". Removing this sub-path from the full path results in the prefix path " / a / b / c / d / e / f / g / h / ".
[0045] The prefix directory data table is a data table that contains partial directory metadata, constructed using the prefix path of the directory metadata as the key and the directory metadata as the value.
[0046] In this embodiment of the disclosure, by maintaining a prefix directory data table and a full directory data table, the operational efficiency of query operations can be effectively improved.
[0047] Specifically, when querying directory metadata based on the query path carried in the directory query request, the query path can be matched with the key of the prefix directory data table first to determine the matching result, and the query result can be determined based on the match and the full directory data table.
[0048] By using the prefix directory data table and the full directory data table, query operations can effectively reduce the number of path resolutions during the query process, thereby improving the efficiency of the query operation.
[0049] In this embodiment of the disclosure, since a full directory data table is maintained, when performing directory metadata change operations such as renaming and deletion, the key of the target node or the parent node of the target node can be directly modified in the full directory data table, resulting in high operation efficiency.
[0050] The solution provided in this disclosure uses a prefix directory data table and a full directory data table to perform directory metadata operations, which can balance improving the query efficiency of directory query operations with reducing the change overhead of directory change operations, effectively improving the efficiency of directory metadata operations and enhancing system performance.
[0051] The method provided in this disclosure, in response to a received directory query request, obtains the query path carried in the directory query request; obtains a pre-built prefix directory data table and a pre-built full directory data table, wherein the key of the prefix directory data table is the prefix path of the directory metadata, the prefix path is the sub-path corresponding to the level of the path corresponding to the directory metadata excluding the last preset level, and the value of the prefix directory data table is the directory metadata; the key of the full directory data table is the identification information of the directory metadata, and the value of the full directory data table is the directory metadata; determines the matching result between the query path and the key of the prefix directory data table; and determines the query result based on the matching result and the full directory data table. In this solution, directory metadata operations are performed using both the prefix directory data table and the full directory data table, which can effectively improve the efficiency of directory metadata operations.
[0052] This solution can be applied to directory tree management in distributed file systems. Directory tree operation requests (such as query requests) can be initiated by the client, and the directory tree server can respond to the directory tree operation requests.
[0053] In this embodiment, the number of levels in the reverse preset hierarchy can be set according to actual conditions. As an example, the number of levels in the reverse preset hierarchy is set to 3. Statistical analysis of directory trees with a large number of directories reveals that, generally, the last three levels of directories account for approximately 95% of the total number of directories, and correspondingly, directories before the last three levels account for approximately 5% of the total number of directories. Therefore, using the prefix subpaths before the last three levels as keys to construct the prefix directory data table incurs relatively low data overhead, and the renaming operation of directories before the last three levels generally accounts for a low percentage, eliminating the need for frequent modifications to the prefix directory data table.
[0054] In one optional approach of this disclosure, determining the matching result between the query path and the key of the prefix directory data table includes:
[0055] Determine whether the prefix path in the key of the prefix directory data table contains the target prefix path. The target prefix path is the sub-path corresponding to the level in the query path, excluding the last preset level.
[0056] If the prefix path contains the target prefix path, the matching result is determined to be a successful match;
[0057] If the target prefix path is not contained within the prefix path, the matching result is determined to be unsuccessful.
[0058] In this embodiment of the disclosure, a target prefix path can be extracted from the query path. The target prefix path is the sub-path corresponding to the level in the query path other than the last preset level.
[0059] As an example, the pre-defined countdown level can be the last three levels. For instance, if the query path is " / a / b / c / d / e / f / ", its last three levels of sub-paths are "d / e / f". Removing this sub-path from the full path gives the target prefix path " / a / b / c / ".
[0060] The target prefix path can be matched against the keys in the prefix directory data table to determine if the target prefix path exists in the prefix paths list. If it exists, the match is successful; otherwise, the match is unsuccessful.
[0061] In one optional approach of this disclosure, in response to a successful match, the query result is determined based on the match result and the full catalog data table, including:
[0062] Determine the prefix directory metadata corresponding to the target prefix path;
[0063] The query results are determined by sequentially querying the metadata of each level of the directory below the prefix directory metadata in the query path from the full directory data table.
[0064] In this embodiment of the disclosure, when the matching result is a successful match, that is, when the target prefix path exists in the prefix directory data table, the directory metadata corresponding to the target prefix path is recorded as the prefix directory metadata. A successful match indicates that the prefix directory metadata exists in the full set of directory metadata.
[0065] Prefix directory metadata can be understood as the directory metadata of the preceding preset levels of the directory metadata to be queried. Once the prefix directory metadata is identified, the query path can be used to search the full directory data table layer by layer, starting from the next level of the prefix directory metadata, using the identifier information of the corresponding level of directory metadata. This process continues until the identifier information of a certain level of directory metadata is not found, or the identifier information of the last level of directory metadata in the query path is found. If the identifier information of a certain level of directory metadata is not found, the query result can be considered a query failure. If the identifier information of the last level of directory metadata in the query path is found, the query is considered successful, and the directory metadata of that last level can be retrieved from the full directory data table as the query result.
[0066] As an example, the pre-defined countdown level can be the last three levels, resulting in a total of ten levels in the query path. If the required directory metadata exists in the full directory data table, directly querying each level in the full directory data table would require ten query operations, involving ten path parsings. However, the solution provided in this embodiment only requires first parsing the target prefix path, querying the prefix directory data table, and then parsing each sub-path after the target prefix path in the query path, querying each level in the full directory data table—that is, involving three path parsings. Therefore, the solution provided in this embodiment only requires a total of four path parsings and four queries, significantly improving query efficiency and system performance compared to the ten path parsings and ten queries performed directly in the full directory data table.
[0067] In one optional approach of this disclosure, in response to a no-match result, the query result is determined based on the matching result and the full catalog data table, including:
[0068] The full catalog data table is queried based on the query path to determine the query results.
[0069] In this embodiment of the disclosure, if the matching result is unsuccessful, meaning the target prefix path does not exist in the prefix directory data table, then if the prefix directory data table contains all prefix paths, the query result can be determined as a query failure. However, if the prefix directory data table does not contain all prefix paths, meaning the information in the prefix directory data table is incomplete, a query can be performed on the full directory data table based on the query path.
[0070] Specifically, the query path contains identifiers for the metadata of each level of the directory. Starting from the first level under the root level of the query path, the system queries the full directory data table level by level using the identifiers of the corresponding level's metadata. It checks if the same identifier exists in the keys of the full directory data table until no identifier for a particular level of directory metadata is found, or until the identifier for the last level of directory metadata in the query path is found. If no identifier for a particular level of directory metadata is found, the query is considered a failure. If the identifier for the last level of directory metadata in the query path is found, the query is considered successful, and the metadata for that last level can be retrieved from the full directory data table as the query result.
[0071] In one optional embodiment of this disclosure, after determining the query results, the method further includes:
[0072] In response to a no-match result and a query result indicating that the directory metadata corresponding to the query path was found, a corresponding entry is created in the prefix directory data table based on the query result.
[0073] In this embodiment of the disclosure, when the matching result is unsuccessful and the query result is that the directory metadata corresponding to the query path is found, it means that the queried directory metadata exists in the full amount of directory metadata, but the prefix path of the path corresponding to the directory metadata does not exist in the prefix directory data table. At this time, a corresponding entry can be created in the prefix directory data table to realize the addition of data in the prefix directory data table.
[0074] Specifically, a prefix path can be extracted from the path corresponding to the queried directory metadata. The extracted prefix path is used as the key, and the directory metadata corresponding to the prefix path is used as the value. A corresponding entry is then created in the prefix directory data table.
[0075] In this embodiment of the disclosure, updating the prefix directory data table based on the query results can greatly reduce the overhead of creating the prefix directory data table.
[0076] In one alternative embodiment of this disclosure, the method further includes:
[0077] The pre-built prefix tree is updated based on the query results. The prefix tree is a subtree of the directory tree corresponding to the directory metadata in the prefix directory data table.
[0078] In this embodiment of the disclosure, a prefix tree can also be maintained. The prefix tree can be understood as a subtree of the directory tree corresponding to all directory metadata in the prefix directory data table.
[0079] Prefix trees can be used to manage the relationships between metadata of each directory in a prefix directory data table, enabling effective maintenance of the prefix directory data table.
[0080] In one optional approach of this disclosure, determining the matching result between the query path and the key of the prefix directory data table includes:
[0081] Retrieve the pre-built directory metadata change record table;
[0082] Determine if a directory change record corresponding to the query path exists in the directory metadata change record table;
[0083] If a directory change record corresponding to the query path exists in the directory metadata change record table, the matching result between the query path and the key in the prefix directory data table is determined to be a failure.
[0084] In this embodiment of the disclosure, a directory metadata change record table can also be maintained, which stores multiple directory change records for directory metadata. The directory change records in the directory metadata change record table include directory renaming records and directory deletion records.
[0085] In this embodiment of the disclosure, when matching the query path with the key of the prefix directory data table, it can be done based on the directory metadata change record table.
[0086] Specifically, the query path can be matched against the directory change record table to determine if there are any change records for the metadata of each level of the queried directory. If so, it indicates that the part of the prefix directory data table corresponding to the query path has changed, and the prefix directory data table is no longer reliable. In this case, the matching result between the query path and the key in the prefix directory data table can be directly determined as a failure. In this situation, after retrieving the directory metadata in the full directory data table based on the query path, the prefix directory data table can be updated to complete the prefix directory data table.
[0087] In this embodiment of the disclosure, all levels of sub-paths of the query path can be determined, and then all levels of sub-paths of the query path can be matched in the directory change record table.
[0088] In this embodiment of the disclosure, after obtaining the query path, when matching the query path with the key of the prefix directory data table, the sub-paths of all levels of the query path can be matched in the directory change record table first. If there is a change record of any level of sub-path in the query path in the directory change record table, the matching result of the query path with the key of the prefix directory data table can be determined as a match failure, thereby quickly determining the matching result and improving processing efficiency.
[0089] In one alternative embodiment of this disclosure, the above also includes:
[0090] In response to receiving a command to delete invalid entries in the prefix directory data table, retrieve directory change records from the directory metadata change record table;
[0091] Determine the changed directory metadata based on the directory change records;
[0092] Determine the associated paths corresponding to the changed directory metadata;
[0093] Delete the changed entries in the prefix directory data table. The prefix path corresponding to the changed entry is the same as the prefix path of the associated path.
[0094] In this embodiment of the disclosure, the invalid entry deletion command of the prefix directory data table is used to instruct the deletion of invalid entries in the prefix directory data table to ensure the validity of the prefix directory data table. The invalid entry deletion command can be initiated at preset intervals, or after detecting that the number of records in the directory metadata change record table has reached a preset value.
[0095] Specifically, all directory change records can be read from the directory metadata change record table, and then the changed directory metadata corresponding to the directory change record can be determined. The associated path of the changed directory metadata is all paths affected by the changed directory metadata. As an example, the associated path of the changed directory metadata is the path that overlaps with the path of the changed directory metadata.
[0096] After identifying the associated paths, the prefix paths of these paths can be extracted. Entries in the prefix directory data table whose prefix paths match the prefix paths of the associated paths are identified as changed entries. These changed entries are then deleted from the prefix directory data table. After deleting the invalid entries from the prefix directory data table, the directory change records in the directory metadata change record table can be deleted.
[0097] In one optional approach disclosed herein, determining the associated path corresponding to the changed directory metadata includes:
[0098] Find the associated path corresponding to the changed directory metadata in the prefix tree.
[0099] In this embodiment of the disclosure, the prefix tree contains the relationship between the metadata of each directory in the prefix directory data table, so the associated path corresponding to the changed directory metadata can be quickly found from the prefix tree.
[0100] In one alternative embodiment of this disclosure, the prefix directory data table, the directory metadata change record table, and the prefix tree are all stored in a cache.
[0101] In this embodiment of the disclosure, the prefix directory data table, the directory metadata change record table, and the prefix tree are all stored in the cache, which can improve access speed.
[0102] The prefix directory data table, directory metadata change record table, and prefix tree can be generated based on user queries of directory metadata each time the system starts, and updated based on user changes to directory metadata. Since the prefix directory data table, directory metadata change record table, and prefix tree are all updated in real time, their consistency with the full directory data table can be effectively maintained without the need for additional resources for consistency maintenance.
[0103] In this embodiment of the disclosure, the full catalog data table can be persistently stored in a database.
[0104] As an example, Figure 2 This is a schematic diagram of the data structure involved in the solution provided in the embodiments of this disclosure.
[0105] like Figure 2 As shown, Directory Remove History 210, i.e., directory change history, is equivalent to the aforementioned directory metadata change record table. Cache Invalidator 220, i.e., invalidation cache, is equivalent to the changed directory metadata matched from the directory metadata change record table. Path Cache Map 230, i.e., path cache data table, is equivalent to the aforementioned prefix directory data table. In this example, the prefix directory data table stores three entries: key " / A", value "pid: 101"; key " / A / C", value "pid: 102"; key " / B", value "pid: 107". Prefix Tree 240, i.e., prefix tree. MemoryHash KV Store, i.e., hash key-value database, is equivalent to the aforementioned full directory data table.
[0106] Lookup refers to the query operation. "Check if the cache uncertain" checks the cache's reliability, essentially determining if the query path and its subpaths contain a directory metadata change record table. If it exists, it indicates the queried directory data has undergone changes, and the cached prefix directory data table is unreliable. If the query path and its subpaths do not contain a directory metadata change record table, it indicates the queried directory data has not changed, and the cached prefix directory data table is reliable. "Get remove directory" retrieves the changed directories, essentially determining the changed directory metadata from the directory metadata change record table. "Invalidate related cache" invalidates related cache entries, effectively disabling entries associated with changed directory metadata. "Build cache after cache miss" rebuilds the cache when it is lost. In this case, the prefix directory data table, directory metadata change record table, and prefix tree can all be stored in the cache and recreated after a system restart.
[0107] As an example, Figure 3 This is a flowchart illustrating a specific implementation of the method provided in this disclosure.
[0108] like Figure 3As shown, Full Path Cache History 310, or full path storage, is equivalent to the system cache, including a directory metadata change record table, a prefix tree, and a subtree. Prefix Tree 320 is the prefix tree. In this example, the prefix tree consists of nodes a, b, c, d, and g. Directory Remove History 340 is the directory change history, equivalent to the aforementioned directory metadata change record table. In this example, the directory metadata change record table includes change records for the directory " / a / b / d". Path table 330 is the path data table, equivalent to the aforementioned prefix directory data table. In this example, the prefix directory data table stores five entries with the keys: key " / a", key " / a / b / d", key " / a / b / d / d", and key " / a / d". DB Dir Tree Table is the directory tree data table database, equivalent to the aforementioned full directory tree data table. In this example, the full directory data table stores five entries, whose keys consist of the parent inode identifier (parentinode id), the inode name (name), and the inode number (inode). The parent inode ids are 0, 1, 2, 3…; the names are / a, b, c…; and the inodes are 1, 2, 3, 4…
[0109] When a lookup request is received from a client, the lookup path "a / b / c / h / i / " is parsed. Step S301: "get inode of "a / b / c" from cache" retrieves "a / b / c" from the cache. Here, "a / b / c" is equivalent to the prefix path of the lookup path "a / b / c / h / i / ". The directory metadata change record table is checked to see if there are any change records related to "a / b / c". In this example, it can be determined that there are no related change records in the directory metadata change record table. Then, the prefix path "a / b / c" is queried in the prefix directory data table, and it is found that "a / b / c" exists in the prefix directory data table.
[0110] Step S302, get inode of h, i, from db, that is, query "h / ic" from the database. After querying "a / b / c" from the prefix directory data table, we can obtain the identifier h, i in the subpath " / h / i / " other than the prefix path "a / b / c" of the query path "a / b / c / h / i / ", and query h, i in the full directory data table to get the query result.
[0111] As an example, Figure 4This is a flowchart illustrating a specific implementation of the method provided in this disclosure.
[0112] like Figure 3 As shown, Full Path Cache History 410, or full path storage, is equivalent to the system cache, including a directory metadata change record table, a prefix tree, and a subtree. Prefix Tree 420 is the prefix tree. In this example, the prefix tree consists of nodes a, b, c, d, and g. Directory Remove History 440 is the directory change history, equivalent to the aforementioned directory metadata change record table. In this example, the directory metadata change record table includes change records for the directory " / a / b / d". Path table 430 is the path data table, equivalent to the aforementioned prefix directory data table. In this example, the prefix directory data table stores five entries with the keys: key " / a", key " / a / b / d", key " / a / b / d / d", and key " / a / d". DB Dir Tree Table is the directory tree data table database, equivalent to the aforementioned full directory tree data table. In this example, the full directory data table stores five entries, whose keys consist of the parent inode identifier (parentinode id), the inode name (name), and the inode number (inode). The parent inode ids are 0, 1, 2, 3…; the names are / a, b, c…; and the inodes are 1, 2, 3, 4…
[0113] When a lookup request is received from a client, the lookup path "a / b / d / e / f / " is parsed. Step S401: "get inode of "a / b / d" from cache" retrieves "a / b / d" from the cache. Here, "a / b / d" is equivalent to the prefix path of the lookup path "a / b / d / e / f / ". The directory metadata change record table is checked to see if there are any change records related to "a / b / d". In this example, the change record for "a / b / d" in the directory metadata change record table can be determined. At this point, the prefix directory data table is no longer reliable.
[0114] Step S402, get inode of a, b, d, e, f, from db, that is, query a, b, d, e, f from the database to get the query results.
[0115] As an example, Figure 5 This is a flowchart illustrating the process of deleting invalid entries in the prefix directory data table in the method provided in this embodiment of the disclosure.
[0116] like Figure 5 As shown, Full Path Cache History 510, or full path storage, is equivalent to the system cache, including a directory metadata change record table, a prefix tree, and a prefix tree. Prefix Tree 520 is the prefix tree. In this example, the prefix tree consists of nodes a, b, c, d, and g. Directory Remove History 540 is the directory change history, equivalent to the aforementioned directory metadata change record table. In this example, the directory metadata change record table includes change records for the directory " / a / b / d". Path table 530 is the path data table, equivalent to the aforementioned prefix directory data table. In this example, the prefix directory data table stores five entries with the keys: key " / a", key " / a / b / d", key " / a / b / d / d", and key " / a / d".
[0117] When a rename request is received from a client, the system parses out the original directory " / a / b / d / " and the new directory " / a / b / e / ". After completing the renaming operation, the change record for the directory " / a / b / d / " is stored in the directory metadata change record table.
[0118] Step S501: delete / a / b / d / *in cache, which deletes the directory " / a / b / d / " and its subdirectories from the cache. This is equivalent to performing the operation of deleting the prefix directory data table change entry when executing the invalid entry deletion command.
[0119] Step S502: Async delete table by tree, which means asynchronously deleting entries from the prefix table based on the prefix tree. In this example, the prefix tree shows that the subdirectory of the directory " / a / b / d / " is " / a / b / d / g". Then, the entries with the key " / a / b / d / " and the entries with the key " / a / b / d / g" can be deleted from the prefix directory data table.
[0120] Based on and Figure 1 The method shown follows the same principle. Figure 6 A schematic diagram of the structure of a directory metadata manipulation device provided in an embodiment of this disclosure is shown, such as... Figure 6 As shown, the directory metadata manipulation device 60 may include:
[0121] The directory query request receiving module 610 is used to respond to the received directory query request and obtain the query path carried in the directory query request;
[0122] The data table acquisition module 620 is used to acquire a pre-built prefix directory data table and a pre-built full directory data table. The key of the prefix directory data table is the prefix path of the directory metadata, and the prefix path is the sub-path corresponding to the level of the path corresponding to the directory metadata except for the last preset level. The value of the prefix directory data table is the directory metadata. The key of the full directory data table is the identification information of the directory metadata, and the value of the full directory data table is the directory metadata.
[0123] The prefix matching module 630 is used to determine the matching results between the query path and the keys of the prefix directory data table;
[0124] The query result determination module 640 is used to determine the query results based on the matching results and the full catalog data table.
[0125] The apparatus provided in this embodiment, in response to a received directory query request, obtains the query path carried in the directory query request; obtains a pre-built prefix directory data table and a pre-built full directory data table, wherein the key of the prefix directory data table is the prefix path of the directory metadata, the prefix path is the sub-path corresponding to the level of the path corresponding to the directory metadata excluding the last preset level, and the value of the prefix directory data table is the directory metadata; the key of the full directory data table is the identification information of the directory metadata, and the value of the full directory data table is the directory metadata; determines the matching result between the query path and the key of the prefix directory data table; and determines the query result based on the matching result and the full directory data table. In this solution, directory metadata operations are performed using both the prefix directory data table and the full directory data table, which effectively improves the efficiency of directory metadata operations.
[0126] Optionally, the prefix matching module is specifically used for:
[0127] Determine whether the prefix path in the key of the prefix directory data table contains the target prefix path. The target prefix path is the sub-path corresponding to the level in the query path, excluding the last preset level.
[0128] If the prefix path contains the target prefix path, the matching result is determined to be a successful match;
[0129] If the target prefix path is not contained within the prefix path, the matching result is determined to be unsuccessful.
[0130] Optionally, the query result determination module is specifically used for:
[0131] In response to a successful match, determine the prefix directory metadata corresponding to the target prefix path;
[0132] The query results are determined by sequentially querying the metadata of each level of the directory below the prefix directory metadata in the query path from the full directory data table.
[0133] Optionally, the query result determination module is specifically used for:
[0134] If the matching result is unsuccessful, the entire catalog data table is queried based on the query path to determine the query result.
[0135] Optionally, the above-mentioned device further includes:
[0136] The prefix directory data table entry creation module is used to create corresponding entries in the prefix directory data table based on the query results after determining the query results. If the matching result is unsuccessful and the query result is that the directory metadata corresponding to the query path was found, the module creates corresponding entries in the prefix directory data table based on the query results.
[0137] Optionally, the above-mentioned device further includes:
[0138] The prefix tree update module is used to update the pre-built prefix tree based on the query results. The prefix tree is a subtree of the directory tree corresponding to the directory metadata in the prefix directory data table.
[0139] Optionally, the query result determination module is specifically used for:
[0140] Retrieve the pre-built directory metadata change record table;
[0141] Determine if a directory change record corresponding to the query path exists in the directory metadata change record table;
[0142] If a directory change record corresponding to the query path exists in the directory metadata change record table, the matching result between the query path and the key in the prefix directory data table is determined to be a failure.
[0143] Optionally, the directory change records in the directory metadata change record table include directory renaming records and directory deletion records.
[0144] Optionally, the above apparatus further includes a prefix directory data table entry deletion module, which is used for:
[0145] In response to receiving a command to delete invalid entries in the prefix directory data table, retrieve directory change records from the directory metadata change record table;
[0146] Determine the changed directory metadata based on the directory change records;
[0147] Determine the associated paths corresponding to the changed directory metadata;
[0148] Delete the changed entries in the prefix directory data table. The prefix path corresponding to the changed entry is the same as the prefix path of the associated path.
[0149] Optionally, the prefix directory data table entry deletion module, when determining the associated path corresponding to the changed directory metadata, is specifically used for:
[0150] Find the associated path corresponding to the changed directory metadata in the prefix tree.
[0151] Optionally, the prefix directory data table, the directory metadata change record table, and the prefix tree are all stored in the cache.
[0152] It is understood that the above-described modules of the directory metadata operation device in the embodiments of this disclosure have the ability to implement... Figure 1 The illustrated embodiment demonstrates the functionality of corresponding steps in the directory metadata manipulation method. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions. These modules can be software and / or hardware, and each module can be implemented individually or integrated from multiple modules. For a detailed description of the functions of each module in the aforementioned directory metadata manipulation device, please refer to [link to relevant documentation]. Figure 1 The corresponding descriptions of the directory metadata operation methods in the illustrated embodiments will not be repeated here.
[0153] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0154] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0155] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a directory metadata manipulation method as provided in the embodiments of this disclosure.
[0156] Compared with existing technologies, this electronic device, in response to a received directory query request, obtains the query path carried in the query request; obtains a pre-built prefix directory data table and a pre-built full directory data table. The keys of the prefix directory data table are the prefix paths of the directory metadata, and the prefix paths are the sub-paths corresponding to the levels other than the last preset level in the path corresponding to the directory metadata. The values of the prefix directory data table are the directory metadata. The keys of the full directory data table are the identification information of the directory metadata, and the values of the full directory data table are the directory metadata. The device then determines the matching result between the query path and the keys of the prefix directory data table; and based on the matching result and the full directory data table, determines the query result. This solution effectively improves the efficiency of directory metadata operations by using both prefix and full directory data tables for directory metadata operations.
[0157] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform directory metadata operation methods as provided in the embodiments of this disclosure.
[0158] Compared with existing technologies, this readable storage medium, in response to a received directory query request, obtains the query path carried in the directory query request; obtains a pre-built prefix directory data table and a pre-built full directory data table, wherein the key of the prefix directory data table is the prefix path of the directory metadata, the prefix path is the sub-path corresponding to the path of the directory metadata except for the last preset level, and the value of the prefix directory data table is the directory metadata; the key of the full directory data table is the identification information of the directory metadata, and the value of the full directory data table is the directory metadata; determines the matching result between the query path and the key of the prefix directory data table; and determines the query result based on the matching result and the full directory data table. In this solution, directory metadata operations are performed using the prefix directory data table and the full directory data table, which can effectively improve the efficiency of directory metadata operations.
[0159] The computer program product includes a computer program that, when executed by a processor, implements the directory metadata manipulation method as provided in the embodiments of this disclosure.
[0160] Compared with existing technologies, this computer program product, in response to a received directory query request, obtains the query path carried in the directory query request; obtains a pre-built prefix directory data table and a pre-built full directory data table, wherein the key of the prefix directory data table is the prefix path of the directory metadata, the prefix path is the sub-path corresponding to the level of the path corresponding to the directory metadata excluding the last preset level, and the value of the prefix directory data table is the directory metadata; the key of the full directory data table is the identification information of the directory metadata, and the value of the full directory data table is the directory metadata; determines the matching result between the query path and the key of the prefix directory data table; and determines the query result based on the matching result and the full directory data table. In this solution, directory metadata operations are performed using the prefix directory data table and the full directory data table, which can effectively improve the efficiency of directory metadata operations.
[0161] Figure 7 A schematic block diagram of an example electronic device 70 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0162] like Figure 7 As shown, the electronic device 70 includes a computing unit 710, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 720 or a computer program loaded from a storage unit 780 into a random access memory (RAM) 730. The RAM 730 may also store various programs and data required for the operation of the device 70. The computing unit 710, ROM 720, and RAM 730 are interconnected via a bus 740. An input / output (I / O) interface 750 is also connected to the bus 740.
[0163] Multiple components in device 70 are connected to I / O interface 750, including: input unit 760, such as keyboard, mouse, etc.; output unit 770, such as various types of monitors, speakers, etc.; storage unit 780, such as disk, optical disk, etc.; and communication unit 790, such as network card, modem, wireless transceiver, etc. Communication unit 790 allows device 70 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0164] The computing unit 710 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 710 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 710 executes the directory metadata manipulation methods provided in the embodiments of this disclosure. For example, in some embodiments, executing the directory metadata manipulation methods provided in the embodiments of this disclosure can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 780. In some embodiments, part or all of the computer program can be loaded and / or installed on device 70 via ROM 720 and / or communication unit 790. When the computer program is loaded into RAM 730 and executed by computing unit 710, one or more steps of the directory metadata manipulation methods provided in the embodiments of this disclosure can be performed. Alternatively, in other embodiments, the computing unit 710 can be configured to execute the directory metadata manipulation methods provided in the embodiments of this disclosure by any other suitable means (e.g., by means of firmware).
[0165] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0166] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0167] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0168] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0169] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0170] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.
[0171] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0172] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A method for manipulating directory metadata, comprising: In response to receiving a directory query request, obtain the query path carried in the directory query request; Obtain a pre-built prefix directory data table and a pre-built full directory data table, wherein the key of the prefix directory data table is the prefix path of the directory metadata, the prefix path is the sub-path corresponding to the level of the path corresponding to the directory metadata except for the last preset level, the value of the prefix directory data table is the directory metadata, the key of the full directory data table is the identification information of the directory metadata, and the value of the full directory data table is the directory metadata. Determine whether the prefix path in the key of the prefix directory data table contains the target prefix path, wherein the target prefix path is the sub-path corresponding to the level in the query path other than the inverse preset level; If the target prefix path is contained within the prefix path, the matching result between the query path and the key of the prefix directory data table is determined to be a successful match; if the target prefix path is not contained within the prefix path, the matching result is determined to be a failed match. Obtain a pre-built directory metadata change record table; determine whether there is a directory change record corresponding to the query path in the directory metadata change record table; in response to the existence of a directory change record corresponding to the query path in the directory metadata change record table, determine that the matching result between the query path and the key of the prefix directory data table is unsuccessful; In response to a successful match, the metadata of the prefix directory corresponding to the target prefix path is determined; the metadata of each level of directory below the prefix directory metadata in the query path is sequentially queried from the full directory data table to determine the query result.
2. The method according to claim 1, wherein, In response to the matching result being unsuccessful, the method further includes: The full catalog data table is queried based on the query path to determine the query results.
3. The method according to claim 2, further comprising, after determining the query result: In response to the matching result being unsuccessful, and the query result being that the directory metadata corresponding to the query path was found, a corresponding entry is created in the prefix directory data table based on the query result.
4. The method according to claim 3, further comprising: The pre-constructed prefix tree is updated based on the query results. The prefix tree is a subtree of the directory tree corresponding to the directory metadata in the prefix directory data table.
5. The method according to claim 1, wherein the directory change records in the directory metadata change record table include directory renaming records and directory deletion records.
6. The method according to claim 4, further comprising: In response to receiving a command to delete invalid entries in the prefix directory data table, retrieve directory change records from the directory metadata change record table; Based on the directory change records, determine the changed directory metadata that has undergone change operations; Determine the associated path corresponding to the changed directory metadata; Delete the changed entries in the prefix directory data table, where the prefix path corresponding to the changed entry is the same as the prefix path of the associated path.
7. The method according to claim 6, wherein, Determining the associated path corresponding to the changed directory metadata includes: Find the associated path corresponding to the changed directory metadata from the prefix tree.
8. The method according to claim 4, wherein, The prefix directory data table, the directory metadata change record table, and the prefix tree are all stored in the cache.
9. A directory metadata manipulation apparatus, comprising: The directory query request receiving module is used to respond to receiving a directory query request and obtain the query path carried in the directory query request; The data table acquisition module is used to acquire a pre-built prefix directory data table and a pre-built full directory data table. The key of the prefix directory data table is the prefix path of the directory metadata, and the prefix path is the sub-path corresponding to the level of the path corresponding to the directory metadata, excluding the last preset level. The value of the prefix directory data table is the directory metadata. The key of the full directory data table is the identification information of the directory metadata, and the value of the full directory data table is the directory metadata. The prefix matching module is used to determine the matching result between the query path and the key of the prefix directory data table; The query result determination module is used to determine the query result based on the matching result and the full catalog data table; The prefix matching module is specifically used for: Determine whether the prefix path in the key of the prefix directory data table contains the target prefix path, wherein the target prefix path is the sub-path corresponding to the level in the query path other than the inverse preset level; If the target prefix path is contained within the prefix path, then the matching result is determined to be a successful match; If the target prefix path is not contained within the prefix path, the matching result is determined to be unsuccessful. The query result determination module is specifically used for: In response to the matching result being successful, the prefix directory metadata corresponding to the target prefix path is determined; The query results are determined by sequentially querying the metadata of each level of the directory below the prefix directory metadata in the query path from the full directory data table. The query result determination module is specifically used for: Retrieve the pre-built directory metadata change record table; Determine whether a directory change record corresponding to the query path exists in the directory metadata change record table; In response to the existence of a directory change record corresponding to the query path in the directory metadata change record table, the matching result of the query path and the key in the prefix directory data table is determined to be unsuccessful.
10. The apparatus according to claim 9, wherein, The query result determination module is specifically used for: In response to the matching result being unsuccessful, the full catalog data table is queried based on the query path to determine the query result.
11. The apparatus of claim 10, further comprising: The prefix directory data table entry creation module is used to create corresponding entries in the prefix directory data table based on the query result after the query result is determined, in response to the matching result being unsuccessful and the query result being that the directory metadata corresponding to the query path was found.
12. The apparatus of claim 11, further comprising: The prefix tree update module is used to update the pre-built prefix tree based on the query result. The prefix tree is a subtree of the directory tree corresponding to the directory metadata in the prefix directory data table.
13. The apparatus according to claim 9, wherein the directory change records in the directory metadata change record table include directory renaming records and directory deletion records.
14. The apparatus of claim 12, further comprising a prefix directory data table entry deletion module, the prefix directory data table entry deletion module being used to: In response to receiving a command to delete invalid entries in the prefix directory data table, retrieve directory change records from the directory metadata change record table; Based on the directory change records, determine the changed directory metadata that has undergone change operations; Determine the associated path corresponding to the changed directory metadata; Delete the changed entries in the prefix directory data table, where the prefix path corresponding to the changed entry is the same as the prefix path of the associated path.
15. The apparatus according to claim 14, wherein, When determining the associated path corresponding to the changed directory metadata, the prefix directory data table entry deletion module is specifically used for: Find the associated path corresponding to the changed directory metadata from the prefix tree.
16. The apparatus according to claim 12, wherein, The prefix directory data table, the directory metadata change record table, and the prefix tree are all stored in the cache.
17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.
18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.
19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.