Method and device for acquiring metadata in file system, equipment and storage medium

Abstract

The embodiment of the invention provides a method and device for obtaining metadata in a file system, equipment and a storage medium, and the method comprises the steps: constructing a metadata index node corresponding to the file system according to file paths of a plurality of files in the file system; analyzing a file path of the target file to obtain a target path section with a hierarchical structure; the target path segment comprises a termination path segment located at the last level; according to the hierarchical structure, traversing the index node corresponding to the target path segment in the metadata index nodes, and determining the index node corresponding to the termination path segment as a target node; and extracting metadata corresponding to the target file from the target node. According to the embodiment of the invention, the metadata can be quickly positioned by constructing the metadata index node, and the metadata index node enables the metadata storage structure of the file system to be convenient for subsequent query and management; moreover, the metadata is acquired based on the metadata index node, so that the time complexity of acquiring the metadata is reduced, and the overall performance of the file system is improved.

Description

Technical Field

[0001] The present application relates to the field of computer storage technology, and in particular to a method and device for obtaining metadata in a file system, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of information technology, the explosive growth of data has put forward higher requirements on the performance of file systems. In today's file system design, metadata index, as an important component of the file system, is directly related to the efficiency of retrieval, positioning and operation of files and specific metadata. However, with the continuous expansion and diversification of storage systems, file systems need to manage a large number of files and metadata more efficiently to ensure the efficiency of retrieval, positioning and operation of files and metadata.

[0003] However, existing metadata indexes often require a large storage space when storing a large number of files, resulting in a waste of storage resources. In addition, in the file system, metadata retrieval, positioning and operation are high-frequency operations. When processing a large number of files, the existing metadata index management method often cannot meet the requirements of real-time and efficiency.

[0004] Obviously, the existing metadata management solutions have deficiencies in space utilization efficiency and operational performance, which will affect the efficiency of obtaining metadata in the file system, and thus lead to a decline in the overall performance of the file system. Therefore, how to improve the efficiency of obtaining metadata in the file system and thus improve the overall performance of the file system is a problem that needs to be urgently solved in this field. Summary of the invention

[0005] The embodiment of the present application provides a method for acquiring metadata in a file system to solve the problem of low efficiency in acquiring metadata in a file system due to deficiencies in a metadata management solution.

[0006] Correspondingly, an embodiment of the present application also provides a device for acquiring metadata in a file system, an electronic device, and a storage medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problem, an embodiment of the present application discloses a method for obtaining metadata in a file system, the method comprising:

[0008] Constructing a metadata index node corresponding to the file system according to the file paths of the plurality of files in the file system;

[0009] Parsing the file path of the target file to obtain a target path segment having a hierarchical structure; the target path segment includes a termination path segment located at the last level;

[0010] According to the hierarchical structure, traverse the index nodes corresponding to the target path segment in the metadata index nodes, and determine the index node corresponding to the termination path segment as the target node;

[0011] The metadata corresponding to the target file is extracted in the target node.

[0012] Optionally, after extracting the metadata corresponding to the target file in the target node, the method further includes:

[0013] Monitoring the incremental log library in the file system; the incremental log library includes a plurality of incremental logs for recording operation behaviors corresponding to the files;

[0014] If the number of the incremental logs reaches a preset number, encoding the metadata in the file system to obtain encoded data;

[0015] The encoded data is persistently stored.

[0016] Optionally, before persistently storing the encoded data, the method further includes:

[0017] Create a temporary data index node;

[0018] If multiple files in the file system need to be modified, it is implemented based on the temporary data index node;

[0019] If multiple files in the file system need to be searched, it is implemented based on the metadata index node and the temporary data index node.

[0020] Optionally, after the encoding data is persistently stored, the method further includes:

[0021] If the metadata index node fails, determining the log sequence numbers corresponding to the multiple incremental logs respectively;

[0022] The multiple incremental logs are executed in sequence according to the log sequence numbers respectively corresponding to the multiple incremental logs, and the metadata in the metadata index node is restored.

[0023] Optionally, the operation behavior includes a new addition behavior, and after extracting metadata corresponding to the target file in the target node, the method further includes:

[0024] When a first target file is newly created in the file system, a first file path corresponding to the first target file is parsed to obtain a first path segment having a first hierarchical structure;

[0025] According to the first hierarchical structure, traversing the metadata index nodes to see whether there is an index node corresponding to the first path segment;

[0026] Determine the first path segment corresponding to which no index node can be found in the metadata index node as the path segment to be inserted;

[0027] Creating a node to be inserted corresponding to the path segment to be inserted, and inserting the node to be inserted into the metadata index node;

[0028] Storing the metadata corresponding to the first target file in the node to be inserted in the metadata index node;

[0029] A first incremental log is generated according to the newly added behavior corresponding to the first target file, and is stored in the incremental log library.

[0030] Optionally, the operation behavior includes a deletion behavior, and after extracting the metadata corresponding to the target file in the target node, the method further includes:

[0031] When a second target file is deleted in the file system, a second file path corresponding to the second target file is parsed to obtain a second path segment having a second hierarchical structure; the second path segment includes a second termination path segment located at the last level;

[0032] According to the second hierarchical structure, traverse the index nodes corresponding to the target path segment in the metadata index nodes;

[0033] Determine the index node corresponding to the second terminal path segment as the node to be deleted;

[0034] Deleting the node to be deleted and metadata corresponding to the second target file in the node to be deleted;

[0035] A second incremental log is generated according to the deletion behavior corresponding to the second target file, and is stored in the incremental log library.

[0036] Optionally, the index node includes a root node and a child node corresponding to the root node, the target path segment also includes a starting path segment located at a first level, and traversing the index node corresponding to the target path segment in the metadata index node according to the hierarchical structure includes:

[0037] Determine a target root node among the root nodes according to the starting path segment;

[0038] According to the hierarchical structure, the sub-nodes corresponding to the target path segment are traversed in the sub-nodes corresponding to the target root node.

[0039] The embodiment of the present application also discloses a device for obtaining metadata in a file system, the device comprising:

[0040] A metadata index building module, used to build a metadata index node corresponding to the file system according to the file paths of multiple files in the file system;

[0041] A file path parsing module, used for parsing the file path of the target file to obtain a target path segment having a hierarchical structure; the target path segment includes a termination path segment located at the last level;

[0042] An index node traversal module, configured to traverse the index nodes corresponding to the target path segment in the metadata index nodes according to the hierarchical structure, and determine the index node corresponding to the termination path segment as the target node;

[0043] The metadata acquisition module is used to extract metadata corresponding to the target file in the target node.

[0044] An embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the method for obtaining metadata in the file system as described in any one of the embodiments of the present application.

[0045] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, a processor executes a method for obtaining metadata in a file system as described in any one of the embodiments of the present application.

[0046] Compared with the prior art, the embodiments of the present application have the following advantages:

[0047] In an embodiment of the present application, a metadata index node corresponding to the file system is constructed according to the file paths of multiple files in the file system; the file path of the target file is parsed to obtain a target path segment with a hierarchical structure; the target path segment includes a terminal path segment located at the last level; according to the hierarchical structure, the index nodes corresponding to the target path segment are traversed in the metadata index node, and the index node corresponding to the terminal path segment is determined as the target node; and the metadata corresponding to the target file is extracted from the target node. The embodiment of the present application can quickly locate the metadata of the target file by constructing the metadata index node corresponding to the file system. The construction of the metadata index node also makes the metadata storage structure of the file system more orderly and efficient, which is convenient for subsequent query and management; and, by parsing the file path of the target file and decomposing it into a target path segment with a hierarchical structure, the metadata corresponding to the target file can be obtained by traversing the metadata index node based on the target path segment, thereby reducing the time complexity of metadata acquisition and improving the overall performance of the file system. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flowchart of the steps of an embodiment of a method for obtaining metadata in a file system of the present application;

[0049] Figure 2 It is a metadata index-based management system of this application;

[0050] Figure 3 It is a schematic diagram of modifying the metadata index during the execution of a snapshot task of the present application;

[0051] Figure 4 This is a schematic diagram of merging the main tree and the backup temporary tree after the snapshot task of the present application is completed;

[0052] Figure 5 This is a schematic diagram of the metadata index before and after the change when a new file is added in the present application;

[0053] Figure 6 It is a structural block diagram of an embodiment of a device for obtaining metadata in a file system of the present application;

[0054] Figure 7 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0056] Reference Figure 1 , is a flowchart of an embodiment of a method for obtaining metadata in a file system of the present application, comprising the following steps:

[0057] Step 101: construct a metadata index node corresponding to the file system according to the file paths of multiple files in the file system;

[0058] The embodiment of the present application uses ARTree (The Adaptive Radix Tree) as the basic data structure to store and manage file path information in the file system, and uses it as an index node to quickly query metadata information of each file.

[0059] Specifically, in an embodiment of the present application, a metadata index for managing metadata in the file system is constructed based on the file path of each file in the file system. The metadata index contains multiple index nodes, and the data structure of the metadata index is ARTree.

[0060] When managing tree nodes (index nodes), ARTree allows index nodes to share the same prefix node. This can significantly save the space required for metadata indexing in scenarios where file paths are used as index nodes in the file system, especially for business applications such as RocksDB where many files in the same directory in the file system use the same prefix name.

[0061] Reference Figure 2 , is a metadata index-based management system of the present application, including a file lock management module, a metadata index ARTree module, a snapshot backup temporary ARTree module and a log management module.

[0062] File lock management module: manages the concurrency of write requests. Through the special structure of ARTree, it can realize file lock and write concurrency control between different levels.

[0063] Metadata index ARTree module: manages metadata index nodes when snapshot tasks are not executed.

[0064] Snapshot standby temporary ARTree module: manages temporary data index nodes during snapshot task execution.

[0065] Log management module: manages incremental logs and needs to be used together with the snapshot backup temporary ARTree module.

[0066] In a conventional file system, different files under the same directory can be implemented, but the file lock management module in the embodiment of the present application can implement file locks different from those of conventional file systems. Instead, it implements path locks, that is, locking sub-files under a certain directory so that other people cannot obtain the lock of this directory when writing or modifying them, thereby avoiding concurrency risks caused by deletion or modification of this directory.

[0067] Specifically, the functions of the file lock management module are as follows:

[0068] 1) Get the path lock

[0069] When a user or process needs to write a subfile in a directory, it first obtains the path lock of the directory.

[0070] 2) Release the path lock

[0071] After the write operation of all sub-files is completed, the user or process releases the path lock.

[0072] 3) Lock hierarchy management

[0073] Through the special structure of ARTree, the file lock management module can manage path locks between different levels to ensure that write operations between different levels do not conflict.

[0074] Step 102: parsing the file path of the target file to obtain a target path segment having a hierarchical structure; the target path segment includes a terminal path segment located at the last level;

[0075] In an embodiment of the present application, if it is necessary to obtain the metadata corresponding to the target file, it is necessary to traverse the metadata index based on the file path of the target file. In order to improve the traversal efficiency, it is necessary to parse the file path of the target file and decompose it into path segments with a hierarchical structure. The path segment refers to each directory level or file name in the file path.

[0076] Specifically, the file path of the target file is decomposed according to the directory hierarchy to obtain a series of path segments, which form a hierarchical structure, wherein each path segment represents a directory level or file name, and the last path segment is the termination path segment, which represents the final location of the target file.

[0077] Step 103: According to the hierarchical structure, traverse the index nodes corresponding to the target path segment in the metadata index nodes, and determine the index node corresponding to the terminal path segment as the target node;

[0078] After determining the target path segment corresponding to the target file, it is necessary to traverse the previously constructed ARTree metadata index according to the hierarchical structure (target path segment) of the parsed target file path to find the index node corresponding to the target path segment.

[0079] Specifically, starting from the root node of ARTree, traverse level by level according to the hierarchical structure of the target file path. For each path segment, find the corresponding index node in the nodes of the current level, and finally find the index node corresponding to the terminal path segment. This node is the target node.

[0080] Step 104: extract metadata corresponding to the target file in the target node.

[0081] Since the target node represents the location where the metadata of the target file is stored, the embodiment of the present application extracts the metadata of the target file in the target node and returns the extracted metadata to the requesting party, thereby completing the operation of obtaining the metadata corresponding to the target file.

[0082] Metadata usually includes file attribute information, such as file size, creation time, modification time, etc.

[0083] The embodiment of the present application can efficiently obtain file system metadata by constructing an ARTree index, parsing the target file path, traversing the index nodes and extracting metadata.

[0084] Specifically, the embodiment of the present application can quickly locate the metadata of the target file by constructing the metadata index node corresponding to the file system. The construction of the metadata index node also makes the metadata storage structure of the file system more orderly and efficient, which is convenient for subsequent query and management; and, by parsing the file path of the target file and decomposing it into target path segments with a hierarchical structure, the metadata corresponding to the target file can be obtained by traversing the metadata index node based on the target path segments, thereby reducing the time complexity of metadata acquisition and improving the overall performance of the file system.

[0085] In one embodiment of the present application, after extracting metadata corresponding to the target file in the target node in step 104, the method further includes:

[0086] Monitoring the incremental log library in the file system; the incremental log library includes a plurality of incremental logs for recording operation behaviors corresponding to the files;

[0087] If the number of the incremental logs reaches a preset number, encoding the metadata in the file system to obtain encoded data;

[0088] The encoded data is persistently stored.

[0089] The metadata index in the embodiment of the present application has a very high efficiency in space utilization. When there are a large number of files and high performance in modifying and querying (obtaining) the metadata index is sought, all the metadata indexes can be placed in the memory.

[0090] Although the full-memory metadata index has greatly improved query and operation performance, it is still necessary to consider the persistent storage of the metadata index and the recovery problem when restarting the service after a failure. Therefore, the embodiment of the present application solves this problem through incremental logs, providing a write-featured chunk (byte stream space, generally 128MB in size) in the storage layer of the storage engine, so that the incremental log can play a higher performance advantage.

[0091] In the embodiment of the present application, by Figure 2 The log management module in manages the incremental logs, and the log management module includes the incremental log library.

[0092] In the file system, for metadata modification and change operations, it is required to write the log successfully before modifying the metadata and metadata index in the memory. When a failure occurs or other reasons require restarting and recovering data, the metadata index can be restored by re-executing the file system request behavior recorded in the log in sequence.

[0093] However, incremental logs will continue to accumulate, resulting in more and more NVM (Non-Volatile Memory) space being occupied, and more and more operational tasks to be performed for restart and recovery when a failure occurs. The consumption of space and time will continue to increase. This problem is particularly serious in scenarios where files are frequently modified in the file system.

[0094] In order to solve this problem, the embodiment of the present application proposes that a snapshot of the metadata in the file system can be taken as a scheduled task, and the number of incremental logs is monitored at all times before executing the snapshot. When the incremental logs in the incremental log library reach a preset number, the snapshot is executed to minimize the write amplification cost of persistent storage. Write amplification refers to the situation in which the amount of data actually written to the storage medium in the storage system is greater than the amount of data requested to be written by the application or operating system. Specifically, the preset number used to determine whether to execute the snapshot needs to be set according to the specifications of the expected number of files in the file system.

[0095] In the embodiment of the present application, when executing a snapshot, the metadata in the file system and its corresponding index are encoded and converted into a format that can be stored on the NVM device to obtain the encoded data, and then the encoded data is written into the NVM device. In addition, in order to improve the writing efficiency, the encoded data is usually divided into multiple chunks, and each chunk is written to the NVM device independently in sequence.

[0096] In one embodiment of the present application, before the encoding data is persistently stored, the method further includes:

[0097] Create a temporary data index node;

[0098] If multiple files in the file system need to be modified, it is implemented based on the temporary data index node;

[0099] If multiple files in the file system need to be searched, it is implemented based on the metadata index node and the temporary data index node.

[0100] Since the snapshot task is a relatively time-consuming process, if the file system metadata is modified directly, it may cause business interruption or data inconsistency in the file system, and multiple concurrent operations (such as file reading and writing, directory creation / deletion, etc.) may conflict with the snapshot task, resulting in data inconsistency or system crash. Therefore, during the snapshot task, the embodiment of the present application proposes to use a temporary ARTree as the data structure for modification and change, that is, when executing the snapshot task, a temporary data index node (temporary ARTree) will be created. Specifically, the modification operation of the metadata index (including creation and deletion) is performed on the temporary ARTree, and the query operation of the metadata index requires the use of the temporary data index node (temporary tree) and the metadata index node (main tree) at the same time.

[0101] Reference Figure 3 , is a schematic diagram of a modification operation on a metadata index during execution of a snapshot task of the present application.

[0102] When a new file or directory is created, the system creates a new node in the temporary tree to represent the new file or directory. Figure 3 The new node will store the metadata of the newly created file or directory (such as file name, size, creation time, permissions, etc.) in the new node.

[0103] When there is an operation to delete a file or directory, the system will search for the node representing the deleted file or directory in the temporary tree, that is, Figure 3 The node's status is marked as deleted by changing the node's "est / file / _test_ / b" in the output.

[0104] When there is an operation to modify a file or directory, the system will search for the node representing the modified file or directory in the temporary tree and update the modified metadata to the corresponding node in the temporary tree. If necessary, the index structure in the temporary tree is updated to ensure that the modified file or directory can correctly reflect the latest status.

[0105] like Figure 3 As shown in the figure, during the snapshot task, the file system created a file " / DBFS / index-10 / log" and deleted a file " / DBFS / test / file_test_ / b". Although the operations were performed on files in multi-level directories, only three nodes were created on the backup temporary tree. The complexity of insertion and deletion operations is low, and the additional computing cost of using the temporary tree is very small.

[0106] In an embodiment of the present application, to ensure metadata consistency and integrity in the file system, after the snapshot task is completed, all nodes on the temporary tree are reinserted into the main tree to complete data synchronization between the main tree and the backup temporary tree.

[0107] Reference Figure 4 , is a schematic diagram of merging the main tree and the backup temporary tree after the snapshot task of the present application is completed.

[0108] For the newly created file " / DBFS / index-10 / log", you need to insert the newly created node in the backup temporary tree into the main tree. Specifically, insert the log node in the / DBFS / index-10 / directory of the main tree.

[0109] For the deleted file " / DBFS / test / file_test_ / b", the node marked as "deleted" in the standby temporary tree needs to be deleted from the main tree. Specifically, the b node in the / DBFS / test / file_test_ / directory of the main tree needs to be deleted.

[0110] For modified files or directories, the modified nodes in the backup temporary tree need to be updated to the main tree.

[0111] A key feature of ARTree is that when inserting a path node on an empty tree, its parent directory node does not need to be considered. This means that when inserting a new node, ARTree can automatically handle all intermediate nodes in the path without explicitly creating or finding these intermediate nodes. In scenarios where the file system includes multiple files, this feature simplifies the synchronization operation between the main tree and the backup temporary tree, optimizes performance, and reduces the time complexity of the operation.

[0112] The embodiment of the present application uses a temporary ARTree to record changes, so that the integrity and consistency of the file system metadata can be maintained during the snapshot task. Since the changes are recorded in a temporary data structure instead of directly modifying the file system metadata, the business can continue to run without being interrupted by the snapshot task. The temporary ARTree can serve as an intermediate layer to manage concurrent operations and ensure that no data conflicts occur during the snapshot task.

[0113] During the snapshot task, all modifications to the file system metadata (such as creation, deletion, modification, etc.) are recorded in the temporary tree, and the file system query operations (such as searching for files and directories) need to be able to reflect the latest metadata status, including all changes during the snapshot task. Therefore, if a query operation is required on the metadata index, the target metadata needs to be searched in the metadata index (main tree) first; if the target metadata is not found in the main tree, or the latest metadata status needs to be confirmed, the target metadata is searched in the temporary tree.

[0114] If a newly created node is found in the temporary tree, the metadata of the node is returned; if a node marked as "deleted" is found in the temporary tree, the status of "not found" or "deleted" is returned; if a modified node is found in the temporary tree, the modified metadata is returned.

[0115] The query results in the temporary tree take precedence over the query results in the main tree. If there is a corresponding node in the temporary tree (whether newly created, deleted or modified), the results in the temporary tree are returned.

[0116] During the execution of the snapshot task, the embodiment of the present application can ensure that the query result can reflect the latest metadata status and maintain data consistency by querying the temporary tree and the main tree at the same time.

[0117] In one embodiment of the present application, when the snapshot task is started, the log sequence number of the current incremental log will be marked. After the snapshot is completed, all incremental logs that are smaller than this log sequence will be cleared to avoid the problem of continuous log accumulation.

[0118] In one embodiment of the present application, after the encoding data is persistently stored, the method further includes:

[0119] If the metadata index node fails, determining the log sequence numbers corresponding to the multiple incremental logs respectively;

[0120] The multiple incremental logs are executed in sequence according to the log sequence numbers respectively corresponding to the multiple incremental logs, and the metadata in the metadata index node is restored.

[0121] Since the file system is a core component for storing and managing data, any failure may result in data loss, inconsistency, or business interruption. Therefore, in the embodiments of the present application, it is also necessary to consider the recovery problem when restarting the service after a failure occurs.

[0122] In an embodiment of the present application, when a failure occurs or other reasons require restarting to recover data, by re-executing the file system request behavior, it can be ensured that the metadata index can be restored to a consistent state after the failure occurs, avoiding data inconsistency and reducing recovery time.

[0123] Specifically, the embodiment of the present application realizes the re-execution of file system request behavior through the combination of snapshot and incremental log.

[0124] Assume that a file system fails at a certain point in time, the most recent snapshot time is T1, and the failure time is T2.

[0125] Select the most recent snapshot T1 as the recovery target; load snapshot T1 into the file system and restore it to the file system state at T1; read the incremental logs from T1 to T2; replay each operation in the incremental logs in chronological order, including creation, deletion, modification, etc.; by replaying the incremental logs, the file system can be restored to the latest state T1; perform a consistency check after recovery to ensure that the file system state is consistent.

[0126] In the embodiment of the present application, the time sequence of the incremental logs is determined based on the log sequence number.

[0127] When replaying the incremental log in the embodiment of the present application, it is first necessary to read the incremental log, obtain all the file system request behaviors recorded in the log, and arrange them in chronological order. Then, each request behavior is executed in sequence, including operations such as creating a new file or directory, deleting a file or directory, and modifying a file or directory. After executing each request behavior, the file system metadata index will be updated to ensure that the index structure can correctly reflect the latest metadata status.

[0128] Assume that the incremental log in the incremental log library records the following operations:

[0129] Create a new file / DBFS / test / file1

[0130] Delete the file / DBFS / test / file2

[0131] Modify the file / DBFS / test / file3

[0132] Then, when a restart is required after a failure, the incremental log is read first to obtain the above operation records, and the following operations are performed in the chronological order of the incremental logs:

[0133] Create a new file / DBFS / test / file1: Create a file1 node in the metadata index.

[0134] Delete file / DBFS / test / file2: Delete the file2 node from the metadata index.

[0135] Modify the file / DBFS / test / file3: Update the metadata of the file3 node in the metadata index.

[0136] After each operation is completed, the file system metadata index is updated and a consistency check is performed to ensure the consistency and integrity of the file system metadata index.

[0137] In addition, by sequentially executing the operations in the log, the system is able to rebuild the file system metadata index to ensure data integrity and consistency.

[0138] In an embodiment of the present application, data consistency and business continuity are ensured by periodically creating snapshots and quickly restoring to the most recent snapshot state after a failure occurs. After testing, the mechanism of introducing timed snapshots during failure recovery performs well in a million-level file system, and can complete failure recovery in seconds, thereby improving the high availability and stability of the system.

[0139] In an embodiment of the present application, the operation behavior includes a new addition behavior. After extracting metadata corresponding to the target file in the target node in step 104, the method further includes:

[0140] When a first target file is newly created in the file system, a first file path corresponding to the first target file is parsed to obtain a first path segment having a first hierarchical structure;

[0141] According to the first hierarchical structure, traversing the metadata index nodes to see whether there is an index node corresponding to the first path segment;

[0142] Determine the first path segment corresponding to which no index node can be found in the metadata index node as the path segment to be inserted;

[0143] Creating a node to be inserted corresponding to the path segment to be inserted, and inserting the node to be inserted into the metadata index node;

[0144] Storing the metadata corresponding to the first target file in the node to be inserted in the metadata index node;

[0145] A first incremental log is generated according to the newly added behavior corresponding to the first target file, and is stored in the incremental log library.

[0146] In an embodiment of the present application, when a new first target file is created in the file system, the file adding operation is completed by parsing the file path, traversing the metadata index node, determining the path segment to be inserted, creating the node to be inserted, storing metadata and generating an incremental log.

[0147] Reference Figure 5 , is a schematic diagram of the metadata index before and after the change when a new file is added in the present application.

[0148] When the first file path of the newly added first target file is / DBFS / test / file_test_A; parse the first file path / DBFS / test / file_test_A to obtain the first path segments DBFS, test, file_test_A; traverse the metadata index in sequence to find whether there are corresponding first path segments DBFS, test, file_test_A; after traversal, it is found that only the index node corresponding to the path segment file_test_A in the metadata index does not exist. At this time, the path segment file_test_A is used as the path segment to be inserted, and the node file_test_A to be inserted is created; then the node file_test_A to be inserted is inserted into the / test / node in the metadata index; after the metadata index operation based on the new addition operation is completed, a first incremental log for recording the operation of the newly created file file_test_A will be generated, and finally, the first incremental log is stored in the incremental log library.

[0149] For example, Figure 5 When adding / DBFS / test / file_test_a and / DBFS / test / file_test_b in , the process is the same as the above process. However, because ARTree allows index nodes to share the same prefix node when managing index nodes, nodes a and b are added. Figure 5 When adding / DBFS / index-10 / , / DBFS / index-11 / , and / DBFS / index-12 / , the process is the same as the previous one and is not repeated here.

[0150] In an embodiment of the present application, the operation behavior includes a deletion behavior. After extracting metadata corresponding to the target file in the target node in step 104, the method further includes:

[0151] When the second target file is deleted in the file system, a second file path corresponding to the second target file is parsed to obtain a second path segment having a second hierarchical structure; the second path segment includes a second termination path segment located at the last level;

[0152] According to the second hierarchical structure, traverse the index nodes corresponding to the target path segment in the metadata index nodes;

[0153] Determine the index node corresponding to the second terminal path segment as the node to be deleted;

[0154] Deleting the node to be deleted and metadata corresponding to the second target file in the node to be deleted;

[0155] A second incremental log is generated according to the deletion behavior corresponding to the second target file, and is stored in the incremental log library.

[0156] In an embodiment of the present application, deleting the index node corresponding to the second target file in the metadata index also requires completing the file deletion operation by parsing the file path, traversing the metadata index nodes, determining the nodes to be deleted, deleting the metadata, and generating an incremental log.

[0157] Assume you need to delete Figure 5 The / DBFS / index-10 / in the changed metadata index is parsed, and the second file path / DBFS / index-10 / is parsed to obtain the second path segment DBFS, index-, 1, 0; at this time, path segment 0 is used as the second termination path segment, and its corresponding index node 0 is the node to be deleted; then the node to be deleted 0 in the metadata index and the metadata stored in the node are deleted together; after the metadata index operation is completed based on the deletion operation, a second incremental log for recording the operation of deleting the file index-10 is generated, and finally, the second incremental log is stored in the incremental log library.

[0158] In an embodiment of the present application, the internal node structure (index node structure) of ARTree has the characteristic of adaptive length. Specifically, the index nodes in the metadata index use a shorter structure during initialization to save space as much as possible. After continuously creating files and inserting metadata indexes during the business process, the index nodes will be gradually expanded to a longer structure.

[0159] There are four kinds of internal node structures in the embodiment of the present application, which are structures containing 4, 16, 64 and 256 nodes respectively, and different search methods are used for different internal node structures. Specifically, for a node structure with a length of no more than 16 nodes, it is sufficient to directly traverse and search for the sorted internal child nodes, and the complexity is O(N); for a node structure with a length of no more than 64 nodes, a binary search method is used to search for internal nodes, and the complexity is O(log N). For a node structure with a length of 256 nodes, most of the characters are already occupied (the maximum number of single characters is 255), and the characters can be directly used as the index of the array for search, and the complexity is only O(1), where N is the number of nodes in the internal node structure.

[0160] The metadata index in the embodiment of the present application improves the space utilization and ensures the performance of metadata index management operations by using an internal node structure with adaptive length. By gradually expanding the node structure and using different search methods under different structures, the metadata index can maintain efficient operating performance under different data volumes.

[0161] In addition, ARTree folds and compresses nodes. Unlike traditional radix trees, where each character occupies one node, ARTree requires that internal nodes be used to distinguish at least two different metadata indexes before creating a new internal node. When an internal node contains only one child node, the node will be deleted and merged into the parent node to achieve a compression effect, further improving space utilization and the efficiency of metadata acquisition.

[0162] The file system uses ARTree as the data structure for metadata index. Although there will be adaptive adjustments to internal nodes and folding and compression of single nodes when creating and deleting files, compared with the self-balancing characteristics of other tree structures, in file systems with a large number of files, ARTree's node addition and deletion operations have significant advantages in computing efficiency, thereby improving the efficiency of metadata acquisition.

[0163] In one embodiment of the present application, the index node includes a root node and a child node corresponding to the root node, the target path segment also includes a starting path segment located at a first level, and the step 103 of traversing the index node corresponding to the target path segment in the metadata index node according to the hierarchical structure includes:

[0164] Determine a target root node among the root nodes according to the starting path segment;

[0165] According to the hierarchical structure, the sub-nodes corresponding to the target path segment are traversed in the sub-nodes corresponding to the target root node.

[0166] In the embodiment of the present application, since ARTree inherits the characteristics of the radix tree, it can efficiently store and retrieve key-value pairs, and the naming rules of the Linux file system path, the nodes in ARTree can reflect the directory hierarchy structure of the file system to a certain extent. Figure 3 , Figure 4 , Figure 5 As shown in the figure, the leaf node " / DBFS / " is a directory node, and all its child nodes are sub-items of this directory, which ensures the continuity of the same directory sub-items in the tree structure. Therefore, using ARTree to manage the file system metadata index does not require additional space to store the directory sub-item content, and can traverse and obtain the directory sub-item content with only a very small computing cost.

[0167] Therefore, when you need to query each index node corresponding to a file in the metadata index, you first need to find the target root node in the root node according to the starting path segment, and then traverse the child nodes corresponding to the target path segment in the child nodes corresponding to the target root node.

[0168] Assume that the file path of the target file is / DBFS / test / file1, and the starting path segment is / DBFS / . At this time, you need to find the DBFS node in the root node and determine that DBFS is the target root node. Then, find the test node in the child node corresponding to the DBFS node, and then find the file1 node in the child node corresponding to the test node.

[0169] When traversing directory sub-items, you can use the upper_bound function to determine the end point of the traversal. The upper_bound function can skip sub-items in sub-items, greatly improving the computational efficiency of traversal search.

[0170] The upper_bound function is a function in the C++ standard library, which is used to find the position of the first element greater than a given value in an ordered container (such as an array, vector, list, etc.). It is usually used in the binary search algorithm to improve the search efficiency.

[0171] In one embodiment of the present application, the directory hierarchy structure of the file system metadata index is ensured by utilizing the radix tree characteristics of ARTree and the naming rules of the Linux file system path. When traversing the target path segment, first determine the target root node in the root node according to the starting path segment, and then traverse the subnodes corresponding to the target path segment in the subnodes corresponding to the target root node. In this way, no additional space is required to store the sub-item content of the directory, and it is possible to traverse and obtain the sub-item content of the directory with only a very small computational cost. In addition, the use of the upper_bound function can further improve the computational efficiency of the traversal search.

[0172] In the embodiment of the present application, ARTree is used as the metadata index data structure for managing the file system. With the characteristics that index nodes in the ARTree data structure can share the same prefix, internal nodes can adaptively adjust the length, and single nodes can be folded and compressed when nodes are deleted, it can not only meet the high performance of index node query, modification, addition and deletion operations, but also achieve the efficiency improvement of metadata acquisition based on metadata index, and realize efficient use of space, so that the metadata index can reside in the memory, which has significant advantages in the scenario of pursuing extreme performance.

[0173] At the same time, the high reliability of the full-memory metadata index is taken into consideration, and the incremental log and snapshot method is used to achieve persistent storage and fault restart recovery of the metadata index. The continuous accumulation of log volume is avoided by starting the snapshot task at a scheduled time. During the execution of the snapshot task, a spare temporary ARTree with low space occupancy and operation cost is used to ensure the normal operation of the business in the file system.

[0174] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0175] On the basis of the above embodiments, this embodiment further provides a device for acquiring metadata in a file system, which is applied to electronic devices such as terminal devices and servers.

[0176] Reference Figure 6 , shows a structural block diagram of an embodiment of a device for obtaining metadata in a file system of the present application, which may specifically include the following modules:

[0177] The metadata index building module 601 is used to build a metadata index node corresponding to the file system according to the file paths of multiple files in the file system;

[0178] A file path parsing module 602, configured to parse the file path of a target file to obtain a target path segment having a hierarchical structure; the target path segment includes a termination path segment at the last level;

[0179] An index node traversal module 603 is used to traverse the index nodes corresponding to the target path segment in the metadata index nodes according to the hierarchical structure, and determine the index node corresponding to the termination path segment as the target node;

[0180] The metadata acquisition module 604 is used to extract metadata corresponding to the target file in the target node.

[0181] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0182] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0183] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Figure 7 An exemplary apparatus 700 that can be used to implement various embodiments described in this application is schematically illustrated.

[0184] For one embodiment, Figure 7 An exemplary apparatus 700 is shown having one or more processors 702, a control module (chip set) 704 coupled to at least one of the processor(s) 702, a memory 706 coupled to the control module 704, a non-volatile memory (NVM) / storage device 708 coupled to the control module 704, one or more input / output devices 710 coupled to the control module 704, and a network interface 712 coupled to the control module 704.

[0185] The processor 702 may include one or more single-core or multi-core processors, and the processor 702 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 700 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0186] In some embodiments, the apparatus 700 may include one or more computer-readable media (e.g., memory 706 or NVM / storage device 708) having instructions 714 and one or more processors 702 combined with the one or more computer-readable media and configured to execute the instructions 714 to implement a module to perform the actions described in the present disclosure.

[0187] For one embodiment, the control module 704 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 702 and / or any suitable device or component in communication with the control module 704 .

[0188] The control module 704 may include a memory controller module to provide an interface to the memory 706. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0189] The memory 706 may be used, for example, to load and store data and / or instructions 714 for the device 700. For one embodiment, the memory 706 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 706 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0190] For one embodiment, control module 704 may include one or more input / output controllers to provide an interface to NVM / storage device 708 and input / output device(s) 710 .

[0191] For example, NVM / storage 708 may be used to store data and / or instructions 714. NVM / storage 708 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0192] NVM / storage device 708 may include storage resources that are physically part of the device on which apparatus 700 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 708 may be accessed via input / output device(s) 710 over a network.

[0193] (One or more) input / output devices 710 may provide an interface for the apparatus 700 to communicate with any other appropriate device, and the input / output device 710 may include a communication component, an audio component, a sensor component, etc. The network interface 712 may provide an interface for the apparatus 700 to communicate through one or more networks, and the apparatus 700 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0194] For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 704. For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers of the control module 704 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704. For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704 to form a system-on-chip (SoC).

[0195] In various embodiments, the apparatus 700 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the apparatus 700 may have more or fewer components and / or a different architecture. For example, in some embodiments, the apparatus 700 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0196] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0197] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0198] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0199] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable file system metadata acquisition terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable file system metadata acquisition terminal device generate instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0200] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable file system metadata acquisition terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0201] These computer program instructions can also be loaded onto a computer or other programmable file system metadata acquisition terminal device, so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0202] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0203] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0204] The above is a detailed introduction to a method and device for obtaining metadata in a file system, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for obtaining metadata in a file system, characterized in that: The method comprises: Constructing a metadata index node corresponding to the file system according to the file paths of the plurality of files in the file system; Parsing the file path of the target file to obtain a target path segment having a hierarchical structure; the target path segment includes a termination path segment located at the last level; According to the hierarchical structure, traverse the index nodes corresponding to the target path segment in the metadata index nodes, and determine the index node corresponding to the termination path segment as the target node; The metadata corresponding to the target file is extracted in the target node.

2. The method according to claim 1, characterized in that After extracting the metadata corresponding to the target file in the target node, the method further includes: Monitoring the incremental log library in the file system; the incremental log library includes a plurality of incremental logs for recording operation behaviors corresponding to the files; If the number of the incremental logs reaches a preset number, encoding the metadata in the file system to obtain encoded data; The encoded data is persistently stored.

3. The method according to claim 2, characterized in that Before the encoding data is persistently stored, the method further includes: Create a temporary data index node; If multiple files in the file system need to be modified, it is implemented based on the temporary data index node; If multiple files in the file system need to be searched, it is implemented based on the metadata index node and the temporary data index node.

4. The method according to claim 2, characterized in that: After the encoding data is persistently stored, the method further includes: If the metadata index node fails, determining the log sequence numbers corresponding to the multiple incremental logs respectively; The multiple incremental logs are executed in sequence according to the log sequence numbers respectively corresponding to the multiple incremental logs, and the metadata in the metadata index node is restored.

5. The method according to claim 2, characterized in that: The operation behavior includes a new addition behavior. After extracting metadata corresponding to the target file in the target node, the method further includes: When a first target file is newly created in the file system, a first file path corresponding to the first target file is parsed to obtain a first path segment having a first hierarchical structure; According to the first hierarchical structure, traversing the metadata index nodes to see whether there is an index node corresponding to the first path segment; Determine the first path segment corresponding to which no index node can be found in the metadata index node as the path segment to be inserted; Creating a node to be inserted corresponding to the path segment to be inserted, and inserting the node to be inserted into the metadata index node; Storing the metadata corresponding to the first target file in the node to be inserted in the metadata index node; A first incremental log is generated according to the newly added behavior corresponding to the first target file, and is stored in the incremental log library.

6. The method according to claim 2, characterized in that The operation behavior includes a deletion behavior. After extracting metadata corresponding to the target file in the target node, the method further includes: When the second target file is deleted in the file system, a second file path corresponding to the second target file is parsed to obtain a second path segment having a second hierarchical structure; the second path segment includes a second termination path segment located at the last level; According to the second hierarchical structure, traverse the index nodes corresponding to the target path segment in the metadata index nodes; Determine the index node corresponding to the second terminal path segment as the node to be deleted; Deleting the node to be deleted and metadata corresponding to the second target file in the node to be deleted; A second incremental log is generated according to the deletion behavior corresponding to the second target file, and is stored in the incremental log library.

7. The method according to claim 2, characterized in that The index node includes a root node and a child node corresponding to the root node, the target path segment also includes a starting path segment located at a first level, and traversing the index node corresponding to the target path segment in the metadata index node according to the hierarchical structure includes: Determine a target root node among the root nodes according to the starting path segment; According to the hierarchical structure, the sub-nodes corresponding to the target path segment are traversed in the sub-nodes corresponding to the target root node.

8. A device for acquiring metadata in a file system, characterized in that: The device comprises: A metadata index building module, used to build a metadata index node corresponding to the file system according to the file paths of multiple files in the file system; A file path parsing module, used for parsing the file path of the target file to obtain a target path segment having a hierarchical structure; the target path segment includes a termination path segment located at the last level; An index node traversal module, configured to traverse the index nodes corresponding to the target path segment in the metadata index nodes according to the hierarchical structure, and determine the index node corresponding to the termination path segment as the target node; The metadata acquisition module is used to extract metadata corresponding to the target file in the target node.

9. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the method for obtaining metadata in a file system as claimed in any one of claims 1 to 7.

10. One or more machine-readable media having executable codes stored thereon, which, when executed, enable a processor to execute the method for obtaining metadata in a file system according to any one of claims 1 to 7.

Images