A method for managing multi-node metadata of a persistent memory file system in a NUMA system

Abstract

The application discloses a kind of NUMA system in persistent memory file system multi-node metadata management method, it is related to data center new type storage technical field.File data access is the mixed process of file metadata and file data, for example, in the process of creating, deleting file, need to access file system global information, to update file system access time, free data block, free inode etc.;In file reading and writing process, need to traverse directory structure, find the file metadata inode structure corresponding to file name, allocate space, copy data, update file system global information and file metadata inode information, complete the reading and writing of file.In current NUMA persistent memory file system design, there is remote access metadata operation in file data access process, which leads to the decline of file reading and writing performance.The application adopts the redundant dynamic attribute metadata management mechanism based on access frequency, centralized directory management mechanism, optimizes the metadata management of persistent memory file system in NUMA system, reduces the influence of cross-node metadata access on file data access performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of new storage technology of data center, and particularly relates to a multi-node metadata management method of a persistent memory file system in a NUMA system. BACKGROUND

[0002] The rapid expansion of Internet and Internet of Things applications leads to an explosive growth of global data, which needs to be stored efficiently. It has become an urgent problem to build a reliable and efficient data storage system. A new paradigm of heterogeneous storage based on hybrid DRAM and persistent memory (PM) has been designed to directly process data within or near the storage device to minimize data movement and improve performance, such as in-memory storage, in-memory computing, etc. Many persistent memory-based file systems are designed to take advantage of the byte-addressable, persistent, and high-speed nature of persistent memory, such as EXT4-DAX, SCMFS, PMFS, NOVA, SIMFS, and HiNFS. These persistent memory file systems use persistent memory as a data storage device, directly copy file data between application user buffers and persistent memory devices, avoid the software overhead of traditional block device file systems through operating system cache and I / O software stack, and obtain better performance improvement.

[0003] To achieve greater throughput, reliability, and economy of computer systems, high-performance computing and cloud computing data center servers commonly use NUMA (Non-Uniform Memory Access) architecture. NUMA architecture systems divide processors and memories into node management, avoiding the scalability problem caused by increasing memory access conflicts when multiple processors address the same memory address space, with lower local memory access delay and better scalability. Many basic software has been specially optimized for NUMA architecture computer systems. For example, the Linux kernel memory management supports the non-uniform memory access model, and the scheduler load balancing takes the NUMA node as the management domain. At the same time, many research works have proposed data allocation strategies for NUMA architecture. For example, JeriFS redesigned the I / O stack to achieve optimal thread and buffer affinity. In addition, enterprises have specially processed virtualization platforms for NUMA architecture. For example, VMWare designed a NUMA-aware virtual machine I / O management scheduler to achieve maximum efficiency from local I / O devices.

[0004] Many studies design high-performance persistent memory file systems for NUMA architecture. NThread proposes a NUMA-aware thread migration module suitable for persistent memory file systems, which designs a NUMA-aware thread migration module without migrating data. J.H.Kim et al. virtualize persistent memory devices across NUMA nodes into a single address space and design a local write-priority memory allocation strategy to reduce remote memory access. HydraFS divides the file system storage space into multiple parts according to the number of NUMA nodes, and schedules threads to the node where the file is located, so that the thread can get the best benefit from local file access.

[0005] The file system uses a directory hierarchy structure to manage file data, and the file system includes file system global information and files, and the file includes file metadata and file data. The file system global information is used to describe the global characteristics of the file system, including the file system size, the file system read-write time, the total number of file metadata inodes, the remaining number of file metadata inodes, the total number of data blocks, the number of allocated blocks, and the remaining number of data blocks. File metadata is used to describe file characteristics, including owner, access time, permission, file identifier, etc. File data access is a hybrid process of file metadata and file data, for example, in the process of creating and deleting files, file system global information needs to be accessed to update file system access time, free data blocks, free inodes, etc. In the process of file read and write, the directory structure needs to be traversed to find the file metadata inode structure corresponding to the file name, allocate space, copy data, update file system global information and file metadata inode information, and complete file read and write. In the current design of NUMA persistent memory file system, the file data access process involves remote access to metadata operations, resulting in a decline in file read and write performance. How to efficiently manage metadata in a multi-node space and reduce remote access to file metadata is an important challenge. SUMMARY

[0006] In view of the problems in the prior art, the present application proposes a NUMA system persistent memory file system multi-node metadata management method, which aims to optimize the metadata management of the persistent memory file system in the NUMA system and reduce the impact of cross-node metadata access on file data access performance.

[0007] To this end, the technical solution adopted by the present application is a NUMA system persistent memory file system multi-node metadata management method, comprising:

[0008] (1) Redundant dynamic attribute metadata management based on access frequency: managing the file system global information and file metadata of the NUMA persistent memory file system, specifically including

[0009] (1.1) File system global information redundancy dynamic attribute management mechanism, managing file system global information on multiple NUMA nodes;

[0010] (1.2) File system global information redundancy dynamic attribute synchronization mechanism, synchronizing file system information dynamic attributes between multiple nodes through active access synchronization and timing synchronization mechanisms;

[0011] (1.3) File metadata redundancy dynamic attribute management mechanism, distributing all file metadata in the file system on a single node, and managing all file metadata attributes through inode table;

[0012] (1.4) File metadata redundancy dynamic attribute synchronization mechanism, synchronizing file metadata dynamic attributes between the master node and the node where the file data is located through an active access synchronization mechanism;

[0013] (2) Centralized directory management: managing the directory structure in the NUMA persistent memory file system, allocating space on a single node to construct a directory tree, and avoiding remote persistent memory access during directory retrieval.

[0014] Further, in step (1.1), all attributes in the file system global information on node 0 are managed, and according to the access frequency of each attribute in the file system information, the file system global information attributes are divided into static attributes and dynamic attributes, and the dynamic attributes are redundantly stored in the starting position of each node. In the file access process, access the redundant file system global information dynamic attributes on the local node to avoid the impact of remote file system global information access on file data access performance. In the file system global information structure of each node, a synchronization flag attribute is added.

[0015] Further, step (1.2) adds a synchronization identification of whether the field is modified in the structure. Whenever the file system global information is modified, the synchronization identification field is set. When there is a modification to the file system global information redundancy dynamic attribute, it is first saved in the redundant backup on the local node. When the user actively accesses the file system global information, the synchronization period is reached, and then the synchronization is performed according to the synchronization identification and modification time modification order to the file system global information on node 0.

[0016] Further, in order to quickly locate the file metadata inode of a specific file in step (1.3), the inode table is managed by an array, and the free inodes in the array are linked by a chain table to realize quick allocation and recovery; a space is reserved on the node where the file data is located for redundantly storing the dynamic attributes of the file metadata on the current node; during the file read-write access process, the read-write operation is completed through the redundant dynamic attributes of the file metadata on the local node, thereby avoiding the influence of remote file metadata access on the file data access performance. Specifically, the metadata attributes of all files are managed on node 0, the file metadata attributes are divided into static attributes and dynamic attributes according to the access frequency of each attribute in the file metadata, and the dynamic attributes with high access frequency in the file access process are redundantly stored in the NUMA node where the file data is located.

[0017] The file metadata redundant dynamic attribute synchronization mechanism synchronizes the dynamic attributes of the file metadata between two nodes (the master node and the node where the file data is located) through an active access synchronization mechanism.

[0018] Further, the multi-node centralized directory tree management mechanism realizes the centralized management of directories in the NUMA persistent memory file system, and avoids the problem of cross-node access to directory files in the vertical directory retrieval process.

[0019] The centralized directory tree management mechanism stores the directory structure in the NUMA persistent memory file system on node 0; when a directory is created, the directory metadata structure and the directory data block are allocated from node 0, and the tree-shaped directory hierarchical management structure is constructed on a single NUMA node, thereby avoiding multiple remote access to traverse the directory among multiple nodes.

[0020] The application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to realize the steps of the multi-node metadata management method of the persistent memory file system in the NUMA system.

[0021] The application also provides a system architecture of the management method, which comprises a file system global information redundancy dynamic attribute management module, a file system global information redundancy dynamic attribute synchronization module, a file metadata redundancy dynamic attribute management module, a file metadata redundancy dynamic attribute synchronization module and a multi-node centralized directory tree management module.

[0022] The NUMA system persistent memory file system multi-node metadata management method provided by the application reduces the influence of cross-node metadata access on file data access performance and improves the parallelism of NUMA persistent memory file system metadata operation through a redundancy dynamic attribute metadata management mechanism based on access frequency and a multi-node centralized directory tree management mechanism. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 NUMA multi-node metadata management method architecture schematic diagram;

[0024] Figure 2 File system global information redundancy dynamic attribute management mechanism schematic diagram;

[0025] Figure 3 File system global information redundancy dynamic attribute synchronization mechanism work flow diagram;

[0026] Figure 4 File metadata redundancy dynamic attribute management mechanism schematic diagram;

[0027] Figure 5 File metadata redundancy dynamic attribute synchronization mechanism work flow diagram;

[0028] Figure 6 Multi-node centralized directory tree management mechanism schematic diagram. DETAILED DESCRIPTION

[0029] Figure 1 The NUMA multi-node metadata management method architecture schematic diagram comprises a file system global information redundancy dynamic attribute management module, a file system global information redundancy dynamic attribute synchronization module, a file metadata redundancy dynamic attribute management module, a file metadata redundancy dynamic attribute synchronization module, and a multi-node centralized directory tree management module. In the NUMA architecture server, processors, DRAMs and persistent memories are divided into node management, and nodes are connected through a high-speed interconnection bus. The persistent memories on multiple nodes are used as data storage devices of the persistent memory file system. The persistent memory file system manages the multi-node persistent memory space, and stores file system global information, file metadata and file data in the persistent memory. The file system global information is used to describe the global characteristics of the file system, and a redundancy dynamic attribute management mechanism of the file system global information manages redundant file system global information attributes on multiple nodes. A redundancy dynamic attribute synchronization mechanism of the file system global information synchronizes dynamic attributes on multiple nodes. The file metadata is used to describe file attribute characteristics, and a redundancy dynamic attribute management mechanism of the file metadata manages redundant file metadata dynamic attributes on multiple NUMA nodes. A redundancy dynamic attribute synchronization mechanism of the file metadata synchronizes dynamic attributes on node 0 and the node where the file is located. A multi-node centralized directory tree management mechanism manages directories and files in the NUMA persistent memory file system, allocates space on a single node to construct a directory tree, and avoids access to remote persistent memories in a directory retrieval process.

[0030] Figure 2 The file system global information redundancy dynamic attribute management mechanism schematic diagram is shown. The file system global information is used to describe the global characteristics of the file system, and comprises a total size of the file system, file system read / write time, data block size, total number of inode file metadata structures, total number of remaining inode file metadata structures, total number of data blocks, total number of allocated blocks, total number of remaining data blocks and the like. According to the access frequency, the attributes are divided into static attributes and dynamic attributes. The static attributes are unchanged in the file system access process, such as the total size of the file system, the data block size, the Volume Name, the log space offset, the inode table offset, the mounting time and the like. The dynamic attributes are constantly changing in the file system access process, such as the file system write time, the total number of allocated blocks, the total number of remaining data blocks, the total number of allocated inodes, the total number of remaining inodes and the like. In addition, the file access process involves operations of dynamic attributes. Based on the above analysis, the file system global information multi-node redundancy dynamic attribute management mechanism is designed for the NUMA multi-node architecture, and the dynamic attributes are redundantly stored at the starting positions of each node, such as Figure 2The file system global information dynamic attribute is stored in the 512-byte space reserved on each NUMA node. In the file access process, the file system global information dynamic attribute on the local node is accessed to avoid performance degradation caused by remote access to the file system global information. A synchronization flag attribute is added to the file system global information structure on each node to implement the synchronization mechanism for the file system global information dynamic attribute on multiple nodes.

[0031] Figure 3 The file system global information redundancy dynamic attribute synchronization mechanism workflow diagram. The file system global information dynamic attribute has a redundant backup on each node, and a synchronization mechanism needs to be designed. Considering that the user accesses the file system global information at a low frequency, active access synchronization and timing synchronization are used to synchronize the multiple-node file system global information redundancy dynamic attribute. To implement the synchronization mechanism for the file system global information dynamic attribute on multiple nodes, a synchronization flag indicating whether the field is modified is added to the structure. Whenever the file system global information is modified, the synchronization flag field is set. The file creation, deletion, and other operations modify the file system global information redundancy dynamic attribute, which is first saved in the local node's redundant backup. When the user actively accesses the file system global information, the synchronization period arrives, and the synchronization flag and modification time are modified according to the order to synchronize to the file system global information on node 0. Figure 3 The file system global information redundancy dynamic attribute synchronization mechanism workflow is illustrated, Figure 3 (a) Periodic synchronization mechanism:

[0032] In step S311, the system creates a periodic synchronization kernel thread when mounting the file system.

[0033] In step S312, the synchronization kernel thread determines whether the time interval period λ has arrived. If the time has not arrived, it continues to wait, otherwise, S313 is executed.

[0034] In step S313, the synchronization kernel thread reads the synchronization flag in the file system global information dynamic attribute structure on each node. The synchronization flag is set to indicate that the dynamic attribute on the node has been updated.

[0035] In step S314, it is determined whether the node with the set synchronization flag is 0. If it is, it indicates that synchronization is not needed, otherwise, S315 is executed to perform synchronization.

[0036] In steps S315 and S316, the values on the master node are updated according to the number of data blocks and inode number in the set dynamic attribute. The time on the master node is updated according to the time in the set dynamic attribute.

[0037] In step S317, it is judged whether to unload the file system, if yes, the kernel thread is terminated, otherwise it is judged whether the synchronization period is reached.

[0038] Figure 3 (b) For active access synchronization:

[0039] In step S321, the user calls VFS layer system call to access file system information.

[0040] In step S322, the NUMA file system receives the file system call request.

[0041] In step S323, the synchronization identifier in the file system global information dynamic attribute structure on each node is read, and the synchronization identifier is set to identify that the dynamic attribute on the node is updated.

[0042] In step S324, it is judged whether the node with the set synchronization identifier is 0, if yes, it indicates that synchronization is not needed, otherwise S325 synchronization operation is executed.

[0043] In steps S325 and S326, the value on the master node is updated according to the data block quantity, inode quantity and other values in the dynamic attribute with the set 1. According to the time in the dynamic attribute with the set 1, the latest time is found, and the time on the master node is updated.

[0044] In step S327, the user request is returned according to the latest file system global information.

[0045] Figure 4 The schematic diagram of file metadata redundancy dynamic attribute management mechanism. File metadata is used to describe the characteristics of a file, including owner, access time, permission, file identifier, etc. Most file accesses need to access file metadata first, and the access and modification frequency of file metadata is much higher than that of file data. The efficiency of file metadata management has an important influence on the I / O performance of the file system. The various attributes of file metadata are different in access characteristics, size and purpose, and are divided into static attributes and dynamic attributes according to the access frequency. Based on the above analysis, the present application stores the dynamic attributes of the file metadata on the current node in the node where the file data is located according to the access frequency of the file metadata attributes. For example Figure 4As shown, all file metadata in the file system is distributed on a single node, such as node 0, and is managed by an index node table. In order to quickly locate the file metadata inode of a specific file, the index node table is managed by an array, and the free index nodes in the array are linked by a linked list to achieve fast allocation and recovery. A space is reserved on the node where the file data is located for storing the metadata dynamic attributes of the file on the current node redundantly. During the file read / write access process, the read / write operation is completed by the redundant file metadata dynamic attributes on the local node, avoiding frequent access to the file metadata on the remote node.

[0046] Figure 5 The working process of the file metadata redundant dynamic attribute synchronization mechanism is shown in the flowchart. On the one hand, the frequency of file metadata access operations is lower than that of file read / write operations in actual application. On the other hand, in order to avoid remote access to file metadata during file access, the file metadata dynamic attributes are redundantly backed up on node 0 and the node where the file data is located. Therefore, active synchronization is adopted between the two copies of file metadata dynamic attributes, and after the file read / write process is completed, the hot attributes of the node where the file is located are actively synchronized to the master node, so as to facilitate the later access operation of file metadata. Figure 5 The working process of the file metadata redundant dynamic attribute synchronization mechanism is illustrated as follows:

[0047] In step S501, the user calls the VFS layer system call to read / write file data.

[0048] In step S502, the NUMA file system receives the file system call request.

[0049] In step S503, the file read / write request operation is completed by the file metadata dynamic attributes on the local node.

[0050] In step S504, the updated file metadata dynamic attributes are remotely synchronized to the master node.

[0051] In step S505, the file read / write request of the user is returned.

[0052] Figure 6A multi-node centralized directory tree management mechanism diagram. The multi-node centralized directory tree management mechanism manages the directory structure in the NUMA persistent memory file system, allocates space on a single node to construct a directory tree, and avoids access to remote persistent memory during the directory retrieval process. The directory tree structure is managed using a B-tree. When mounting the file system, allocate directory inodes and directory file data pages on the master node, for example, node 0, to create the root directory of the file system. When the user calls the VFS system call to create a directory, allocate directory inodes and directory file data pages from the master node to build a B-tree directory structure. When the user creates a file, allocate a file inode on the master node, allocate a file metadata dynamic attribute structure on the target node by polling, and copy the dynamic attributes in the master node file inode to the metadata dynamic attribute structure on the target node. This achieves the purpose of separating directory retrieval and file metadata access from file data read and write.

[0053] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the purpose and scope of the technical solutions, and all should be covered in the scope of the claims of the present application.

Claims

1. A method for managing multi-node metadata of a persistent memory file system in a NUMA system, characterized in that, Comprise: (1) Redundant dynamic attribute metadata management based on access frequency: managing file system global information and file metadata of NUMA persistent memory file system, specifically including (1.1) File system global information redundant dynamic attribute management mechanism, managing file system global information on multiple NUMA nodes; according to the access frequency of each attribute in the file system information, the file system global information attributes are divided into static attributes and dynamic attributes, and the dynamic attributes are redundantly stored in the starting position of each node, and in the file access process, the redundant file system global information dynamic attributes on the local node are accessed to avoid the influence of remote file system global information access on file data access performance, and a synchronization flag attribute is added in the file system global information structure of each node; (1.2) File system global information redundant dynamic attribute synchronization mechanism, synchronizing file system information dynamic attributes between multiple nodes through active access synchronization and timing synchronization mechanisms; a synchronization identification of whether the field is modified is added in the structure, and the synchronization identification field is set whenever the file system global information is modified; when there is a modification to the file system global information redundant dynamic attributes, save it in the redundant backup of the local node first, and then synchronize it to the file system global information of node 0 according to the synchronization identification and modification time modification order when the user actively accesses the file system global information or the synchronization period arrives; (1.3) File metadata redundant dynamic attribute management mechanism, distributing all file metadata in the file system on a single node, and managing all file metadata attributes through the inode table; in order to quickly locate the file metadata inode of a specific file, the inode table is managed by an array, and the free inodes in the array are linked by a linked list to realize fast allocation and recovery; a space is reserved in the node where the file data is located for redundantly storing the dynamic attributes of the file metadata on the current node; during file read and write access, read and write are completed through the redundant file metadata dynamic attributes on the local node; (1.4) File metadata redundant dynamic attribute synchronization mechanism, synchronizing file metadata dynamic attributes between the master node and the node where the file data is located through active access synchronization mechanism; Specifically including Step S501, the user calls VFS layer system call to read and write file data; Step S502, the NUMA file system receives the file system call request; Step S503, complete the file read and write request operation through the file metadata dynamic attributes on the local node; Step S504, remotely synchronize the updated file metadata dynamic attributes to the master node; Step S505, return the file read and write request of the user; (2) Centralized directory management: managing the directory structure in the NUMA persistent memory file system, allocating space on a single node to construct a directory tree, and avoiding directory retrieval process access to remote persistent memory; The directory tree is managed by B-tree, and when the file system is mounted, the directory inode and directory file data page are allocated on the master node to create the root directory of the file system; When a user calls a VFS system call to create a directory, a directory inode and directory file data pages are allocated from the master node to build a B-tree directory structure; when a user creates a file, a file inode is allocated on the master node, and a file metadata dynamic attribute structure is allocated on the target node in a polling manner, and dynamic attributes in the master node file inode are copied to the metadata dynamic attribute structure on the target node. 2.The method of claim 1, wherein: The static attributes are constant during the file system access process, including system total size, data block size, Volume Name, log space offset, inode table offset, and mounting time; the dynamic attributes are constantly changing during the file system access process, including file system write time, allocated block quantity, remaining data block quantity, allocated inode quantity, and remaining inode quantity.

3. The method of claim 1, wherein the NUMA system is a persistent memory file system multi-node metadata management method. The timing synchronization in step (1.2) comprises: In step S311, the system creates a periodic synchronization kernel thread when mounting the file system; In step S312, the synchronization kernel thread judges whether the time interval period λ arrives, if the time does not arrive, it continues to wait, otherwise, it executes S313; In step S313, the synchronization kernel thread reads the synchronization identifier in the dynamic attribute structure of the file system global information on each node, and the synchronization identifier is set to indicate that the dynamic attribute on the node is updated; In step S314, it is judged whether the node with the set synchronization identifier is 0, if yes, it indicates that synchronization is not needed, otherwise, S315 synchronization operation is executed; In step S315, the values on the master node are updated according to the data block quantity, inode quantity and other values in the set 1 dynamic attribute; In step S316, the latest time is found according to the time in the set 1 dynamic attribute, and the time on the master node is updated; In step S317, it is judged whether the file system is unloaded, if yes, the kernel thread is terminated, otherwise, it is continued to judge whether the synchronization period arrives; The active access synchronization comprises: In step S321, a user calls a VFS layer system call to access file system information; In step S322, the NUMA file system receives a file system call request; In step S323, the synchronization identifier in the dynamic attribute structure of the file system global information on each node is read, and the synchronization identifier is set to indicate that the dynamic attribute on the node is updated; In step S324, it is judged whether the node with the set synchronization identifier is 0, if yes, it indicates that synchronization is not needed, otherwise, S325 synchronization operation is executed; In step S325, the values on the master node are updated according to the data block quantity, inode quantity and other values in the set 1 dynamic attribute; In step S326, the latest time is found according to the time in the set 1 dynamic attribute, and the time on the master node is updated; In step S327, the latest file system global information is returned to the user request.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the NUMA system persistent memory file system multi-node metadata management method in any one of claims 1 to 3.

5. A system architecture suitable for use in the management method of any one of claims 1 to 3, characterized by: The file system global information redundancy dynamic attribute management module, the file system global information redundancy dynamic attribute synchronization module, the file metadata redundancy dynamic attribute management module, the file metadata redundancy dynamic attribute synchronization module and the multi-node centralized directory tree management module are used for dividing processors, DRAMs and persistent memories into node management in a NUMA architecture server, connecting nodes through a high-speed interconnection bus, using the persistent memories on multiple nodes as data storage devices of the persistent memory file system, managing the multi-node persistent memory space by the persistent memory file system, and storing file system global information, file metadata and file data in the persistent memories. The file system global information redundancy dynamic attribute management module is used for managing redundant file system global information attributes on multiple nodes, the file system global information redundancy dynamic attribute synchronization module is used for synchronizing dynamic attributes on multiple nodes, the file metadata redundancy dynamic attribute management module is used for managing redundant file metadata dynamic attributes on multiple NUMA nodes, the file metadata redundancy dynamic attribute synchronization module is used for synchronizing dynamic attributes on node 0 and a file node, and the multi-node centralized directory tree management module is used for managing directories and files in the NUMA persistent memory file system.

Images