File system metadata management method, terminal device and computer storage medium

Abstract

Disclosed in the present application are a file system metadata management method, a terminal device and a computer-readable storage medium. The data file management method of the present application comprises: on the basis of a path name corresponding to a file needing to be stored, performing encoding to obtain an initial key value format corresponding to the file; allocating a corresponding pre-allocated index value to the initial key value format so as to obtain a pre-allocated key value format; on the basis of the pre-allocated key value format, executing a merging operation on a pre-allocated index management space and a parent directory index management space corresponding to the pre-allocated key value format to obtain a target index management space; and obtaining a target key value format, and, on the basis of the target key value format, storing the file into a target storage structure.

Description

File system metadata management method, terminal device and computer storage medium

[0001] This application claims priority to Chinese patent application No. 202310695689.3, filed on June 12, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of data processing technology, and in particular to a metadata management method for a file system, a terminal device, and a computer-readable storage medium. Background Art

[0003] Currently, big data processing frameworks such as MapReduce and Spark often need to frequently create and update file information when faced with scenarios where a large number of files are created. As a result, the file management system needs to read a large number of disk files, making it difficult to meet users' low-latency requirements. Metadata read and write latency is a key factor affecting application performance in many scenarios.

[0004] Therefore, how to make the file management system meet users' low-latency requirements for reading and writing data when facing scenarios where a large number of files are created has become a technical problem that needs to be urgently solved in the industry. Technical issues

[0005] The main purpose of this application is to provide a metadata management method, apparatus, terminal device and computer-readable storage medium for a file system, aiming to enable a file management system to store file data continuously and adjacently on a file data storage device when storing file data, thereby reducing the number of times the device accesses the storage device when reading file data, improving the performance of the file management system, and further enabling the file management system to meet the user's low-latency requirements when facing scenarios of large-scale file creation and deletion. Technical Solutions

[0006] To achieve the above objectives, the present application provides a metadata management method for a file system, the metadata management method for a file system comprising the following steps:

[0007] Obtain the path name corresponding to the file to be stored, and encode the path name to obtain the initial key value format corresponding to the file;

[0008] Performing a pre-allocation operation through a preset pre-allocated index management space to allocate a corresponding pre-allocated index value to the initial key value format, thereby obtaining a pre-allocated key value format;

[0009] Performing a merge operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the pre-allocated key value format to obtain a target index management space;

[0010] The initial key value format is assigned a corresponding target index value according to the index value contained in the target index management space to obtain a target key value format, and the file is stored in a target storage structure based on the target key value format.

[0011] Furthermore, the step of encoding based on the path name to obtain the initial key value format corresponding to the file includes:

[0012] Determine the file parent directory, file name and file attribute information corresponding to the path name;

[0013] The initial key value format corresponding to the file is obtained by encoding based on the parent directory index value corresponding to the parent directory of the file, the file name and the file attribute information.

[0014] Furthermore, the step of performing a merging operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format to obtain a target index management space based on the pre-allocated key value format includes:

[0015] Comparing the pre-allocated index value contained in the pre-allocated key value format with the parent directory index value to obtain a first comparison result;

[0016] If it is determined that the first comparison result is that the difference between the pre-allocated index value and the parent directory index value is greater than a preset first threshold, a merge operation is performed on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the parent directory index value to obtain the target index management space.

[0017] Furthermore, the step of performing a merging operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the parent directory index value to obtain a target index management space includes:

[0018] Determining whether a space allocation value corresponding to the pre-allocated index management space is less than a preset second threshold;

[0019] If it is determined that the space allocation value is less than the second threshold, the pre-allocated index management space is merged into the parent directory index management space to obtain a target index management space.

[0020] Furthermore, the step of performing a merging operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format to obtain a target index management space based on the parent directory index value further includes:

[0021] Determine whether the reserved space corresponding to the parent directory index management space is used up;

[0022] If it is determined that the reserved space is exhausted, determining the number of allocated nodes corresponding to the pre-allocated index management space and the parent directory index management space;

[0023] A node balancing operation is performed on the pre-allocated index management space and the parent directory index management space based on the number of each allocated node, and the pre-allocated index management space after the balancing operation is performed is determined as a target index management space.

[0024] Furthermore, after the step of storing the file in the target storage structure based on the target key-value format, the method further includes:

[0025] detecting a foreground load upon receiving a file deletion instruction, and determining whether the foreground load is greater than or equal to a preset third threshold;

[0026] If it is determined that the foreground load is greater than or equal to the third threshold, inserting a preset all-delete key into the target file directory corresponding to the file deletion instruction;

[0027] If it is determined that the foreground load is less than the third threshold, a preset delete key is recursively inserted into the target file directory and each sub-file directory corresponding to the target file directory.

[0028] Furthermore, after the step of storing the file in the target storage structure based on the target key-value format, the method further includes:

[0029] Upon receiving a file access instruction, determining a target file according to the file access instruction, and detecting an access frequency corresponding to the target file;

[0030] Determining whether the data type corresponding to the target file is hot data based on the access frequency;

[0031] If it is determined that the data type is the hot data, determining the target index management space corresponding to the target file;

[0032] Based on the target index management space, similar hot data files corresponding to the target file are determined, and the similar hot data files are inserted into a preset log merging storage structure tree.

[0033] Furthermore, before the step of performing a pre-allocation operation using a preset pre-allocated index management space to allocate a corresponding pre-allocated index value to the initial key-value format to obtain the pre-allocated key-value format, the method further includes:

[0034] Determining duplicate files that are duplicates of the file based on the initial key-value format;

[0035] A merging and storing operation is performed on the file and each of the duplicate files.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a terminal device, which includes: a memory, a processor, and a metadata management program of a file system stored on the memory and runnable on the processor, and when the metadata management program of the file system is executed by the processor, the steps of the metadata management method of the file system as described above are implemented.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, on which a metadata management program of a file system is stored. When the metadata management program of the file system is executed by a processor, the steps of the metadata management method of the file system as described above are implemented.

[0038] The metadata management method, apparatus, terminal device and computer medium of the file system provided by the embodiments of the present application obtain the path name corresponding to the file to be stored, and encode based on the path name to obtain the initial key value format corresponding to the file; perform a pre-allocation operation through a preset pre-allocated index management space to assign a corresponding pre-allocated index value to the initial key value format to obtain the pre-allocated key value format; perform a merge operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the pre-allocated key value format to obtain a target index management space; assign a corresponding target index value to the initial key value format according to the index value contained in the target index management space to obtain a target key value format, and store the file in a target storage structure based on the target key value format.

[0039] In this embodiment, when the terminal device is running, the file management system in the terminal device first obtains the file to be stored and determines the path name corresponding to the file, and then encodes the file based on the file name to obtain the initial key value format corresponding to the file. Afterwards, the file management system obtains the preset pre-allocated index management space, and performs a pre-allocation operation on the initial key value format corresponding to the file to be stored based on the pre-allocated index value space, thereby assigning the corresponding pre-allocated index value to the initial key value format to generate a pre-allocated key value format. Afterwards, the file management system performs a merge operation on the pre-allocated index management space and the parent directory index management space based on the pre-allocated index value and the parent directory index value contained in the pre-allocated key value format to generate a target index management space. Finally, the file management system assigns the corresponding target index value to the initial key value format corresponding to the file according to the index value contained in the target index management space to generate the corresponding target key value format. The file management system then saves the file in the storage structure corresponding to the storage device according to the target key value format. Beneficial effects

[0040] This application achieves the technical effect of reducing the number of times the device accesses the storage device during the process of reading file data and improving the performance of the file management system by allowing the file management system to store files in continuous locations within the storage structure based on the continuous index values ​​contained in the index management space. This enables the file management system to meet the user's low-latency requirements when facing scenarios where a large number of files are created and deleted. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] FIG1 is a schematic diagram of the structure of a terminal device in a hardware operating environment according to an embodiment of the present application;

[0042] FIG2 is a flowchart of a first embodiment of a metadata management method for a file system of the present application;

[0043] FIG3 is a schematic diagram of the layout of the index management space involved in an embodiment of the metadata management method for a file system of the present application;

[0044] FIG4 is a schematic diagram of a system structure involved in an embodiment of a metadata management method for a file system of the present application.

[0045] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. Modes for Carrying Out the Invention

[0046] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0047] Refer to Figure 1, which is a schematic diagram of the terminal device structure of the hardware operating environment involved in the embodiment of the present application.

[0048] It should be noted that Figure 1 is a schematic diagram of the hardware operating environment of a terminal device. The terminal device of the embodiment of the present application can be a device that executes the metadata management method of the file system of the present application, and the terminal device can specifically be a mobile terminal, a data storage control terminal, a PC, or a portable computer.

[0049] As shown in Figure 1, the terminal device may include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and memory 1005. Communication bus 1002 is used to enable communication between these components. User interface 1003 may include a display and an input unit, such as a keyboard. User interface 1003 may also include a standard wired interface or a wireless interface. Network interface 1004 may include a standard wired interface or a wireless interface (such as a wireless fidelity (WI-FI) interface). Memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as disk storage. Memory 1005 may also be a storage device independent of processor 1001.

[0050] Those skilled in the art will understand that the structure shown in FIG1 does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0051] As shown in FIG. 1 , the memory 1005 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module, and a metadata management program of a file system.

[0052] In the terminal device shown in Figure 1, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the terminal device of the present application can be set in the terminal device, and the terminal device calls the metadata management program of the file system stored in the memory 1005 through the processor 1001, and executes the metadata management method of the file system provided in the embodiment of the present application.

[0053] Based on the above-mentioned terminal device, various embodiments of the metadata management method of the file system of the present application are provided.

[0054] It's important to note that the local file system for storage devices is the foundation of all data storage and management systems. For example, mainstream databases are built on file systems, and distributed file systems must rely on local file systems to organize data on devices. Big data storage systems like HBase and Dynamo are either built on distributed file systems or directly use local file systems to read and write data on storage devices. Therefore, the performance of local file systems has a crucial impact on all data storage and management systems.

[0055] The local file system can be broadly divided into two structural components: namespace management and address space management. Address space management is primarily responsible for organizing data on the storage device. Generally speaking, regardless of the technology used to implement the storage device, the device manufacturer abstracts the device's physical address space into a linear logical address space. The address space management module is responsible for organizing user data and file system metadata within this linear logical address space. Therefore, the primary function of namespace management is to organize and manage file and directory names, provide users with a file system interface, and implement the path resolution process from file name to file information.

[0056] Common classifications of namespace management include: hierarchical, flat, hash, and so on. Among them, a hierarchical file system refers to a parent-child relationship between files and directories, forming a tree structure, and each node of the tree represents a file or directory. This structure allows files and directories in the file system to be easily organized and managed. In this structure, each node has a unique path name that is used to identify the position of the node in the tree structure. For example, / usr / local / bin is a path name, which represents the bin directory under the local directory under the usr directory under the root directory. When performing path resolution, it is necessary to start from the root node of the tree, access the intermediate directory files from top to bottom, and finally obtain the target file information;

[0057] Similarly, a flat file system places all files and directories on the same level, without parent-child relationships. A hash file system uses a hash function to calculate a unique value for each file name, and then uses this value to store and search for files. Hierarchical namespace management. Compared to hierarchical namespace management, it simplifies path resolution, eliminating the need for sequential resolution of each level. However, it does not preserve the relationships between adjacent files at the same level, making operations involving multiple files more complex, such as renaming files and listing directory file information.

[0058] Different file systems employ different namespace management methods. For example, the Ext family of file systems (such as Ext2, Ext3, and Ext4) uses a hierarchical structure to manage namespaces, defining directory-type files to store information about files at the same level. During path lookup, for each level in the target path, the Ext family of file systems must search the target directory's inode (index node) and parse the directory contents to obtain the inode number of the next level. This results in a significant amount of disk I / O (input / output) during path lookup, not only due to directory resolution but also due to its inode prefetching mechanism. However, this mechanism is ineffective when resolving deep paths, as Ext struggles to ensure the vertical locality of metadata on disk. Ext directory files store file information linearly on the storage device. When the number of subfiles in a directory is large, searching for a single subfile requires traversing all files, significantly degrading query performance. Ext3 and later optimized the arrangement of subfiles stored in directory files, using hashing and B+ trees to organize subfiles, accelerating file lookups at the same level. However, when facing write-intensive workloads, the performance of the Ext file system will be greatly reduced due to the influence of a large number of random small writes. In addition, when there are many files and the file paths are deep, the performance of path resolution will also be reduced. For example, if the file depth is N, N disk I / Os are required, which greatly reduces the performance of path resolution.

[0059] Similarly, BtrFS is a new copy-on-write (CoW) Linux file system that uses a B-tree to store directory entries, making file searches within a given directory highly efficient. To meet the requirements of readdir, BtrFS inserts a directory entry keyed by a hash value and a separate directory entry index each time a new file is created. This directory entry index uses a sequence number as the B-tree key. The sequence number increases linearly with each new file creation, ensuring that readdir reads the files in order. While BtrFS's file search efficiency within a given directory is very high, the B-tree grows deeper with a large number of files and long file paths, increasing the latency of searching the target file within the B-tree and degrading file access performance. Furthermore, similar to the Ext family of file systems, performance can be significantly degraded under write-intensive workloads, impacted by large numbers of random small writes.

[0060] Similarly, TableFS is an LSM-Tree-based file system, stacked on top of the underlying file system. It ensures that metadata is written to disk in a non-overwriting, sorted log format. Due to the write-friendly nature of LSM trees, TableFS is highly suitable for write-intensive scenarios, allowing random, small writes to disk in a sequential manner. However, LSM-Trees are inherently read-insensitive, so using them directly as metadata storage structures results in poor read performance. TableFS's file read operations (such as stat) are significantly inferior to those of common file systems like ext4. This is because path resolution requires multiple LSM-Tree reads, particularly when files on a path are not in the same SSTable. Multiple disk accesses are required, resulting in poor read performance and high latency. Consequently, current major data processing frameworks often require frequent file creation and updates when dealing with large file creation scenarios. This necessitates a large number of disk reads, making it difficult to meet user requirements for low latency. Metadata read and write latency is a critical factor affecting application performance in many scenarios. Therefore, how to design a local file system that is write-intensive and has low latency in file path resolution when a large number of files are created has become a technical problem that needs to be solved urgently in the industry.

[0061] In response to the above technical problems, the present application proposes a metadata management method for a file system, a terminal device, and a computer-readable storage medium, which obtains the path name corresponding to the file to be stored and encodes the path name to obtain the initial key value format corresponding to the file; performs a pre-allocation operation through a preset pre-allocated index management space to assign a corresponding pre-allocated index value to the initial key value format to obtain a pre-allocated key value format; performs a merge operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the pre-allocated key value format to obtain a target index management space; assigns a corresponding target index value to the initial key value format according to the index value contained in the target index management space to obtain a target key value format, and stores the file in a target storage structure based on the target key value format. In this way, the present application achieves the technical effect of reducing the number of times the device accesses the storage device during the process of reading file data and improving the performance of the file management system by allowing the file management system to store files in consecutive locations within the storage structure based on the consecutive index values ​​contained in the index management space, thereby enabling the file management system to meet the user's low latency requirements when facing scenarios where a large number of files are created or deleted.

[0062] Please refer to FIG. 2 , which is a flowchart of a first embodiment of a metadata management method for a file system of the present application.

[0063] It should be understood that although a logical order is shown in the flowchart, in some cases, the metadata management method of the file system of the present application can of course also execute the steps shown or described in a different order than here.

[0064] In this embodiment, the metadata management method of the file system of the present application may include the following steps:

[0065] Step S10: obtaining a path name corresponding to a file to be stored, and encoding the path name to obtain an initial key value format corresponding to the file;

[0066] In this embodiment, when the terminal device is running, it first obtains the file to be stored, and the file management system in the terminal device determines the path name corresponding to the file, and encodes it based on the preset encoding format and the path name to obtain the initial key value format.

[0067] For example, when the terminal device is running, it first obtains the file to be stored, and the file management system in the terminal device determines the path name corresponding to the file. After that, the file management system encodes the path name according to the preset key value format to obtain the initial key value format corresponding to the file:<pinode:filename,stat> .

[0068] It is understandable that the above key-value format is more easily accepted than the format in the current technology that uses the full path of the directory or file as the key, because it has lower overhead for performing move or rename operations.

[0069] Furthermore, in a feasible embodiment, the step of "encoding the file to obtain the initial key-value format corresponding to the file" in the above step S10 may specifically include:

[0070] Step S101: Determine the file parent directory, file name, and file attribute information corresponding to the path name;

[0071] Step S102: Encoding the parent directory index value corresponding to the parent directory of the file, the file name, and the file attribute information to obtain an initial key value format corresponding to the file;

[0072] For example, the file management system first determines the file parent directory corresponding to the file according to the path name, thereby determining the parent directory index value pinode corresponding to the parent directory of the file. At the same time, the file management system determines the file name filename corresponding to the file. At the same time, the file management system determines the file attribute information stat corresponding to the file. After that, the file management system encodes the obtained parent directory index value pinode, file name filename and file attribute information stat to obtain the initial key value format corresponding to the file.<pinode:filename,stat> .

[0073] Step S20: performing a pre-allocation operation through a preset pre-allocated index management space to allocate a corresponding pre-allocated index value to the initial key value format, thereby obtaining a pre-allocated key value format;

[0074] Further, please refer to Figures 3 and 4, where Figure 3 is a schematic diagram of the layout of the index management space involved in an embodiment of the metadata management method of the file system of the present application, and Figure 4 is a schematic diagram of the system structure involved in an embodiment of the metadata management method of the file system of the present application; the index management space is a space for managing index values ​​(i.e., inode_number), where the size corresponding to an index value should be set to 64 bits; thus, the index management space can identify 2 64 As shown in Figure 4, since the size of the key value corresponding to a file is 256B, and a 4KiB storage block can accommodate 16 key-value pairs, the default size of the index management space is 16, and a part of the space will be reserved for subsequent merging.

[0075] In addition, the index values ​​in the index management space should be similar in lexicographic order, and each index value in the index management space should contain a minimum index value min_inode_number and a maximum index value max_inode_number. In this way, when a new file is created, the file management system can determine an index value in the index management space inode_space and assign the index node inode corresponding to the index value to the newly created file. When all the index node inodes in the index management space inode_space have been allocated, the file management system can create a new index management space inode_space.

[0076] As shown in Figure 3, the above-mentioned index management space inode_space is divided into an index table in the memory (i.e., the DRAM in Figure 3) and a linear table on the storage device (i.e., the SSD in Figure 3). Among them, the table entries of the linear table include the root node inode number (i.e., the root inode in Figure 3), the maximum inode number max_inode that can be allocated by the index management space inode_space, the total number of reserved inodes step, and the currently allocated number count. It can be understood that the total number of reserved inodes step is the reserved space corresponding to the index management space inode_space, and the allocated number count corresponds to the allocated nodes corresponding to the index management space inode_space.

[0077] In this embodiment, the file management system obtains a preset pre-allocated index management space for performing a pre-allocation operation, and performs an index value pre-allocation operation based on the pre-allocated index management space to allocate a corresponding pre-allocated index value to the generated initial key value format, and the file management system then writes the pre-allocated index value into the initial key value format to generate a pre-allocated key value format. For example, the file management system first obtains a preset pre-allocated index management space inode_space1 for performing a pre-allocation operation, and performs a pre-allocation operation based on the pre-allocated index management space inode_space1 to allocate a corresponding pre-allocated index value to the generated initial key value format.<pinode:filename,stat> Assign the corresponding pre-allocated index value cur_inode, and then the file management system writes the allocated pre-allocated index value cur_inode into the initial key value format to obtain the pre-allocated key value format:<pinode:file_name,cur_inode…> .

[0078] It's understandable that the pre-allocated key-value format ensures that the keys of subfiles in the same directory all begin with the inode of the parent directory, allowing subfiles in the same directory to be stored contiguously on the storage device. Since storage devices typically read and write at the KB level, such as the minimum read granularity of an SSD at 4KiB, a single I / O can read metadata for files located adjacently on multiple storage devices, significantly improving the performance of file system range queries.

[0079] Step S30: performing a merge operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the pre-allocated key value format to obtain a target index management space;

[0080] In this embodiment, the file management system performs a merge operation on the pre-allocated index management space corresponding to the pre-allocated key value format and the parent directory index management space based on the pre-allocated index value and the parent directory index value contained in the pre-allocated key value format, thereby obtaining a target index management space. For example, the file management system performs a merge operation on the pre-allocated index management space corresponding to the pre-allocated key value format and the parent directory index management space based on the pre-allocated key value format.<pinode:file_name,cur_inode …> The parent directory index management space inode_space2 corresponding to the pre-allocated index value cur_inode and the parent directory index value pinode contained therein is merged with the pre-allocated index management space inode_space1 to obtain the target index management space inode_space.

[0081] Furthermore, in a feasible embodiment, the above step S30 may specifically include:

[0082] Step S301: Compare the pre-assigned index value contained in the pre-assigned key value format with the parent directory index value to obtain a first comparison result;

[0083] In this embodiment, the file management system first determines the preallocated index value and the parent directory index value contained in the preallocated key value format, and compares the preallocated index value with the parent directory index value to obtain the difference between the preallocated index value and the parent directory index value. At the same time, the file management system obtains a first threshold value preset by a technician, and compares the difference with the first threshold value to obtain a first comparison result. The file management system then determines whether the difference between the preallocated index value and the parent directory index value is too large based on the first comparison result.

[0084] Step S302: If it is determined that the first comparison result is that the difference between the pre-allocated index value and the parent directory index value is greater than a preset first threshold, a merge operation is performed on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the parent directory index value to obtain a target index management space;

[0085] In this embodiment, if the file management system determines that the first comparison result is that the above-mentioned difference is greater than or equal to the preset first threshold, it determines that the difference between the pre-allocated index value and the parent directory index value is too large, and thus performs a merge operation on the parent directory index management space and the pre-allocated index management space based on the above-mentioned parent directory index value to obtain the target index management space.

[0086] For example, the file management system first determines the pre-assigned key format<pinode:file_name,cur_inode …> The pre-allocated index value cur_inode and the parent directory index value pinode contained in the file management system are compared with each other to obtain the difference between the two. At the same time, the file management system obtains a preset first threshold value and compares the difference with the first threshold value to obtain a first comparison result. Afterwards, if the file management system determines that the first comparison result is that the difference is greater than or equal to the first threshold value, it is determined that the difference between the pre-allocated index value cur_inode and the parent directory index value pinode is too large. The file management system then compares the pre-allocated key value format based on the pre-allocated key value format.<pinode:file_name,cur_inode …> The parent directory index management space inode_space2 corresponding to the internal parent directory index value pinode and the pre-allocated index management space inode_space1 corresponding to the pre-allocated index value cur_inode are merged to obtain the target index management space inode_space.

[0087] In addition, in another embodiment, the file management system can also calculate the idle rate α=1-count / (i max_inode-root_inode) corresponding to the pre-allocated index management space inode_space1, and compare the idle rate with the preset idle rate threshold. Afterwards, if the file management system determines that the idle rate is higher than the idle rate threshold, it determines to perform a merge operation.

[0088] Furthermore, in a feasible embodiment, the step of "merging the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the parent directory index value to obtain the target index management space" in step S302 may specifically include:

[0089] Step S3021: determining whether the space allocation value corresponding to the pre-allocated index management space is less than a preset second threshold;

[0090] Step S3022: If it is determined that the space allocation value is less than the second threshold, the pre-allocated index management space is merged into the parent directory index management space to obtain a target index management space;

[0091] Exemplarily, for example, when the file management system determines to perform a merge operation, it first obtains the second threshold preset by the technician, and compares the corresponding space allocation number in the pre-allocated index management space inode_space1 with the second threshold to obtain a second comparison result. Afterwards, if the file management system determines that the second comparison result is that the space allocation number is less than the second threshold, it determines that there are fewer subfiles in the index management space inode_space1. Therefore, there is no need to maintain the locality within the index management space inode_space1. The file management system then merges the pre-allocated index management space inode_space1 into the above-mentioned parent directory index management space inode_space2, thereby using the merged parent directory index management space inode_space2 as the target index management space inode_space.

[0092] Furthermore, in a feasible embodiment, the step of "merging the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the parent directory index value to obtain a target index management space" in step S302 may further include:

[0093] Step S3023: Determine whether the reserved space corresponding to the parent directory index management space is used up;

[0094] In this embodiment, when the file management system determines to perform a merge operation, it first determines whether the reserved space contained in the parent directory index management space is used up.

[0095] Step S3024: If it is determined that the reserved space is exhausted, the number of allocated nodes corresponding to the pre-allocated index management space and the parent directory index management space is determined;

[0096] In this embodiment, if the file management system determines that the reserved space has been used up, it determines that a split operation needs to be performed on the parent directory index management space, and then the pre-allocated index management space and the parent directory index management space are detected respectively to determine the number of allocated nodes corresponding to the pre-allocated index management space and the parent directory index management space.

[0097] Step S3025: performing a node balancing operation on the pre-allocated index management space and the parent directory index management space based on the number of each allocated node, and determining the pre-allocated index management space after performing the balancing operation as the target index management space;

[0098] In this embodiment, the file management system performs node balancing operations on the parent directory index management space and the pre-allocated index management space based on the obtained number of allocated nodes, thereby migrating some nodes in the parent directory index management space to the pre-allocated index management space to balance the number of nodes in the two index management spaces. The file management system then determines the pre-allocated index management space after the balancing operation as the target index management space.

[0099] For example, when the file management system determines to perform a merge operation, it first determines the working status corresponding to the reserved space contained in the parent directory index management space inode_space2, and determines whether the reserved space is used up based on the working status. Afterwards, if the file management system determines that the reserved space is used up, it determines that it is necessary to perform a split operation of the index management space, and detects the number of allocated nodes corresponding to the pre-allocated index management space inode_space1 and the parent directory index management space inode_space2, thereby determining the inode node range [x, y] that needs to be migrated. At the same time, the file management system determines that the storage Level involved in the migration operation is L k and L k-1 , thus L0 to L in the file management system k Search between layers for key-value pairs that satisfy the condition x:*<= Key<= y:*, and modify the inode of the key-value pair to new_inode. In this way, the file management system determines the average number of nodes corresponding to the pre-allocated index management space inode_space1 and the parent directory index management space inode_space2 based on the obtained number of allocated nodes, and based on the average number of nodes, migrates some nodes contained in the parent directory index management space inode_space2 to the pre-allocated index management space inode_space1 according to the above migration operation to complete the node balancing operation, thereby determining the balanced pre-allocated index management space inode_space1 as the target index management space and updating the corresponding index table information.

[0100] Step S40: assigning a corresponding target index value to the initial key value format according to the index value contained in the target index management space to obtain a target key value format, and storing the file in a target storage structure based on the target key value format;

[0101] In this embodiment, after completing the merge operation, the file management system performs an index allocation operation based on the index value contained in the obtained target index management space, thereby assigning the corresponding target index value to the initial key value format to generate a target key value format, and the file management information then stores the file in the corresponding storage structure according to the target key value format.

[0102] For example, after completing the merge operation, the file management system performs an inode allocation operation based on the target index management space inode_space, thereby providing the initial key value format<pinode:filename,stat> The corresponding target index value cur_inode is allocated to obtain a target key value format. Afterwards, the file management system stores the corresponding file in the target storage structure based on the target key value format.

[0103] In this embodiment, when the terminal device is running, it first obtains the file to be stored, and the file management system in the terminal device determines the path name corresponding to the file, and encodes the file according to a preset encoding format and the path name to obtain an initial key value format. Afterwards, the file management system obtains a preset pre-allocated index management space for performing a pre-allocation operation, and performs an index value pre-allocation operation based on the pre-allocated index management space to assign a corresponding pre-allocated index value to the generated initial key value format. The file management system then writes the pre-allocated index value into the initial key value format to generate a pre-allocated key value format. Afterwards, the file management system performs a merge operation on the pre-allocated index management space corresponding to the pre-allocated key value format and the above-mentioned parent directory index management space based on the pre-allocated index value and the parent directory index value contained in the pre-allocated key value format to obtain a target index management space. Finally, after completing the merge operation, the file management system performs an index allocation operation based on the index value contained in the obtained target index management space to assign a corresponding target index value to the initial key value format to generate a target key value format. The file management information then stores the file in the corresponding storage structure according to the target key value format.

[0104] Thus, by having the file management system store files in consecutive locations within the storage structure based on the consecutive index values ​​contained in the index management space, the present application achieves the technical effect of reducing the number of times a device accesses the storage device when reading file data, thereby improving the performance of the file management system. This, in turn, enables the file management system to meet the user's low-latency requirements when faced with scenarios involving the creation and deletion of large numbers of files. Furthermore, based on the first embodiment of the metadata management method for the file system of the present application, a second embodiment of the metadata management method for the file system of the present application is proposed.

[0105] After step S40, the metadata management method of the file system of the present application may further include the following steps:

[0106] Step A10: detecting the foreground load upon receiving the file deletion instruction, and determining whether the foreground load is greater than or equal to a preset third threshold;

[0107] In this embodiment, when the file management system receives a file deletion instruction, it obtains a third threshold value preset by a technician. At the same time, the file management system detects the terminal device based on the file deletion instruction, thereby obtaining the current foreground load of the terminal device, and compares the foreground load with the third threshold value to obtain a third comparison result.

[0108] Step A20: If it is determined that the foreground load is greater than or equal to the third threshold, inserting a preset all-delete key into the target file directory corresponding to the file deletion instruction;

[0109] In this embodiment, if the file management system determines that the third comparison result is that the foreground load is greater than or equal to the above-mentioned third threshold, it determines the target file directory to be deleted according to the file deletion instruction, and inserts the preset full delete key value into the target file directory, so that when the file management system performs the merge operation, the full delete key performs the file recovery operation to complete the deletion operation of the target file directory.

[0110] Step A30: If it is determined that the foreground load is less than the third threshold, recursively inserting a preset delete key into the target file directory and each sub-file directory corresponding to the target file directory;

[0111] In this embodiment, if the file management system determines that the third comparison result is that the foreground load is less than the above-mentioned third threshold, it determines the target file directory to be deleted and the subdirectories under the target file directory based on the file deletion instruction. The file management system then recursively inserts a delete key under the target file directory and each subdirectory to remove the file directory marked with the delete key during the merge operation to complete the deletion operation.

[0112] Exemplarily, for example, when a user needs to delete a file, a file deletion instruction can be generated through a terminal device, and the terminal device then sends the file deletion instruction to the file management system. Later, when the file management system receives the file deletion instruction, it first obtains the third threshold preset by the technician and determines the target file directory corresponding to the target file to be deleted. At the same time, the file management system detects the foreground load corresponding to the terminal device, and the file management system then compares the foreground load with the third threshold to obtain a third comparison result. Thereafter, if the file management system determines that the third comparison result is that the foreground load is greater than or equal to the third threshold, it obtains the preset full delete key full_delete_key and inserts the full delete key full_delete_key into the target file directory, so that the file management system can search the key directory for the file containing the full delete key full_delete_key in the storage structure SSTable, and then when executing the above-mentioned index management space merge operation, the file directory corresponding to the file containing the full delete key full_delete_key is directly deleted;

[0113] If the file management system determines that the third comparison result is that the foreground load is less than the third threshold, it determines the target file directory corresponding to the target file to be deleted and the subdirectories under the target file directory according to the file deletion instruction, and the file management system then recursively inserts the preset deletion key into each of the subdirectories to remove these deletion keys and files with the deletion key from the storage device during the subsequent merging process.

[0114] In this embodiment, when the file management system receives a file deletion instruction, it obtains a third threshold value preset by a technician. At the same time, the file management system detects the terminal device based on the file deletion instruction to obtain the current foreground load of the terminal device, and compares the foreground load with the third threshold value to obtain a third comparison result. Afterwards, if the file management system determines that the third comparison result is that the foreground load is greater than or equal to the above-mentioned third threshold value, it determines the target file directory to be deleted according to the file deletion instruction, and inserts the preset full delete key value into the target file directory, so that when the file management system performs the merge operation, the full delete key performs a file recovery operation to complete the deletion operation of the target file directory. Finally, if the file management system determines that the third comparison result is that the foreground load is less than the above-mentioned third threshold value, it determines the target file directory to be deleted according to the file deletion instruction, and each subdirectory under the target file directory. The file management system then recursively inserts the delete key under the target file directory and each of the subdirectories to remove the file directory marked with the delete key during the merge operation to complete the deletion operation.

[0115] In this way, the present application determines the foreground load of the terminal device, and when the foreground load is higher than a preset threshold, inserts a full delete key into the target file directory to be deleted, and when the foreground load is lower than the threshold, recursively inserts the delete key into the target file directory and the corresponding subdirectories. This enables the file management system to select a more efficient file deletion scheme according to the foreground load when executing the file deletion operation, thereby achieving the goal of quickly recovering space on the storage device and improving the performance of the file management system.

[0116] Furthermore, based on the first embodiment and / or the second embodiment of the metadata management method for the file system of the present application, a third embodiment of the metadata management method for the file system of the present application is proposed herein.

[0117] After step S40, the metadata management method of the file system of the present application may further include the following steps:

[0118] Step B10: upon receiving a file access instruction, determining a target file according to the file access instruction, and detecting an access frequency corresponding to the target file;

[0119] In this embodiment, when the file management system receives a file access instruction, it counts the access frequency of the target file corresponding to the file access instruction through a preset time window.

[0120] Step B20: determining whether the data type corresponding to the target file is hot data based on the access frequency;

[0121] In this embodiment, the file management system obtains a preset fourth threshold and compares the obtained access frequency with the fourth threshold to obtain a fourth comparison result. The file management system then determines whether the data type corresponding to the target file is hot data based on the fourth comparison result.

[0122] Step B30: If it is determined that the data type is hot data, determining the target index management space corresponding to the target file;

[0123] In this embodiment, if the file management system determines that the fourth comparison result is that the access frequency is greater than or equal to the fourth threshold, it determines that the data type corresponding to the target file is hot data, and then determines the corresponding target index value based on the above-mentioned target key value format corresponding to the target file, and searches for the target index management space corresponding to the target index value.

[0124] Step B40: determining each similar hot data file corresponding to the target file based on the target index management space, and inserting each similar hot data file into a preset log merging storage structure tree;

[0125] In this embodiment, the file management system determines each similar hot data file corresponding to the target metadata based on multiple index values ​​contained in the target index management space, and then inserts each similar hot data file into the surface storage structure in the file management system. Exemplarily, for example, when the file management system receives a file access instruction, it opens a time window in the system background, and uses the time window to count the access frequency freq of the target file corresponding to the file access instruction. At the same time, the file management system obtains a preset fourth threshold, and compares the access frequency freq with the fourth threshold to obtain a fourth comparison result. Afterwards, if the file management system determines that the fourth comparison result is that the access frequency freq is greater than or equal to the fourth threshold, then it determines that the data type corresponding to the target file is a hot file filehot, and the file management system further determines the target index value inode corresponding to the target file. hot , and determine the target index value inode hot Corresponding target index management space inode_space hot After that, the file management system will manage the space inode_space based on the index hot Determine each similar hot data file corresponding to the target file, and insert each similar hot data file into a lower-level SSTable in the file management system so that users can access each hot data file faster.

[0126] In this embodiment, when the file management system receives a file access instruction, it counts the access frequency of the target file corresponding to the file access instruction through a preset time window. Afterwards, the file management system obtains a preset fourth threshold value, and compares the obtained access frequency with the fourth threshold value to obtain a fourth comparison result. The file management system then determines whether the data type corresponding to the target file is hot data based on the fourth comparison result. Then, if the file management system determines that the fourth comparison result is that the access frequency is greater than or equal to the fourth threshold value, it determines that the data type corresponding to the target file is hot data, and then determines the corresponding target index value according to the above-mentioned target key value format corresponding to the target file, and searches for the target index management space corresponding to the target index value. Finally, the file management system determines each similar hot data file corresponding to the target metadata based on the multiple index values ​​contained in the target index management space, and then inserts each similar hot data file into the surface storage structure within the file management system.

[0127] Thus, the present application adopts a method of determining each similar hot data file corresponding to the hot data based on the target index value corresponding to the hot data when the accessed data is determined to be hot data, and then inserting each similar hot data file into the surface storage structure within the system, thereby achieving the purpose of improving the access performance of hot data and reducing the delay incurred by users when reading hot data. Furthermore, based on the various embodiments of the metadata management method of the file system of the present application described above, a fourth embodiment of the metadata management method of the file system of the present application is proposed.

[0128] Before the above step S20, the metadata management method of the file system of the present application may further include the following steps:

[0129] Step C10: determining duplicate files that are duplicates of the file based on the initial key value format;

[0130] In this embodiment, the file management system compares the initial key-value formats corresponding to each file to obtain a fifth comparison result, and determines the duplicate files corresponding to the file based on the fifth comparison result.

[0131] Step C20: performing a merge storage operation on the file and each of the duplicate files;

[0132] In this embodiment, the file management system performs a merge storage operation on the file and the duplicate files corresponding to the file.

[0133] For example, the file management system first sets the initial key value format corresponding to each file.<pinode:filename, stat> The fifth comparison results are obtained by performing comparisons respectively. After that, the file management system determines the duplicate files with the same initial key-value format based on the fifth comparison results, and merges the files and the duplicate files for storage, so that the memory table in the file management system can store more key-value pairs.

[0134] In addition, the present application also provides a terminal device having a file system metadata management program that can be run on a processor. When the terminal device executes the file system metadata management program, the steps of the file system metadata management method described in any of the above embodiments are implemented. The specific embodiments of the terminal device of the present application are basically the same as the embodiments of the file system metadata management method described above and are not further described here.

[0135] In addition, the present application also provides a computer-readable storage medium storing a file system metadata management program. When executed by a processor, the file system metadata management program implements the steps of the file system metadata management method described in any of the above embodiments. The specific embodiments of the computer-readable storage medium of the present invention are substantially the same as the above embodiments of the file system metadata management method and are not further described here.

[0136] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0137] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0138] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions for enabling a terminal device (which can be a device that executes the metadata management method of the file system of the present application, and the terminal device can specifically be a mobile terminal, a data storage control terminal, a PC or a portable computer, etc.) to execute the methods described in each embodiment of the present application.

[0139] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A metadata management method for a file system, wherein: The metadata management method of the file system comprises the following steps: Obtain the path name corresponding to the file to be stored, and encode based on the path name to obtain the initial key value format corresponding to the file; Performing a pre-allocation operation through a preset pre-allocated index management space to allocate a corresponding pre-allocated index value to the initial key value format to obtain a pre-allocated key value format; Based on the pre-allocated key value format, a merge operation is performed on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format to obtain a target index management space; The initial key value format is assigned a corresponding target index value according to the index value contained in the target index management space to obtain a target key value format, and the file is stored in a target storage structure based on the target key value format.

2. The metadata management method of a file system according to claim 1, wherein: The step of encoding based on the path name to obtain the initial key value format corresponding to the file includes: Determine the file parent directory, file name and file attribute information corresponding to the path name; The initial key value format corresponding to the file is obtained by encoding based on the parent directory index value corresponding to the parent directory of the file, the file name and the file attribute information.

3. The metadata management method of the file system according to claim 2, wherein: The step of performing a merging operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format to obtain a target index management space based on the pre-allocated key value format comprises: Comparing the pre-allocated index value contained in the pre-allocated key value format with the parent directory index value to obtain a first comparison result; If it is judged that the first comparison result is that the difference between the pre-allocated index value and the parent directory index value is greater than a preset first threshold, a merge operation is performed on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the parent directory index value to obtain the target index management space.

4. The metadata management method of the file system according to claim 3, wherein: The step of performing a merging operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the parent directory index value to obtain a target index management space comprises: Determine whether the space allocation value corresponding to the pre-allocated index management space is less than a preset second threshold; If it is determined that the space allocation value is less than the second threshold, the pre-allocated index management space is merged into the parent directory index management space to obtain a target index management space.

5. The metadata management method of the file system according to claim 4, wherein: The step of performing a merge operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the parent directory index value to obtain a target index management space also includes: Determine whether the reserved space corresponding to the parent directory index management space is used up; If it is determined that the reserved space is exhausted, determining the number of allocated nodes corresponding to the pre-allocated index management space and the parent directory index management space respectively; A node balancing operation is performed on the pre-allocated index management space and the parent directory index management space based on the number of each allocated node, and the pre-allocated index management space after the balancing operation is performed is determined as a target index management space.

6. The metadata management method of a file system according to claim 1, wherein: After the step of storing the file in the target storage structure based on the target key-value format, the method further comprises: When receiving the file deletion instruction, detecting the foreground load, and determining whether the foreground load is greater than or equal to a preset third threshold; If it is determined that the foreground load is greater than or equal to the third threshold, inserting a preset full delete key into the target file directory corresponding to the file deletion instruction; If it is determined that the foreground load is less than the third threshold, a preset delete key is recursively inserted into the target file directory and each sub-file directory corresponding to the target file directory.

7. The metadata management method of a file system according to claim 1, wherein: After the step of storing the file in the target storage structure based on the target key-value format, the method further comprises: When receiving a file access instruction, determining a target file according to the file access instruction, and detecting an access frequency corresponding to the target file; Determining whether the data type corresponding to the target file is hot data based on the access frequency; If it is determined that the data type is the hot data, determining the target index management space corresponding to the target file; Based on the target index management space, similar hot data files corresponding to the target file are determined, and the similar hot data files are inserted into a preset log merging storage structure tree.

8. The metadata management method of a file system according to claim 1, wherein: Before the step of performing a pre-allocation operation through a preset pre-allocated index management space to allocate a corresponding pre-allocated index value to the initial key value format to obtain a pre-allocated key value format, the method further includes: Determine each duplicate file that is duplicate of the file based on the initial key value format; A merging and storing operation is performed on the file and each of the duplicate files.

9. A terminal device, wherein: The terminal device includes: a memory, a processor, and a metadata management program of a file system stored in the memory and executable on the processor. When the metadata management program of the file system is executed by the processor, the steps of the metadata management method of the file system as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores a metadata management program for a file system, and when the metadata management program for the file system is executed by a processor, the steps of the metadata management method for a file system as claimed in any one of claims 1 to 8 are implemented.

Images