Metadata management method of file system, terminal device and computer storage medium

Abstract

The application discloses a metadata management method of a file system, a terminal device and a computer readable storage medium, and relates to the technical field of data processing. The data file management method comprises the following steps: acquiring a path name corresponding to a file to be stored, and encoding the path name to obtain an initial key-value format corresponding to the file; performing a pre-allocation operation on a pre-allocation index management space to allocate a corresponding pre-allocation index value to the initial key-value format, so as to obtain a pre-allocation key-value format; performing a merging operation on the pre-allocation index management space and a parent directory index management space corresponding to the pre-allocation key-value format based on the pre-allocation key-value format, so as to obtain a target index management space; allocating a corresponding target index value to the initial key-value format according to an index value contained in the target index management space, so as to obtain a target key-value format; and storing the file in a target storage structure based on the target key-value format.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a metadata management method of a file system, a terminal device and a computer readable storage medium. BACKGROUND

[0002] Currently, when a large data processing framework such as MapReduce or Spark faces a large number of file creation scenarios, it often needs to frequently create and update file information. In this way, the file management system needs to read a large number of disk files, so it is difficult to meet the low-latency needs of users, and the metadata read-write delay is a key factor affecting application performance in many scenarios.

[0003] Therefore, how to enable the file management system to meet the low-latency needs of users for reading and writing data when facing a large number of file creation scenarios has become a technical problem that needs to be solved in the industry. SUMMARY

[0004] The main purpose of the present application is to provide a metadata management method, device, terminal equipment and computer readable storage medium of a file system, which aims to enable the file management system to store file data continuously and adjacently on a file data storage device when storing file data, thereby reducing the number of times of accessing the storage device in the process of reading file data, improving the performance of the file management system, and thus enabling the file management system to meet the low-latency needs of users when facing a large number of file creation and deletion scenarios.

[0005] To achieve the above purpose, the present application provides a metadata management method of a file system, which comprises the following steps:

[0006] Obtaining a path name corresponding to a file to be stored, and encoding the path name to obtain an initial key-value format corresponding to the file;

[0007] Performing a pre-allocation operation on a pre-allocation index management space to allocate a corresponding pre-allocation index value to the initial key-value format to obtain a pre-allocation key-value format;

[0008] Performing a merging operation on the pre-allocation index management space and a parent directory index management space corresponding to the pre-allocation key-value format based on the pre-allocation key-value format to obtain a target index management space;

[0009] Allocating a corresponding target index value to the initial key-value format according to the index values contained in the target index management space to obtain a target key-value format, and storing the file in a target storage structure based on the target key-value format.

[0010] Further, the step of encoding based on the path name to obtain the initial key-value format corresponding to the file includes:

[0011] determining file parent directory, file name and file attribute information corresponding to the path name;

[0012] encoding based on the parent directory index value corresponding to the file parent directory, the file name and the file attribute information to obtain the initial key-value format corresponding to the file.

[0013] Further, the step of performing merging operation based on the pre-allocated key-value format on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key-value format to obtain the target index management space includes:

[0014] comparing the pre-allocated index value and the parent directory index value contained in the pre-allocated key-value format to obtain a first comparison result;

[0015] If it is judged that the first comparison result is that the difference between the pre-allocated index value and the parent directory index value is greater than a preset first threshold value, performing merging operation based on the parent directory index value on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key-value format to obtain the target index management space.

[0016] Further, the step of performing merging operation based on the parent directory index value on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key-value format to obtain the target index management space includes:

[0017] judging whether the space allocation value corresponding to the pre-allocated index management space is less than a preset second threshold value;

[0018] If it is judged that the space allocation value is less than the second threshold value, merging the pre-allocated index management space into the parent directory index management space to obtain the target index management space.

[0019] Further, the step of performing merging operation based on the parent directory index value on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key-value format to obtain the target index management space further includes:

[0020] judging whether the reserved space corresponding to the parent directory index management space is used up;

[0021] If it is judged that the reserved space is used up, determining the number of allocated nodes corresponding to the pre-allocated index management space and the parent directory index management space respectively;

[0022] Perform a node balancing operation on the pre-allocation index management space and the parent directory index management space based on the number of allocated nodes, and determine the pre-allocation index management space after the balancing operation as a target index management space.

[0023] Further, after the step of storing the file in a target storage structure based on the target key-value format, the method further comprises:

[0024] Detect a foreground load when a file deletion instruction is received, and determine whether the foreground load is greater than or equal to a preset third threshold value;

[0025] If it is determined that the foreground load is greater than or equal to the third threshold value, insert a preset full deletion key into a target file directory corresponding to the file deletion instruction;

[0026] If it is determined that the foreground load is less than the third threshold value, recursively insert a preset deletion key into the target file directory and each sub-file directory corresponding to the target file directory.

[0027] Further, after the step of storing the file in a target storage structure based on the target key-value format, the method further comprises:

[0028] Upon receiving a file access instruction, determine a target file according to the file access instruction, and detect an access frequency corresponding to the target file;

[0029] Determine whether a data type corresponding to the target file is hot data based on the access frequency;

[0030] If it is determined that the data type is the hot data, determine the target index management space corresponding to the target file;

[0031] Determine each similar hot data file corresponding to the target file based on the target index management space, and insert each similar hot data file into a preset log-merge storage structure tree.

[0032] Further, before the step of performing a pre-allocation operation on the initial key-value format to allocate a corresponding pre-allocation index value to the initial key-value format to obtain a pre-allocation key-value format through a preset pre-allocation index management space, the method further comprises:

[0033] Determine each duplicate file duplicated with a file based on the initial key-value format;

[0034] Perform a merge storage operation on the file and each duplicate file.

[0035] In addition, to achieve the above object, the present application also provides a terminal device, comprising a memory, a processor, and a metadata management program of a file system stored in the memory and executable on the processor, wherein the metadata management program of the file system, when executed by the processor, implements the steps of the metadata management method of the file system.

[0036] In addition, to achieve the above object, the present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a metadata management program of a file system, wherein the metadata management program, when executed by a processor, implements the steps of the metadata management method of the file system.

[0037] The metadata management method, device, terminal device and computer medium of the file system provided by the embodiments of the present application obtain the path name corresponding to the file to be stored, and encode the path name to obtain the initial key-value format corresponding to the file; perform pre-allocation operation on the pre-allocated index management space to allocate the pre-allocated index value corresponding to the initial key-value format to obtain the pre-allocated key-value format; perform merging operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key-value format based on the pre-allocated key-value format to obtain the target index management space; allocate the target index value corresponding to the initial key-value format according to the index value contained in the target index management space to obtain the target key-value format, and store the file in the target storage structure based on the target key-value format.

[0038] In the embodiments, when the terminal device is running, the file management system in the terminal device first obtains the file to be stored, determines the path name corresponding to the file, encodes the file based on the file name to obtain the initial key-value format corresponding to the file, then obtains the pre-allocated index management space, performs pre-allocation operation on the initial key-value format corresponding to the file to be stored based on the pre-allocated index value space, allocates the pre-allocated index value corresponding to the initial key-value format to generate the pre-allocated key-value format, then performs merging operation on the pre-allocated index management space and the parent directory index management space based on the pre-allocated index value and the parent directory index value contained in the pre-allocated key-value format to generate the target index management space, finally allocates the target index value corresponding to the initial key-value format of the file according to the index value contained in the target index management space to generate the corresponding target key-value format, and the file management system further saves the file in the storage structure corresponding to the storage device according to the target key-value format.

[0039] Thus, the present application achieves the technical effect of reducing the number of times the device accesses the storage device during reading of file data and improving the performance of the file management system by storing files in the file management system in a continuous position in the storage structure based on the index management space managing the continuous index values contained in the space, thereby enabling the file management system to meet the low-latency requirements of users when facing a large number of file creation and deletion scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a structural schematic diagram of a terminal device of a hardware running environment related to an embodiment of the present application;

[0041] Figure 2 is a flowchart of a first embodiment of a metadata management method of a file system of the present application;

[0042] Figure 3 is a layout schematic diagram of an index management space related to an embodiment of a metadata management method of a file system of the present application;

[0043] Figure 4 is a system structure schematic diagram related to an embodiment of a metadata management method of a file system of the present application.

[0044] The implementation of the object of the present application, the functional features and the advantages will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0046] Reference Figure 1 , Figure 1 is a structural schematic diagram of a terminal device of a hardware running environment related to an embodiment of the present application.

[0047] It should be noted that Figure 1 may be a structural schematic diagram of a hardware running environment of a terminal device. The terminal device of the embodiment of the present application can be a device executing the metadata management method of the file system of the present application, and the terminal device can be specifically a mobile terminal, a data storage control terminal, a PC or a portable computer, etc.

[0048] For example Figure 1As shown, the terminal device can include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between the components. The user interface 1003 can include a display, an input unit such as a keyboard, and can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 can be a high-speed random access memory (RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0049] Those skilled in the art can understand that Figure 1 The structure shown in the figure does not constitute a limitation on the terminal device, and can include more or fewer components than the figure, or combine certain components, or different component arrangements.

[0050] As Figure 1 As shown, the memory 1005 as a storage medium can include an operating system, a data storage module, a network communication module, a user interface module, and a metadata management program of a file system.

[0051] In Figure 1 In the terminal device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the terminal device of the present application can be arranged in the terminal device, and the terminal device calls the metadata management program of the file system stored in the memory 1005 through the processor 1001, and executes the metadata management method of the file system provided by the present application.

[0052] Based on the terminal device described above, each embodiment of the metadata management method of the file system of the present application is provided.

[0053] It should be noted that the local file system facing the storage device is the basis of all data storage and management systems. For example, mainstream databases are built on file systems, and distributed file systems must rely on local file systems to organize data on devices. Among them, big data storage systems represented by Hbase and Dynamo are built on distributed file systems or directly call local file systems to read and write data on storage devices. Therefore, the performance of the local file system has a crucial impact on all data storage and management systems.

[0054] Among them, the local file system can be roughly divided into two components in terms of structure: namespace management and address space management. Address space management is mainly responsible for the organization of data on storage devices. Generally speaking, regardless of the technical means used to implement the storage device, the device manufacturer will abstract the physical address space of the device into a linear logical address space, and the address space management module is responsible for organizing user data and file system metadata in the linear logical address space. Therefore, the main function of the namespace management is to organize and manage file and directory names, provide users with file system operation interfaces, and implement path resolution from file name to file information.

[0055] The common classification of namespace management includes hierarchical, flat, and hash type. Among them, the hierarchical file system refers to the parent-child relationship between files and directories, forming a tree structure, and each node of the tree represents a file or directory. This structure makes it easy to organize and manage files and directories in the file system. In this structure, each node has a unique path name to identify the node's position in the tree structure. For example, / usr / local / bin is a path name, indicating the bin directory under the local directory under the usr directory under the root directory. When performing path resolution, you need to start from the root node of the tree and access the intermediate directory files from top to bottom, and finally get the target file information.

[0056] Similarly, the flat file system refers to all files and directories at the same level without parent-child relationships. The hash file system refers to calculating a unique value for the file name through a hash function, and then storing and searching for the file according to this value. Hierarchical namespace management. Compared with hierarchical management, its path resolution is simpler and does not need to be parsed in layers. However, it does not preserve the relationship between adjacent files at the same level, and the processing operation involving multiple files is more complex, such as file renaming and listing directory file information operations.

[0057] Different file systems adopt different namespace management methods. For example, the Ext series file systems (such as Ext2, Ext3, Ext4) adopt a hierarchical structure to manage the namespace, and the information of files in the same hierarchy is saved by defining a directory type file, in the path lookup process, for each layer of the target path, the inode (i.e., index node) of the target directory must be searched and the directory content must be parsed to obtain the inode number of the next layer directory. Thus, the Ext system file system will generate a large amount of disk I / O (i.e., input / output) in the path lookup process, not only because of directory parsing, but also because of its index node prefetch mechanism. However, this mechanism is not effective when parsing a deep path, because Ext cannot guarantee the vertical locality of metadata on the disk. The file information stored in the Ext directory file is arranged directly and linearly in the storage device, and when the number of subfiles in the same directory is too large, searching for a subfile requires traversing all the files, which will cause a serious decline in query performance. After Ext3, the arrangement of subfiles stored in the directory file is optimized, and it uses Hash and B+ tree to organize subfiles to speed up the file query performance in the same layer. However, in the face of write-intensive workloads, the performance of the Ext file system will be greatly reduced due to a large number of random small writes; in addition, when the number of files is large and the path of the file is deep, the performance of path parsing will also decline, for example, if the file depth is N, N times of disk I / O is required, which greatly reduces the performance of path parsing.

[0058] Similarly, BtrFS is a new copy-on-write (CoW) Linux file system, which itself uses B-tree to store directory items, so the efficiency of searching files in a given directory is very high. In order to meet the needs of readdir, BtrFS inserts another directory item index in addition to the directory item with hash value as Key when creating a new file each time, and the Key of the directory item index is sequence number (i.e., sequence number) as the key value of B-tree, which is linearly increased each time a new file is created, so that the files read out by readdir are in order. Although BtrFS has very high efficiency in searching files in a given directory, when the number of files involved is large and the path of the file is deep, the depth of the B-tree will become deeper and the delay of searching the target file in the B-tree will become larger, thereby causing a decline in file access performance. Furthermore, similar to the Ext series file system, in the face of write-intensive workloads, the performance of the file system will be greatly reduced due to a large number of random small writes.

[0059] Similarly, TableFS is an LSM-Tree based file system stacked on the underlying file system, which ensures that the metadata is written to the disk in a non-overlapping, sorted log form. Due to the write-friendly characteristics of the LSM tree, TableFS is very friendly to write-intensive scenarios, and can aggregate random small writes into sequential writes to the disk. However, the LSM-Tree itself is not friendly to reading, so when the LSM-Tree is directly used as a metadata storage structure, the performance of the read operation of the file system is poor. The performance of the read file operation (such as stat) of TableFS is much lower than that of common file systems such as ext4, because the path resolution needs to read the LSM-Tree multiple times, especially when the files on the path are not in the same SSTable, multiple disk accesses are required, resulting in low read performance and high latency.

[0060] In summary, when facing a large number of file creation scenarios, the current major data processing framework often needs to frequently create and update file information, so the file management system needs to read a large number of disk files, thereby it is difficult to meet the low latency demand of users, and the metadata read-write delay is a key factor affecting the application performance in many scenarios. Therefore, how to design a local file system that is still friendly to write-intensive and has low latency in file path resolution when facing a large number of file creation scenarios has become a technical problem that needs to be solved in the industry.

[0061] In view of the above technical problems, the present application provides a metadata management method of a file system, a terminal device and a computer readable storage medium, the method comprising: obtaining a path name corresponding to a file to be stored, and encoding the path name to obtain an initial key-value format corresponding to the file; performing a pre-allocation operation on a pre-allocated index management space to allocate a corresponding pre-allocated index value to the initial key-value format to obtain a pre-allocated key-value format; performing a merging operation on the pre-allocated index management space and a parent directory index management space corresponding to the pre-allocated key-value format based on the pre-allocated key-value format to obtain a target index management space; allocating a corresponding target index value to the initial key-value format according to the index values contained in the target index management space to obtain a target key-value format, and storing the file in a target storage structure based on the target key-value format.

[0062] In this way, the present application stores the file in a continuous position in the storage structure based on the continuous index values contained in the index management space by the file management system, thereby reducing the number of times of accessing the storage device during the reading of the file data, improving the performance of the file management system, and further enabling the file management system to meet the low latency demand of users when facing a large number of file creation and deletion scenarios.

[0063] Please refer to Figure 2, Figure 2 The flowchart of the metadata management method of the file system of the present application.

[0064] It should be understood that, although the logical order is shown in the flowchart, in some cases, the metadata management method of the file system of the present application can of course also perform the steps shown or described in a different order than here.

[0065] In the present embodiment, the metadata management method of the file system of the present application can include the following steps:

[0066] Step S10: obtaining the path name corresponding to the file to be stored, and encoding based on the path name to obtain the initial key-value format corresponding to the file;

[0067] In the present embodiment, the terminal device, when running, first obtains the file to be stored, and determines the path name corresponding to the file by the file management system in the terminal device, and encodes based on the preset encoding format and the path name to obtain the initial key-value format.

[0068] Illustratively, for example, the terminal device, when running, first obtains the file to be stored, and determines the path name corresponding to the file by the file management system in the terminal device, and then the file management system encodes based on the preset key-value format and the path name to obtain the initial key-value format corresponding to the file: <pinode:filename, stat>.

[0069] It can be understood that the key-value format described above is more easily accepted than the format of using the complete path of the directory or file as the key in the current technology, because it has lower overhead for performing move or rename operations.

[0070] Further, in a possible embodiment, the step of "encoding the file to obtain the initial key-value format corresponding to the file" in the above step S10 can specifically include:

[0071] Step S101: determining the file parent directory, file name and file attribute information corresponding to the path name;

[0072] Step S102: encoding based on the parent directory index value corresponding to the file parent directory, the file name and the file attribute information to obtain the initial key-value format corresponding to the file;

[0073] Exemplarily, for example, the file management system firstly determines the file parent directory corresponding to the file according to the path name, thereby determining the parent directory index value pinode corresponding to the file parent directory, meanwhile, the file management system determines the file name filename corresponding to the file, meanwhile, the file management system determines the file attribute information stat corresponding to the file, then, the file management system encodes based on the obtained parent directory index value pinode, the file name filename and the file attribute information stat, thereby obtaining the initial key value format <pinode:filename, stat> corresponding to the file.

[0074] Step S20: performing pre-allocation operation on the preset pre-allocation index management space to allocate the corresponding pre-allocation index value for the initial key value format, thereby obtaining the pre-allocation key value format;

[0075] Further, please refer to Figure 3 and Figure 4 , wherein, Figure 3 is the index management space layout schematic diagram involved in an embodiment of the metadata management method of the file system of the present application, Figure 4 is the system structure schematic diagram involved in an embodiment of the metadata management method of the file system of the present application; the index management space is the space used for managing the index value (i.e. inode_number), wherein the size corresponding to one index value should be set as 64 bits; in this way, the index management space can identify 2 64 files, as shown in Figure 4 , since the size of the key value corresponding to one file is 256B, and one 4KiB storage block can accommodate 16 key value pairs, therefore, the size of the index management space is set as 16 by default, and a part of the space will be reserved for subsequent merging.

[0076] In addition, the index values in the index management space should be similar in lexicographical order, and each index value in the index management space should contain a minimum index value min_inode_number and a maximum index value max_inode_number; in this way, when a new file is created, the file management system can determine an index value in the index management space inode_space, and allocate the index node inode corresponding to the index value to the newly created file, and when the index nodes inode in the index management space inode_space have all been allocated, the file management system can create a new index management space inode_space.

[0077] As shown in Figure 3 , the above index management space inode_space is divided into in-memory (i.e. Figure 3an index table of the DRAM) and a storage device (i.e. Figure 3 a linear table on the SSD), wherein the table entry of the linear table contains a root inode number (i.e. Figure 3 the root inode in the inode_space), the maximum inode number max inode that the index management space inode_space can allocate, the total number of reserved allocated inodes step, and the current allocated number count. It can be understood that the total number of reserved allocated inodes step is the reserved space corresponding to the index management space inode_space, and the allocated number count corresponds to the allocated node corresponding to the index management space inode_space.

[0078] In the embodiment, the file management system obtains a preset pre-allocation index management space for performing a pre-allocation operation, and performs an index value pre-allocation operation based on the pre-allocation index management space to allocate a corresponding pre-allocation index value to the generated initial key value format, and the file management system further writes the pre-allocation index value to the initial key value format to generate a pre-allocation key value format.

[0079] For example, the file management system first obtains a preset pre-allocation index management space inode_space1 for performing a pre-allocation operation, and performs a pre-allocation operation based on the pre-allocation index management space inode_space1 to allocate a corresponding pre-allocation index value cur inode to the initial key value format <pinode:filename, stat>, and then the file management system writes the allocated pre-allocation index value cur inode to the initial key value format to obtain a pre-allocation key value format <pinode:file_name, cur_inode…>.

[0080] It can be understood that based on the pre-allocation key value format, the keys of the subfiles under the same directory can all start with the inode of the parent directory, so that the subfiles under the same directory are stored contiguously and adjacently on the storage device. Since the general read-write granularity of the storage device is at the KB level, for example, the minimum read granularity of the SSD is 4KiB, and a plurality of file metadata information adjacently arranged on the storage device can be read at a time I / O, so that the performance of the file system range query can be greatly improved.

[0081] Step S30: performing a merging operation on the pre-allocation index management space and the parent directory index management space corresponding to the pre-allocation key value format based on the pre-allocation key value format to obtain a target index management space;

[0082] In the embodiment, the file management system performs a merging operation on the pre-allocated index management space corresponding to the pre-allocated key-value format and the parent directory index management space based on the pre-allocated index value and the parent directory index value contained in the pre-allocated key-value format to obtain a target index management space.

[0083] For example, the file management system performs a merging operation on the pre-allocated index management space inode_space1 and the parent directory index management space inode_space2 corresponding to the pre-allocated index value cur_inode and the parent directory index value pinode contained in the pre-allocated key-value format <pinode:file_name, cur_inode…> to obtain a target index management space inode_space.

[0084] Further, in a feasible embodiment, the step S30 can specifically include:

[0085] Step S301: comparing the pre-allocated index value and the parent directory index value contained in the pre-allocated key-value format to obtain a first comparison result.

[0086] In the embodiment, the file management system first determines the pre-allocated index value and the parent directory index value contained in the pre-allocated key-value format, compares the pre-allocated index value and the parent directory index value to obtain a difference between the pre-allocated index value and the parent directory index value, obtains a first threshold value preset by a technician, compares the difference with the first threshold value to obtain a first comparison result, and further judges whether the pre-allocated index value and the parent directory index value are too different based on the first comparison result.

[0087] Step S302: if it is judged that the first comparison result is that the difference between the pre-allocated index value and the parent directory index value is greater than the first threshold value, performing a merging operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key-value format based on the parent directory index value to obtain a target index management space.

[0088] In the embodiment, if the file management system judges that the first comparison result is that the difference is greater than or equal to the first threshold value, it is determined that the pre-allocated index value and the parent directory index value are too different, and a merging operation is performed on the parent directory index management space and the pre-allocated index management space based on the parent directory index value to obtain a target index management space.

[0089] For example, the file management system first determines the pre-allocated index value cur inode and the parent directory index value pinode contained in the pre-allocated key-value format <pinode: file_name, cur inode…>, and compares the pre-allocated index value cur inode and the parent directory index value pinode to obtain a difference therebetween. Meanwhile, the file management system obtains a preset first threshold value, and compares the difference with the first threshold value to obtain a first comparison result. Then, if the file management system determines that the first comparison result is that the difference is greater than or equal to the first threshold value, it is determined that the pre-allocated index value cur inode and the parent directory index value pinode differ too much. The file management system further performs a merging operation on the parent directory index management space inode_space2 corresponding to the parent directory index value pinode in the pre-allocated key-value format <pinode: file_name, cur inode…> and the pre-allocated index management space inode_space1 corresponding to the pre-allocated index value cur inode, thereby obtaining a target index management space inode_space.

[0090] In addition, in another embodiment, the file management system can also calculate the free rate α = 1 - count / (imax inode-root inode) of the pre-allocated index management space inode_space1, and compare the free rate with a preset free rate threshold value. Then, if the file management system determines that the free rate is higher than the free rate threshold value, it is determined to perform the merging operation.

[0091] Further, in a feasible embodiment, the step of “performing a merging operation on the pre-allocated index management space and the parent directory index management space corresponding to the parent directory index value in the pre-allocated key-value format to obtain a target index management space” in step S302 can specifically include:

[0092] Step S3021: determining whether the space allocation value corresponding to the pre-allocated index management space is less than a preset second threshold value;

[0093] Step S3022: if it is determined that the space allocation value is less than the second threshold value, merging the pre-allocated index management space into the parent directory index management space to obtain a target index management space;

[0094] Exemplarily, for example, when the file management system determines to perform the merging operation, first, the second threshold value preset by the technician is acquired, and the corresponding space allocation number in the pre-allocated index management space inode_space1 is compared with the second threshold value to obtain a second comparison result. Then, if the file management system judges that the second comparison result is that the space allocation number is less than the second threshold value, it is determined that there are less word files in the index management space inode_space1, and thus, the locality of the index management space inode_space1 does not need to be maintained. Therefore, the file management system merges the pre-allocated index management space inode_space1 into the parent directory index management space inode_space2, and takes the merged parent directory index management space inode_space2 as the target index management space inode_space.

[0095] Further, in a feasible embodiment, the step of "performing a merging operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated key value format based on the parent directory index value" in the step S302 can further include:

[0096] Step S3023: judging whether the reserved space corresponding to the parent directory index management space is used up;

[0097] In the embodiment, when the file management system determines to perform the merging operation, first, it is judged whether the reserved space contained in the parent directory index management space is used up.

[0098] Step S3024: if it is judged that the reserved space is used up, the allocated node numbers corresponding to the pre-allocated index management space and the parent directory index management space are determined;

[0099] In the embodiment, if the file management system judges that the reserved space has been used up, it is determined that the splitting operation needs to be performed on the parent directory index management space, and then the pre-allocated index management space and the parent directory index management space are detected respectively, so as to determine the allocated node numbers corresponding to the pre-allocated index management space and the parent directory index management space respectively.

[0100] Step S3025: performing a node balancing operation on the pre-allocated index management space and the parent directory index management space based on the allocated node numbers, and taking the pre-allocated index management space after the balancing operation as the target index management space;

[0101] In the embodiment, the file management system performs a node balancing operation on the parent directory index management space and the pre-allocated index management space based on the obtained allocated node numbers of each, so as to migrate part of the nodes in the parent directory index management space to the pre-allocated index management space, so as to balance the node numbers in the two index management spaces, and the file management system further determines the pre-allocated index management space after the balancing operation as the target index management space.

[0102] For example, when the file management system determines to perform the merging operation, it first judges the working state of the reserved space in the parent directory index management space inode_space2, and judges whether the reserved space is used up based on the working state. Then, if the file management system judges that the reserved space is used up, it determines that the splitting operation of the index management space needs to be performed, and detects the allocated node numbers of the pre-allocated index management space inode_space1 and the parent directory index management space inode_space2 respectively, so as to determine the inode node range [x, y] that needs to be migrated, and the file management system determines that the storage level involved in the migration operation is L k and L k-1 , so as to search for the key-value pair satisfying the condition x:*<=Key<=y:* between L0 to L k , and modify the inode of the key-value pair to new_inode. In this way, the file management system determines the average node numbers of the pre-allocated index management space inode_space1 and the parent directory index management space inode_space2 based on the obtained allocated node numbers of each, and migrates part of the nodes in the parent directory index management space inode_space2 to the pre-allocated index management space inode_space1 according to the average node numbers to complete the node balancing operation, so as to determine the pre-allocated index management space inode_space1 after balancing as the target index management space and update the corresponding index table information.

[0103] Step S40: allocating a target index value corresponding to the initial key value format according to the index value in the target index management space to obtain a target key value format, and storing the file in the target storage structure based on the target key value format;

[0104] In the embodiment, after the file management system completes the merging operation, it performs an index allocation operation based on the obtained index value in the target index management space, so as to allocate a target index value corresponding to the initial key value format to generate a target key value format, and the file management information further stores the file in the corresponding storage structure according to the target key value format.

[0105] For example, after the merging operation is completed, the file management system performs an inode allocation operation based on the target index management space to allocate a target index value corresponding to the initial key-value format <pinode:filename, stat> to obtain a target key-value format, and then stores the corresponding file in the target storage structure based on the target key-value format.

[0106] In this embodiment, when the terminal device is running, the terminal device first acquires a file to be stored, and the file management system in the terminal device determines a path name corresponding to the file, encodes the path name based on a preset encoding format to obtain an initial key-value format, acquires a pre-allocation index management space preset for performing a pre-allocation operation, performs an index value pre-allocation operation based on the pre-allocation index management space to allocate a pre-allocation index value corresponding to the generated initial key-value format, writes the pre-allocation index value into the initial key-value format to generate a pre-allocation key-value format, performs a merging operation on the pre-allocation index management space corresponding to the pre-allocation key-value format and the parent directory index management space based on the pre-allocation index value and the parent directory index value included in the pre-allocation key-value format to obtain a target index management space, and finally, after the merging operation is completed, performs an index allocation operation based on the index value included in the obtained target index management space to allocate a target index value corresponding to the initial key-value format to generate a target key-value format, and then stores the file in the corresponding storage structure according to the target key-value format.

[0107] In this way, the file management system stores the file in a continuous position in the storage structure based on the continuous index values included in the index management space, thereby reducing the number of times of accessing the storage device during reading of file data, improving the performance of the file management system, and meeting the low-latency requirement of the user when facing a large number of file creation and deletion scenarios.

[0108] Further, based on the first embodiment of the metadata management method of the file system of the present application, a second embodiment of the metadata management method of the file system of the present application is provided.

[0109] After the step S40, the metadata management method of the file system of the present application can further include the following steps:

[0110] Step A10: detecting a foreground load when the file deletion instruction is received, and determining whether the foreground load is greater than or equal to a preset third threshold value;

[0111] In the embodiment, when receiving the file deletion instruction, the file management system acquires the third threshold value preset by the technician, and detects the terminal device based on the file deletion instruction to acquire the foreground load of the terminal device currently, and compares the foreground load with the third threshold value to obtain a third comparison result.

[0112] Step A20: if it is judged that the foreground load is greater than or equal to the third threshold value, inserting a preset full deletion key into a target file directory corresponding to the file deletion instruction;

[0113] In the embodiment, if the file management system judges that the third comparison result is that the foreground load is greater than or equal to the third threshold value, the target file directory to be deleted is determined according to the file deletion instruction, and a preset full deletion key value is inserted into the target file directory, so that the file management system performs a file recycling operation by the full deletion key when performing a merging operation, thereby completing the deletion operation of the target file directory.

[0114] Step A30: if it is judged that the foreground load is less than the third threshold value, recursively inserting a preset deletion key into the target file directory and each sub-file directory corresponding to the target file directory;

[0115] In the embodiment, if the file management system judges that the third comparison result is that the foreground load is less than the third threshold value, the target file directory to be deleted and each sub-directory under the target file directory are determined according to the file deletion instruction, and the file management system further recursively inserts a deletion key into the target file directory and each sub-directory, so that the file directory marked with the deletion key is removed in the merging operation process, thereby completing the deletion operation.

[0116] Exemplarily, for example, when a user needs to delete a file, a file deletion instruction can be generated by a terminal device, which in turn sends the file deletion instruction to the file management system. After the file management system receives the file deletion instruction, it first acquires the third threshold value preset by the technical personnel and determines the target file directory corresponding to the target file to be deleted. At the same time, the file management system detects the foreground load corresponding to the terminal device, and then compares the foreground load with the third threshold value to obtain a third comparison result. After that, if the file management system determines that the third comparison result is that the foreground load is greater than or equal to the third threshold value, it acquires the preset full delete key full_delete_key and inserts the full delete key full_delete_key under the target file directory, so that the file management system searches for files whose key directories contain the full delete key full_delete_key in the storage structure SSTable, and then directly deletes the file directories corresponding to the files containing the full delete key full_delete_key when performing the above index management space merging operation.

[0117] If the file management system determines that the third comparison result is that the foreground load is less than the third threshold value, it determines the target file directory corresponding to the target file to be deleted according to the file deletion instruction, and the subdirectories under the target file directory. The file management system then recursively inserts the preset delete key into each of the subdirectories, so that these delete keys and the files with the delete keys are removed from the storage device in the subsequent merging process.

[0118] In the present embodiment, when the file management system receives a file deletion instruction, it acquires the third threshold value preset by the technical personnel. At the same time, the file management system detects the terminal device based on the file deletion instruction to obtain the current foreground load of the terminal device, and compares the foreground load with the third threshold value to obtain a third comparison result. After that, if the file management system determines that the third comparison result is that the foreground load is greater than or equal to the third threshold value, it determines the target file directory to be deleted according to the file deletion instruction, and inserts the preset full delete key value under the target file directory, so that the file management system performs a file recycling operation by the full delete key when performing the merging operation, thereby completing the deletion operation of the target file directory. Finally, if the file management system determines that the third comparison result is that the foreground load is less than the third threshold value, it determines the target file directory to be deleted according to the file deletion instruction, and the subdirectories under the target file directory. The file management system then recursively inserts the delete key under the target file directory and each of the subdirectories, so that the file directories marked with the delete key are removed in the merging operation process, thereby completing the deletion operation.

[0119] Therefore, the application determines foreground load condition of the terminal device, inserts a full deletion key into a target file directory which needs to be deleted when the foreground load condition is higher than a preset threshold, and inserts a deletion key into the target file directory and corresponding subdirectories recursively when the foreground load condition is lower than the threshold, so that the file management system can select a more efficient file deletion scheme according to the foreground load condition when performing a file deletion operation, thereby quickly recycling space on the storage device and improving the performance of the file management system.

[0120] Further, based on the first embodiment and / or the second embodiment of the metadata management method of the file system of the application, the third embodiment of the metadata management method of the file system of the application is provided.

[0121] After the step S40, the metadata management method of the file system of the application can further include the following steps:

[0122] Step B10: receiving a file access instruction, determining a target file according to the file access instruction, and detecting an access frequency corresponding to the target file;

[0123] In this embodiment, when the file management system receives a file access instruction, the access frequency of the target file corresponding to the file access instruction is counted through a preset time window.

[0124] Step B20: judging whether the data type corresponding to the target file is hot data based on the access frequency;

[0125] In this embodiment, the file management system obtains a fourth threshold value, compares the obtained access frequency with the fourth threshold value to obtain a fourth comparison result, and further judges whether the data type corresponding to the target file is hot data based on the fourth comparison result.

[0126] Step B30: if it is judged that the data type is the hot data, determining a target index management space corresponding to the target file;

[0127] In this embodiment, if the file management system determines that the fourth comparison result is that the access frequency is greater than or equal to the fourth threshold value, it is determined that the data type corresponding to the target file is hot data, and further determines a target index value corresponding to the target file according to the target key value format of the target file, and finds a target index management space corresponding to the target index value.

[0128] Step B40: determining each similar hot data file corresponding to the target file based on the target index management space, and inserting each similar hot data file into a preset log merge storage structure tree;

[0129] In the embodiment, the file management system determines the similar hot data files corresponding to the target metadata based on the plurality of index values contained in the target index management space, and inserts the similar hot data files into the surface storage structure in the file management system.

[0130] For example, the file management system opens a time window in the background of the system when receiving the file access instruction, and counts the access frequency freq of the target file corresponding to the file access instruction through the time window. Meanwhile, the file management system obtains a preset fourth threshold value, compares the access frequency freq with the fourth threshold value to obtain a fourth comparison result, and then determines that the data type of the target file is a hot file filehot if the file management system determines that the fourth comparison result is that the access frequency freq is greater than or equal to the fourth threshold value. The file management system further determines the target index value inode hot corresponding to the target file, and determines the target index management space inode_space hot corresponding to the target index value inode hot Then, the file management system determines the similar hot data files corresponding to the target file based on the index management space inode_space hot , and inserts the similar hot data files into the lower level SSTable in the file management system, so that the user can access the hot data files more quickly.

[0131] In the embodiment, the file management system counts the access frequency of the target file corresponding to the file access instruction through a preset time window when receiving the file access instruction. Then, the file management system obtains a preset fourth threshold value, compares the obtained access frequency with the fourth threshold value to obtain a fourth comparison result, and further determines whether the data type of the target file is hot data based on the fourth comparison result. Then, the file management system determines that the data type of the target file is hot data if it is determined that the fourth comparison result is that the access frequency is greater than or equal to the fourth threshold value. Then, the file management system determines the target index value corresponding to the target file according to the target key value format of the target file, and finds the target index management space corresponding to the target index value. Finally, the file management system determines the similar hot data files corresponding to the target metadata based on the plurality of index values contained in the target index management space, and inserts the similar hot data files into the surface storage structure in the file management system.

[0132] Therefore, the application adopts a manner of determining each similar hot data file corresponding to the hot data based on a target index value corresponding to the hot data when the accessed data is determined as hot data, and then inserting each similar hot data file into a surface storage structure in the system, so as to improve the access performance of the hot data and reduce the delay generated by the user when reading the hot data.

[0133] Further, based on each embodiment of the metadata management method of the file system of the application, the fourth embodiment of the metadata management method of the file system of the application is provided.

[0134] Before the step S20, the metadata management method of the file system of the application can further include the following steps:

[0135] Step C10: determining each duplicate file corresponding to the file based on the initial key value format.

[0136] In the embodiment, the file management system compares the initial key value format corresponding to each file to obtain a fifth comparison result, and determines each duplicate file corresponding to the file based on the fifth comparison result.

[0137] Step C20: performing a merge storage operation on the file and each duplicate file.

[0138] In the embodiment, the file management system performs a merge storage operation on the file and each duplicate file corresponding to the file.

[0139] For example, the file management system first compares the initial key value format <pinode:filename, stat> corresponding to each file to obtain a fifth comparison result, then determines each duplicate file with the same initial key value format based on each fifth comparison result, and performs a merge storage operation on the file and each duplicate file, so that the memory table in the file management system can store more key value pairs.

[0140] In addition, the application also provides a terminal device having a metadata management program of a file system that can run on a processor, and the terminal device implements the steps of the metadata management method of the file system as described in any one of the above embodiments when executing the metadata management program of the file system.

[0141] The specific embodiments of the terminal device of the application are basically the same as those of the metadata management method of the file system, and are not repeated here.

[0142] In addition, the present application also provides a computer readable storage medium, and the computer readable storage medium stores a metadata management program of a file system, and the metadata management program of the file system realizes the steps of the metadata management method of the file system according to any one of the above embodiments when executed by a processor.

[0143] The specific embodiments of the computer readable storage medium of the present application are basically the same as the above-mentioned embodiments of the metadata management method of the file system, and are not described here.

[0144] It should be noted that in this paper, the term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or system. Without more limitations, the element defined by the sentence "includes a" does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0145] The above-mentioned serial numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a plurality of instructions for making a terminal device (which can be a device executing the metadata management method of the file system of the present application, and the terminal device can be a mobile terminal, a data storage control terminal, a PC or a portable computer, etc.) execute the method described in each embodiment of the present application.

[0147] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A metadata management method of a file system, characterized by, The metadata management method of the file system comprises the following steps: Obtaining a path name corresponding to a file to be stored, and encoding based on the path name to obtain an initial key-value format corresponding to the file, the initial key-value format comprising a parent directory index value, a file name and file attribute information; Performing a pre-allocation operation on a pre-allocation index management space to allocate a corresponding pre-allocation index value to the initial key-value format to obtain a pre-allocation key-value format, wherein the pre-allocation key-value format comprises a parent directory index value, a file name, a pre-allocation index value and file attribute information, wherein the pre-allocation index value is a temporary index value pre-allocated from the pre-allocation index management space, and the pre-allocation key-value formats of all child files under the same parent directory all start with the same parent directory index value to ensure that the child files of the same parent directory are continuously and adjacently distributed in a target storage structure during subsequent storage; Performing a merging operation on the pre-allocation index management space and a parent directory index management space corresponding to the pre-allocation key-value format based on the pre-allocation key-value format to obtain a target index management space; Allocating a corresponding target index value to the initial key-value format according to the index values contained in the target index management space to obtain a target key-value format, and storing the file in a target storage structure based on the target key-value format.

2. The metadata management method of a file system according to Claim 1, wherein, The step of encoding based on the path name to obtain the initial key-value format corresponding to the file comprises: Determining a file parent directory, a file name and file attribute information corresponding to the path name; Encoding based on a parent directory index value corresponding to the file parent directory, the file name and the file attribute information to obtain the initial key-value format corresponding to the file.

3. The metadata management method of a file system according to Claim 2, wherein, The step of performing a merging operation on the pre-allocation index management space and the parent directory index management space corresponding to the pre-allocation key-value format based on the pre-allocation key-value format to obtain a target index management space comprises: Comparing the pre-allocation index value and the parent directory index value contained in the pre-allocation key-value format to obtain a first comparison result; If it is determined that the first comparison result is that the difference between the pre-allocation index value and the parent directory index value is greater than a preset first threshold value, performing a merging operation on the pre-allocation index management space and the parent directory index management space corresponding to the pre-allocation key-value format based on the parent directory index value to obtain a target index management space.

4. The metadata management method of a file system according to Claim 3, wherein, The step of performing a merging operation on the pre-allocation index management space and the parent directory index management space corresponding to the pre-allocation key-value format based on the parent directory index value to obtain a target index management space comprises: Determining whether a space allocation value corresponding to the pre-allocation index management space is less than a preset second threshold value; If it is determined that the space allocation value is less than the second threshold value, merging the pre-allocation index management space into the parent directory index management space to obtain a target index management space.

5. The metadata management method of a file system according to Claim 4, wherein, The step of performing a merge operation on the pre-allocated index management space and the parent directory index management space corresponding to the pre-allocated index value based on the parent directory index value to obtain a target index management space further comprises: determining whether the reserved space corresponding to the parent directory index management space is used up; if it is determined that the reserved space is used up, determining the number of allocated nodes corresponding to the pre-allocated index management space and the parent directory index management space respectively; performing a node balancing operation on the pre-allocated index management space and the parent directory index management space based on the number of allocated nodes, and determining the pre-allocated index management space after the balancing operation as the target index management space.

6. The metadata management method of a file system according to Claim 1, wherein, After the step of storing the file in the target storage structure based on the target key-value format, the method further comprises: detecting a foreground load when a file deletion instruction is received, and determining whether the foreground load is greater than or equal to a preset third threshold value; if it is determined that the foreground load is greater than or equal to the third threshold value, inserting a preset full deletion key into a target file directory corresponding to the file deletion instruction; if it is determined that the foreground load is less than the third threshold value, recursively inserting a preset deletion key into the target file directory and each sub-file directory corresponding to the target file directory.

7. The metadata management method of a file system according to Claim 1, wherein, After the step of storing the file in the target storage structure based on the target key-value format, the method further comprises: when a file access instruction is received, determining a target file according to the file access instruction, and detecting an access frequency corresponding to the target file; determining whether a data type corresponding to the target file is hot data based on the access frequency; if it is determined that the data type is the hot data, determining the target index management space corresponding to the target file; determining each similar hot data file corresponding to the target file based on the target index management space, and inserting each similar hot data file into a preset log merge storage structure tree.

8. The metadata management method of a file system according to Claim 1, wherein, Before the step of performing a pre-allocation operation on a preset pre-allocated index management space to allocate a corresponding pre-allocated index value to the initial key-value format to obtain a pre-allocated key-value format, the method further comprises: determining each duplicate file duplicated with the file based on the initial key-value format; performing a merge storage operation on the file and each duplicate file.

9. A terminal device, comprising: The terminal device comprises a memory, a processor, and a file system metadata management program stored on the memory and executable on the processor, and the file system metadata management program, when executed by the processor, implements the steps of the file system metadata management method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a file system metadata management program, and the file system metadata management program, when executed by a processor, implements the steps of the file system metadata management method according to any one of claims 1 to 8.

Images