Non-volatile memory-based file system indexing method and system

Abstract

The application provides a new index method and system for a non-volatile memory-based file system, comprising: simultaneously performing reverse indexing and regular indexing, searching the regular index and the reverse index simultaneously by multi-threading until the corresponding file is found; the reverse index traverses the file path in reverse from the end file name of the file path to perform directory indexing until the corresponding file is obtained; and the regular index starts from the root directory and indexes by the directory names level by level until the corresponding file is obtained. The application fully utilizes the performance advantage of the non-volatile memory by simultaneously searching the regular index and the reverse index by multi-threading when indexing.

Description

Technical Field

[0001] This invention relates to the field of file system indexing methods, and more specifically, to a file system indexing method and system based on non-volatile memory. Background Technology

[0002] Non-volatile memory is a storage medium that combines the low latency and high throughput of traditional memory with the non-volatile nature of traditional storage devices after power failure. Furthermore, it can be addressed at the byte level, allowing the processor to directly access the contents of non-volatile memory using load / store instructions, just like with traditional memory.

[0003] Current file systems are typically organized in a hierarchical structure. To read or write a file, you need to locate the file based on its path. If an absolute path is used, the file system indexes itself layer by layer from the root directory: the file system first locates the root directory, then finds the corresponding directory in the root directory's index structure based on the first-level directory name, then finds the next-level directory in the index structure based on the second-level directory name, and so on.

[0004] Because non-volatile memory offers significantly higher read / write performance compared to traditional block storage devices (HDDs and SSDs), the performance bottleneck for file data read / write operations shifts from data read / write to metadata operations. In particular, for reading small files with deep directory hierarchies, the time overhead of hierarchical indexing starting from the root directory is considerable compared to the actual data read / write operations. Many social networking platform systems need to store a large number of user avatars and other small files, and require rapid indexing when needed. However, current indexing performance is limited by the root directory-based indexing design, which cannot fully utilize the high performance of non-volatile memory.

[0005] Patent document CN110413724B (application number: 201910527961.0) discloses a data retrieval method and apparatus, comprising: synchronizing a mapping table and high-frequency documents in a retrieval engine to a non-volatile internal memory; in response to receiving different retrieval conditions, performing a reverse indexing in the volatile internal memory through the mapping table to obtain different retrieved documents; redetermining high-frequency documents based on their respective retrieval frequencies; obtaining new index documents and writing them into the non-volatile internal memory; updating the mapping table in the non-volatile internal memory based on the new index documents; and feeding back the redetermined high-frequency documents, the new index documents, and the updated mapping table to the retrieval engine.

[0006] Therefore, we need an indexing system designed for file systems using non-volatile memory to eliminate the indexing overhead of small files in deep directories, thereby making full use of the high performance of non-volatile memory. Summary of the Invention

[0007] In view of the deficiencies in the prior art, the purpose of this invention is to provide an indexing method and system for a file system based on non-volatile memory.

[0008] An indexing method for a file system based on non-volatile memory provided by the present invention includes: simultaneously performing reverse indexing and regular indexing, and simultaneously searching the regular index and reverse index using multiple threads until the corresponding file is found;

[0009] The reverse index starts from the last filename of the file path and traverses the file path backward to index the directory until the corresponding file is obtained.

[0010] The conventional index starts from the root directory and proceeds through directory names until the corresponding file is found.

[0011] Preferably, the reverse index includes a multi-level index. When the file system uses the file path for reverse indexing, it first uses the file name at the end of the path to search in the first-level index. If the result obtained by the first-level index contains multiple files, it then searches in the second-level index again using the directory name at the next end of the path, until the corresponding file is found.

[0012] Preferably, each level of the reverse index is composed of a hash table. The first-level index uses the filename at the end to perform a hash search to locate the hash table entry. If the hash table entry corresponds to only one file, it points to the file's reverse index node. The current reverse index node stores the complete path of the file and the file index node. The complete path is used to perform a complete match with the search path, and the file index node points to the actual location of the file. If the hash table entry corresponds to multiple files, it points to another hash table as the next-level index. A hash search is performed by the file directory of the filename's parent directory, and so on.

[0013] Preferably, the method of simultaneously searching the regular index and the reverse index through multiple threads until the corresponding file is found is as follows: when querying the regular index and the reverse index simultaneously, if neither index finds the file corresponding to the file path, the current file is directly returned as not existing; if either index finds the corresponding file based on the file path, the current file is directly returned.

[0014] Preferably, when performing file system directory operations, when modifying the regular file system index, the reverse index needs to be modified simultaneously to ensure that the regular index and the reverse index contain the same file system directory information.

[0015] An indexing system for a file system based on non-volatile memory, provided by the present invention, includes: simultaneously performing reverse indexing and regular indexing, and simultaneously searching the regular index and reverse index using multiple threads until the corresponding file is found;

[0016] The reverse index starts from the last filename of the file path and traverses the file path backward to index the directory until the corresponding file is obtained.

[0017] The conventional index starts from the root directory and proceeds through directory names until the corresponding file is found.

[0018] Preferably, the reverse index includes a multi-level index. When the file system uses the file path for reverse indexing, it first uses the file name at the end of the path to search in the first-level index. If the result obtained by the first-level index contains multiple files, it then searches in the second-level index again using the directory name at the next end of the path, until the corresponding file is found.

[0019] Preferably, each level of the reverse index is composed of a hash table. The first-level index uses the filename at the end to perform a hash search to locate the hash table entry. If the hash table entry corresponds to only one file, it points to the file's reverse index node. The current reverse index node stores the complete path of the file and the file index node. The complete path is used to perform a complete match with the search path, and the file index node points to the actual location of the file. If the hash table entry corresponds to multiple files, it points to another hash table as the next-level index. A hash search is performed by the file directory of the filename's parent directory, and so on.

[0020] Preferably, the method of simultaneously searching the regular index and the reverse index through multiple threads until the corresponding file is found is as follows: when querying the regular index and the reverse index simultaneously, if neither index finds the file corresponding to the file path, the current file is directly returned as not existing; if either index finds the corresponding file based on the file path, the current file is directly returned.

[0021] Preferably, when performing file system directory operations, when modifying the regular file system index, the reverse index needs to be modified simultaneously to ensure that the regular index and the reverse index contain the same file system directory information.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. This invention utilizes multiple threads to simultaneously retrieve both regular and inverted indexes during indexing to fully leverage the performance advantages of non-volatile memory;

[0024] 2. By using multiple threads to retrieve data from two indexes simultaneously, query latency can be reduced and query performance improved.

[0025] 3. The advantage of reverse indexes is that there are fewer duplicate file names in the file system, which can quickly narrow down the search scope;

[0026] 4. The advantage of parallel queries is that in extreme cases, such as when there are many files with the same name in the file system, the query efficiency of reverse index may be lower than that of regular index. In this case, it can ensure that the performance of the indexing method is no lower than that of the traditional index (i.e., only regular index), thereby making full use of the high performance advantage of non-volatile memory and accelerating the indexing of files with deep directory levels.

[0027] 5. If any index returns query results (file exists or does not exist), the results are returned to the user directly without waiting for the query results of another index;

[0028] 6. This invention can improve the performance of reading and writing small files with deep directory hierarchies. Attached Figure Description

[0029] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0030] Figure 1 This is an example of a specific structure for a reverse index in this invention.

[0031] Figure 2 This is an example of the query method in the dual index design of this invention. Detailed Implementation

[0032] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0033] To address the need for fast indexing of small files with deep directory hierarchies in file systems based on non-volatile memory, this invention aims to provide an indexing method for such systems. This method fully considers the performance advantages and characteristics of non-volatile memory, combines the fast indexing requirements of user-space systems for small files with deep directory hierarchies, utilizes a reverse index starting from the filename at the end of the file path, and performs parallel queries with the regular index. This avoids the situation where the reverse index performs worse than the regular index in extreme cases, reducing overall index query latency and thus improving the indexing performance of the file system for small files with deep directory hierarchies.

[0034] Example 1

[0035] The present invention provides an indexing method for a file system based on non-volatile memory, comprising: simultaneously performing reverse indexing and regular indexing, and simultaneously searching the regular index and reverse index through multiple threads until the corresponding file is found to fully utilize the performance advantages of non-volatile memory; the advantage of parallel querying is that in extreme cases, such as when the queried file has a large number of files with the same name in the file system, the query efficiency of reverse indexing may be lower than that of regular indexing. In this case, it can be guaranteed that the performance of the indexing method is not lower than that of traditional indexing (i.e., only regular indexing), thereby fully utilizing the advantages of high performance and low latency access characteristics of non-volatile memory to accelerate the indexing of files with deep directory levels.

[0036] The reverse index starts by traversing the file path in reverse order from the last filename in the file path until the corresponding file is found; that is, it starts indexing from the last filename in the file path and traverses the file path in reverse order until the corresponding file is found.

[0037] The conventional index starts from the root directory and proceeds through directory names until the corresponding file is found.

[0038] Specifically, the inverted index consists of multiple levels. When the file system uses a file path for inverted indexing, it first searches the first-level index using the filename at the very end of the path. If the results from the first-level index contain multiple files, it then searches the second-level index using the directory name at the next end of the path, and so on, until the corresponding file is found. The advantage of this design is that there are fewer cases of duplicate file names in the file system, which can quickly narrow down the search scope. In contrast, conventional indexes are limited by the file system's hierarchical structure and cannot quickly locate files. Therefore, inverted indexes are generally faster than conventional indexes in terms of query speed.

[0039] Furthermore, each level of the inverted index is composed of a hash table. The first-level index uses the filename at the end to perform a hash search to locate the hash table entry. If the hash table entry corresponds to only one file, it points to the file's inverted index node. The current inverted index node stores the complete path of the file and the file index node. The complete path is used to perform a complete match with the search path, and the file index node points to the actual location of the file. If the hash table entry corresponds to multiple files, it points to another hash table as the next-level index, and performs a hash search by the file directory of the filename's parent directory, and so on.

[0040] Specifically, the method of simultaneously searching the regular index and the reverse index using multiple threads until the corresponding file is found employs the following approach: when querying both the regular index and the reverse index simultaneously, if either index returns a query result (file exists or does not exist), the result is directly returned to the user without waiting for the query result from the other index. That is, the reverse index, like the regular index, contains all file information from the file system.

[0041] Specifically, when performing file system directory operations (creating, deleting files, etc.), after modifying the regular file system index, the reverse index also needs to be modified to ensure that the regular index and the reverse index contain the same file system directory information.

[0042] The file system contains both regular indexes and inverted indexes. When performing a query, multiple threads are used to retrieve both indexes simultaneously to reduce query latency and improve query performance.

[0043] The present invention provides an indexing system for a file system based on non-volatile memory, comprising: simultaneously performing reverse indexing and regular indexing; and simultaneously searching the regular index and reverse index using multiple threads until the corresponding file is found to fully utilize the performance advantages of non-volatile memory. The advantage of parallel querying is that in extreme cases, such as when the queried file has a large number of files with the same name in the file system, the query efficiency of the reverse index may be lower than that of the regular index. In this case, it can be guaranteed that the performance of the indexing method is not lower than that of the traditional index (i.e., only the regular index), thereby fully utilizing the high performance advantage of non-volatile memory and accelerating the indexing of files with deep directory levels.

[0044] The reverse index starts by traversing the file path in reverse order from the last filename in the file path until the corresponding file is found; that is, it starts indexing from the last filename in the file path and traverses the file path in reverse order until the corresponding file is found.

[0045] The conventional index starts from the root directory and proceeds through directory names until the corresponding file is found.

[0046] Specifically, the inverted index consists of multiple levels. When the file system uses a file path for inverted indexing, it first searches the first-level index using the filename at the very end of the path. If the results from the first-level index contain multiple files, it then searches the second-level index using the directory name at the next end of the path, and so on, until the corresponding file is found. The advantage of this design is that there are fewer cases of duplicate file names in the file system, which can quickly narrow down the search scope. In contrast, conventional indexes are limited by the file system's hierarchical structure and cannot quickly locate files. Therefore, inverted indexes are generally faster than conventional indexes in terms of query speed.

[0047] Furthermore, each level of the inverted index is composed of a hash table. The first-level index uses the filename at the end to perform a hash search to locate the hash table entry. If the hash table entry corresponds to only one file, it points to the file's inverted index node. The current inverted index node stores the complete path of the file and the file index node. The complete path is used to perform a complete match with the search path, and the file index node points to the actual location of the file. If the hash table entry corresponds to multiple files, it points to another hash table as the next-level index, and performs a hash search by the file directory of the filename's parent directory, and so on.

[0048] Specifically, the method of simultaneously searching the regular index and the reverse index using multiple threads until the corresponding file is found employs the following approach: when querying both the regular index and the reverse index simultaneously, if either index returns a query result (file exists or does not exist), the result is directly returned to the user without waiting for the query result from the other index. That is, the reverse index, like the regular index, contains all file information from the file system.

[0049] Specifically, when performing file system directory operations (creating, deleting files, etc.), after modifying the regular file system index, the reverse index also needs to be modified to ensure that the regular index and the reverse index contain the same file system directory information.

[0050] The file system contains both regular indexes and inverted indexes. When performing a query, multiple threads are used to retrieve both indexes simultaneously to reduce query latency and improve query performance.

[0051] The term "memory" as used in this invention refers to "non-volatile memory" or "persistent memory," which is memory that can be accessed via a memory interface and can persistently store data.

[0052] Example 2

[0053] Example 2 is a preferred example of Example 1.

[0054] This invention addresses the indexing of file systems based on non-volatile memory, encompassing the following two aspects, with specific implementation methods as follows:

[0055] Reverse index design: A reverse index consists of multiple levels of indexes. When the file system uses a file path for reverse indexing, it first uses the file name at the end of the path to search in the first-level index. If the result obtained by the first-level index contains multiple files, it then searches in the second-level index using the directory name at the next end of the path, and so on, until the corresponding file is found.

[0056] In an inverted index, each level consists of a hash table. The first-level index uses the filename at the end of the filename to perform a hash search to locate the hash table entry. If the hash table entry corresponds to only one file, it points to the file's inverted index node. This node stores the file's full path and the file index node. The full path is used to perform a complete match with the search path, and the file index node points to the actual location of the file. If the hash table entry corresponds to multiple files (due to duplicate names or hash collisions), it points to another hash table as the next-level index. The hash search is then performed using the file directory at the parent directory of the filename, and so on. Figure 1 As shown, the file system contains four files with paths / a, / b / b, / c / b, and / d / b / b. Since there is only one file named 'a' in the file system, the hash entry for 'a' in the first-level index points to the inverted inode of file / a. Since there are three files named 'b', the hash entry for 'b' in the first-level index points to a new hash table used as the second-level index. Since there is only one file ending in / c / b, the hash entry for 'c' in the second-level index points to the inverted inode of file / c / b. Since there are two files ending in / b / b, the hash entry for 'b' in the second-level index still points to another hash table used as the third-level index.

[0057] Dual index design: The file system's regular index starts from the root directory and indexes directory names level by level until the corresponding file is found. The index also includes both a reverse index and a regular index. During indexing, multiple threads simultaneously retrieve both the regular and reverse indexes to fully utilize the performance advantages of non-volatile memory. For example... Figure 2 As shown, when querying a file using a dual-index approach, a new thread is first created to query the file using the regular index, and then the file is queried using the inverted index within the current thread. Specifically, querying a file using the inverted index involves the following steps:

[0058] Step 1: Locate the hash table of the first-level index and use the file name at the end of the file path as the key of the hash table.

[0059] Step 2: Locate the corresponding hash entry in the hash table based on the hash key.

[0060] Step 3: If the hash item is empty, return a result indicating that the file does not exist.

[0061] Step 4: If the hash entry points to the hash table of the next level index, then set the directory name of the parent directory in the file path as the key of the hash table and return to Step 2.

[0062] Step 5: If the hash item points to an inverted index node, compare the file path in the inverted index node with the queried file path.

[0063] Step 6: If they match, return the file pointed to by the file inode in the reverse inode.

[0064] Step 7: If there is a discrepancy, return a result indicating that the file does not exist.

[0065] If the regular indexing thread has already found the result at any step, it will directly return the found result (file does not exist or the corresponding file is missing).

[0066] In summary, the indexing method for file systems based on non-volatile memory proposed in this invention fully considers the performance advantages and characteristics of non-volatile memory, combines the fast indexing requirements of user-space systems for small files with deep directory hierarchies, utilizes a reverse index that starts from the filename at the end of the file path, and performs parallel queries with the regular index. This avoids the situation where the performance of the reverse index is lower than that of the regular index in extreme cases, reduces the overall index query latency, and thus improves the indexing performance of the file system for small files with deep directory hierarchies.

[0067] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0068] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. An indexing method for a file system based on non-volatile memory, characterized in that, include: Simultaneously perform reverse indexing and regular indexing, and use multiple threads to search the regular index and reverse index until the corresponding file is found; The reverse index starts from the last filename of the file path and traverses the file path backward to index the directory until the corresponding file is obtained. The conventional index starts from the root directory and proceeds through directory names until the corresponding file is found. The reverse index includes a multi-level index. When the file system uses the file path for reverse indexing, it first uses the file name at the end of the path to search in the first-level index. If the result obtained by the first-level index contains multiple files, it then searches in the second-level index again using the directory name at the next end of the path, until the corresponding file is found. Each level of the inverted index consists of a hash table. The first-level index uses the filename at the end of the filename to perform a hash search to locate the hash table entry. If the hash table entry corresponds to only one file, it points to the file's inverted index node. The current inverted index node stores the complete path of the file and the file index node. The complete path is used to perform a complete match with the search path, and the file index node points to the actual location of the file. If the hash table entry corresponds to multiple files, it points to another hash table as the next-level index. A hash search is performed using the file directory of the filename's parent directory, and so on.

2. The indexing method for a file system based on non-volatile memory according to claim 1, characterized in that, The method of simultaneously searching the regular index and the reverse index using multiple threads until the corresponding file is found is as follows: when querying the regular index and the reverse index at the same time, if neither index finds the file corresponding to the file path, the current file is directly returned as not existing; if either index finds the corresponding file based on the file path, the current file is directly returned.

3. The indexing method for a file system based on non-volatile memory according to claim 1, characterized in that, When performing file system directory operations, when modifying the regular file system index, the reverse index must also be modified to ensure that the regular index and the reverse index contain the same file system directory information.

4. An indexing system for a file system based on non-volatile memory, characterized in that, include: Simultaneously perform reverse indexing and regular indexing, and use multiple threads to search the regular index and reverse index until the corresponding file is found; The reverse index starts from the last filename of the file path and traverses the file path backward to index the directory until the corresponding file is obtained. The conventional index starts from the root directory and proceeds through directory names until the corresponding file is found. The reverse index includes a multi-level index. When the file system uses the file path for reverse indexing, it first uses the file name at the end of the path to search in the first-level index. If the result obtained by the first-level index contains multiple files, it then searches in the second-level index again using the directory name at the next end of the path, until the corresponding file is found. Each level of the inverted index consists of a hash table. The first-level index uses the filename at the end of the filename to perform a hash search to locate the hash table entry. If the hash table entry corresponds to only one file, it points to the file's inverted index node. The current inverted index node stores the complete path of the file and the file index node. The complete path is used to perform a complete match with the search path, and the file index node points to the actual location of the file. If the hash table entry corresponds to multiple files, it points to another hash table as the next-level index. A hash search is performed using the file directory of the filename's parent directory, and so on.

5. The indexing system for a file system based on non-volatile memory according to claim 4, characterized in that, The method of simultaneously searching the regular index and the reverse index using multiple threads until the corresponding file is found is as follows: when querying the regular index and the reverse index at the same time, if neither index finds the file corresponding to the file path, the current file is directly returned as not existing; if either index finds the corresponding file based on the file path, the current file is directly returned.

6. The indexing system for a file system based on non-volatile memory according to claim 4, characterized in that, When performing file system directory operations, when modifying the regular file system index, the reverse index must also be modified to ensure that the regular index and the reverse index contain the same file system directory information.

Images