File system directory moving method and device based on flat structure namespace
Through technologies such as delayed execution and compressed prefix trees, the problem of inefficient movement of directory directories in flat structures is solved, and efficient directory movement operations are achieved, which reduces response delays and improves system performance.
Patent Information
- Application Number
- CN202411226816.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-12-27
AI Technical Summary
Directory movement operations in flat structure namespaces are inefficient, especially when moving multiple files, with time complexity up to O(N), resulting in performance and availability issues.
The directory movement method that delays execution is used, first complete the position movement of the target directory itself, record relevant parameters, and update the longest old directory name prefix to delay the movement of files and subdirectories in the target directory until the file metadata access instruction is received. Complete relocation and movement, and use compressed prefix trees and linked list structure to improve search efficiency.
It effectively reduces the response delay of directory movement operations, improves the overall system throughput, enables applications to make full use of system resources, and the directory movement efficiency is close to the hierarchical structure namespace.
Smart Images
Figure CN119203078A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of namespace management, and in particular to a file system directory movement method and device based on a flat structure namespace. Background Art
[0002] The namespace is the interface provided by the file system to the upper layer to access file metadata. It is responsible for converting readable, user-defined file identifiers into computer-recognizable storage addresses. In the namespace, each file has a unique name or path, and applications can use these identifiers to access the file's metadata without knowing the specific storage details of the underlying layer.
[0003] A classic way to organize the file system namespace is to use a hierarchical structure based on a directory tree, such as Figure 1 As shown. In the directory tree, all files are grouped into different directories, and a directory can also contain another directory (the latter is called a subdirectory). At the top of the directory tree is a top-level directory called the "root directory", which serves as the starting point of the entire namespace, and all other files and subdirectories are nested under it. When searching for a file, the file system will start from the root directory and search layer by layer until the target file is found on the leaf node. During the search process, the names of all nodes (root directory, subdirectory, and target file) are connected together to form the "full path" of the file.
[0004] As the performance requirements for file systems change, a flat structure namespace based on full path indexing has been designed, such as Figure 2 As shown. The most significant difference from the hierarchical namespace is that the flat namespace does not group files. All files are in the same single-level space and are managed using "key-value" pairs. The "key" is the full path of the file, and the "value" is the metadata of the file or the storage address of the metadata. In the key-value storage system, an index structure is needed to translate the "key" into the corresponding "value". Common index structures include: B+ tree, hash table, radix tree, etc.
[0005] Compared with hierarchical namespaces, flat namespaces are more efficient in finding individual files. This is because in a flat namespace, all files are located at the same top level, while in a hierarchical namespace, different files may be located in storage hierarchies at different depths. However, the drawback of a flat namespace is that it is extremely inefficient when moving multiple files (such as moving directories). The fundamental reason is that each file in a flat namespace is a separate individual, so it needs to be moved one by one, while in a hierarchical namespace, related files are grouped together through directories, so multiple files can be moved at the same time just by moving a single directory.
[0006] Figure 3 and Figure 4 Examples of completing a directory move operation in a hierarchical namespace and a flat namespace are shown respectively. In a hierarchical namespace, the directory move operation is divided into three steps: (1) find the target directory to be moved; (2) remove the target directory from its parent directory; (3) add the target directory to the specified new parent directory. The entire process involves only one move operation. In a flat namespace, the directory move operation first needs to complete the same move of the target directory itself as in the hierarchical namespace, and then move the files in the target directory to the new location one by one: (1) determine the range of all files in the target directory; (2) find the first file in the range; (3) remove the file from the old location; (4) add the file to the new location; (5) perform steps (2), (3), and (4) on the next file in the range until all files in the range are moved to the new location. From a theoretical analysis, for a directory containing N files, the time complexity of moving the directory in a hierarchical namespace is only O(1), while the time complexity of moving the directory in a flat namespace is as high as O(N). In the specific experimental analysis, it was also confirmed that the directory movement operation of the flat structure namespace is inefficient. Summary of the invention
[0007] The main purpose of the present invention is to overcome the defect of low efficiency of directory movement operation in flat structure namespace in the prior art, and to propose a file system directory movement method and device based on flat structure namespace, which can improve the actual performance and availability of the file system based on flat structure namespace.
[0008] The present invention adopts the following technical solution:
[0009] On the one hand, a method for moving a file system directory based on a flat structure namespace is characterized by comprising:
[0010] S1, after receiving the directory move operation instruction sent by the application, only complete the position move of the target directory itself, delay the movement of files and subdirectories under the target directory, record the relevant parameters of this operation, and update the longest old directory name prefix; wherein the relevant parameters of this operation include the old directory name before the move and the new directory name after the move; the longest old directory name prefix is to organize all leaf nodes storing the old directory name together in a linked list and maintain the common longest prefix of all old directory names;
[0011] S2, after receiving the file metadata access instruction sent by the application, searches for the matching structure based on the longest old directory name prefix, and relocates the access request to the correct storage location of the file or subdirectory that has not been moved; after completing the relocation, complete the movement of the file or subdirectory that has not been moved according to the old directory name before the move and the new directory name after the move.
[0012] Preferably, after receiving the directory moving operation instruction, the method further includes:
[0013] A timer is set for this directory move operation instruction and started. After the preset time has passed, the background thread checks whether the directory move has been completed. If not, the move operation is performed on all files that have not been moved to prevent the move of these files from being delayed indefinitely.
[0014] After the target directory's own move operation is completed, feedback is given to the application that the directory move operation has been completed.
[0015] Preferably, the file system directory movement method based on a flat structure namespace further includes:
[0016] During the target directory movement process, the old directory name and the new directory name are stored in a compressed prefix tree in memory; the contents of all intermediate nodes on a single path of the compressed prefix tree are combined to form a new directory name, and the leaf node stores the old directory name corresponding to the new directory name.
[0017] Preferably, the S2 specifically includes:
[0018] S21, after receiving the file metadata access instruction sent by the application, first match the parameter full path with the longest old directory name prefix, if the match fails, go to S23; if the match succeeds, go to S22;
[0019] S22, matching the full path of the parameter with all the old directory names, if the match fails, searching in the namespace using the full path of the parameter, and then going to S23; if the match succeeds, checking whether the length of the full path of the parameter and the old directory name are equal, if not, returning to the application that the file does not exist; if they are equal, first completing the entire directory move operation corresponding to the old directory name, and then going to S23;
[0020] S23, searching in the namespace using the full path of the parameter, if the search is successful, returning metadata information of the target file to the application; if the search is unsuccessful, go to S24;
[0021] S24, matching the parameter full path with all new directory names, if the matching fails, returning to the application that the file does not exist; if the matching succeeds, completing the file move operation, and going to S25;
[0022] S25, replace the matching part in the parameter full path, that is, the new directory name, with the corresponding old directory name, so as to obtain the original full path of the file before the directory move operation occurs; use the original full path to find the file in the namespace, remove the file from the old location, and add it to the new location corresponding to the parameter full path.
[0023] On the other hand, a file system directory moving device based on a flat structure namespace includes:
[0024] The target directory moving module is used to complete the position movement of the target directory itself after receiving the directory moving operation instruction sent by the application, delay the movement of files and subdirectories under the target directory, record the relevant parameters of this operation, and update the longest old directory name prefix; wherein the relevant parameters of this operation include the old directory name and the new directory name; the longest old directory name prefix is to organize all leaf nodes storing the old directory name together in a linked list and maintain the common longest prefix of all old directory names;
[0025] The delayed execution move module is used to, after receiving the file metadata access instruction sent by the application, find the matching structure based on the longest old directory name prefix, and relocate the access request to the correct storage location of the file or subdirectory that has not been moved; after completing the relocation, complete the movement of the file or subdirectory that has not been moved according to the old directory name before the move and the new directory name after the move.
[0026] Compared with the prior art, the present invention has the following beneficial effects:
[0027] After receiving the directory move operation instruction sent by the application, the present invention only completes the position movement of the target directory itself. After receiving the file metadata access instruction sent by the application, the present invention first completes the file / subdirectory relocation, and then realizes the file / subdirectory movement, so as to evenly distribute the movement cost of the entire directory to each file access operation, which can effectively reduce the response delay of the directory move operation of the file system based on the flat structure namespace, and at the same time enable the application to make more full use of system resources, thereby improving the overall throughput of the system and further improving the file system namespace technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 A structural diagram of the hierarchical namespace;
[0029] Figure 2 A structural diagram of the namespace for a flat structure;
[0030] Figure 3 This is an example diagram of completing a directory move operation in a hierarchical namespace;
[0031] Figure 4 This is an example diagram of completing a directory move operation in a flat structure namespace;
[0032] Figure 5 A brief flow chart of a file system directory moving method based on a flat structure namespace according to an embodiment of the present invention;
[0033] Figure 6 This is a rendering of a method for implementing directory movement based on delayed execution according to an embodiment of the present invention;
[0034] Figure 7 A detailed flow chart of a file system directory moving method based on a flat structure namespace according to an embodiment of the present invention;
[0035] Figure 8 A structural diagram of a compressed prefix tree according to an embodiment of the present invention;
[0036] Fig. 9 It is a structural block diagram of a file system directory moving device based on a flat structure namespace according to an embodiment of the present invention. DETAILED DESCRIPTION
[0037] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only intended to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art may make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.
[0038] See also Figure 5As shown, a file system directory moving method based on a flat structure namespace is characterized by comprising:
[0039] S1, after receiving the directory move operation instruction sent by the application, only complete the position move of the target directory itself, delay the movement of files and subdirectories under the target directory, record the relevant parameters of this operation, and update the longest old directory name prefix; wherein the relevant parameters of this operation include the old directory name before the move and the new directory name after the move; the longest old directory name prefix is to organize all leaf nodes storing the old directory name together in a linked list and maintain the common longest prefix of all old directory names;
[0040] S2, after receiving the file metadata access instruction sent by the application, searches for the matching structure based on the longest old directory name prefix, and relocates the access request to the correct storage location of the file or subdirectory that has not been moved; after completing the relocation, complete the movement of the file or subdirectory that has not been moved according to the old directory name before the move and the new directory name after the move.
[0041] As described above, the present invention is based on delayed directory movement, and the effect of the method is as follows: Figure 6 As shown, the specific principles are as follows.
[0042] (1) In UNIX-like operating systems, directory movement is accomplished by the system call rename(), which receives two parameters: the old directory name of the target directory to be moved, and the new directory name after the target directory is moved to the new location;
[0043] (2) After the operating system executes the rename() system call, the application can only observe the changes of the target directory itself, and the changes of all files and subdirectories contained in it are hidden. Therefore, delaying the update of these files and subdirectories will not affect the execution of the rename() system call itself;
[0044] (3) After the rename() system call is successfully executed, the application will think that all files and subdirectories contained in the target directory have also been moved to the new location, so it will use the new path name to access these files and subdirectories. However, since these files and subdirectories are moved late, they are still in the old location and cannot be accessed through the new directory name;
[0045] (4) In order to allow delayed moved files and subdirectories to be correctly accessed, this method adds records of two operation parameters to the execution process of the rename() system call, and stores these records in a data structure that supports efficient search and matching. When the application finds that a file or subdirectory cannot be accessed using the current path name, it will search for all rename() parameter records and check whether they can match, and obtain the original path name of the file or subdirectory through prefix replacement to achieve correct access. This process is called "relocation";
[0046] (5) After the "relocation", all the parameters required to complete the move of the file or subdirectory are available, so the move operation can be completed immediately with very little overhead, thus spreading the larger overhead required to move the entire directory to the access of each file or subdirectory in it;
[0047] (6) Experimental results show that after using this method, the response time of directory move operations in a flat namespace is greatly reduced, reaching the same level as that in a hierarchical namespace.
[0048] See also Figure 7 As shown, the specific execution process of a file system directory movement method based on a flat structure namespace will be described in detail as follows.
[0049] (1) After receiving the instruction of the directory move operation, the directory to be moved is moved first; then the two parameters of this directory move operation, "old directory name" and "new directory name", are combined into a single log message and added to the persistent log on the disk; finally, with "new directory name" as the key and "old directory name" as the value, this key-value pair is inserted into the compressed prefix tree in the memory.
[0050] In the present invention, the use of a compressed prefix tree is involved. A compressed prefix tree is an optimized prefix tree structure that can merge multiple intermediate nodes with only one child node in the original prefix tree into one intermediate node, thereby improving the search efficiency of the prefix tree. Figure 8 Specifically, the contents of all the intermediate nodes on a certain path of the compressed prefix tree are combined to form a "new directory name", and the leaf node stores its corresponding "old directory name". In addition, the present invention also organizes all these leaf nodes together in a linked list, and maintains the common longest prefix of all the "old directory names" in a leaf node, which is called the "longest old directory name prefix".
[0051] (2) Update the “longest old directory name prefix” according to the current “old directory name”.
[0052] (3) Set a timer for this directory move operation instruction and start it. After the preset time has passed, the background thread will check whether the directory move has been completed and perform the move operation on all files that have not been moved. When the directory move is completed, the relevant records in the compressed prefix tree and the persistent log will be removed (because the files or subdirectories related to this record can be directly accessed without "relocation"), and the "longest old directory name prefix" will be updated accordingly.
[0053] (4) The moving operation of the target directory to be moved is completed, and then feedback is sent to the upper-level system that the directory moving operation has been completed.
[0054] (5) When receiving the relevant instructions for file metadata access, first check whether the parameter full path (for example, for the metadata access instruction stat(" / home / liming / study.txt"), / home / liming / study.txt is the parameter full path of the instruction) can be successfully matched with the "longest old directory name prefix". The judgment condition for successful matching is that the "longest old directory name prefix" is a prefix of the parameter full path. Assuming that the current "longest old directory name prefix" is " / home / liming", if the parameter full path is " / home / liming / sport / plan.txt", the match is successful; if the parameter full path is " / tmp / 20240901.data", the match is unsuccessful.
[0055] (6) If the match in (5) is successful, then traverse all the "old directory names" stored in the compressed prefix tree, and use the parameter full path to match these "old directory names". The judgment condition for successful matching is the same as the matching of the parameter full path and the "longest old directory name prefix" in (5). If there is no successful match, proceed to the next step. If there is a successful match, it means that the file being accessed does not exist, and different processing is required according to the matching situation: if the length of the parameter full path is greater than the "old directory name", it is sufficient to directly return to the upper-level system that the file does not exist; if the length of the parameter full path is equal to the "old directory name", it is necessary to block other operations, and the delayed directory move operation corresponding to the "old directory name" is completed immediately, and then continue to the next step.
[0056] (7) If the match in (5) is unsuccessful, the normal access process can continue, using the full path of the parameter to search in the namespace.
[0057] (8) If the search in (7) is successful, the relevant information of the file metadata can be obtained, and the required data specified in the instruction is returned to the upper-level system.
[0058] (9) If the search in (7) fails, the full path of the parameter is used to search in the compressed prefix tree to check whether there is a "new directory name" in the compressed prefix tree that can match the full path of the parameter. The judgment condition for successful matching is the same as the matching of the full path of the parameter with the "longest old directory name prefix" in (5). If there is no successful match, the upper system is directly fed back that the file does not exist; if there is a successful match, the next step is continued.
[0059] (10) Replace the matching part (i.e., "new directory name") in the parameter full path with the corresponding "old directory name" to obtain the original full path of the file before the directory move operation occurs. For example, for the parameter full path " / home / liming / sport / plan.txt", assuming that the matching "new directory name" is " / home / liming", and the "old directory name" corresponding to the "new directory name" is " / home / hanmei", then the original full path is " / home / hanmei / sport / plan.txt". Using the original full path, the file can be correctly found in the namespace, removed from the old location, and added to the new location corresponding to the parameter full path. In this way, the previously delayed move operation of the file will be completed, and then jump to (7) to continue execution.
[0060] See also Fig. 9 As shown, the present invention also discloses a file system directory moving device based on a flat structure namespace, comprising:
[0061] The target directory moving module is used to complete the position movement of the target directory itself after receiving the directory moving operation instruction sent by the application, delay the movement of files and subdirectories under the target directory, record the relevant parameters of this operation, and update the longest old directory name prefix; wherein the relevant parameters of this operation include the old directory name and the new directory name; the longest old directory name prefix is to organize all leaf nodes storing the old directory name together in a linked list and maintain the common longest prefix of all old directory names;
[0062] The delayed move module is used to, after receiving the file metadata access instruction sent by the application, find the matching structure based on the longest old directory name prefix, and relocate the access request to the correct storage location of the file or subdirectory that has not been moved; after completing the relocation, complete the movement of the file or subdirectory that has not been moved according to the old directory name before the move and the new directory name after the move.
[0063] Specific implementation of a file system directory moving device based on a flat structure namespace The same file system directory moving method based on a flat structure namespace is not repeatedly described in this embodiment.
[0064] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.
Claims
1. A method for moving a file system directory based on a flat structure namespace, characterized in that: include: S1, after receiving the directory move operation instruction sent by the application, only complete the position move of the target directory itself, delay the movement of files and subdirectories under the target directory, record the relevant parameters of this operation, and update the longest old directory name prefix; wherein the relevant parameters of this operation include the old directory name before the move and the new directory name after the move; the longest old directory name prefix is to organize all leaf nodes storing the old directory name together in a linked list and maintain the common longest prefix of all old directory names; S2, after receiving the file metadata access instruction sent by the application, searches for a matching structure based on the longest old directory name prefix and relocates the access request to the correct storage location of the file or subdirectory that has not been moved; After the relocation is completed, the files or subdirectories that have not been moved are moved according to the old directory name before the move and the new directory name after the move.
2. The file system directory moving method based on a flat structure namespace according to claim 1, characterized in that: After receiving the directory move operation instruction, it also includes: A timer is set for this directory move operation instruction and started. After the preset time has passed, the background thread checks whether the directory move has been completed. If not, the move operation is performed on all files that have not been moved to prevent the move of these files from being delayed indefinitely. After the target directory's own move operation is completed, feedback is given to the application that the directory move operation has been completed.
3. The file system directory moving method based on a flat structure namespace according to claim 1, characterized in that: Also includes: During the target directory movement process, the old directory name and the new directory name are stored in a compressed prefix tree in memory; the contents of all intermediate nodes on a single path of the compressed prefix tree are combined to form a new directory name, and the leaf node stores the old directory name corresponding to the new directory name.
4. According to the file system directory movement method based on flat structure namespace according to claim 1, said S2 specifically comprises: S21, after receiving the file metadata access instruction sent by the application, first match the parameter full path with the longest old directory name prefix, if the match fails, go to S23; If the match is successful, go to S22; S22, matching the full path of the parameter with all the old directory names, if the match fails, searching in the namespace using the full path of the parameter, and then going to S23; If the match succeeds, check whether the length of the parameter full path and the old directory name are equal. If they are not equal, return to the application that the file does not exist; if they are equal, complete the entire directory move operation corresponding to the old directory name first, and then go to S23; S23, searching in the namespace using the full path of the parameter, and if the search is successful, returning metadata information of the target file to the application; If the search is unsuccessful, go to S24; S24, matching the parameter full path with all new directory names, if the matching fails, returning to the application that the file does not exist; if the matching succeeds, completing the file move operation, and going to S25; S25, replace the matching part in the parameter full path, that is, the new directory name, with the corresponding old directory name, so as to obtain the original full path of the file before the directory move operation occurs; use the original full path to find the file in the namespace, remove the file from the old location, and add it to the new location corresponding to the parameter full path.
5. A file system directory moving device based on a flat structure namespace, characterized in that: include: The target directory moving module is used to complete the position movement of the target directory itself after receiving the directory moving operation instruction sent by the application, delay the movement of files and subdirectories under the target directory, record the relevant parameters of this operation, and update the longest old directory name prefix; wherein the relevant parameters of this operation include the old directory name and the new directory name; the longest old directory name prefix is to organize all leaf nodes storing the old directory name together in a linked list and maintain the common longest prefix of all old directory names; The delayed execution move module is used to, after receiving the file metadata access instruction sent by the application, find the matching structure based on the longest old directory name prefix, and relocate the access request to the correct storage location of the file or subdirectory that has not been moved; after completing the relocation, complete the movement of the file or subdirectory that has not been moved according to the old directory name before the move and the new directory name after the move.