A method to improve the performance of metadata loading in Lustre HSM tiered storage

Abstract

The present invention relates to a method for improving the performance of loading metadata of Lustre HSM tiered storage. The method obtains a set of imported metadata based on namespace segmentation. The client asks the MDS server to verify whether the target directory imported into Lustre meets the import condition. If so, the subsequent file import process is set to be prohibited, and then the directory of each file meets the import condition. If so, the client sends a file import request to the MDS server. The client performs batch file import in parallel through the Lustre HSM import API based on the parallel metadata import mechanism of namespace segmentation. The client re-enables the function of checking whether the directory of the file meets the import condition during the file import process. The number of RPC calls required for the import of a single file and the number of RPC calls required for the import of a large number of files are reduced, and multi-node parallelization greatly improves the metadata import performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of tiered storage data loading, and in particular relates to a method for improving the performance of Lustre HSM tiered storage metadata loading. Background Art

[0002] Lustre HSM provides a means to integrate Lustre with other storage to provide tiered storage. The standard approach is to use Lustre as the HSM front end and other storage as the HSM back end. When loading data from the back-end object storage to the Lustre file system through HSM, you first need to import the file metadata information to Lustre through the HSM import mechanism. Then load the data. The data loading methods include HSM on demand loading and HSMrestore command loading. HSM on demand loading is triggered when the user accesses the imported file (file attribute released), and the HSM coordiantor notifies the HSM agent (copytool) to load the file data from the HSM back end. HSM restore command loading Through this command, the HSM coordiantor notifies the HSM agent (copytool) to load the file data from the HSM back end. In both of the above methods, to use HSM to load data, you must first import the file metadata information to Lustre through HSM import. For example, Lustre high-performance all-flash file system uses HSM and high-cost-effective large-capacity HDD object storage to form tiered storage. For typical application scenarios such as AI training, data is usually stored in the capacity layer and is imported into the performance layer before use. Therefore, import performance is very critical.

[0003] The Lustre HSM import function in the prior art is implemented through the client, and the client implements the import of a single file by sending several RPC calls to the Lustre MDS. The file IO process of Lustre DoM is to send an RPC request to the MDS through the client to obtain the file metadata information. When the MDS finds that the DoM function is enabled and the file meets the DoM configuration rules, it directly returns the data and metadata to the client on the MDT. Even for the import of a single file metadata, due to the excessive number of RPC calls involved in Lustre import, each RPC call involves round-trip network delay, and the RPC calls are interdependent serial execution mode, which seriously affects the performance of import and achieves extremely low file import efficiency. If the number of files involved in import is tens of thousands, the import performance will be severely limited, and for the import operation of millions of file metadata required for AI training, the import time will be very long. Moreover, due to the RPC performance limitation from a single client to MDS, it is not possible to simply increase the number of import operation worker threads to improve the import performance in a concurrent manner. According to Microsoft's report at LUG 2022, import performance was saturated before the concurrency reached 16, and the improvement in import performance by increasing the number of threads was extremely limited. In fact, our tests in an all-flash environment showed that even if the concurrency was increased from 16 to 256, the import performance remained almost unchanged, and the CPU and network loads were extremely low, almost in the single digit.

[0004] Therefore, how to improve the file metadata loading process of Lustre HSM import and enhance its execution efficiency and data loading performance is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The object of the present invention is to provide a method for improving the performance of loading metadata of Lustre HSM tiered storage, so as to improve the file metadata loading process of Lustre HSM import and enhance its execution efficiency and data loading performance.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0007] A method for improving Lustre HSM tiered storage metadata loading performance includes the following steps:

[0008] S1: obtain the set of imported metadata based on namespace segmentation;

[0009] S2: The client asks the MDS server to verify whether the target directory for importing Lustre meets the import conditions. If so, set to prohibit the subsequent file import process, and then check whether the directory of each file meets the import conditions. If so, execute step S3;

[0010] S3: The client sends a file import request to the MDS server. The client uses the parallel metadata import mechanism based on namespace segmentation to perform batch file import in parallel through the Lustre HSM import API.

[0011] S4: The client re-enables the function of checking whether the directory of the file meets the import conditions during the file import process, and executes steps S2-S3 in a loop;

[0012] The set of imported metadata in step S1 is a set of names and attributes of files to be imported; the Lustree HSM import API in step S3 is a library function llapi_hsm_import provided by Lustre, which is used to perform metadata import operations to the target directory according to the file name and file attributes.

[0013] Preferably, an import target verification function and an interface function for opening / closing the import target verification logic of the MDS server are provided in the storage system, including a need_check flag, a disable_check() function and an enable_check() function;

[0014] The disable_check() function is used to set the need_check flag to false, indicating that the import target verification logic of the MDS server is turned off, and the enable_check() function is used to set the need_check flag to true, indicating that the import target verification logic of the MDS server is turned on.

[0015] Preferably, the storage system is further provided with a parent_check() function, a parent_lock() function and a parent_unlock() function, wherein the parent_check() function is used to verify whether the target directory to be imported into the file satisfies the import condition; the parent_lock() function is used to lock the import target directory; and the parent_unlock() function is used to unlock the import target directory.

[0016] Preferably, step S2 is to verify whether the target directory where the file is imported from the HSM backend to Lustre meets the import condition based on the import target verification function. The specific process is as follows:

[0017] S21: Call parent_lock() function to lock the import target directory;

[0018] S22: Call the parent_check() function to verify whether the target directory where the file is to be imported meets the import conditions;

[0019] S23: calling the disable_check() function, setting the flag bit to false, and disabling the import target verification logic of the MDS server;

[0020] S24: Import the file into the target directory;

[0021] S25: calling the enable_check() function, setting the flag bit to true, and enabling the import target verification logic of the MDS server;

[0022] S26: Call the parent_unlock() function to unlock the import target directory.

[0023] Preferably, the parallel metadata import mechanism based on namespace segmentation in step S3 includes the following specific processes:

[0024] S31: Processing segmentation subsets through import tools;

[0025] S32: performing data import to the same Lustre target file system for different partition subsets on multiple clients;

[0026] S33: Use data sharding to assign different sharding_id to each node;

[0027] S34: When importing files, automatically skip the imported files that do not match your sharding_id to achieve parallel file import.

[0028] Preferably, the implementation process of data sharding in step S33 is:

[0029] Sum all the characters in the file name to get the value of sum_name in the file name;

[0030] Calculate the remainder of the value of sum_name and the number of concurrently executed client nodes;

[0031] When the remainder is the same as the sharding_id of the client node shard, the file is imported for the client node;

[0032] If the remainder is different from the sharding_id of the client node shard, the execution is skipped.

[0033] Preferably, in the process of implementing data sharding, a number is obtained by summing up all characters in the file name, and the number is set to N, and the remainder is a number from 0 to N-1. The client node obtains the number and compares it with its own sharding_id. If the remainder is the same as its own sharding_id, the file is imported for the client node. If the remainder is different from its own sharding_id, the execution is skipped.

[0034] Preferably, the parallel metadata import mechanism based on namespace segmentation is a mechanism that executes a hash function based on the client node ID and the name of the file to be imported, and the function satisfies that the N client nodes can completely segment the entire namespace and that the same file name does not exist on any two client nodes.

[0035] The beneficial effects of the present invention include:

[0036] The method for improving the metadata loading performance of Lustre HSM tiered storage provided by the present invention obtains a set of imported metadata based on namespace segmentation; the client asks the MDS server to verify whether the target directory imported into Lustre meets the import condition, if so, sets to prohibit the subsequent file import process, and then checks whether the directory of each file meets the import condition, if so, the client sends a file import request to the MDS server, and the client performs batch file import in parallel through the Lustre HSM import API based on the parallel metadata import mechanism of namespace segmentation; the client re-enables the function of checking whether the directory of the file meets the import condition during the file import process.

[0037] Migrating the existing Lustre HSM import implementation from the client to the MDS server reduces the number of RPC calls required to import a single file.

[0038] Batch mode RPC allows a single RPC call to handle multiple file imports, thereby reducing the number of RPC calls required to import a large number of files.

[0039] By adding an import condition check switch, for the same data import target, only one import condition check needs to be performed for batch data import, rather than performing a condition check for each file import, eliminating the inspection overhead for massive file imports.

[0040] To meet the needs of allocating inodes and dentry for new files involved in batch import, we break through the limitation that the VFS layer can only handle single inode and dentry allocation, adopt the idea of ​​batch allocation, and reduce the inode and dentry allocation overhead associated with the massive file creation process.

[0041] For the scenario of Lustre HSM importing to the same directory, this solution solves the problem of poor performance associated with the existing solution that the import operation of one target directory can only be performed on one node. Through multi-node parallelization, the metadata import performance is greatly improved.

[0042] This method breaks through the limitation of POSIX file system that creates new files and sets file attributes separately. When creating a new file, the file attributes are set directly, which reduces the system call overhead by at least half. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a structural diagram of the composition of the method for improving the Lustre HSM tiered storage metadata loading performance of the present invention.

[0044] Figure 2 The figure is a schematic diagram of the client HSM metadata import process in the prior art of the present invention.

[0045] Figure 3 The figure is a schematic diagram of three operation flows executed on the client side for calling the HSM import API for importing a single file in the prior art of the present invention.

[0046] Figure 4 The figure is a schematic diagram of three operation flow charts executed on the server side by calling the HSM import API in the prior art of the present invention.

[0047] Figure 5 The figure is a schematic diagram of the operation flow of calling the HSM import API for importing a single file on the server side according to the present invention.

[0048] Figure 6 Schematic diagram of a single client import process of the present invention

[0049] Figure 7 It is a schematic diagram of the operation flow of parallel import of the present invention. DETAILED DESCRIPTION

[0050] The following is combined with Figure 1 to Figure 7 The present invention is further described in detail:

[0051] See also Figure 2-Figure 4This application is based on the batching of HSM import operations to improve file loading performance. There are two ways. The first method is to maintain the existing import logic unchanged, batching RPC calls for creating files + batching RPC calls for modifying file attributes + batching RPC calls for closing files. When the Meta DataWriteback Cache function of the Lustre community is officially released, by modifying the client code and adding a batchimport interface, batch import of files can be achieved by calling batch RPC three times for three operations inside the function.

[0052] Method 2 is to create a new import logic, and treat the three actions of creating a file + modifying file attributes + closing a file as one action, and directly hand it over to MDS for execution through an RPC (import RPC). Compared with method 1, the number of RPC calls is further reduced. Even if the Lustre community cannot release the Meta Data WritebackCache function as scheduled, this method can be applied to the existing Lustre architecture and significantly improve the import performance of Lustre HSMimport. When this method is used in combination with the batchmodeRPC provided by the Meta Data Writeback Cache function, the import operation performance can be maximized. However, the combination of the three operations means that the workflow is different from that of Lustre (or a typical file system), and the MDS code needs to be significantly modified to implement special logic for import files, which makes the code implementation much more complicated. The improvement of method 2 is based on the C++ constructor list parameter. The C++ constructor introduces the logic before the list parameter. The default constructor constructs the object and completes the initialization of the object member variables. In the constructor with parameters, the object member variable assignment is performed. The logic after the C++ constructor introduces the list parameter is to directly use the values ​​given by the list parameter in the parameterized constructor to initialize the object member variables.

[0053] The logic of HSM import is to create a file through RPC call, assign default attributes to the new file, modify the default attributes of the newly created file through RPC call, and close the file through RPC call, which ensures that the previous attribute modification is 100% persistent. Therefore, the reason for HSMimport and C++ constructor to improve performance is similar, modifying the attributes again after creation. The cost of modifying the attributes here is much higher than that of the C++ constructor to modify the default value of the member variable with the initialization value, because the latter is just a variable copy, while here is an RPC call involving the network. The current import logic is based on the client as its implementation core, and the import function can be implemented with minimal changes to Lustre, at the cost of performance loss. The performance loss here was not significant in the past, because the traditional application of Lustre parallel file system is HPC scientific computing, which faces large files. In recent years, as new loads such as AI and ML have gradually increased and become the main loads of HPC applications, HSMimport faces a large number of small files, which greatly magnifies the shortcomings of insufficient import performance based on the client.

[0054] For users of the HSM import function, the file is closed after the import is completed. Therefore, the import can be designed to be similar to the C++ constructor initialization list. When MDS creates a directory entry and allocates an inode for the imported file, the file attribute initialization is completed directly, and the file is closed when the import call ends. This means that even for a single file import, only one RPC call is needed to complete the file metadata import. Compared with the current implementation, the performance is doubled.

[0055] Before the Lustre community introduced metadata batch submission, it was very difficult to implement batch import submission to the MDS server. However, through the above process optimization of executing HSM import on a single file, for the import of massive batch files, although it is still necessary to call RPC separately for each file import, the number of RPC calls is greatly reduced compared to the current implementation, and the performance is improved several times. In our test, it takes at least 4 hours to import the metadata of 40 million files in a Lustre configuration in an all-flash environment. After migrating the import operation from the client to the MDS server, the RPC call overhead is greatly reduced, the performance is improved exponentially, and the time is expected to be reduced to a fraction of the current time.

[0056] When the Lustre community introduces batch submission of metadata, combined with improvements in import execution from the client to the MDS server, the RPC execution performance of metadata operations will be improved by more than an order of magnitude. That is, the metadata loading time of 40 million small files will be reduced from the current more than 4 hours to a few minutes.

[0057] During the file import process, the Lustre file system ensures the legality of the import for safe import, while the user of the file system needs to ensure the legality of the import for unsafe import, otherwise it may cause some damage to the file system. The advantage is that since the file system no longer strictly checks certain operations during the import process, the performance of importing large amounts of files is greatly improved. Taking the simplest check of whether the import target directory exists as an example, in the current implementation (safe import), it is necessary to check whether the target directory exists for each imported file. For batch loading of large amounts of files, since the import target of these files is the same directory, the check of whether the directory exists can be completely placed outside the batch import execution.

[0058] Example 1

[0059] Based on the above theoretical foundation, this embodiment proposes a method for improving the performance of Lustre HSM tiered storage metadata loading. Figure 1 as well as Figure 5-Figure 7 As shown, the following steps are included:

[0060] S1: obtain the set of imported metadata based on namespace segmentation;

[0061] The metadata collection here is to confirm what kind of data to import, that is, which files need to be imported. It is a collection of file names and attributes. Each record in the collection is a file name and the related attributes of this file. In the subsequent import, each file in this collection will be used to perform the actual metadata import. The so-called metadata import is to create an empty file in the import directory based on the relevant information of the metadata, and set the relevant attributes of this empty file;

[0062] S2: The client asks the MDS server to verify whether the target directory of the Lustre import meets the import conditions based on the import target verification function. If so, set to prohibit the subsequent file import process, and then check whether the directory of each file meets the import conditions. If so, execute step S3;

[0063] Target verification based on the import target verification function is the relevant inspection work for the directory where the file is located when the Lustre file system executes the metadata import of a certain file. In the prior art, verification is for each file, because in the conventional import API call, it is not ensured that the target directory of each import is the same, so each file needs to be verified independently. In the hierarchical storage scenario processed by this application, the target directory imported from the HSM backend to Lustre is unique, which means that only one verification is required for the entire import process. After the verification is passed, all subsequent file imports can no longer perform the target directory check. For the import of 10 million files, it means that the original 10 million import directory checks are reduced to 1 time. Similarly, to add a file to the directory, it is necessary to obtain the access lock of the directory before adding it, and unlock the directory after adding it; in the scenario we are dealing with, since the target directory is unique, it is only necessary to lock the directory once, and then unlock it after adding all the files, instead of performing the "directory shackles-add files-directory unlock" action for each file. For the import of 10 million files, it means that the original directory "lock-unlock" operation when importing the directory 10 million times is reduced from 10 million times to 1 time.

[0064] S3: The client sends a file import request to the MDS server. The client uses the parallel metadata import mechanism based on namespace segmentation to perform batch file import in parallel through the Lustre HSM import API.

[0065] The parallel metadata import mechanism of namespace segmentation enables the use of import tools to process segmentation subsets, which means that data import to the same Lustre target file system can be performed on multiple clients for different segmentation subsets, improving performance by N times, where N is the number of clients. The Lustree HSM import API, that is, the library function llapi_hsm_import provided by Lustre, is used to perform metadata import operations to the target directory based on the file name and file attributes.

[0066] By adding sharding, i.e. data sharding, multiple nodes can execute an import task at the same time, as long as the client node is sharded with different sharding_ids, where sharding_id is the number of the client node shard. The client node will automatically skip the import files that do not match its own sharding_id, thereby achieving parallel import and solving the problem of low import performance of a single node due to the low latency overhead of RPC calls. Data sharding is to sum all the characters of the file name, i.e. the sum_name value, and then find the remainder of the sum_name value and the number of concurrently executed nodes. If the remainder is the same as the sharding_id, it means that the client node needs to execute it, otherwise it will skip the execution. Therefore, the performance is improved through parallelization. The more nodes there are, the higher the parallelization and the more obvious the performance improvement. That is, the execution time of N nodes importing a target directory at the same time is 1 / N of the execution time of a single node.

[0067] S4: The client re-enables the function of checking whether the directory of the file meets the import condition during the file import process, and executes steps S2-S3 in a loop.

[0068] In the above scheme, an import target verification function and an interface function for opening / closing the import target verification logic of the MDS server are provided in the storage system, including a need_check flag, a disable_check() function and an enable_check() function; wherein the disable_check() function is used to set the need_check flag to false, indicating closing the import target verification logic of the MDS server, and the enable_check() function is used to set the need_check flag to true, indicating opening the import target verification logic of the MDS server. The storage system is also provided with a parent_check() function, a parent_lock() function and a parent_unlock() function, wherein the parent_check() function is used to verify whether the target directory to be imported by the file meets the import conditions; the parent_lock() function is used to lock the import target directory; and the parent_unlock() function is used to unlock the import target directory.

[0069] On the basis of Example 1, step S2 is to verify whether the target directory where the file is imported from the HSM backend to Lustre meets the import conditions based on the import target verification function. The specific process is as follows:

[0070] S21: Call parent_lock() function to lock the import target directory;

[0071] S22: Call the parent_check() function to verify whether the target directory where the file is to be imported meets the import conditions;

[0072] S23: calling the disable_check() function, setting the flag bit to false, and disabling the import target verification logic of the MDS server;

[0073] S24: Import the file into the target directory;

[0074] S25: calling the enable_check() function, setting the flag bit to true, and enabling the import target verification logic of the MDS server;

[0075] S26: Call the parent_unlock() function to unlock the import target directory.

[0076] Based on the fact that the target directory of HSM import is the same, the relevant checks are moved outward, and there is only one call in the loop body, that is, parent_check(), parent_lock(), and parent_unlock() are no longer called repeatedly for each file, but are only called once for the import operation of a directory, which greatly improves the execution efficiency.

[0077] The parallel metadata import mechanism based on namespace segmentation in step S3 includes the following specific processes:

[0078] S31: Processing segmentation subsets through import tools;

[0079] S32: performing data import to the same Lustre target file system for different partition subsets on multiple clients;

[0080] S33: using data sharding to assign different sharding_id to each node, summing up all characters in the file name to obtain the value of sum_name in the file name; calculating the remainder of the value of sum_name and the number of concurrently executed client nodes; when the remainder is the same as the sharding_id of the client node shard, import the file for the client node; when the remainder is different from the sharding_id of the client node shard, skip the execution;

[0081] S34: When importing files, automatically skip the imported files that do not match your sharding_id to achieve parallel file import.

[0082] In the process of implementing data sharding, a number is obtained by summing up all the characters in the file name, and the number is set to N, and the remainder is a number from 0 to N-1. The client node obtains the number and compares it with its own sharding_id. If the remainder is the same as its own sharding_id, the file is imported for the client node. If the remainder is different from its own sharding_id, the execution is skipped.

[0083] The parallel metadata import mechanism based on namespace segmentation is a mechanism that executes a hash function based on the client node ID and the name of the file to be imported. The function satisfies the requirement that the N client nodes can completely segment the entire namespace and that there is no identical file name on any two client nodes.

[0084] When a client performs a metadata acquisition operation on the namespace to be imported into the HSM backend, it obtains all the metadata. Assuming there are M metadata records, the time consumption is T0. The metadata in the namespace to be imported from the HSM backend is obtained by querying the HSM backend storage to obtain the data to be imported into the Lustre file system of the HSM front end. For each metadata record, the import is performed through the Lustre HSM import API, that is, the import function is called M times. Assuming the total time is T1, the import is to create a file in the Lustre file system and set the attributes of this file to the archived+ release state, that is, the backend has data and the front-end file content is empty.

[0085] When there are N clients, each client performs a metadata acquisition operation on the namespace to be imported by the HSM backend, and only acquires the metadata that it should process, and the time consumption is T0. Each client only acquires the metadata that it should process, and filters out the metadata that does not belong to its own processing, but these metadata still need to be downloaded to the client, so there is no difference in time between N clients and the single client currently used, and the time cannot be reduced. If each client performs import in parallel through the Lustre HSM import API, that is, each client has M / N import function calls, the time consumption of each client is approximately T1 / N. Since the import time T1 is much greater than the time T0 required for metadata acquisition, the entire time overhead can be approximately regarded as T1 and T1 / N. Therefore, the import time is greatly reduced, that is, the file import performance is greatly improved. Therefore, the present invention improves the import performance by using more clients to perform import at the same time, and each client is only responsible for a part of the total data.

[0086] Taking N file imports as an example, the file import logic in the prior art is:

[0087] Import file 1 (import target directory verification, execute the action of importing file 1);

[0088] Import file 2 (import target directory verification, execute the action of importing file 2);

[0089] …

[0090] Import file N (import target directory verification, execute the action of importing file N).

[0091] The file import logic of the present invention is: verifying the legality of the import target through the import target verification function, and closing the import target verification logic of the MDS server.

[0092] Import file 1 (execute the action of importing file 1);

[0093] Import file 2 (execute the action of importing file 2);

[0094] …

[0095] Import file N (execute the action of importing file N);

[0096] By turning on the import target verification logic of the MDS server, the number of import target checks is greatly reduced, and the performance is significantly improved for scenarios where massive files are imported.

[0097] In summary, the method for improving the metadata loading performance of Lustre HSM tiered storage provided by the present invention obtains a set of imported metadata based on namespace segmentation; the client asks the MDS server to verify whether the target directory imported into Lustre meets the import conditions, and if so, sets to prohibit the subsequent file import process, and then checks whether the directory of each file meets the import conditions, and if so, the client sends a file import request to the MDS server, and the client performs batch file import in parallel through the Lustre HSM import API based on the parallelized metadata import mechanism of namespace segmentation; the client re-enables the function of checking whether the directory of the file meets the import conditions during the file import process.

[0098] Migrate the existing Lustre HSM import implementation from the client to the MDS server, reducing the number of RPC calls required for a single file import. With batch mode RPC, a single RPC call can handle multiple file imports, thereby reducing the number of RPC calls required for a large number of file imports. By adding an import condition check switch, only one import condition check is required for batch data import, rather than a condition check for each file import, eliminating the check overhead for massive file imports. For the needs of allocating inodes and dentry for new files involved in batch import, the limitation that the VFS layer can only handle single inode and dentry allocation is broken, and the idea of ​​batch allocation is adopted to reduce the inode and dentry allocation overhead associated with the massive file creation process. For the scenario of Lustre HSM import to the same directory, the existing solution can only perform the import operation of a target directory on one node, which is accompanied by poor performance. Through multi-node parallelization, the metadata import performance is greatly improved. Breaking the limitation of the POSIX file system that creates new files and sets file attributes separately, directly sets file attributes when creating new files, reducing at least half of the system call overhead.

Claims

1. A method for improving the performance of loading metadata for Lustre HSM tiered storage, characterized in that: The following steps are involved: S1: obtain the set of imported metadata based on namespace segmentation; S2: The client asks the MDS server to verify whether the target directory to be imported into Lustre meets the import conditions. If so, set to prohibit the subsequent file import process, and then check whether the directory of each file meets the import conditions. If so, execute step S3; S3: The client sends a file import request to the MDS server. The client uses the parallel metadata import mechanism based on namespace segmentation to perform batch file import in parallel through the Lustre HSM import API. S4: The client re-enables the function of checking whether the directory of the file meets the import conditions during the file import process, and executes steps S2-S3 in a loop; The set of imported metadata in step S1 is a set of names and attributes of files to be imported; the Lustree HSM import API in step S3 is a library function llapi_hsm_import provided by Lustre, which is used to perform metadata import operations to the target directory according to the file name and file attributes.

2. The method for improving the metadata loading performance of Lustre HSM tiered storage according to claim 1, characterized in that: The storage system is provided with an import target verification function and an interface function for opening / closing the import target verification logic of the MDS server, including the need_check flag, the disable_check() function and the enable_check() function; The disable_check() function is used to set the need_check flag to false, indicating that the import target verification logic of the MDS server is turned off, and the enable_check() function is used to set the need_check flag to true, indicating that the import target verification logic of the MDS server is turned on.

3. A method for improving Lustre HSM tiered storage metadata loading performance according to claim 2, characterized in that: The storage system is also provided with a parent_check() function, a parent_lock() function and a parent_unlock() function, wherein the parent_check() function is used to verify whether the target directory to which the file is to be imported satisfies the import condition; The parent_lock() function is used to lock the import target directory; the parent_unlock() function is used to unlock the import target directory.

4. A method for improving Lustre HSM tiered storage metadata loading performance according to claim 3, characterized in that: Step S2 is to verify whether the target directory where the file is imported from the HSM backend to Lustre meets the import conditions based on the import target verification function. The specific process is as follows: S21: Call parent_lock() function to lock the import target directory; S22: Call the parent_check() function to verify whether the target directory where the file is to be imported meets the import conditions; S23: calling the disable_check() function, setting the flag bit to false, and disabling the import target verification logic of the MDS server; S24: Import the file into the target directory; S25: calling the enable_check() function, setting the flag bit to true, and enabling the import target verification logic of the MDS server; S26: Call the parent_unlock() function to unlock the import target directory.

5. The method for improving the metadata loading performance of Lustre HSM tiered storage according to claim 1, characterized in that: The parallel metadata import mechanism based on namespace segmentation in step S3 includes the following specific processes: S31: Processing segmentation subsets through import tools; S32: performing data import to the same Lustre target file system for different partition subsets on multiple clients; S33: Use data sharding to assign different sharding_id to each node; S34: When importing files, automatically skip the imported files that do not match their sharding_id to achieve parallel file import.

6. A method for improving Lustre HSM tiered storage metadata loading performance according to claim 5, characterized in that: The implementation process of data sharding in step S33 is as follows: Sum all the characters in the file name to get the value of sum_name in the file name; Calculate the remainder of the value of sum_name and the number of concurrently executed client nodes; When the remainder obtained is the same as the sharding_id of the client node shard, the file is imported for the client node; If the remainder obtained is not the same as the sharding_id of the client node shard, the execution is skipped.

7. A method for improving Lustre HSM tiered storage metadata loading performance according to claim 6, characterized in that: In the process of implementing data sharding, a number is obtained by summing up all the characters in the file name, and the number is set to N, and the remainder is a number from 0 to N-1. The client node obtains the number and compares it with its own sharding_id. If the remainder is the same as its own sharding_id, the file is imported for the client node. If the remainder is different from its own sharding_id, the execution is skipped.

8. The method for improving the Lustre HSM tiered storage metadata loading performance according to claim 7, characterized in that: The parallel metadata import mechanism based on namespace segmentation is a mechanism that executes a hash function based on the client node ID and the name of the file to be imported. The function satisfies the requirement that the N client nodes can completely segment the entire namespace and that there is no identical file name on any two client nodes.

Images