A method and system for improving Lustre small file management and performance

Abstract

The present invention belongs to the technical field of Lustre file management, and specifically relates to a method and system for improving Lustre small file management and performance, including a nested file system, a three-level index module, a cache index module and an adaptive cache module; the nested file system realizes the management and storage of small files; the three-level index module includes a small file global metadata index, a container file internal index and a cache index, which are used to optimize the retrieval and storage management of small files; the cache index module is used to manage small files with high frequency access and reduce repeated I / O requests; the adaptive cache module, the client dynamically adjusts the cache strategy of small files according to the frequency and mode of file access to improve the reading speed of high frequency access files and reduce network I / O operations. The performance bottleneck of the Lustre file system when processing small files is solved by the nested file system, and the access burden on the metadata server and the object storage server is reduced by the three-level index architecture.

Description

Technical Field

[0001] The present invention belongs to the technical field of Lustre file management, and in particular to a method and system for improving Lustre small file management and performance. Background Art

[0002] Lustre file system is one of the widely used parallel file systems in high performance computing (HPC) environments. It is an open source parallel distributed file system designed to support large-scale data storage and access in high performance computing environments. The main components of the Lustre file system include: Management Server (MGS): Provides configuration information for the entire file system; Manager Target (MGT): MGS is a block device that stores Lustre file system configuration information; Metadata Server (MDS): Manages file system namespace and provides metadata services, such as finding file names, directory information, file layout and access permissions; Metadata Target (MDT): MDS is a block device that stores Lustre file system metadata; Object Storage Server (OSS): Stores file data objects and provides file data access to clients; Object Storage Target (OST): OSS is a block device that stores user file data; Client: Mounts the Lustre file system and provides users with a unified file system namespace.

[0003] However, when faced with a large number of small files, Lustre's performance will drop significantly. The main reasons include:

[0004] First, frequent metadata operations: Each small file requires metadata creation, update, and query operations, which will cause a sharp increase in the load of the metadata server (MDS).

[0005] Second, high metadata overhead: one file requires the Lustre file system to allocate one inode. As the number of small files increases, the inode usage will also increase. When the inodes in the system are exhausted, even if there is still disk space left, no new files can be created.

[0006] Third, high I / O overhead: A large number of small file read and write operations will generate frequent I / O requests, increase network transmission and disk seek time, and reduce overall I / O performance.

[0007] Fourth, storage fragmentation: small files are scattered and stored on different storage nodes, which easily leads to storage space fragmentation and affects storage utilization and performance.

[0008] The above factors work together to severely limit the performance and scalability of the Lustre file system when processing small files, affecting the efficiency and user experience of high-performance computing (HPC) applications.

[0009] Therefore, how to improve the existing Lustre file system to improve the performance and scalability of the Lustre file system when processing small files is a technical problem that needs to be solved urgently. Summary of the invention

[0010] The purpose of the present invention is to provide a method and system for improving Lustre small file management and performance, and to improve the existing Lustre file system so as to improve the performance and scalability of the Lustre file system when processing small files.

[0011] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0012] A system for improving Lustre small file management and performance, including a nested file system, a three-level index module, a cache index module, and an adaptive cache module;

[0013] The nested file system is deployed on the client and is an intermediate layer between the application and the underlying Lustre file system, and is used to manage and store small files.

[0014] The three-level index module includes a small file global metadata index, a container file internal index, and a cache index, which are used to optimize the retrieval and storage management of small files;

[0015] The cache index module is used to manage small files with high frequency of access and reduce repeated I / O requests;

[0016] The adaptive cache module allows the client to dynamically adjust the cache strategy for small files according to the frequency and mode of file access to improve the reading speed of frequently accessed files and reduce network I / O operations.

[0017] Preferably, the nested file system includes an intermediate layer function module and an aggregation mechanism module, wherein the intermediate layer function module is located between the application and the Lustre file system to intercept read and write requests for all small files;

[0018] The aggregation mechanism module is used to aggregate multiple small files into a larger container file to reduce the overhead of metadata management and I / O operations.

[0019] Preferably, the specific process of the aggregation mechanism module aggregating multiple small files into a larger container file is as follows:

[0020] Creation and management of container files: The Lustre client maintains a container file pool and implements a pre-allocation strategy to avoid creating new container files on the IO path when small files are written, which causes additional overhead; container file creation follows the general large file creation process, and the metadata server maintains its metadata; the client assigns a unique container file identifier to each container file;

[0021] File aggregation: The Lustre client automatically merges multiple small files into a container file in the background;

[0022] Block aggregation: The nested file system sets the aggregation threshold based on the file size and file quantity parameters; when the set threshold is reached, the nested file system selects a new container file.

[0023] Preferably, the small file global metadata index introduces a distributed key-value database for tracking the container file identifier of each small file and the position offset of the small file in the container file;

[0024] The container file internal index is an index maintained by the nested file system on the client, stored in the container file, and records the specific location, size and metadata information of each small file in the container file;

[0025] The cache index is used to manage small files that are frequently accessed, reduce repeated I / O requests, is stored locally on the client, and tracks information about currently cached small files.

[0026] Preferably, the small file global metadata index includes a small file path, a container file identifier, a small file offset and a small file size;

[0027] When the client writes a new small file, the Lustre client updates the small file global metadata index and records the path, container file identifier and offset of the new file. When reading, when the client reads a small file, it first searches for the container file and small file offset through the small file global metadata index, and then extracts the required small file from the object storage server.

[0028] Preferably, the container file internal index includes the small file name, the offset of the small file in the container file, the small file size and the small file metadata;

[0029] When writing, when the client writes a new small file into the container file, the system updates the internal index of the container file and records the location and size of the new file; when reading, the client locates the small file in the container file through the internal index and extracts the file from it.

[0030] Preferably, the cache index includes small file path, cache location and cache status;

[0031] When the cache hits, a small file has been cached, and the client reads the file directly from the local cache, reducing I / O requests to the object storage server OSS; when the cache fails, the cached file has expired or does not exist, the client will read the file from the object storage server OSS and update the cache service.

[0032] In a second aspect, a method for improving Lustre small file management and performance is provided, based on any one of the system implementations for improving Lustre small file management and performance, including a process in which a small file is updated to become a large file and a large file is updated to become a small file;

[0033] The process of updating a small file into a large file includes the following:

[0034] Incremental update and transaction processing: When updating a file, only the changed part is written. The atomicity of the file update is ensured through the transaction mechanism, and any failed operation can be safely rolled back.

[0035] Dynamic migration process: When the size of a small file exceeds the preset threshold, the system automatically migrates it out of the container file of the small file aggregation and stores it in the large file area of ​​Lustre;

[0036] After the migration is completed, the data index is updated in a timely manner, and the enlarged files are removed from the small data global index, container file index, and cache index. For the files that have become larger after the update, the system dynamically adjusts the cache strategy;

[0037] The process of large files becoming small files after updating:

[0038] Dynamic re-aggregation mechanism: When a large file is reduced to a certain preset threshold, the system re-aggregates it into a small file container file to reduce the management and storage overhead of a single large file; re-aggregation is performed asynchronously in the background, and the system selects a specified container file and moves the reduced large file data into it;

[0039] Index management: After reaggregation, the small file global metadata index and the container file internal index are updated to reflect the new location of the file; the small file global index updates the file path and offset to relocate the small file to the aggregate file, and removes the index record of the file on the metadata server.

[0040] Preferably, the I / O process is included:

[0041] File writing process:

[0042] The user or application initiates a file write request;

[0043] The virtual file system layer intercepts write requests;

[0044] Determine whether the file is a small file based on a preset file size threshold;

[0045] If it is a small file, enter the small file processing process:

[0046] Small file processing flow: select the specified container file, write the file data into the data area of ​​the container file, update the internal index of the container file, update the global index of small files, and return the write operation result to the user or application;

[0047] If it is a large file, enter the large file processing process:

[0048] Directly call the underlying file interface to perform write operations;

[0049] Return the write operation result to the user or user program;

[0050] File reading process:

[0051] The user or application initiates a file read request;

[0052] If the cache hits, the data is returned directly to the user or application;

[0053] The virtual file system layer intercepts read requests;

[0054] Determine whether the file is a small file based on a preset file size threshold;

[0055] If it is a small file, enter the small file processing process:

[0056] Read the global index of small files to determine the container file and internal position of the file; read the file data from the data area of ​​the container file according to the offset and length of the container file index table; return the read data to the user or application;

[0057] If it is a large file, enter the large file processing flow: directly call the underlying file interface to read data from the independent large file, and return the read data to the user or application.

[0058] Preferably, the file reading process is also included:

[0059] The user or application initiates a file read request;

[0060] If the cache hits, the data is returned directly to the user or application;

[0061] The virtual file system layer intercepts read requests;

[0062] Determine whether the file is a small file based on a preset file size threshold;

[0063] If it is a small file, enter the small file processing process:

[0064] Read the global index of small files to determine the container file and internal position of the file; read the file data from the data area of ​​the container file according to the offset and length of the container file index table; return the read data to the user or application;

[0065] If it is a large file, enter the large file processing flow: directly call the underlying file interface to read data from the independent large file, and return the read data to the user or application.

[0066] The beneficial effects of the present invention include:

[0067] The method and system for improving Lustre small file management and performance provided by the present invention include a nested file system, a three-level index module, a cache index module and an adaptive cache module; the nested file system realizes the management and storage of small files; the three-level index module includes a small file global metadata index, a container file internal index and a cache index, which are used to optimize the retrieval and storage management of small files; the cache index module is used to manage small files with high frequency access and reduce repeated I / O requests; the adaptive cache module, the client dynamically adjusts the cache strategy of small files according to the frequency and mode of file access to improve the reading speed of high frequency access files and reduce network I / O operations. The three-level index architecture reduces the access burden on metadata servers and object storage servers.

[0068] First, by introducing a nested file system in the Lustre client as an intermediate layer between the application and the underlying Lustre file system, the client aggregates small files, including automatic file merging and aggregated file management operations. All processes are transparent to the client application. Users can access small files like accessing ordinary file systems, without additional mounting and unmounting operations, simplifying management. It is responsible for the efficient management and storage of small files, solving the performance bottleneck of the Lustre file system when processing small files.

[0069] Secondly, through the three-level index architecture, a global metadata index for small files, an internal index for container files, and a cache index are set up to efficiently manage and retrieve the location and status of small files, solving the problem of small file locating and reducing the frequent access to the metadata server, thereby reducing the access burden on the metadata server (MDS) and object storage server (OSS).

[0070] Thirdly, by introducing a distributed key-value database to store global index data of small files, the distributed key-value database runs on the Lustre distributed file system cluster nodes, which can be achieved without changing the original logic of the metadata server (MDS).

[0071] Thirdly, by setting up an adaptive cache module, the client dynamically adjusts the cache strategy of small files according to the frequency and mode of file access to improve the reading speed of frequently accessed files, reduce network I / O operations, and optimize the reading performance by combining cache indexes. In addition, by combining small files for storage, the storage fragmentation problem is reduced, and the performance impact caused by the excessive size of aggregated files is avoided by block management. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a schematic diagram of the integrated structure of the nested file system of the present invention.

[0073] Figure 2 It is a schematic diagram of the file writing process of the present invention.

[0074] Figure 3 It is a schematic diagram of the file reading process of the present invention. DETAILED DESCRIPTION

[0075] The following is combined with Figure 1~Figure 3 The present invention is further described in detail:

[0076] Example 1

[0077] See attached Figure 1 As shown, a system for improving Lustre small file management and performance includes a nested file system, a three-level index module, a cache index module, and an adaptive cache module. Figure 1 The nested file system is deployed on the client and is an intermediate layer between the application and the underlying Lustre file system, which is used to manage and store small files. By introducing a nested file system in the Lustre client as an intermediate layer between the application and the underlying Lustre file system, the client aggregates small files, including automatic file merging and aggregated file management operations. All processes are transparent to the client application. Users can access small files like accessing ordinary file systems, without additional mounting and unmounting operations, simplifying management. Responsible for the efficient management and storage of small files, it solves the performance bottleneck of the Lustre file system when processing small files.

[0078] The three-level index module includes a global metadata index for small files, an internal index for container files, and a cache index, which are used to optimize the retrieval and storage management of small files. The global metadata index for small files, the internal index for container files, and the cache index are set through a three-level index architecture to efficiently manage and retrieve the location and status of small files, solve the problem of small file location, and reduce the frequent access to the metadata server, thereby reducing the access burden on the metadata server (MDS) and the object storage server (OSS).

[0079] The cache index module is used to manage small files that are frequently accessed and reduce repeated I / O requests. In the adaptive cache module, the client dynamically adjusts the cache strategy of small files according to the frequency and mode of file access to improve the reading speed of frequently accessed files and reduce network I / O operations. The adaptive cache mechanism dynamically adjusts the cache strategy according to the access frequency and mode of small files, efficiently caches small files that are frequently accessed, and optimizes the reading performance in combination with the cache index. By merging small files for storage, the storage fragmentation problem is reduced, and at the same time, the performance impact caused by excessively large aggregate files is avoided through block management.

[0080] By introducing nested file system settings in the Lustre client to automatically merge and aggregate file management operations, all processes are transparent to client applications. Users can access small files like accessing ordinary file systems, without additional mounting and unmounting operations, simplifying management. It is responsible for the efficient management and storage of small files, solving the performance bottleneck of the Lustre file system when processing small files.

[0081] Example 2

[0082] Based on Example 1, the nested file system includes an intermediate layer function module and an aggregation mechanism module. The intermediate layer function module is located between the application and the Lustre file system to intercept read and write requests for all small files; the aggregation mechanism module is used to aggregate multiple small files into a larger container file to reduce the overhead of metadata management and I / O operations.

[0083] The specific process of the aggregation mechanism module aggregating multiple small files into a larger container file is as follows:

[0084] Creation and management of container files: The Lustre client maintains a container file pool and implements a pre-allocation strategy to avoid creating new container files on the IO path when small files are written, which causes additional overhead; container file creation follows the general large file creation process, and the metadata server maintains its metadata; the client assigns a unique container file identifier to each container file. File aggregation: The Lustre client automatically merges multiple small files into a container file in the background. Block aggregation: The nested file system sets the aggregation threshold based on the file size and file number parameters; when the set threshold is reached, the nested file system selects a new container file to maintain the balance and efficiency of the file system. The system automatically manages the file aggregation process, reducing the burden on administrators and the risk of human operational errors.

[0085] In this embodiment, the small file global metadata index introduces a distributed key-value database to track the container file identifier of each small file and the position offset of the small file in the container file; the container file internal index is an index maintained by the nested file system on the client, stored in the container file, and records the specific location, size and metadata information of each small file in the container file; the cache index is used to manage high-frequency access small files, reduce repeated I / O requests, and is stored locally on the client to track the currently cached small file information. The small file global metadata index includes the small file path, container file identifier, small file offset and small file size.

[0086] When the client writes a new small file, the Lustre client updates the small file global metadata index and records the path, container file identifier and offset of the new file. When reading, when the client reads a small file, it first searches for the container file and small file offset through the small file global metadata index, and then extracts the required small file from the object storage server.

[0087] In this embodiment, the internal index of the container file includes the small file name, the offset of the small file in the container file, the small file size and the small file metadata; when writing, when the client writes a new small file to the container file, the system updates the internal index of the container file and records the location and size of the new file; when reading, the client locates the location of the small file in the container file through the internal index and extracts the file from it. The cache index includes the small file path, cache location and cache status; when the cache hits, a small file has been cached, and the client reads the file directly from the local cache, reducing the I / O request to the object storage server OSS; when the cache fails, the cache file has expired or does not exist, and the client will read the file from the object storage server OSS and update the cache service.

[0088] Example 3

[0089] Based on Example 1 or Example 2, a method for improving Lustre small file management and performance is implemented based on a system for improving Lustre small file management and performance described in any one of the above, including a process in which a small file is updated to become a large file and a large file is updated to become a small file;

[0090] The process of updating a small file into a large file includes the following:

[0091] Incremental update and transaction processing: When updating a file, only the changed part is written. The atomicity of the file update is ensured through the transaction mechanism, and any failed operation can be safely rolled back.

[0092] Dynamic migration process: When the size of a small file exceeds the preset threshold, the system automatically migrates it out of the container file of the small file aggregation and stores it in the large file area of ​​Lustre;

[0093] After the migration is completed, the data index is updated in a timely manner, and the enlarged files are removed from the small data global index, container file index, and cache index. For the files that have become larger after the update, the system dynamically adjusts the cache strategy;

[0094] The process of large files becoming small files after updating:

[0095] Dynamic re-aggregation mechanism: When a large file is reduced to a certain preset threshold, the system re-aggregates it into a small file container file to reduce the management and storage overhead of a single large file; re-aggregation is performed asynchronously in the background, and the system selects a specified container file and moves the reduced large file data into it;

[0096] Index management: After reaggregation, the small file global metadata index and the container file internal index are updated to reflect the new location of the file; the small file global index updates the file path and offset to relocate the small file to the aggregate file, and removes the index record of the file on the metadata server.

[0097] A method for improving Lustre small file management and performance of the present invention also includes an I / O process:

[0098] See also Figure 2 The file writing process shown:

[0099] The user or application initiates a file write request;

[0100] The virtual file system layer intercepts write requests;

[0101] Determine whether the file is a small file based on a preset file size threshold;

[0102] If it is a small file, enter the small file processing process:

[0103] Small file processing flow: select the specified container file, write the file data into the data area of ​​the container file, update the internal index of the container file, update the global index of small files, and return the write operation result to the user or application;

[0104] If it is a large file, enter the large file processing process:

[0105] Directly call the underlying file interface to perform write operations;

[0106] Return the write operation result to the user or user program;

[0107] See also Figure 3 The file reading process shown:

[0108] The user or application initiates a file read request;

[0109] If the cache hits, the data is returned directly to the user or application;

[0110] The virtual file system layer intercepts read requests;

[0111] Determine whether the file is a small file based on a preset file size threshold;

[0112] If it is a small file, enter the small file processing process:

[0113] Read the global index of small files to determine the container file and internal position of the file; read the file data from the data area of ​​the container file according to the offset and length of the container file index table; return the read data to the user or application;

[0114] If it is a large file, enter the large file processing flow: directly call the underlying file interface to read data from the independent large file, and return the read data to the user or application.

[0115] Preferably, the file reading process is also included:

[0116] The user or application initiates a file read request;

[0117] If the cache hits, the data is returned directly to the user or application;

[0118] The virtual file system layer intercepts read requests;

[0119] Determine whether the file is a small file based on a preset file size threshold;

[0120] If it is a small file, enter the small file processing process:

[0121] Read the global index of small files to determine the container file and internal position of the file; read the file data from the data area of ​​the container file according to the offset and length of the container file index table; return the read data to the user or application;

[0122] If it is a large file, enter the large file processing flow: directly call the underlying file interface to read data from the independent large file, and return the read data to the user or application.

[0123] The present invention dynamically adjusts the storage layout of files according to file size and access mode, avoiding the performance overhead caused by fixed-size image files. Reduce network communication: Through intelligent layout and caching, the number of network communications is reduced and the overall performance is improved. The file layout can also be dynamically adjusted according to the actual usage of the file system to adapt to different workloads. Scalability: Supports large-scale cluster environments and can be expanded as the cluster scales, providing better scalability. This technical solution can be seamlessly integrated into the existing Lustre file system. Users do not need to change the existing workflow, and users do not need to understand the underlying technical details and can focus on business needs.

[0124] In summary, the system for improving Lustre small file management and performance provided by the present invention includes a nested file system, a three-level index module, a cache index module and an adaptive cache module; the nested file system realizes the management and storage of small files; the three-level index module includes a global metadata index of small files, an internal index of container files and a cache index to optimize the retrieval and storage management of small files; the cache index module is used to manage small files with high frequency access and reduce repeated I / O requests; the adaptive cache module, the client dynamically adjusts the cache strategy of small files according to the frequency and mode of file access to improve the reading speed of high frequency access files and reduce network I / O operations. The three-level index architecture reduces the access burden on metadata servers and object storage servers.

[0125] By introducing a nested file system in the Lustre client as an intermediate layer between the application and the underlying Lustre file system, the client aggregates small files, including automatic file merging and aggregated file management operations. All processes are transparent to the client application. Users can access small files like accessing ordinary file systems, without additional mounting and unmounting operations, simplifying management. It is responsible for the efficient management and storage of small files, solving the performance bottleneck of the Lustre file system when processing small files.

[0126] The three-level index architecture is used to set up a global metadata index for small files, an internal index for container files, and a cache index to efficiently manage and retrieve the location and status of small files, solving the problem of locating small files and reducing the frequent access to the metadata server, thereby reducing the access burden on the metadata server (MDS) and the object storage server (OSS). By introducing a distributed key-value database to store global index data for small files, the distributed key-value database runs on the nodes of the Lustre distributed file system cluster, which can achieve the original logic of the metadata server (MDS) without changing it. By setting up an adaptive cache module, the client dynamically adjusts the cache strategy of small files according to the frequency and mode of file access to improve the reading speed of high-frequency access files, reduce network I / O operations, and optimize the reading performance in combination with cache indexes. In addition, by combining small files for storage, the problem of storage fragmentation is reduced, and the performance impact caused by excessively large aggregated files is avoided through block management.

Claims

1. A system for improving Lustre small file management and performance, characterized in that: It includes nested file system, three-level index module, cache index module and adaptive cache module; The nested file system is deployed on the client and is an intermediate layer between the application and the underlying Lustre file system, and is used to manage and store small files. The three-level index module includes a small file global metadata index, a container file internal index, and a cache index, which are used to optimize the retrieval and storage management of small files; The cache index module is used to manage small files with high frequency of access and reduce repeated I / O requests; The adaptive cache module allows the client to dynamically adjust the cache strategy for small files based on the frequency and mode of file access to improve the reading speed of frequently accessed files and reduce network I / O operations; The nested file system includes an intermediate layer function module and an aggregation mechanism module. The intermediate layer function module is located between the application and the Lustre file system to intercept read and write requests for all small files. The aggregation mechanism module is used to aggregate multiple small files into a larger container file to reduce the overhead of metadata management and I / O operations; The specific process of the aggregation mechanism module aggregating multiple small files into a larger container file is as follows: Creation and management of container files: The Lustre client maintains a container file pool and implements a pre-allocation strategy to avoid creating new container files on the IO path when small files are written, which causes additional overhead; The container file is created using the general large file creation process, and its metadata is maintained by the metadata server; The client assigns each container file a unique container file identifier; File aggregation: The Lustre client automatically merges multiple small files into a container file in the background; Block aggregation: The nested file system sets the aggregation threshold based on the file size and file quantity parameters; when the set threshold is reached, the nested file system selects a new container file; This includes the process of small files becoming large files after being updated and large files becoming small files after being updated; The process of updating a small file into a large file includes the following: Incremental update and transaction processing: When updating a file, only the changed part is written. The atomicity of the file update is ensured through the transaction mechanism, and any failed operation can be safely rolled back. Dynamic migration process: When the size of a small file exceeds the preset threshold, the system automatically migrates it out of the container file of the small file aggregation and stores it in the large file area of ​​Lustre; After the migration is completed, the data index is updated in a timely manner, and the enlarged files are removed from the small file global metadata index, container file internal index, and cache index. For the files that have become larger after the update, the system dynamically adjusts the cache strategy; The process of large files becoming small files after updating: Dynamic re-aggregation mechanism: When a large file is reduced to a certain preset threshold, the system re-aggregates it into a small file container file to reduce the management and storage overhead of a single large file; re-aggregation is performed asynchronously in the background, and the system selects a specified container file and moves the reduced large file data into it; Index management: After reaggregation, both the small file global metadata index and the container file internal index are updated to reflect the new location of the file; The global metadata index of small files updates the file path and offset to relocate the small files to the aggregate file, and removes the index record of the file on the metadata server; Including I / O process: File writing process: The user or application initiates a file write request; The virtual file system layer intercepts write requests; Determine whether the file is a small file based on a preset file size threshold; If it is a small file, enter the small file processing process: Small file processing flow: select the specified container file, write the file data into the data area of ​​the container file, update the internal index of the container file, update the global metadata index of the small file, and return the write operation result to the user or application; If it is a large file, enter the large file processing process: Directly call the underlying file interface to perform write operations; Return the write operation result to the user or user program; File reading process: The user or application initiates a file read request; If the cache hits, the data is returned directly to the user or application; The virtual file system layer intercepts read requests; Determine whether the file is a small file based on a preset file size threshold; If it is a small file, enter the small file processing process: Read the global metadata index of the small file to determine the container file and internal position of the file; read the file data from the data area of ​​the container file according to the offset and length of the internal index table of the container file; Return the read data to the user or application; If it is a large file, enter the large file processing flow: directly call the underlying file interface to read data from the independent large file, and return the read data to the user or application.

2. A system for improving Lustre small file management and performance according to claim 1, characterized in that: The small file global metadata index introduces a distributed key-value database for tracking the container file identifier of each small file and the position offset of the small file in the container file; The container file internal index is an index maintained by the nested file system on the client, stored in the container file, and records the specific location, size and metadata information of each small file in the container file; The cache index is used to manage small files that are frequently accessed, reduce repeated I / O requests, is stored locally on the client, and tracks information about currently cached small files.

3. A system for improving Lustre small file management and performance according to claim 2, characterized in that: The small file global metadata index includes a small file path, a container file identifier, a small file offset and a small file size; When the client writes a new small file, the Lustre client updates the small file global metadata index and records the path, container file identifier and offset of the new file. When reading, when the client reads a small file, it first searches for the container file and small file offset through the small file global metadata index, and then extracts the required small file from the object storage server.

4. A system for improving Lustre small file management and performance according to claim 2, characterized in that: The container file internal index includes the small file name, the offset of the small file in the container file, the small file size and the small file metadata; When writing, when the client writes a new small file into the container file, the system updates the internal index of the container file and records the location and size of the new file; when reading, the client locates the small file in the container file through the internal index and extracts the file from it.

5. A system for improving Lustre small file management and performance according to claim 2, characterized in that: The cache index includes small file path, cache location and cache status; When the cache hits, a small file has been cached, and the client reads the file directly from the local cache, reducing I / O requests to the object storage server OSS; when the cache fails, the cached file has expired or does not exist, the client will read the file from the object storage server OSS and update the cache service.

Images