Lustre file system write processing method and apparatus

Abstract

The application provides a Lustre file system write processing method and device, the method comprising: in response to a write data request initiated by an application layer for a target file, detecting whether an accelerated write condition is met; if the detection result is yes, copying to-be-written data to a preset cache area in a kernel state of a Lustre client and returning write success to the application layer, applying a distributed write lock corresponding to a file offset range of the to-be-written data to a metadata service by using a background thread, and after holding the lock, writing the to-be-written data in batches into a page cache; if the detection result is no, writing the to-be-written data into the page cache page by page, including applying a distributed write lock corresponding to a file offset range of the page data to the metadata service by using a background thread, and after holding the lock, writing the page data into the page cache; and persistently storing data in the page cache to an object storage server. Compared with related technologies, the application avoids multiple lock applications and improves data write efficiency.

Description

Technical Field

[0001] This invention relates to the field of Lustre file system technology, and in particular to a method and apparatus for writing Lustre file system. Background Technology

[0002] Lustre is an open-source distributed file system designed for high-performance computing (HPC) and artificial intelligence training scenarios. All its services run in kernel mode, avoiding the overhead of switching between user and kernel modes and meeting low latency requirements down to the microsecond level. Lustre's core components include: a metadata service (MDT) for maintaining the file system directory tree and providing distributed locks; an object storage server (OST) for persistent storage of actual data; and a Lustre client that provides applications with a POSIX-compliant file system interface.

[0003] In AI model training scenarios, after training is completed, model parameters, optimizer status, etc. need to be written to the Checkpoint file. During this process, Lustre uses a byte-range lock mechanism to ensure data consistency when multiple clients write concurrently.

[0004] However, as the number of model parameters increases dramatically, the size of the checkpoint file also expands. The speed of writing checkpoint files directly affects the training efficiency of the model. The current data writing mechanism is time-consuming, with the writing of a single checkpoint file taking minutes, which seriously restricts training efficiency. Therefore, how to improve data writing efficiency while maintaining the original data consistency of Lustre is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] This application aims to at least partially address one of the technical problems in the related art.

[0006] Therefore, the first aspect disclosed in this application provides a Lustre file system write processing method, including the following steps: In response to a write data request for a target file initiated by the application layer, it is checked whether the conditions for accelerated writing are met. The write data request carries the data to be written and the corresponding file offset range. When the detection result is yes, the data to be written is copied to the preset cache area in the Lustre client kernel mode, and a write success is returned to the application layer. The background thread requests a distributed write lock from the metadata service corresponding to the file offset range of the data to be written. After holding the lock, the data to be written is written to the page cache in batches. If the detection result is negative, the data to be written is written to the page cache page by page in memory pages. The process of writing each page of data to the page cache includes: using a background thread to request a distributed write lock from the metadata service corresponding to the file offset range of the page data, and writing the page data to the page cache after holding the lock. The data in the page cache is persistently stored to the object storage server through the operating system's write-back mechanism.

[0007] Optionally, the step of copying the data to be written to the preset cache area in the Lustre client kernel space may include: The Lustre client uses its native LNet heartbeat mechanism to initiate a lock pre-lease request for the file offset range of the data to be written to the metadata service. The metadata service verifies the lock pre-lease request to determine whether the file offset range has been held by an exclusive lock by another client. If no exclusive lock is held, the service grants a temporary pre-lease to the Lustre client, marks the file offset range as pre-locked, and returns a temporary pre-lease grant response to the Lustre client via a heartbeat response message. Upon receiving a temporary pre-lease grant response, the step of copying the data to be written to a preset cache in the Lustre client kernel mode is executed.

[0008] Optionally, the step of initiating a lock pre-lease request to the metadata service for the file offset range of the data to be written includes: Using the memory blocks in the preset cache where the data to be written is distributed as leaf nodes, calculate the Merkle root hash of the data to be written; A lock pre-lease request containing a file identifier, a Merkle root hash, and the file offset range corresponding to the data to be written is sent to the metadata service. After the metadata service verifies the lock pre-lease request, it writes the file identifier, the file offset range corresponding to the data to be written, the Merkle root hash, and the pre-lease validity period into the metadata extended attributes of the target file, and initializes the corresponding global state to uncommitted. The metadata extended attributes are stored in the index node maintained by the metadata service for each file.

[0009] Optionally, the Lustre file system write processing method further includes: Calculate the hash value of each memory block in the preset cache area, construct a hash index table and store it in the kernel space memory of the Lustre client. The hash index table records the file offset range and hash value corresponding to each memory block. In response to a read data request initiated by the application layer, the hash index table is used to query whether there is corresponding valid data in the preset cache area according to the file offset range corresponding to the read data request; If valid data exists, copy the valid data from the preset cache to the application layer and return a read success message; If no valid data is found, the corresponding data is read from the page cache or object storage server and returned.

[0010] Optionally, the step of writing the data to be written in batches to the page cache after holding the lock includes: The first processing step is used to write the data to be written in batches to the page cache, and at the same time send a transaction preparation request carrying the Merkle root hash to the object storage server. The object storage server records the Merkle root hash and returns a successful preparation. The second processing step is used to send transaction completion information carrying the Merkle root hash to the metadata service after the data to be written in the page cache is persisted to the object storage server through the operating system's write-back mechanism. In response to the transaction completion information, the metadata service releases the distributed write lock and the temporary pre-lease of the file offset range, and updates the global state recorded in the metadata extended attribute of the target file from uncommitted to committed. The third processing step is used to delete the data to be written in the preset cache area and return a write failure to the application layer when the batch writing of the data to be written to the page cache fails or no response of preparation success is received.

[0011] Optionally, after calculating the Merkle root hash of the data to be written, the method further includes: Create a transaction record containing the Merkle root hash, the transaction record marking the local state of the current transaction as uncommitted, and write the transaction record to the transaction log area in the local memory of the Lustre client; After the data in the page cache is persistently stored to the object storage server through the operating system's write-back mechanism, the data to be written in the preset cache area is deleted, and the hash value corresponding to the data to be written is removed from the hash index table.

[0012] Optionally, the Lustre file system write processing method includes: After an abnormal restart, the Lustre client scans the transaction records in the transaction log area, obtains the Merkle root hash of the incomplete transaction as the local Merkle root hash, and the local state based on the transaction records. The local state includes uncommitted and committed. Query the metadata extended attributes of the corresponding file from the metadata service to obtain the global state of each transaction of the file and the global Merkle root hash corresponding to each transaction. The global state includes uncommitted and committed. For each incomplete transaction, the local Merkle root hash is compared with the global Merkle root hash, and the local state is compared with the global state; Based on the comparison results, perform operations such as cleaning up completed transactions, reapplying for pre-leases to continue complete transactions, or rolling back damaged transactions.

[0013] Optionally, the metadata service maintains a global file view of the pre-lease status, file offset range, and Merkle root hash of each Lustre client for the same file through the file's metadata extended attributes. When the metadata service receives a lock pre-lease request, it uses the global file view to verify the lock pre-lease request.

[0014] Optionally, the Lustre file system write processing method further includes: Monitor the global free memory ratio of the computing node where the Lustre client is located. When the global free memory ratio is lower than the preset low watermark, increase the data flushing rate of the background thread and pause copying newly written data to the preset cache area until the global free memory ratio recovers to the preset high watermark.

[0015] Based on the same inventive concept, a second aspect of this disclosure provides a Lustre file system write processing apparatus, comprising: The detection module is used to respond to a write data request for a target file initiated by the application layer and detect whether the conditions for accelerated writing are met. The write data request carries the data to be written and the corresponding file offset range. The accelerated data writing module is used to copy the data to be written to the preset cache area in the Lustre client kernel mode when the detection result is yes, and return a successful write to the application layer. It uses a background thread to apply for a distributed write lock corresponding to the file offset range of the data to be written from the metadata service, and after holding the lock, it writes the data to be written in batches to the page cache. The regular data writing module is used to write the data to be written to the page cache page by page in memory pages when the detection result is negative. The process of writing each page of data to the page cache includes: using a background thread to request a distributed write lock from the metadata service corresponding to the file offset range of the page data, and writing the page data to the page cache after holding the lock. The data in the page cache is persistently stored to the object storage server through the operating system's write-back mechanism.

[0016] Beneficial effects The Lustre file system write processing method provided in this application sets a preset cache in the Lustre client kernel mode, and when the accelerated write conditions are met, directly copies the data to be written to the preset cache and immediately returns success to the application layer. This removes the distributed lock request operation from the application write path and processes it asynchronously in batches by a background thread. Since the application write path only contains one memory copy, it avoids the network round-trip delay of multiple lock requests and greatly improves data write efficiency. When the preset cache does not meet the accelerated write conditions, it automatically degrades to the native Lustre page-by-page write process, ensuring the stable operation of the system under memory pressure. Attached Figure Description

[0017] Figure 1 This is a schematic diagram illustrating the principle of writing data in the Lustre system in related technologies; Figure 2 A flowchart of an embodiment of the Lustre file system write processing method provided in this disclosure; Figure 3 A schematic diagram illustrating the principle of an embodiment of the Lustre file system write processing method provided in this disclosure; Figure 4 This is a schematic diagram illustrating another embodiment of the Lustre file system write processing method provided in this disclosure. Detailed Implementation

[0018] To better understand this application, various aspects of this application will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely illustrative of exemplary embodiments of this application and are not intended to limit the scope of this application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0019] It should be noted that, where there is no conflict, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] The inventors analyzed the data writing mechanism of the existing Lustre system and found that when an application in the application layer (user space) initiates a write operation (e.g., saving a 1GB Checkpoint file), the Lustre client's processing is as follows: Figure 1As shown, the process is as follows: After the application calls the write() system call, the operating system kernel copies the data to be written into the page cache one by one, in units of memory pages (usually 4KB). The page cache is a disk data buffer managed by the kernel, and its basic operation unit is a memory page. It cannot atomically process large blocks of continuous data across pages. Before each page of data is written to the page cache, the Lustre client must request a distributed write lock for the file offset range corresponding to the page from the metadata service (MDT). After obtaining the lock, the client writes the page of data to the page cache and then immediately releases the lock. Dirty pages in the page cache are eventually asynchronously persisted to the object storage server (OST) by the operating system background thread through the writeback mechanism.

[0021] In the above process, a single 1GB data write is broken down into approximately 260,000 independent loops (1GB / 4KB) of "requesting a lock → writing to the page cache → releasing the lock". On the one hand, since each lock request includes a complete network round trip (RTT, usually 50~100 microseconds) and MDT conflict detection, writing to checkpoints in AI model training scenarios takes a long time. On the other hand, when multiple clients write to different areas of the same file at the same time, since the writing ranges of each client may overlap and each page lock request needs to go through MDT conflict detection, as the number of clients increases, lock conflicts will also increase significantly, reducing data writing efficiency.

[0022] To address at least one of the aforementioned technical problems, this disclosure provides a Lustre file system write processing method, such as... Figure 2 As shown, it includes the following steps: Step S101: In response to a write data request for the target file initiated by the application layer, detect whether the current conditions for accelerated writing are met. The write data request carries the data to be written and the corresponding file offset range.

[0023] The target file refers to the file specified by the application layer, i.e., the application, where data needs to be written, such as the checkpoint file in AI training.

[0024] A write request is a write operation initiated by an application through the write() system call. The request carries the data to be written, its starting offset and length in the target file. The starting offset and length together define the file offset range for this write operation.

[0025] Accelerated write conditions refer to the predefined conditions for enabling the accelerated write data mode.

[0026] Optionally, the accelerated write conditions include: the preset cache has been initialized and there are sufficient free memory blocks in the preset cache. The accelerated write data mode will only be enabled when the accelerated write conditions are met, i.e., step S102.

[0027] In some embodiments, the accelerated write condition further includes: the current target file has been marked as an accelerable file. Specifically, when a file is created or opened for the first time, the Lustre client sets an "allow acceleration" flag for a specific file (such as a Checkpoint file) or all files in a specific directory through extended attributes or mount options. When the application layer initiates a write data request, the client first checks the flag: if the flag is true and there are sufficient free memory blocks in the preset cache, the accelerated write path is entered (step S102); if the flag is false, even if there is free memory in the preset cache, the normal write path is directly followed (step S103). This setting enables acceleration only for files suitable for batch writing (such as Checkpoint files in AI training), avoiding small files or random writes from consuming valuable preset cache resources.

[0028] Compared with traditional data writing techniques, the improvements of this disclosed embodiment include: adding a preset cache tier (Client Cache Tier, CCT) in the Lustre client kernel space, specifically as follows: Figure 3 As shown, it is a memory region located in the Lustre client kernel mode, which is divided into contiguous physical memory blocks and maintains a list of pre-allocated free memory blocks. Each free memory block has a fixed size (e.g., 16KB) and is used to temporarily store data to be written.

[0029] In the application layer write() system call processing path, the pre-defined cache CCT is located after data is copied from user space to kernel space and before it is written to the page cache. It is independent of the operating system's page cache and is directly managed by the Lustre client, playing the role of a data transfer and batch aggregation.

[0030] In step S102, when the detection result is yes, the data to be written is copied to the preset cache area in the Lustre client kernel mode, and a write success is returned to the application layer. A background thread is used to apply for a distributed write lock corresponding to the file offset range of the data to be written from the metadata service. After holding the lock, the data to be written is written in batches to the page cache. The data in the page cache is persistently stored to the object storage server through the operating system's write-back mechanism.

[0031] Step S102 accelerates data writing: the data to be written from the application layer is first copied to the memory block of the preset cache CCT, and a write success is immediately returned. This removes the "lock acquisition" operation from the application layer's write path and delays it to the background thread. The background thread can aggregate multiple memory blocks belonging to the same file in the preset cache CCT into a batch, and only request a distributed write lock (denoted as Y1) for the entire batch. The coverage of this distributed write lock Y1 is the union of the file offset ranges of all memory blocks where the data to be written is distributed. Through this mechanism, the number of lock requests is reduced from once per page to once per batch. It should be noted that the application layer returns immediately after completing only one memory copy. Time-consuming operations such as lock acquisition, lock release, and batch flushing are all executed asynchronously by the background thread. The application write path completely avoids network round trips and lock conflict detection, and its latency is only the memory copy time (in microseconds). Compared with the Lustre native process, where each page write requires synchronous waiting for the lock RTT, this solution significantly reduces the application-perceived write latency by converting synchronous waiting into asynchronous batch processing, thereby significantly improving the write efficiency of the target file.

[0032] Step S103: If the detection result is negative, the data to be written is written to the page cache page by page in memory pages. The process of writing each page of data to the page cache includes: using a background thread to request a distributed write lock from the metadata service corresponding to the file offset range of the page data, and writing the page data to the page cache after holding the lock. The data in the page cache is persistently stored to the object storage server through the operating system's write-back mechanism.

[0033] In particular, step S103 implements the traditional Lustre file system's data writing process, that is, when the conditions for accelerated writing are not met, such as when the preset cache memory is insufficient, the native data writing process is adopted, that is, writing to the page cache page by page in memory pages.

[0034] To facilitate understanding of the relationships between the components in this embodiment, the structure of the Lustre client, operating system kernel, default cache (CCT), page cache, and background threads is described uniformly. The Lustre client itself is a kernel module running in the operating system kernel mode. The default cache (CCT) is a memory area managed internally by the Lustre client kernel module, allocated, reclaimed, and maintained by the Lustre client, and does not depend on the operating system's page cache. The background thread is a kernel thread created by the Lustre client kernel module, also running in kernel mode, responsible for requesting distributed write locks from the metadata service, writing data in batches to the page cache, and handling transaction commits. The page cache is a disk data buffer globally managed by the operating system kernel, independent of the Lustre client.

[0035] Compared with related technologies, the Lustre file system write processing method of this disclosure sets a preset cache in the Lustre client kernel mode, and when the accelerated write conditions are met, directly copies the data to be written to the preset cache and immediately returns success to the application layer. This removes the distributed lock request operation from the application write path and processes it asynchronously in batches by a background thread. Since the application write path only contains one memory copy, it avoids the network round-trip delay of multiple lock requests and greatly improves the data write efficiency. When the preset cache does not meet the accelerated write conditions, it automatically degrades to the native Lustre page-by-page write process, ensuring the stable operation of the system under memory pressure.

[0036] Regarding the accelerated data writing mode shown in step S102 above, the inventors further discovered that the data to be written is first copied to a preset cache area, and a write success message is returned to the application layer. Only then does the background thread request a distributed write lock from the metadata service corresponding to the file offset range of the data to be written. There is a time window between the data to be written falling into the preset cache area and the distributed write lock being officially acquired. During this period, the metadata service's lock table has not yet recorded that the file offset range is locked by any client. Therefore, if other clients request a distributed write lock for the same file offset range, the metadata service will consider it conflict-free and grant it. Other clients that obtain the distributed write lock can legally write their data to be written to the page cache and persist it, thereby overwriting the data to be written that is temporarily stored in the preset cache area but has not yet been flushed. At this time, the client that initiated the write data request has already received a write success return, and its application layer believes that the data has been safely written, but in reality, it may be overwritten or discarded, resulting in silent data corruption.

[0037] To address the aforementioned technical issues, this disclosure introduces a pre-lease mechanism before copying the data to be written to a preset cache area. By introducing the pre-lease mechanism, this application can perform lightweight intention registration. It does not provide mutual exclusion protection and is only used to inform the client in advance of the file offset range to be written to the metadata service.

[0038] In one possible implementation, the step of copying the data to be written to the preset buffer in the Lustre client kernel mode further includes: Step A1: Using the native LNet heartbeat message mechanism of the Lustre client, a lock pre-lease request for the file offset range of the data to be written is initiated to the metadata service. The metadata service verifies the lock pre-lease request to determine whether the file offset range has been held by an exclusive lock by another client. If no exclusive lock is held, a temporary pre-lease is granted to the Lustre client, the file offset range is marked as pre-locked, and a temporary pre-lease grant response is returned to the Lustre client via a heartbeat response message.

[0039] The LNet heartbeat message is a keep-alive message periodically sent between nodes in the Lustre network layer to detect node liveness and synchronize configuration information. This invention encapsulates a lock pre-lease request in the extended TLV field of the LNet heartbeat message, achieving in-band transmission with zero additional network round trips. Upon receiving the request, the metadata service returns the grant result via a heartbeat response message, again without adding extra RTT. The lock pre-lease request includes not only the file offset range but also information such as the file identifier.

[0040] In this embodiment of the disclosure, the exclusive lock includes two types: one is a formal distributed write lock, that is, a lock natively provided by Lustre for mutually exclusive writes; the other is a temporary pre-lease that has been granted to other clients. The temporary pre-lease, as an intention registration, will mark the corresponding file offset range as a pre-locked state. During the period when any client holds the exclusive lock, other clients cannot obtain any type of distributed write lock or temporary pre-lease for the same file offset range.

[0041] Step A2 involves copying the data to be written to a preset cache area in the Lustre client kernel mode after receiving a temporary pre-lease grant response.

[0042] The Lustre client only performs the operation of copying the data to be written to the preset cache after receiving the temporary pre-lease grant response. Subsequently, within the validity period of the temporary pre-lease, the background thread requests a formal distributed write lock from the metadata service. Since the metadata service already has a pre-lock record of the client's file offset range, the metadata service directly grants the formal distributed write lock without further conflict detection. After holding the lock, the background thread writes the data to be written in batches to the page cache. After the data is fully persisted to the object storage server through the operating system's write-back mechanism, the background thread releases the formal distributed write lock and notifies the metadata service to release the corresponding pre-locked state, thus completing the entire write process.

[0043] Through the aforementioned pre-lease mechanism, this embodiment advances lock acquisition from the time of writing to the page cache (i.e., the flush phase) to before the data is copied to the preset cache area. This allows the metadata service to know the client's writing intention in advance and lock the corresponding file offset range in advance, thereby eliminating the time window that exists in the above embodiments and avoiding silent data overwriting caused by other clients acquiring conflicting locks during the window period. At the same time, since the lock pre-lease application reuses the LNet heartbeat message, no additional network round trips are added, and the performance of the application write path is not affected.

[0044] In one possible approach, the step of initiating a lock pre-lease request to the metadata service for the file offset range of the data to be written includes: Step B1: Using the memory blocks where the data to be written is distributed in the preset cache as leaf nodes, calculate the Merkle root hash of the data to be written.

[0045] In this embodiment of the disclosure, before initiating a lock pre-lease application, the Lustre client calculates the Merkle root hash of the data to be written. The calculation method is as follows: the data to be written is divided into multiple data blocks according to the size of a single memory block in a preset cache (e.g., 16KB), with each data block corresponding to a memory block. The hash value of the data in each memory block is calculated as a leaf node. The hash values ​​of two adjacent leaf nodes are concatenated and hashed again. This process is repeated until a unique root hash value, namely the Merkle root hash, is obtained. This root hash can uniquely identify the complete data to be written. Any change in data bits will lead to inconsistency in the Merkle root hash.

[0046] Step B2: A lock pre-lease request containing a file identifier, a Merkle root hash, and the file offset range corresponding to the data to be written is sent to the metadata service. After the metadata service verifies the lock pre-lease request, it writes the file identifier, the file offset range corresponding to the data to be written, the Merkle root hash, and the pre-lease validity period into the metadata extended attributes of the target file, and initializes the corresponding global state to uncommitted. The metadata extended attributes are stored in the index node maintained by the metadata service for each file.

[0047] In the Lustre file system, each file corresponds to a unique inode in the metadata service. Besides storing standard attributes such as file size, permissions, and timestamps, this inode also provides an extended attribute area for storing custom metadata related to the file. This embodiment extends the metadata service by adding an anchor area to the extended attribute area. This anchor area records the transaction status, Merkle root hash, and temporary pre-lease information for each batch write transaction. The transaction status recorded in the metadata extended attributes is also the global status, and the Merkle root hash is the global Merkle root hash. When a client initiates a lock pre-lease request, a step of calculating and carrying the Merkle root hash is added, which binds the data integrity fingerprint to the pre-lease.

[0048] This embodiment, through the improvement of adding an anchoring area to the metadata extended attributes, enables the integrity fingerprint (Merkel root hash) and transaction state of each batch of data to be written to be persistently stored on the metadata service side. Subsequently, when the client performs batch writes, the metadata service can verify the integrity of the committed data based on the Merkel root hash recorded in the metadata extended attributes. In the recovery process after a client crash, the integrity of the data and whether the transaction has been committed can also be determined by comparing the Merkel root hash in the local transaction log with the global Merkel root hash in the extended attribute area. Therefore, by embedding the Merkel root hash into the pre-lease application, this embodiment makes the data integrity fingerprint a component of the pre-lease, providing a foundation for atomic commits and crash recovery.

[0049] In the above embodiments, after the data to be written is copied to the preset cache area CCT, a write success message is immediately returned to the application layer. At this time, if the application layer immediately initiates a read data request to read data within the same file offset range, since the data has not yet been flushed to the page cache or object storage server, the Lustre native read path cannot read the latest data from the page cache or object storage server, which does not conform to the read after write semantics required by the POSIX standard (i.e., the data just written should be able to be read immediately after a successful write).

[0050] To address the above issues, in some embodiments, the Lustre file system write processing method further includes: Step C1: Calculate the hash value of each memory block in the preset cache area, construct a hash index table and store it in the kernel-mode memory of the Lustre client. The hash index table records the file offset range and hash value corresponding to each memory block.

[0051] The Lustre client calculates the hash value of the data stored in each memory block in the preset cache CCT, and constructs a hash index table using the file offset range as the key and the memory block address and hash value as the value. This index table is stored in the client kernel-mode memory, allowing for fast access. Whenever data is written to a memory block, the corresponding file offset range and hash value are immediately inserted into the index table. Once the data in the memory block has been flushed and persisted, the corresponding entry is deleted from the index table.

[0052] Step C2: In response to a read data request initiated by the application layer, the hash index table is used to query whether there is corresponding valid data in the preset cache area according to the file offset range corresponding to the read data request.

[0053] Step C3: If valid data exists, copy the valid data from the preset cache to the application layer and return a read success message.

[0054] Step C4: If no valid data exists, read the corresponding data from the page cache or object storage server and return it.

[0055] In this embodiment of the disclosure, when the application layer initiates a read data request, the Lustre client first checks the hash index table for valid data covering the file offset range carried in the read request (i.e., checks if the file offset range of any memory block overlaps with the read request range). If valid data is found, the data is directly copied from the corresponding memory block to the application layer buffer and a read success is returned; the entire process does not involve a page cache or object storage server. If no valid data is found, the process falls back to the native Lustre read flow, reading data from the page cache or object storage server and returning it.

[0056] Through the hash index table described above, this embodiment ensures that while the data to be written is copied to a preset cache for temporary storage, but has not yet been flushed to the page cache or object storage server, subsequent read requests can retrieve the latest written data, satisfying POSIX's read-after-write semantics. Simultaneously, since reading data involves only one memory copy, read latency is significantly reduced, improving performance in mixed read-write scenarios.

[0057] In some embodiments, after the background thread writes the data to be written in batches to the page cache, if a system crash occurs during the operating system's writeback persistence process, some data may have already been written to the object storage server, while some data remains in the page cache (i.e., a half-written state), resulting in file corruption. To ensure the atomicity of the data to be written, this embodiment employs a two-phase commit protocol in the batch write-down step, specifically including a first processing step, a second processing step, and a third processing step.

[0058] Optionally, the step of writing the data to be written in batches to the page cache after holding the lock includes: The first processing step S1 is used to write the data to be written in batches to the page cache, and at the same time send a transaction preparation request carrying the Merkle root hash to the object storage server. The object storage server records the Merkle root hash and returns a successful preparation.

[0059] The first processing step S1 can be understood as the preparation phase. During this phase, after the background thread writes the data to be written in batches to the page cache, it sends a transaction preparation request carrying a Merkle root hash to the object storage server. Upon receiving the request, the object storage server records the Merkle root hash of the transaction in its local (i.e., OST-side) persistent storage and marks the transaction's status as "Preparing." This "Preparing" status indicates that the object storage server has received the preparation request and promised to accept subsequent commit or rollback instructions, but has not yet actually persisted the data to the final storage location. After recording, the object storage server returns a successful preparation response to the client. Thereafter, the transaction remains in the "Preparing" state on the object storage server side until it receives a commit instruction (at which point the status is updated to "Committed") or a rollback instruction (at which point the status is updated to "Rollback"), or is automatically cleaned up due to timeout.

[0060] The second processing step S2 is used to send transaction completion information carrying the Merkle root hash to the metadata service after the data to be written in the page cache is persisted to the object storage server through the operating system's write-back mechanism. In response to the transaction completion information, the metadata service releases the distributed write lock and the temporary pre-lease of the file offset range, and updates the global state recorded in the metadata extended attributes of the target file from uncommitted to committed.

[0061] In the second processing step S2, also known as the commit phase, since the object storage server has already recorded the Merkle root hash of the transaction during the preparation phase, and the data has been actually written through the write-back mechanism, even if the client crashes later, the "committed" status in the metadata service can still mark that the corresponding data to be written has been completely persisted.

[0062] The third processing step S3 is used to delete the data to be written in the preset cache area and return a write failure to the application layer when the batch writing of the data to be written to the page cache fails or the response of preparation success is not received.

[0063] During the preparation phase, if bulk writing to the page cache fails (e.g., page cache allocation fails), or if the client does not receive a successful preparation response from the object storage server (e.g., network timeout or object storage server rejection), the preparation phase is considered a failure. In this case, the client immediately deletes the data to be written from the preset cache (releasing the corresponding memory block) and returns a write failure to the application layer. The transaction state on the metadata service side remains "uncommitted," and records on the object storage server side in the "preparing" state are automatically cleaned up due to timeout. Since the object storage server has not yet performed actual data writing, no half-written data is generated, and the file retains its original complete state.

[0064] Through the above three processing steps, the metadata service will mark the transaction as committed only after the preparation phase is successful and the data is persisted through writeback. If any step fails, no data will be written to the object storage server, thus avoiding partial file writing and corruption due to system crashes and ensuring the atomicity of data writing.

[0065] In some embodiments, when a kernel crash or host power failure occurs during the batch writing process of the client, the transaction state and data cache information in the local memory will be completely lost. After the client restarts, it will be unable to distinguish which data has been persisted and which data has not yet been committed, which can easily lead to duplicate writing, data loss or file corruption. It will also cause memory leaks in the preset cache area and invalid entries remaining in the hash index table.

[0066] In one possible implementation, the step of calculating the Merkle root hash of the data to be written further includes: Step E1: Create a transaction record containing the Merkle root hash, the transaction record marking the local state of the current transaction as uncommitted, and write the transaction record to the transaction log area in the local memory of the Lustre client.

[0067] In this embodiment of the disclosure, after calculating the Merkle root hash of the data to be written, the Lustre client creates a transaction record containing that Merkle root hash. This transaction record also includes information such as a transaction ID (monotonically incremented by the client), creation time, and file offset range, and marks the current transaction's local state as uncommitted. Then, this transaction record is appended to the transaction log area in the Lustre client's local memory. This step enables the client to scan the transaction log area to obtain information on incomplete transactions after a crash and restart, providing a local anchor point for subsequent recovery.

[0068] Step E2: After the data in the page cache is persistently stored to the object storage server through the operating system's write-back mechanism, the data to be written in the preset cache area is deleted, and the hash value corresponding to the data to be written is removed from the hash index table.

[0069] Once the data to be written is persisted to storage, the Lustre client performs resource cleanup: it deletes the data to be written corresponding to the transaction from the preset cache to release the corresponding memory block, and removes the hash value corresponding to the transaction from the hash index table. Furthermore, if a transaction commit fails (e.g., triggered by the third processing step S3), local resources also need to be cleaned up immediately, and the corresponding entry removed from the hash index table to prevent subsequent read requests from hitting invalid data that has already been flushed or rolled back.

[0070] Through the above steps, the transaction log area provides Lustre clients with local data for crash recovery, the linkage between resource cleanup and transaction commit / rollback ensures that the preset cache area is not occupied for a long time, and the timely maintenance of the hash index table guarantees read consistency.

[0071] In one possible implementation, when the Lustre client crashes abnormally (e.g., process crash or kernel error) and restarts, there may be incomplete transactions remaining in the preset cache area CCT. The global true state of each transaction cannot be determined by the local transaction log area alone. This disclosure embodiment achieves transaction state identification by comparing the two-end Merkle root hashes and combining the global state in the metadata service anchor area.

[0072] Optionally, the method further includes: Step F1: After an abnormal restart, the Lustre client scans the transaction records in the transaction log area, obtains the Merkle root hash of the incomplete transaction as the local Merkle root hash and the local status based on the transaction records, and the local record status includes uncommitted and committed. Each transaction record corresponds to a data write request or write transaction, and contains the Merkle root hash and local state (uncommitted or committed) of that transaction. These records are written during the normal write process via step E1. Since the transaction log area only retains transactions that have not yet been cleaned up (i.e., incomplete or completed but not cleaned up transactions), the records obtained from the scan are incomplete transactions.

[0073] Step F2: Query the metadata extended attributes of the corresponding file from the metadata service to obtain the global status of each transaction of the file and the global Merkle root hash corresponding to each transaction. The global status includes uncommitted and committed. Here, the corresponding file refers to the file targeted by the incomplete transaction.

[0074] Step F3: For each incomplete transaction, compare the local Merkle root hash with the global Merkle root hash, and compare the local state with the global state; Step F4 involves performing operations based on the comparison results: cleaning up completed transactions, reapplying for a pre-lease to continue the complete transaction, or rolling back damaged transactions.

[0075] For example, if the comparison result shows that the global status is committed, it is determined that the transaction has been persisted and stored. The operation of cleaning up the completed transaction is then performed, such as deleting the transaction record of the transaction in the transaction log area and releasing the memory block occupied by the transaction in the preset cache area. If the comparison result is that the global state is not committed, the local state is not committed, and the local Merkle root hash is consistent with the global Merkle root hash, the step of repeatedly writing data is executed, for example, by initiating a lock pre-lease application for the transaction to the metadata service. If the comparison result shows that the global status is uncommitted, the local status is uncommitted, and the local Merkle root hash is inconsistent with the global Merkle root hash, it indicates that the data in the local preset cache has been corrupted (e.g., memory bit errors, or accidentally overwritten by other clients) and cannot be used for recovery or resumption. The transaction needs to be rolled back, and operations such as deleting the transaction record in the transaction log area, releasing the memory block occupied by the transaction in the preset cache area, and discarding the data to be written corresponding to the transaction need to be performed.

[0076] In this embodiment of the disclosure, through the above-described Merkle root hash double-end comparison recovery process, the client can automatically determine the true state of each incomplete transaction after a crash. Committed transactions are only cleaned up with local resources, complete data transmission is completed, and corrupted data is rolled back, thereby ensuring data consistency and zero loss.

[0077] In one possible implementation, the metadata service maintains the pre-lease status, file offset range, and Merkle root hash of each Lustre client for the same file through the file's metadata extended attributes, forming a global file view. When the metadata service receives a lock pre-lease request, it uses the global file view to verify the lock pre-lease request.

[0078] Specifically, the metadata service maintains a pre-lease information table in the extended attribute area of ​​each file with accelerated write enabled. This table records all currently granted temporary pre-leases. Each record includes: the client identifier holding the pre-lease, the locked file offset range (starting offset and length), and the Merkle root hash corresponding to the pre-lease. Whenever the metadata service successfully grants a temporary pre-lease, it inserts the record into the table; when the pre-lease is released or the corresponding transaction is committed, the record is deleted from the table. Thus, the metadata service can obtain a global view of all active pre-leases on that file in real time. When the metadata service receives a new lock pre-lease request, it first parses the file offset range in the request, and then queries the global file view to see if this range overlaps with existing pre-lease records. The query can use a range tree or a list sorted by starting offset for quick detection. If any overlap is detected, a conflict is determined, and the verification fails; otherwise, there is no conflict, and the verification passes.

[0079] In this embodiment, through a global file view, the metadata service can accurately detect file offset range conflicts between different client pre-leases and advance the conflict detection time from the formal lock application stage to the pre-lease application stage, so that conflicts can be detected before data is written to the cache, avoiding silent data overwriting; at the same time, by merging multiple formal lock applications into a small number of pre-lease applications, the number of lock request processing times of the metadata service is reduced, and the total overhead of conflict detection is reduced.

[0080] In one possible implementation, the Lustre file system write processing method further includes: Monitor the global free memory ratio of the computing node where the Lustre client is located. When the global free memory ratio is lower than the preset low watermark, increase the data flushing rate of the background thread and pause copying newly written data to the preset cache area until the global free memory ratio recovers to the preset high watermark.

[0081] The Lustre client monitors the global free memory ratio of its compute nodes in real time, calculated as the node's current free physical memory divided by the node's total physical memory. When the global free memory ratio falls below a preset low watermark, indicating significant system memory pressure, the client performs the following actions: Increase the data flushing rate of background threads: for example, by increasing the number of concurrent flushing threads, shortening the flushing interval (from the default 50ms to 10ms), or increasing the scheduling priority of flushing threads in the kernel; Pause copying newly written data to the preset cache: for newly arriving write data requests, the native write data process is directly used, without using the preset cache for acceleration; Continuously monitor the global free memory ratio, and once it recovers above the high watermark, restore the normal accelerated write path (i.e., re-allow new writes to use the preset cache) and restore the flushing rate to the default value. Through watermark scheduling, the system automatically accelerates flushing and pauses new writes to the cache when system memory is tight, effectively preventing the risk of OOM (Out of Memory) due to excessive memory consumption by the preset cache, and automatically restores the accelerated path when memory is sufficient, ensuring system stability and high performance under high load.

[0082] In a specific example, such as Figure 4 As shown, a complete data writing process is executed according to the following flow: At time t1, the application initiates a write data request; At time t2, the Lustre client initiates a lock pre-lease request to the Metadata Service (MDT) via an LNet heartbeat message, carrying the file identifier, file offset range, and the calculated Merkle root hash.

[0083] At time t3, after the MDT verification is conflict-free, a temporary pre-lease is granted, and the file offset range, Merkle root hash, and global status of the transaction are recorded in the metadata extended attributes and set to uncommitted. At time t4, after obtaining a temporary pre-lease, the client copies the data to be written to a free memory block in the pre-defined cache (CCT) and immediately (at time t5) returns a write success message to the application layer. Simultaneously, a local transaction record is created and written to the transaction log area, marking the local state as uncommitted.

[0084] At time t6, a formal distributed write lock is requested from the metadata service MDT (granted directly based on the pre-lease), and the grant information is returned at time t7; At time t8, after acquiring the lock, the data to be written is copied to the inbound page cache in one go. At the same time, a transaction preparation request carrying the Merkle root hash is sent to the object storage server (OST). After the OST records the request, it returns a successful preparation at time t9. At time t10, after receiving a successful preparation notification, the data in the page cache is persisted to the OST via a write-back. At time t11, after the data persistence is completed, a transaction completion message is sent to the MDT. The MDT updates the global state to committed and releases the locks and pre-leases.

[0085] Based on the same inventive concept, a second aspect of this disclosure provides a Lustre file system write processing apparatus, comprising: The detection module is used to respond to a write data request for a target file initiated by the application layer and detect whether the conditions for accelerated writing are met. The write data request carries the data to be written and the corresponding file offset range. The accelerated data writing module is used to copy the data to be written to the preset cache area in the Lustre client kernel mode when the detection result is yes, and return a successful write to the application layer. It uses a background thread to apply for a distributed write lock corresponding to the file offset range of the data to be written from the metadata service, and after holding the lock, it writes the data to be written in batches to the page cache. The regular data writing module is used to write the data to be written to the page cache page by page in memory pages when the detection result is negative. The process of writing each page of data to the page cache includes: using a background thread to request a distributed write lock from the metadata service corresponding to the file offset range of the page data, and writing the page data to the page cache after holding the lock. The data in the page cache is persistently stored to the object storage server through the operating system's write-back mechanism.

[0086] It is understandable that the specific implementation principles of the above detection module, accelerated data writing module, and regular data writing module can be found in [reference needed]. Figures 1 to 4 The relevant descriptions of the illustrated embodiments will not be repeated here.

[0087] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0088] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0089] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this application should be determined by the appended claims.

[0090] The above-described order of steps for the method is for illustrative purposes only, and the steps of the method of this application are not limited to the order specifically described above, unless otherwise specifically stated. Furthermore, in some embodiments, this application may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the method according to this application. Therefore, this application also covers recording media storing programs for performing the method according to this application.

[0091] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A write processing method for a Lustre file system, characterized in that, Includes the following steps: In response to a write data request for a target file initiated by the application layer, it is checked whether the conditions for accelerated writing are met. The write data request carries the data to be written and the corresponding file offset range. When the detection result is yes, the data to be written is copied to the preset cache area in the Lustre client kernel mode, and a write success is returned to the application layer. The background thread requests a distributed write lock from the metadata service corresponding to the file offset range of the data to be written. After holding the lock, the data to be written is written to the page cache in batches. If the detection result is negative, the data to be written is written to the page cache page by page in memory pages. The process of writing each page of data to the page cache includes: using a background thread to request a distributed write lock from the metadata service corresponding to the file offset range of the page data, and writing the page data to the page cache after holding the lock. The data in the page cache is persistently stored to the object storage server through the operating system's write-back mechanism.

2. The Lustre file system write processing method according to claim 1, characterized in that, Before the step of copying the data to be written to the preset cache area in the Lustre client kernel mode, the following steps are also included: The Lustre client uses its native LNet heartbeat mechanism to initiate a lock pre-lease request for the file offset range of the data to be written to the metadata service. The metadata service verifies the lock pre-lease request to determine whether the file offset range has been held by an exclusive lock by another client. If no exclusive lock is held, the service grants a temporary pre-lease to the Lustre client, marks the file offset range as pre-locked, and returns a temporary pre-lease grant response to the Lustre client via a heartbeat response message. Upon receiving a temporary pre-lease grant response, the step of copying the data to be written to a preset cache in the Lustre client kernel mode is executed.

3. The Lustre file system write processing method according to claim 2, characterized in that, The steps of initiating a lock pre-lease request to the metadata service for the file offset range of the data to be written include: Using the memory blocks in the preset cache where the data to be written is distributed as leaf nodes, calculate the Merkle root hash of the data to be written; A lock pre-lease request containing a file identifier, a Merkle root hash, and the file offset range corresponding to the data to be written is sent to the metadata service. After the metadata service verifies the lock pre-lease request, it writes the file identifier, the file offset range corresponding to the data to be written, the Merkle root hash, and the pre-lease validity period into the metadata extended attributes of the target file, and initializes the corresponding global state to uncommitted. The metadata extended attributes are stored in the index node maintained by the metadata service for each file.

4. The Lustre file system write processing method according to claim 3, characterized in that, The Lustre file system write processing method also includes: Calculate the hash value of each memory block in the preset cache area, construct a hash index table and store it in the kernel space memory of the Lustre client. The hash index table records the file offset range and hash value corresponding to each memory block. In response to a read data request initiated by the application layer, the hash index table is used to query whether there is corresponding valid data in the preset cache area according to the file offset range corresponding to the read data request; If valid data exists, copy the valid data from the preset cache to the application layer and return a read success message; If no valid data is found, the corresponding data is read from the page cache or object storage server and returned.

5. The Lustre file system write processing method according to claim 4, characterized in that, The steps of writing the data to be written in batches to the page cache after holding the lock include: The first processing step is used to write the data to be written in batches to the page cache, and at the same time send a transaction preparation request carrying the Merkle root hash to the object storage server. The object storage server records the Merkle root hash and returns a successful preparation. The second processing step is used to send transaction completion information carrying the Merkle root hash to the metadata service after the data to be written in the page cache is persisted to the object storage server through the operating system's write-back mechanism. In response to the transaction completion information, the metadata service releases the distributed write lock and the temporary pre-lease of the file offset range, and updates the global state recorded in the metadata extended attribute of the target file from uncommitted to committed. The third processing step is used to delete the data to be written in the preset cache area and return a write failure to the application layer when the batch writing of the data to be written to the page cache fails or no response of preparation success is received.

6. The Lustre file system write processing method according to claim 5, characterized in that, The step of calculating the Merkle root hash of the data to be written also includes: Create a transaction record containing the Merkle root hash, the transaction record marking the local state of the current transaction as uncommitted, and write the transaction record to the transaction log area in the local memory of the Lustre client; After the data in the page cache is persistently stored to the object storage server through the operating system's write-back mechanism, the data to be written in the preset cache area is deleted, and the hash value corresponding to the data to be written is removed from the hash index table.

7. The Lustre file system write processing method according to claim 6, characterized in that, The Lustre file system write processing method includes: After an abnormal restart, the Lustre client scans the transaction records in the transaction log area, obtains the Merkle root hash of the incomplete transaction as the local Merkle root hash, and the local state based on the transaction records. The local state includes uncommitted and committed. Query the metadata extended attributes of the corresponding file from the metadata service to obtain the global state of each transaction of the file and the global Merkle root hash corresponding to each transaction. The global state includes uncommitted and committed. For each incomplete transaction, the local Merkle root hash is compared with the global Merkle root hash, and the local state is compared with the global state; Based on the comparison results, perform operations such as cleaning up completed transactions, reapplying for pre-leases to continue complete transactions, or rolling back damaged transactions.

8. The Lustre file system write processing method according to claim 3, characterized in that, The metadata service maintains a global file view of each Lustre client for the same file by using the file's metadata extended attributes to form a pre-lease status, file offset range, and Merkle root hash. When the metadata service receives a lock pre-lease request, it uses the global file view to verify the lock pre-lease request.

9. The Lustre file system write processing method according to claim 1, characterized in that, The Lustre file system write processing method also includes: Monitor the global free memory ratio of the computing node where the Lustre client is located. When the global free memory ratio is lower than the preset low watermark, increase the data flushing rate of the background thread and pause copying newly written data to the preset cache area until the global free memory ratio recovers to the preset high watermark.

10. A Lustre file system write processing device, characterized in that, include: The detection module is used to respond to a write data request for a target file initiated by the application layer and detect whether the conditions for accelerated writing are met. The write data request carries the data to be written and the corresponding file offset range. The accelerated data writing module is used to copy the data to be written to the preset cache area in the Lustre client kernel mode when the detection result is yes, and return a successful write to the application layer. It uses a background thread to apply for a distributed write lock corresponding to the file offset range of the data to be written from the metadata service, and after holding the lock, it writes the data to be written in batches to the page cache. The regular data writing module is used to write the data to be written to the page cache page by page in memory pages when the detection result is negative. The process of writing each page of data to the page cache includes: using a background thread to request a distributed write lock from the metadata service corresponding to the file offset range of the page data, and writing the page data to the page cache after holding the lock. The data in the page cache is persistently stored to the object storage server through the operating system's write-back mechanism.

Images