A method, apparatus, device, and medium for modifying metadata.
By recording and writing log entries in memory, the metadata consistency problem caused by file system crashes is resolved, ensuring the consistency of metadata modifications on physical storage space.
Patent Information
- Application Number
- CN202611611101.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-19
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, file system crashes cause inconsistent metadata modifications on physical storage space.
Metadata modification operations are recorded in memory to generate log entries, which are then written to the log area of the physical storage space. When the file system crashes, the metadata on the physical storage space is modified again based on the log entries in the log area to ensure the consistency of the modifications.
Even when the file system crashes, metadata modifications on the physical storage space remain consistent, preventing the loss of some modifications.
Smart Images

Figure CN123114867A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of disk data modification technology, specifically to a method, apparatus, device, and medium for modifying metadata. Background Technology
[0002] In a file system, metadata modifications must maintain consistency. This means that changes to metadata must either take effect entirely on physical storage (such as a disk) or not at all. In other words, changes to metadata requiring consistency must be consistent across all locations on the physical storage space. Current technology modifies metadata first in the file system's memory and then synchronizes the modified metadata to the physical storage space. However, during this synchronization process, if the file system experiences a power outage or crash, the memory is cleared. The remaining modifications to metadata not yet synchronized to physical storage cannot be synchronized, resulting in only partial metadata changes being stored on the physical storage space. This leads to a failure to maintain consistency across all locations on the physical storage space.
[0003] In summary, existing technologies can lead to inconsistent metadata modifications on physical storage space due to file system crashes.
[0004] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention
[0005] To address the aforementioned technical problems, this invention provides a method, apparatus, device, and medium for modifying metadata, which solves the problem in the prior art where metadata modification loses consistency in physical storage space due to file system crashes.
[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for modifying metadata, comprising: Obtain all modifications to the original metadata to be performed, and generate log entries in memory with the modification information corresponding to each modification operation; Once all modifications to the original metadata have been completed, the log entries corresponding to each modification are written to the log area on the physical storage space. When the file system where the physical storage space and the memory reside crashes, the original metadata of the data area is modified based on the log entries in the log area. The data area is the region on the physical storage space used to store metadata.
[0007] In one implementation, the process includes obtaining all modification operations to be performed on the original metadata and generating the log entries in memory with the modification information corresponding to each modification operation, further comprising: Estimate the log capacity and data area capacity required for the data generated by the modification operation of the original metadata; Based on the estimated log capacity and data area capacity, log space is reserved in the log area and physical blocks are reserved in the data area, respectively. The physical blocks are storage units of the physical storage space.
[0008] In one implementation, the log item includes a base class field and a content field. The base class field includes the log item's mount node in the transaction linked list, the modification type of the original metadata, the modification status flag of the original metadata, and the log sequence number corresponding to the most recent modification of the original metadata. The transaction linked list is used to record the modification order of the original metadata. The mount node of the log item in the transaction linked list is used to represent the modification sequence number of the modification corresponding to the log item among various modifications. The modification status flag is used to represent the timeliness of the modification of the original metadata. The content field includes a buffer block of the original metadata in memory and an intra-block bitmap of the buffer block, wherein the intra-block bitmap is used to record the modifications made to the original metadata.
[0009] In one implementation, writing the log entries corresponding to the modifications at various locations to the log area on physical storage space includes: Move the log entries to the list of ready items in memory; Get the write mode identifier of whether the transaction is synchronously written to the log area on the physical storage space. The transaction is a modification transaction of each original metadata. Synchronization means the synchronization between the completion of all modifications of the original metadata and the writing of all log entries corresponding to each original metadata to the log area. Based on the write mode identifier, the log entries corresponding to each transaction in the ready item list are written to the log area on the physical storage space.
[0010] In one implementation, based on the write mode identifier, the log entries corresponding to each transaction in the ready item list are written to the log area in the physical storage space, including: When the write mode is set to synchronous, the log entry corresponding to each transaction in the ready item list is written separately to the log area on the physical storage space. When the write mode is set to asynchronous, the number of transactions in the ready item list is counted until the number of transactions reaches a threshold. Then, the log entries corresponding to all transactions in the ready item list are written to the log area in the physical storage space.
[0011] In one implementation, the log entries corresponding to each transaction on the ready list are written to the log area in the physical storage space, and then the process further includes: Create a new blank ready item list, which is used to store the log items corresponding to subsequent transaction items.
[0012] In one implementation, modifying the original metadata of the data area based on the log entries in the log area includes: Retrieve log records from the log area that were being processed at the time of the last system crash, where each log record includes all log entries generated by the same metadata modification; The log records in the log area are read back into the memory, and the original metadata is written into the buffer block in the memory; On the memory, based on the log records, the original metadata of the buffer block is modified to obtain the modified metadata; The modified metadata is written from the buffer block to the data area to complete the modification of the original metadata of the data area.
[0013] In one implementation, the metadata is a free space management structure, which is used to record free physical blocks on the physical storage space. The free space management structure includes a BNO tree and a CNT tree.
[0014] Secondly, embodiments of the present invention also provide a metadata modification apparatus, wherein the apparatus comprises the following components: The log item generation module is used to obtain the various modification operations to be performed on the original metadata, and generate the log items in memory with the modification information corresponding to each modification operation. The writing module is used to write the log entries corresponding to each modification to the original metadata to the log area on the physical storage space when all modification operations have been completed. The modification module is used to modify the original metadata of the data area based on the log entries in the log area, wherein the data area is a region on the physical storage space used to store metadata.
[0015] Thirdly, embodiments of the present invention also provide a terminal device, wherein the terminal device includes a memory, a processor, and a metadata modification program stored in the memory and executable on the processor, wherein when the processor executes the metadata modification program, it implements the steps of the metadata modification method described above.
[0016] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a metadata modification program, wherein when the metadata modification program is executed by a processor, it implements the steps of the metadata modification method described above.
[0017] Beneficial Effects: This invention modifies the original metadata in memory and records these modifications using memory log entries. Once all modifications are complete, the log entries are written to the log area of the physical storage space. Even if the file security system containing the memory and physical storage crashes, causing memory to be zeroed out, the log area of the physical storage space will still retain the original metadata modification information. Therefore, the log entries (used to record the information to be modified in various parts of the metadata) can be written back to memory, as can the original metadata in the data area. Memory then modifies the metadata again based on the log entries until all modifications are complete. Finally, the modified metadata is written to the data area of the physical storage space. From the above analysis, this invention stores the information or content to be modified in the metadata in the log area of the physical storage space. When the file security system crashes, the memory can modify the metadata again based on the log entries in the log area until all modifications are complete before writing the modified metadata to the physical storage space. This ensures that the modifications to the metadata in the physical storage space are synchronized, preventing a situation where one part of the metadata is modified while others remain unchanged, ultimately ensuring consistency in the modifications to the metadata in the physical storage space. Attached Figure Description
[0018] Figure 1 This is an overall flowchart of the present invention; Figure 2 A structural diagram of the metadata modification device provided by the present invention; Figure 3 This is a block diagram illustrating the internal structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation
[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0020] Research has revealed that certain metadata modifications in a file system must maintain consistency. This means that metadata modifications must either take full effect on physical storage (e.g., disk) or not at all. In other words, modifications to metadata requiring consistency must be consistent across all locations on the physical storage space. Current technology modifies metadata first in the file system's memory and then synchronizes the modified metadata to physical storage. However, during this synchronization process, if the file system experiences a power outage or crash, the memory is cleared. The remaining modifications to metadata not yet synchronized to physical storage cannot be synchronized, resulting in only partial metadata changes being stored on physical storage. Consequently, metadata modifications on physical storage fail to maintain consistency across all locations.
[0021] To address the aforementioned technical problems, this invention provides a method, apparatus, device, and medium for modifying metadata, resolving the issue of inconsistent metadata modifications in physical storage space caused by file system crashes in existing technologies. Specifically, each modification operation to be performed on the original metadata is generated as a log entry in memory; once all modification operations on the original metadata have been completed, the log entries corresponding to each modification are written to a log area in the physical storage space; subsequently, based on the log entries in the log area, the original metadata in the data area is modified.
[0022] For example, the original metadata consists of two sub-data items, which are stored in a data area on physical storage space (such as a hardware disk). When these two sub-data items are modified in the data area of the physical storage space, they must maintain consistency; that is, both sub-data items must either be modified or not modified at all. To ensure that modifications to these two sub-data items maintain consistency in the data area, this invention adopts the following technical solution: First, the modification information for these two sub-data items is recorded in the memory log. These modifications represent two modifications to the original metadata. Once all the modification information is recorded in the memory log, the corresponding log entries in memory are written to the log area of the physical storage space. Since the original metadata cannot be directly modified in the physical storage space, it is written to memory. Memory then modifies this original metadata based on the log entries. During this modification process, if the file system residing in the memory and physical storage space crashes due to power failure or other reasons, the log entries in memory, some modified data, and the original metadata will be cleared. Because the log area of the physical storage space stores the log entries, these entries and the original metadata can be written to memory again, allowing memory to re-modify the two sub-data items contained in the original metadata based on the log entries. After both sub-data items have been modified, memory writes the modified sub-data items to the data area of the physical storage space to overwrite the original two sub-data items. Because this invention modifies both sub-data items in memory before writing them to the physical storage space, the two sub-data items in the physical storage space always maintain consistency. If the file system crashes during the modification process, since the incomplete modifications are all in memory, the file system crash will clear the memory. Therefore, the incomplete or inconsistent modified data will not be written to the data area, thus ensuring that the modifications of the two sub-data in the data area are always consistent.
[0023] Example 1: This example provides a metadata modification method. This metadata modification method can be applied to a terminal device, which can be a terminal product with data processing capabilities, such as a computer. In this example, as... Figure 1 As shown, the metadata modification method specifically includes the following steps: S110, obtain the various modification operations to be performed on the original metadata, and generate the modification information corresponding to each modification operation into the log entries in memory.
[0024] S120, when all modification operations of the original metadata have been completed, the log entries corresponding to each modification are written to the log area on the physical storage space.
[0025] S130, when the file system where the physical storage space and the memory are located crashes, the original metadata of the data area is modified based on the log entries in the log area, wherein the data area is the area on the physical storage space used to store metadata.
[0026] The purpose of step S110 is to record the modification information of the original metadata in the form of log entries. Each log entry consists of a base class field and a content field. The base class field includes the log entry's mount node in the transaction list, the modification type of the original metadata, the modification status flag of the original metadata, and the log sequence number corresponding to the most recent modification of the original metadata. Each modification of the metadata is recorded in one log entry. The in-memory transaction list is used to record the order of modifications to the original metadata. Therefore, the mount node of a log entry in the transaction list represents the order of modifications corresponding to that log entry, that is, the modification order of that log entry among all modifications to the original metadata.
[0027] Modification types include the inode (Inode) of the file containing the original metadata (Inode represents file identifier). The file's inode corresponds to the allocation group (AG) in the physical storage space used to store the file's internal data; that is, the inode is used to match the allocation group where the original metadata resides. Modification types also include the identifier of the buffer block, which identifies which buffer block in memory modified the original metadata.
[0028] The modification status flag is used to characterize the timeliness of the modification of the original metadata on the memory side. It is only used for the memory side to evict or merge log entries. It will not abandon the replay of log entries already written to disk due to the timeout of this flag. The modification status flag does not interfere with the replay logic of WAL (Write-Ahead Logging). WAL guarantees the persistence order of writing logs first and then updating data areas.
[0029] Since the metadata records the content to be modified in a log entry before each modification, the log sequence number corresponding to the most recent modification of the original metadata represents the content of the last modification of the original metadata.
[0030] The content field includes a buffer block of the original metadata in memory and an intra-block bitmap of the buffer block, wherein the intra-block bitmap is used to record the modifications made to the original metadata.
[0031] In this embodiment, each modification to metadata is not directly written back to the final location (i.e., the data area in physical storage space), but is first registered in memory as a "log entry". The base class fields of the log entry record the following information: the mount node of the log entry in the transaction chain, the mount node of the log entry in the committed item list (the committed item list is the ready item list; log entries in the ready item list can be written to the log area in physical storage space, while log entries not in the ready item list cannot be written to the log area), the type of the log entry (such as modified inode, modified buffer block, quota update, etc.), the status flag, and the log sequence number of the most recently modified log entry. This embodiment also uses an operation function table to define how the log entry is serialized into a log vector, how the corresponding object is locked before the log is written to disk, and how it is unlocked after writing to disk. Log entries modified by inodes and log entries modified by buffer blocks are specified by different subtypes.
[0032] When metadata modification involves a disk buffer block, the system first checks if the buffer block is already associated with a corresponding buffer block (located in memory) log entry. If not, a log entry is allocated and bound to the buffer block. The scope of the modification is recorded in the internal bitmap of the entry using a bitmap format based on "intra-block offset and length" (only the actually modified part is recorded to avoid recording the entire block and save log space). The entry is then added to the current transaction's log chain table, and the buffer block is marked as dirty (dirty means the latest data in memory has been updated, but the content of the buffer block in physical storage has not). The log entry modified by the innode records the target innode block number, the range of fields to be updated, and whether the mandatory access control label has changed. This minimal recording method, "by object, by range," is the core trade-off between achieving high throughput and low log write amplification. If the log entry structure itself fails to allocate on an abnormal path, it triggers a file system abort: the log transaction engine refuses to allocate new transactions (i.e., metadata modification transactions).
[0033] When all the modification operations of the original metadata in step S120 have been completed, that is, when all the places in the original metadata that need to be consistently modified have been registered as a log item, it means that all the modification operations of the original metadata have been completed.
[0034] For example, the original metadata includes CNT trees and BNO trees. CNT trees and BNO trees record free areas on physical storage space in different forms. Therefore, modifications to CNT trees and BNO trees must be consistent in order to obtain the same search results for free areas through CNT trees and BNO trees. CNT trees are Count Btrees, which are used to record the free areas where free physical blocks are located according to the length of consecutive free physical blocks. BNO trees are Block Number Btrees, which are used to record free areas according to the size of the starting physical block number of the free area.
[0035] When a modification to the CNT tree is recorded as a log entry, and a modification to the BNO tree is also recorded as a log entry, it means that all modification operations on the original metadata have been completed.
[0036] Step S120, which involves writing the log entries corresponding to the modifications to the log area on the physical storage space, includes the following specific steps S121, S122, and S123: S121, the log entry is transferred to the ready list in memory for submission.
[0037] In other words, after each transaction has completed all modifications, all log entries corresponding to the modifications made by that transaction are transferred from the current transaction list to the ready list. This transfer operation is also called enqueuing the commit list. Log entries transferred to the ready list represent log entries that can be written to the physical storage space log area, while log entries that are not transferred to the ready list cannot be written to the physical storage space log area.
[0038] Before the transfer, this embodiment calls the locking function in the log entry operation function table to lock the corresponding memory object, preventing the virtual file system or page reclamation mechanism from prematurely flushing dirty pages into the data area during this period. After the transfer, the unused ticket space reserved by the transaction itself is released. The entire committed item list is protected by a global spin lock, enabling multiple transactions to complete their respective object modifications without locks and concurrently push log entries into the global list.
[0039] S122, obtain the write mode identifier of whether the transaction is synchronously written to the log area on the physical storage space, wherein the transaction is a modification transaction of each original metadata, and synchronization represents the synchronization of writing all the log entries corresponding to each original metadata to the log area when all modifications of the original metadata are completed.
[0040] A transaction represents all modifications to a piece of original metadata. Synchronization means that after all modifications to a piece of original metadata have been logged in memory, all log entries corresponding to that original metadata are synchronized to the log area of the physical storage space, without waiting for modifications to other original metadata in memory to be logged.
[0041] S123, based on the write mode identifier, write the log entries corresponding to each transaction in the ready item list to the log area on the physical storage space.
[0042] That is, when the write mode is identified as synchronous, the log entry corresponding to each transaction in the ready item list is written separately to the log area on the physical storage space.
[0043] When the write mode is set to synchronous, once all modifications to the original metadata have been registered as log entries and transferred to the ready list, the log entry corresponding to the modification transaction of the original metadata in the ready list is directly written to the log area on the physical storage space without waiting for modifications from other transactions.
[0044] When the write mode is set to asynchronous, the number of transactions in the ready item list is counted until the number of transactions reaches a threshold. Then, the log entries corresponding to all transactions in the ready item list are written to the log area in the physical storage space.
[0045] When the write mode is set to asynchronous, even if all modifications to one of the original metadata have been registered as log entries and transferred to the ready list, the log entries corresponding to the modification transactions of that original metadata in the ready list will not be immediately written to the log area of the physical storage space. Instead, it will wait for modifications by other transactions until the number of transactions in the ready list reaches a threshold before writing the log entries corresponding to all transactions in the ready list to the log area of the physical storage space.
[0046] The disk flushing strategy (flushing, i.e., writing to the log area of physical storage space) of the ready item list in this embodiment is on-demand batching. On-demand batching means that if the transaction to be committed is marked as synchronous (such as from a system call with flushing semantics), the commit interface immediately calls the ready item list push interface to synchronously flush to disk and waits for all modifications of the transaction to be completed; if synchronization is not required, it checks whether the accumulated data in the ready item list exceeds a preset threshold. If it does, it wakes up a background thread to asynchronously execute the push; otherwise, it returns directly. The commit interface does not force all transactions to be aggregated, nor does it aggregate indiscriminately. Instead, it performs selective aggregation based on two trigger conditions: "push if synchronous, otherwise push according to threshold." This takes into account both the latency of the synchronous path and the throughput of the asynchronous path. When there are few log items and asynchronous commits are made, multiple small transactions will naturally accumulate in the committed item list until they reach the threshold before being flushed to disk in a unified manner; when there are many log items or synchronous commits are made, they are flushed to disk immediately.
[0047] In this embodiment, while writing the log entries in the ready item list to the log area on the physical storage space, a new blank ready item list is created. The blank ready item list is used to store the log entries corresponding to subsequent transaction items.
[0048] This embodiment assigns a new global log sequence number to the blank ready item list, iterates through each log item in the old list (the old list being the ready item list where log items will be written to the log area), and calls the formatting function in the log item operation function table to serialize the log item into discrete memory vectors. Multiple vectors are then assembled sequentially into a complete log record (containing the log magic number, loop write count, valid data length, log sequence number, and cyclic redundancy check value for the entire record, where the log magic number characterizes the log's validity). The entire record is then constructed as an input / output request according to the verification and written sequentially to the disk (physical storage space) log area. Synchronous waiting for input / output completion is followed by calling the unlock function in the log item operation function table to unlock the corresponding memory objects, allowing the virtual file system to flush these objects to the final data area. This entire process ensures that logs are written to disk before data is written to the disk. In the synchronous path, if input / output fails, an input / output error is returned and the file system is aborted; in the asynchronous path, the background thread performs the same processing on failed input / output.
[0049] S130 modifies the original metadata of the data area based on the log entries in the log area, including the following specific steps S131, S132, S133, and S134: S131, retrieve the log records in the log area that the file system was processing at the time of the last system crash, where each log record includes all the log entries generated by the same metadata modification.
[0050] Before the file system crash, all modifications to the metadata corresponding to log records that had been processed were completed, so there was no need to repeat the processing of metadata that had already been fully modified. However, when the system crashed, the metadata being modified was still in memory, and since the system crash would clear memory, the memory needed to re-modify the metadata that had not been modified before the crash based on the log records in the log area.
[0051] S132, read the log records in the log area back to the memory, and write the original metadata into the buffer block in the memory.
[0052] Since a system crash only clears the memory, but not the physical storage space, the log records are still fully preserved in the physical storage space. Therefore, these log records can be read back into memory from the physical storage space's log area. Because the original metadata cannot be directly modified in the physical storage space, but only in memory, the unmodified original metadata must also be written back from the physical storage space's data area to the memory's buffer block.
[0053] S133, on the memory, based on the log record, modify the original metadata of the buffer block to obtain the modified metadata.
[0054] Because the log records complete information about all the modifications that are to be made to the original metadata, memory can modify the original metadata on its buffer block based on the reread log records to obtain the modified metadata.
[0055] S134, the modified metadata is written from the buffer block to the data area to complete the modification of the original metadata of the data area.
[0056] The modified metadata is written from the buffer block to the data area, overwriting the original metadata in the data area, to complete the modification of the metadata in the data area.
[0057] In another embodiment, the start and end log sequence numbers recorded during the last normal uninstallation of the file security system are obtained. Starting from the start log sequence number, each log entry is scanned sequentially, verifying its magic number and cyclic redundancy check value. Scanning stops when a check fails or the cycle count is inconsistent with the current context. This position is the true end of the log (the end being the last log entry processed before the system crash). Any corruption discovered at this stage must halt recovery, refuse to mount the file system, and prompt the user to use repair tools or scrap the system. This corruption includes physical media damage, underlying hardware violating disk flushing or forced write unit semantics, corrupted silent log data, incorrect input / output of target data blocks, and malicious tampering or out-of-bounds overwriting of the log space, all of which can lead to system crashes.
[0058] Starting from the initial log sequence number and scanning towards the end, each complete log record is read, and its multiple log entries are parsed. The contents of these log entries are applied to the final data area according to their type. Specifically, for buffer block entries, the physical block of the target physical storage space (a physical block is the physical unit storing metadata) is read into memory, overwritten according to the offset and content of the log entry, and marked as dirty. For inode entries, the target inode block is read into memory, the corresponding fields (such as size, modification time, and mandatory access control labels) are overwritten, and marked as dirty. After looping to the end, the virtual file system flushes the dirty pages to disk, writing them back to the final data area. If an incomplete transaction is found during the scan, and the transaction's characteristics include a truncated log record or a failed validation, the transaction and all subsequent records are discarded. Only transactions that have been fully committed and passed validation are replayed, thus ensuring that inconsistent states are not replayed into the data area.
[0059] The determination of a valid log sequence number uniformly adopts a combination of the following three criteria: the magic number of the record must match; the cyclic redundancy check value of the entire record must pass; and the cycle count field of the record must be consistent with the current recovery context. Only records that meet these three criteria and whose log sequence number falls between the starting sequence number of the last normal unload and the tail located by the recovery scan are considered valid. The log area as a whole is a circular buffer. When the tail catches up with the starting position, the data area must be flushed to disk first (checkpoint), pushing the starting sequence number towards the tail to release log space.
[0060] Example 2: This example provides a metadata modification method based on a reservation method, including the following steps S210, S220, S230, S240, and S250: S210, Estimate the log capacity and data area capacity required for the data generated by the modification operation of the original metadata; S220, based on the estimated log capacity and data area capacity, reserve log space in the log area and reserve physical blocks in the data area, respectively, wherein the physical blocks are storage units of the physical storage space.
[0061] S230, obtain the various modification operations to be performed on the original metadata, and generate the modification information corresponding to each modification operation into the log entries in memory.
[0062] S240, when all modification operations of the original metadata have been completed, the log entries corresponding to each modification are written to the reserved log area on the physical storage space. S250, based on the log entries in the log area, modify the original metadata of the data area, where the data area is a region on the physical storage space used to store metadata, and the data area is composed of the physical blocks reserved in step S220.
[0063] S210 estimates the log space and data area capacity required for the data generated by the modification operation of the original metadata. This means estimating the maximum number of bytes that might be needed based on the amount of data potentially involved in the metadata modification, and reserving a certain amount of log space and data area capacity based on that maximum number of bytes. Alternatively, a certain amount of log space and data area capacity can be reserved based on the maximum log space and data area capacity used in historical metadata modifications.
[0064] Creating a file on physical storage space is equivalent to modifying the metadata on that storage space. Taking file creation as an example, the detailed process of reserving log space and data area physical blocks is explained below: Creating a file may involve creating a directory block, allocating an innode block, modifying an AG header, and modifying a superblock; therefore, a corresponding number of tickets can be reserved for both log space and physical block quotas. Successful reservation requires both the remaining space in the global log area and the available global physical blocks to be satisfied; otherwise, the current thread enters sleep mode, waiting for background disk flushing to release space or for the upper-level user space to actively relinquish it. When reservation ultimately fails, a corresponding resource exhaustion error is returned. On abnormal paths, if the allocation of the transaction structure itself fails, an "out of memory" error is returned; if log space reservation times out, an I / O error is returned; if physical block reservation fails, no available space is returned.
[0065] The reserved log and data area capacity in this embodiment can prevent the inability to roll back from the fact that the log area or physical block quota is insufficient when the transaction is halfway through execution.
[0066] S230, S240, and S250 in this embodiment are the same as S110, S120, and S130 in Embodiment 1, respectively, and will not be described again here.
[0067] Example 3 provides a method for modifying metadata related to a free space management structure. The free space management structure includes a CNT tree and a BNO tree. Both the CNT tree and the BNO tree are used to record free physical block data within a space region on the physical storage space. Modifying the CNT tree must simultaneously modify the BNO tree, and vice versa; otherwise, inconsistencies in the free physical block data recorded by the CNT tree and the BNO tree will occur. The method for modifying the free space management structure in this example includes the following steps: S310, determine the remaining free physical blocks formed after the free physical blocks in the free area are occupied by data; obtain the newly released free physical blocks; S320, determine whether the starting block number of the free area recorded in the BNO tree, the block number of the new free physical block, and the block number of the remaining free physical blocks are consecutive, and obtain the continuity judgment result; S330, based on the continuity judgment result, obtain the operations to be modified for each of the BNO tree and CNT tree; S340, the modification information corresponding to the modification operations of the BNO tree and CNT tree is used to generate the log entries in memory.
[0068] The modifications to the BNO tree and the CNT tree constitute two changes to the metadata of the free space management structure.
[0069] S350, when the modification operations of the BNO tree and the CNT tree have been completed, the log entries corresponding to the modifications of the BNO tree and the CNT tree are written to the log area on the physical storage space. S360, based on the log entries in the log area, modify the BNO tree and CNT tree in the data area.
[0070] The remaining free physical blocks in S310 are the free physical blocks remaining after the free physical blocks in the free area of the physical storage space have been occupied. The new free physical blocks in S310 are the free physical blocks that have become free physical blocks after the previously occupied physical blocks have released the data on them.
[0071] Whether the number of free physical blocks decreases due to being occupied or increases due to the release of new free physical blocks, the distribution and number of free physical blocks in the allocation group are changed. Therefore, the BNO tree and CNT tree used to manage free physical blocks in the allocation group need to be updated so that the updated BNO tree and CNT tree can accurately record the information of free physical blocks within the allocation group.
[0072] The above allocation groups represent groups of physical blocks on physical storage space. Free areas consist of contiguous free physical blocks.
[0073] In S320 of this embodiment, the BNO tree records several free areas. Each free area has a corresponding start number of a free physical block. If the block number of the new free physical block and the block number of the remaining free physical blocks are consecutive with the start block numbers of the several free areas, the new free physical block and the remaining free physical blocks are merged into the free area. The BNO tree is then updated based on the merged free area, and the free areas of the updated BNO tree are mapped to the CNT tree.
[0074] Example 4, based on Example 3, provides a data management method based on a modified BNO tree and CNT tree, including the following steps: S410, determine the target allocation group corresponding to the data to be stored, the allocation group represents the grouping of physical blocks on the physical storage space; S420: The CPU core searches in parallel for the free space management structure of each target allocation group to obtain the target free physical block in the target allocation group that matches the data to be stored. S430, each of the central processing unit cores stores each group of data to be stored in parallel to its corresponding target free physical block.
[0075] In this embodiment, the data to be stored consists of several groups. A central processing unit core is responsible for allocating a target free physical block to a group of data to be stored and for storing the group of data to be stored into the target free physical block.
[0076] In this embodiment, the determination of the target allocation group corresponding to the data to be stored in step S410, where the allocation group represents the grouping of physical blocks on the physical storage space, includes the following specific steps S411, S412, and S413: S411, obtain the file index node to which the data to be stored needs to be written, and obtain the index node of the allocation group on the physical storage space.
[0077] Physical storage space is a hardware storage space composed of several physical blocks, which can be a disk. The file system divides the physical storage space into allocation groups of the same size, meaning that each allocation group contains the same number of physical blocks.
[0078] S412, based on the file index node and the index node of the allocation group, select a pre-selected allocation group from the allocation group.
[0079] A file is a folder at the software level that needs to be written, and each file has a corresponding inode (Inode represents a file identifier). In the file system, each inode is permanently fixed in one of the allocation groups AG (AG stands for Allocation Group) when it is created. This allocation group AG is the allocation group AG where the file's inode is located.
[0080] Since the header of the allocation group AG records its inode, matching the inode of the allocation group with the file inode allows us to match the allocation group configured by the file system for the file being written to. This allocation group AG is only fixedly configured for the file based on its inode, but its available capacity may be insufficient to accommodate the data to be stored. Therefore, the allocation group matched based on the inode is only a pre-selected allocation group. The subsequent decision on whether to use this pre-selected allocation group as the target allocation group depends on its available capacity; that is, whether to store the data to be stored in this pre-selected allocation group.
[0081] S413, when the available capacity of the free physical blocks contained in the preselected allocation group meets the storage capacity required by the data to be stored, the preselected allocation group is used as the target allocation group.
[0082] The header of the allocation group AG also stores a free space management structure, which records the number of free physical blocks within the allocation group AG. Knowing the number of free physical blocks allows us to determine the available capacity of the allocation group AG. When the available capacity of the pre-selected allocation group is greater than or equal to the storage capacity required by the data to be stored, the pre-selected allocation group is directly used as the target allocation group for the data to be stored.
[0083] When the available capacity of the free physical blocks contained in the preselected allocation group is insufficient to meet the storage capacity required for the data to be stored, the target allocation group is selected from the remaining allocation groups based on the available capacity of the remaining allocation groups. The remaining allocation groups are the allocation groups after removing the preselected allocation group from all the allocation groups.
[0084] That is, according to the allocation group number, query whether the available capacity of the next allocation group (that is, the remaining allocation group) of the pre-selected allocation group meets the storage capacity required by the data to be stored. If it does not meet the requirements, continue to query the next allocation group until the target allocation group with available capacity that meets the storage capacity required by the data to be stored is found.
[0085] In this embodiment, when the preselected allocation group is not the target allocation group, the data block pointer inside the inode of the preselected allocation group needs to be modified so that the preselected allocation group corresponding to the file can be located based on the inode, and then the target allocation group can be located based on the modified data block pointer of the preselected allocation group, thereby finding the data stored in the target allocation group.
[0086] For example, the inode of the file test.log (where test.log represents the filename) is fixed in allocation group AG1. Allocation group AG1 is the pre-selected allocation group for the file test.log. When more data than the available capacity of allocation group AG1 needs to be written to the file test.log, since the available capacity (i.e., remaining space) of allocation group AG1 is insufficient, allocation group AG2 is polled. It is found that the available capacity of allocation group AG2 can accommodate the data to be stored, so allocation group AG2 is selected as the target allocation group. The data block pointer list inside the inode of allocation group AG1 is updated with "Data from 50MB to 100MB is located in allocation group AG2", where 50MB to 100MB represents the space occupied by the data to be stored.
[0087] In this embodiment, step S420, which involves parallel searching of the free space management structure of each target allocation group by the central processing unit core to obtain the target free physical blocks within each target allocation group that are compatible with the data to be stored, includes the following specific steps S421, S422, and S423: S421, the CNT tree of each of the target allocation groups is searched in parallel by the central processing unit core.
[0088] The CNT tree and BNO tree constitute the free space management structure; that is, the free space management structure includes both the CNT tree and the BNO tree. Each allocation group (AG) has a CNT tree and a BNO tree at its head. The CNT tree is used to sort the free areas containing free physical blocks according to the length of consecutive free physical blocks, while the BNO tree is used to sort the free areas according to the size of the starting physical block number of the free area.
[0089] For example, the block numbers of the free physical blocks within the target allocation group include 150 to 155, 200 to 208, 250 to 257, 300 to 302, and 400 to 404. That is, there are five free areas within the target allocation group. The five free areas are E1 (150,6), E2 (200,9), E3 (250,8), E4 (300,3), and E5 (400,5). The BNO tree sorts the above five free areas by the starting free physical block number of the free area: E1 (150), E2 (200), E3 (250), E4 (300), and E5 (400). The CNT tree sorts the above five free areas by the length of the free area: E4 (3), E5 (5), E1 (6), E3 (8), and E2 (9).
[0090] S422, based on the length of the free area recorded in the CNT tree, find a target free area from the free area that matches the storage capacity required for the data to be stored. The free area is a physical space region composed of consecutive free physical blocks, and the length of the free area corresponds to the available capacity of the free area.
[0091] The storage capacity of each free physical block is fixed. Therefore, the available capacity of the free area can be determined based on the length of the free area (the length of the free area corresponds to the number of consecutive free physical blocks included in the free area). Thus, the target free area suitable for storing the data to be stored can be selected from the free areas based on the length of the free area.
[0092] Continuing the example above, if the data to be stored requires 4 free physical blocks, then based on the CNT tree, we can find that 5 is greater than 4. Therefore, E5 is chosen as the target free area for the data to be stored, meaning the data can be written to free area E5. Indexing the target free area according to its length ensures that the free area's length matches the data to be stored, rather than using a free area whose length is much greater than the required capacity. Using a free area that is much larger than the required capacity would lead to more fragmentation, resulting in discontinuous free physical blocks. Discontinuous free physical blocks are not suitable for storing large amounts of data later.
[0093] S423, Select target free physical blocks that match the data to be stored from the free physical blocks in the target free area.
[0094] In other words, a certain length of free physical block is extracted from the starting free physical block of the target free area as the target free physical block for the data to be stored, and the length of the extracted block is adapted to the storage amount required for the data to be stored.
[0095] In step S430 of this embodiment, each of the central processing unit (CPU) cores stores each group of data to be stored in parallel to its corresponding target free physical block. In other words, each CPU core simultaneously stores its own data to be stored onto the target free physical block allocated to its respective data. Since each CPU core has the function of writing data to physical blocks in parallel, it can improve the data writing speed.
[0096] After the data to be stored is stored in the target free physical block, the target free physical block is occupied. At this time, it is necessary to modify the CNT tree and simultaneously modify the BNO tree so that the modified CNT tree and BNO tree can correctly record the current free physical block.
[0097] This embodiment modifies and updates the CNT tree and BNO tree, including the following specific steps: determining the remaining free physical blocks formed after the target free physical blocks in the target free area are occupied by the data to be stored; forming a new free area based on the remaining free physical blocks; updating the BNO tree based on the starting block number of the new free area; and updating the CNT tree based on the length of the new free area.
[0098] When the data to be stored does not occupy all the free physical blocks in the target free area, there will be remaining free physical blocks in the target free area. Therefore, the number of free physical blocks in the target free area has changed. Thus, the BNO tree and CNT tree need to be modified so that the BNO tree and CNT tree can correctly record the free physical blocks in the current free area.
[0099] Continuing with the example above, the data to be stored is stored in the free area E5, occupying blocks 400 to 402, leaving only two free physical blocks 403 and 404 in E5. Let e5 represent the modified E5, that is, e5 is the new free area, and the new free area e5: blocks 403 to 404.
[0100] The original BNO tree is modified based on the starting block number of the new free area e5 (which is 403). The free areas recorded in the modified BNO tree are sorted as follows: E1 (150), E2 (200), E3 (250), E4 (300), e5 (403). The original CNT tree is updated based on the length of the new free area e5. The free areas recorded in the updated CNT tree are sorted as follows: e5 (2), E4 (3), E1 (6), E3 (8), E2 (9).
[0101] In this embodiment, after the target free physical block in the target free area of the target allocation group is occupied, the BNO tree and CNT tree inside the target allocation group are modified so that the BNO tree and CNT tree are updated in a timely manner to synchronize the information of the remaining free physical blocks in the target allocation group.
[0102] In summary, this invention solidifies metadata modifications into the log area in log-first order and replays them in batches in the background, enabling the secure file system to achieve near-memory write throughput while ensuring crash consistency. The batch flushing mechanism triggered on demand for the committed item list amortizes the overhead of multiple flushes for small transactions into a single flush for a large transaction; minimum log entries recorded by object and range reduce log write amplification; two-phase recovery and tear transaction processing ensure that inconsistencies will never occur during mounting; mandatory access control labels and encrypted metadata also benefit from this mechanism.
[0103] Example 5: This example also provides a metadata modification device, such as... Figure 2 As shown, the device comprises the following components: Log item generation module 01 is used to obtain various modification operations to be performed on the original metadata, and generate log items in memory by generating the modification information corresponding to each modification operation. The writing module 02 is used to write the log entries corresponding to each modification to the original metadata into the log area on the physical storage space when all modification operations have been completed. Modification module 03 is used to modify the original metadata of the data area based on the log entries in the log area, wherein the data area is the area on the physical storage space used to store metadata.
[0104] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 3As shown, the terminal device includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a metadata modification method. The display screen of the terminal device can be a liquid crystal display (LCD) or an electronic ink display.
[0105] Those skilled in the art will understand that Figure 3 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.
[0106] In one embodiment, a terminal device is provided, the terminal device including a memory, a processor, and a metadata modification program stored in the memory and executable on the processor. When the processor executes the metadata modification program, it implements the following operation instructions: Obtain all modifications to the original metadata to be performed, and generate log entries in memory with the modification information corresponding to each modification operation.
[0107] Once all modifications to the original metadata have been completed, the log entries corresponding to each modification are written to the log area on the physical storage space. Based on the log entries in the log area, modify the original metadata of the data area, where the data area is a region on the physical storage space used to store metadata.
[0108] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for modifying metadata, characterized in that, include: The system acquires all the modification operations to be performed on the original metadata and generates log entries in memory for the modification information corresponding to each modification operation. The metadata is a free space management structure, which is used to record free physical blocks on the physical storage space. The free space management structure includes a BNO tree and a CNT tree. Once all modifications to the original metadata have been completed, the log entries corresponding to each modification are written to the log area on the physical storage space. When the file system where the physical storage space and the memory reside crashes, the original metadata of the data area is modified based on the log entries in the log area. The data area is the region on the physical storage space used to store metadata.
2. The metadata modification method as described in claim 1, characterized in that, The process includes acquiring all modification operations to be performed on the original metadata, generating in-memory log entries with the modification information corresponding to each modification operation, and prior to this: Estimate the log capacity and data area capacity required for the data generated by the modification operation of the original metadata; Based on the estimated log capacity and data area capacity, log space is reserved in the log area and physical blocks are reserved in the data area, respectively. The physical blocks are storage units of the physical storage space.
3. The metadata modification method as described in claim 1, characterized in that, The log entry includes a base class field and a content field. The base class field includes the log entry's mount node in the transaction chain, the modification type of the original metadata, the modification status flag of the original metadata, and the log sequence number corresponding to the most recent modification of the original metadata. The transaction chain is used to record the modification order of the original metadata. The mount node of the log entry in the transaction chain is used to represent the modification sequence number of the modification corresponding to the log entry in various modifications. The modification status flag is used to represent the timeliness of the modification of the original metadata. The content field includes a buffer block of the original metadata in memory and an intra-block bitmap of the buffer block, wherein the intra-block bitmap is used to record the modifications made to the original metadata.
4. The metadata modification method as described in claim 2, characterized in that, Write the corresponding log entries for each modification to the log area on the physical storage space, including: Move the log entries to the list of ready items in memory; Get the write mode identifier of whether the transaction is synchronously written to the log area on the physical storage space. The transaction is a modification transaction of each original metadata. Synchronization means the synchronization between the completion of all modifications of the original metadata and the writing of all log entries corresponding to each original metadata to the log area. Based on the write mode identifier, the log entries corresponding to each transaction in the ready item list are written to the log area on the physical storage space.
5. The metadata modification method as described in claim 4, characterized in that, Based on the write mode identifier, the log entries corresponding to each transaction in the ready item list are written to the log area in the physical storage space, including: When the write mode is set to synchronous, the log entry corresponding to each transaction in the ready item list is written separately to the log area on the physical storage space. When the write mode is set to asynchronous, the number of transactions in the ready item list is counted until the number of transactions reaches a threshold. Then, the log entries corresponding to all transactions in the ready item list are written to the log area in the physical storage space.
6. The metadata modification method as described in claim 4, characterized in that, Write the log entries corresponding to each transaction on the ready list to the log area on the physical storage space, and then the process includes: Create a new blank ready item list, which is used to store the log items corresponding to subsequent transaction items.
7. The metadata modification method as described in claim 3, characterized in that, Based on the log entries in the log area, modify the original metadata of the data area, including: Retrieve log records from the log area that were being processed at the time of the last system crash, where each log record includes all log entries generated by the same metadata modification; The log records in the log area are read back into the memory, and the original metadata is written into the buffer block in the memory; On the memory, based on the log records, the original metadata of the buffer block is modified to obtain the modified metadata; The modified metadata is written from the buffer block to the data area to complete the modification of the original metadata of the data area.
8. A metadata modification device, characterized in that, The device comprises the following components: The log entry generation module is used to obtain the various modification operations to be performed on the original metadata, and generate log entries in memory with the modification information corresponding to each modification operation. The metadata is a free space management structure, which is used to record free physical blocks on the physical storage space. The free space management structure includes a BNO tree and a CNT tree. The writing module is used to write the log entries corresponding to each modification to the original metadata to the log area on the physical storage space when all modification operations have been completed. The modification module is used to modify the original metadata of the data area based on the log entries in the log area, wherein the data area is a region on the physical storage space used to store metadata.
9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a metadata modification program stored in the memory and executable on the processor. When the processor executes the metadata modification program, it implements the steps of the metadata modification method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a metadata modification program, which, when executed by a processor, implements the steps of the metadata modification method as described in any one of claims 1-7.
