A Cooperative File System Garbage Collection Method and System Based on Computational Storage Devices
By employing a cooperative file system garbage collection method in flash memory devices, and utilizing the collaborative work of a garbage collection offloading module and a separate flash conversion layer, the problem of collaborative garbage collection between the host and storage devices is solved, improving performance and device lifespan, reducing write amplification, and increasing data migration efficiency.
Patent Information
- Application Number
- CN202510628439.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies lack garbage collection methods that allow for collaboration between the host and storage devices in flash memory, resulting in inefficient data migration and impacting performance and lifespan.
A cooperative file system garbage collection method based on computing storage devices is adopted. The garbage collection unloading module prepares GC requests and performs data migration in the computing storage device. Combined with a separate flash conversion layer, garbage collection of the file system and flash conversion layer is completed. Data migration requests are submitted asynchronously, and metadata is synchronized between the host and the storage device.
It improves garbage collection performance, reduces storage system tail latency, reduces host data processing pressure, reduces write amplification, increases flash memory lifespan and data migration efficiency, and optimizes file read and write efficiency.
Smart Images

Figure CN123114356A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer system architecture technology, and more specifically, to a method and system for garbage collection in a cooperative file system based on computing storage devices. Background Technology
[0002] With the development of flash storage technology, Log-structured File System (LFS) has become a widely used storage management method for flash devices. LFS transforms random write operations into sequential writes, thereby improving performance. However, LFS requires periodic garbage collection (GC) to release space occupied by invalid data. The GC process reclaims space by migrating valid data, but this process can lead to write amplification and negatively impact I / O performance.
[0003] F2FS (Flash-Friendly File System) is an LFS optimized for flash memory. It divides data into two categories: "data segments" and "node segments," and allocates data blocks to different segments based on the frequency of data usage.
[0004] Popularity-based grouping strategies reduce write amplification by storing data blocks with similar usage frequency in the same segment, thus decreasing the amount of data migrated during garbage collection. However, because the Flash Translation Layer (FTL) cannot recognize popularity grouping at the file system level, the distribution of data on the storage device may be disrupted, affecting optimization performance.
[0005] In-storage migration technology offloads data migration tasks directly to the storage device, avoiding unnecessary data transfer between the host and storage device and reducing the host's I / O burden. Nevertheless, when data blocks are distributed, in-storage migration can still incur high communication costs.
[0006] Existing technologies either rely on hot-data grouping at the host level, which incurs additional data transfer overhead, or perform intra-storage migration, ignoring file system hot-data awareness and leading to performance degradation. Therefore, there is a lack of a GC method that allows the host and storage devices to work together to reduce data migration while fully utilizing file system hot-data information.
[0007] The relevant technical terms in this field are as follows:
[0008] Log-structured file system (LFS): A file system that writes data to a sequential log, used to improve the write performance of flash memory devices.
[0009] Garbage collection (GC): In a file system, GC is the process of reclaiming space occupied by invalid data, which usually requires migrating valid data.
[0010] Write amplification (WA): In a file system, due to data migration caused by garbage collection, the actual amount of data written exceeds the original amount of data.
[0011] Flash Translation Layer (FTL): A layer that manages the data mapping between flash storage devices and file systems, typically hiding the physical characteristics of flash memory. Summary of the Invention
[0012] To address the shortcomings of existing technologies, this invention provides a cooperative file system garbage collection method and system based on computing storage devices.
[0013] According to the present invention, a method and system for garbage collection of a cooperative file system based on a computing storage device are provided, the solution of which is as follows:
[0014] Firstly, a cooperative file system garbage collection method based on computing storage devices is provided, the method comprising:
[0015] Step S1: Prepare GC requests, construct data migration requests, and offload them to the compute storage device CSD executor through the garbage collection offloading module of the cooperative file system;
[0016] Step S2: Perform data migration operations in the compute storage device (CSD) based on the split flash translation layer (sFTL) in the cooperative file system, and simultaneously complete garbage collection of the file system layer and the flash translation layer (FTL).
[0017] Step S3: The garbage collection offloading module receives dirty metadata from the compute storage device (CSD) and synchronizes it with the file system metadata in memory.
[0018] Preferably, step S1 includes:
[0019] Step S1.1: Identify the data segments to be migrated and initialize GC requests;
[0020] Step S1.2: Determine valid data blocks and pre-allocate destination addresses for them;
[0021] Step S1.3: Package the pre-allocated address, victim segment identifier, and valid bitmap into a request and offload it to the compute storage device (CSD).
[0022] Preferably, step S2 includes:
[0023] Step S2.1: The executor in the computational storage device (CSD) receives the GC request, unpacks the incoming request, and obtains the pre-allocated address, victim segment identifier, and valid bitmap;
[0024] Step S2.2: The executor migrates valid data blocks through the provided address, migrates physical flash pages based on the direct logical address to physical address L2P mapping of the storage block in the split flash translation layer sFTL, and completes garbage collection of the file system layer and the flash translation layer FTL layer at the same time.
[0025] Step S2.2.1: The actuator identifies the valid block source address through the data segment identifier and the valid bitmap;
[0026] Step S2.2.2: The executor obtains the valid block target address from the pre-allocated address;
[0027] Step S2.2.3: The executor asynchronously submits a data migration request and continues processing the next valid block source address;
[0028] Step S2.2.4: The scheduler merges consecutive migration requests using a delayed submission strategy. The migrated addresses are converted to physical flash page addresses through the direct on-block logical address to physical address L2P mapping of the split flash translation layer sFTL, and then submitted to the data transfer engine. After the data transfer is completed, the scheduler notifies the executor.
[0029] Step S2.2.5: After the executor submits all migration requests, it waits for the migration requests to be completed.
[0030] Step S2.3: While migrating the data, the executor performs lazy updates on the metadata corresponding to the data blocks. That is, the modifications to these metadata are temporarily stored in a designated area in the CSD memory and then sent back to the host after the data migration is completed.
[0031] Step S2.4: After the executor completes the data migration, it packages the execution status and the metadata modified during the execution process stored in memory, and then sends it back to the garbage collection and unloading module in the host.
[0032] Preferably, step S3 includes:
[0033] Step S3.1: The garbage collection unloading module receives the GC completion information packet sent back by CSD, unpacks it to obtain the metadata modified during the GC process;
[0034] Step S3.2: The garbage collection unloading module performs batch synchronization of dirty metadata with file system metadata. If the metadata is cached in memory, it directly modifies the memory; otherwise, it reads the metadata from storage into memory first and then modifies it.
[0035] Step S3.2.1: Modify the metadata of the log header segment to record the file index block number corresponding to the newly allocated data block and its offset in the index block;
[0036] Step S3.2.3: Update the cache of the mapping from consecutive file logical blocks to consecutive file system logical blocks, and delete the extent cache based on the old address;
[0037] Step S3.2.2: Modify the block number corresponding to the logical block in the file index block, changing it from the original address to the new address after data migration.
[0038] Secondly, a cooperative file system garbage collection system based on computing storage devices is provided, the system comprising: a garbage collection offloading module, a garbage collection execution module, and a separate flash conversion layer;
[0039] The garbage collection unloading module is used to prepare garbage collection requests, unload garbage collection requests, and perform metadata synchronization after the request is completed.
[0040] Garbage collection execution module: Used to calculate the specific execution of garbage collection on storage devices;
[0041] Split flash translation layer: Used to manage the translation of logical and physical addresses of data blocks and device-level garbage collection.
[0042] Preferably, the garbage collection unloading module includes: a request preparation module;
[0043] The request preparation module is used to prepare garbage collection requests and unload them to the garbage collection execution module. This module includes a valid block identification unit, a metadata acquisition unit, an address pre-allocation unit, and a data packaging unit.
[0044] When the file system issues a garbage collection request, the request will be handed over to the request preparation module. In the request preparation module, the valid block identification unit obtains the data segment validity bitmap from the file system metadata in memory according to the identifier of the data segment to be reclaimed. The offset of the valid block in the data segment is identified through the bitmap, and the logical address of the valid block is confirmed.
[0045] The metadata acquisition unit searches the reverse mapping table from logical block address to index block address in the metadata cache in memory based on the logical address, and obtains the identifier of the index block corresponding to the logical address. If the cache does not contain the required metadata, it will first read it from storage into memory.
[0046] The address pre-allocation unit allocates destination block addresses available for migration in batches from the data header data segment based on the number of valid blocks by modifying the data segment validity bitmap, while pre-setting the old logical addresses of the valid blocks to invalid;
[0047] The data packaging unit packages the data collected by the first three units into a request, including a migration segment identifier, a valid block bitmap, an index block identifier, and a pre-allocated address vector. This request is transmitted to the garbage collection execution module of the computing storage device through the kernel's I / O stack.
[0048] Preferably, the garbage collection unloading module further includes: a metadata synchronization module;
[0049] The metadata synchronization module is used to synchronize dirty metadata to the metadata cache in the host memory. The module includes a data unpacking unit, a metadata update unit, and a resource release unit.
[0050] After the garbage collection execution module completes the garbage collection request, it sends a metadata synchronization request back to the host. This request wakes up the metadata synchronization module through a series of interrupt handling functions and callback functions in the I / O stack.
[0051] In the metadata synchronization module, the data unpacking unit unpacks the metadata that needs to be synchronized from the received data, including the reverse mapping table from the block address of the migration target segment to the index block address, and the index entries that need to be modified.
[0052] The metadata update unit updates the reverse mapping table to the metadata of the log header data segment, updates the dirty index entries to the index pages in the metadata, and invalidates the cached segments involved.
[0053] The resource release unit releases previously held resources, including locks on index pages and data pages, as well as semaphores that block file system operations, thereby restoring the file system to normal operation.
[0054] Preferably, the garbage collection execution module includes: a request sending and receiving module, a data migration module, and a migration scheduling module;
[0055] The request sending and receiving module is responsible for receiving requests from the uninstallation module and sending a metadata synchronization request to the uninstallation module after the request is executed.
[0056] The request transceiver module first processes the reception of garbage collection requests. After receiving the NVMe instruction, it identifies whether the instruction is a garbage collection request by checking whether it contains a special identifier. If so, it sets up the DMA controller so that the metadata carried by the request is transferred to a specific area of the controller memory and puts the request into the processing queue of the data migration module.
[0057] The data migration module retrieves a request from the processing queue and begins execution. First, it searches for the migration segment identifier, valid block bitmap, and allocation address vector in the request metadata, and quickly calculates the source and target addresses of the valid blocks to be migrated. Second, it sends a migration request to the migration scheduling module, which is executed asynchronously. Subsequently, the data migration module updates the index reverse mapping table of the target data segment and obtains the index entries that need to be modified based on the index reverse mapping table of the source data segment, and then records their modified values.
[0058] After all valid data blocks that need to be migrated have been processed, the data migration module waits for all submitted migration requests to be executed asynchronously. After the requests are completed, the data migration module puts a metadata synchronization request into its completion queue. The request sending and receiving module reads the completion queue and constructs a metadata synchronization request when it receives the request. This request contains the metadata that needs to be updated and is sent to the metadata synchronization module in the garbage collection unloading module.
[0059] The migration scheduling module is responsible for interacting with the storage medium and executing the migration of specific data blocks from the source address to the destination address. This module maintains a time window to merge consecutive migration requests and improve migration efficiency. The migration scheduling module periodically merges consecutive migration requests from the time window and executes them, and then moves the window forward to the next unexecuted request.
[0060] Preferably, the split flash conversion layer divides the physical storage space into an on-site update IPU area and an off-site update OPU area, including an IPU management module, an OPU management module, and a flash GC module;
[0061] The IPU management module is responsible for managing the IPU area. In the IPU area, the logical address to physical address L2P mapping of storage blocks is a non-direct page-level mapping. This mapping adopts a round-robin strategy to allocate physical write addresses for new logical write addresses, allowing the file system to update the metadata area in-situ.
[0062] The OPU management module is responsible for managing the OPU area. In the OPU area, the logical address to physical address L2P mapping of storage blocks is a direct mapping, and there is no additional device-level garbage collection module. The device side of the OPU area does not require an additional log storage structure.
[0063] The flash GC module is responsible for device-level garbage collection in the IPU area; it needs to maintain the log structure of the device layer through device garbage collection in order to support in-place updates.
[0064] Preferably, when the discrete flash memory conversion layer receives a read / write request, it first determines whether the request is located in the OPU area based on the read / write address;
[0065] If the read / write operation is located in the OPU area, the physical address is obtained from the logical block address through direct L2P mapping, and then the specified data transfer operation is performed on that physical address.
[0066] If located in the IPU region, the L2P mapping table of the IPU region is searched first to obtain the corresponding physical address through the logical block address. Secondly, the separate flash conversion layer determines whether the remaining available physical space is lower than a preset threshold. If so, the execution of the read / write request is blocked, the flash GC is triggered, and the flash GC module selects the blocks to be cleaned from the flash blocks of the IPU, performs valid data migration, and updates the L2P mapping table. If not, the read / write request continues to be executed, and the specified data transmission operation is performed on the specified physical address.
[0067] Compared with the prior art, the present invention has the following beneficial effects:
[0068] 1. This invention addresses the garbage collection problem in journal-based file systems, effectively improving garbage collection performance and reducing storage system tail latency;
[0069] 2. This invention enables the file system to leverage the hardware advantages of both the host and storage devices to achieve efficient data migration, reduce the data processing pressure on the host, reduce write amplification caused by garbage collection, improve the lifespan of flash memory devices, optimize data migration efficiency, reduce tail latency of file read and write, and improve the read and write efficiency of the storage system, thus having broad application value.
[0070] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description
[0071] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0072] Figure 1 This is a schematic diagram of the overall modules in an embodiment of the present invention;
[0073] Figure 2 This is a schematic diagram of the garbage collection and unloading module in an embodiment of the present invention;
[0074] Figure 3 This is a schematic diagram of the garbage collection execution module in an embodiment of the present invention;
[0075] Figure 4 This is a flowchart illustrating the process of reading and writing requests by the discrete flash memory conversion layer in an embodiment of the present invention. Detailed Implementation
[0076] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.
[0077] This invention provides a cooperative file system garbage collection method based on computing storage devices, the method specifically including:
[0078] Step S1: Prepare a garbage collection (GC) request through the garbage collection offloading module in the cooperative file system, construct a data migration request through scatter-gather, and offload it to the compute storage device (CSD) executor.
[0079] Step S1 includes:
[0080] Step S1.1: Select the data segment with the fewest valid blocks using a greedy algorithm, designate this data segment as the data segment to be migrated (victim segment), and initialize the GC request;
[0081] Step S1.2: Determine the valid data blocks based on the valid bitmap of the data segment and pre-allocate destination addresses for them;
[0082] Step S1.3: Package the pre-allocated address, victim segment identifier, and valid bitmap into a request and offload it to the compute storage device (CSD).
[0083] Step S2: Perform data migration operations in the compute storage device (CSD) based on the split flash translation layer (sFTL) in the cooperative file system, and simultaneously complete garbage collection of the file system layer and the flash translation layer (FTL).
[0084] Step S2 specifically includes:
[0085] Step S2.1: The executor in the computational storage device (CSD) receives the GC request, unpacks the incoming request, and obtains the pre-allocated address, victim segment identifier, and valid bitmap;
[0086] Step S2.2: The executor migrates valid data blocks through the provided address, migrates physical flash pages based on the direct logical address to physical address L2P mapping of the storage block in the split flash translation layer sFTL, and completes garbage collection of the file system layer and the flash translation layer FTL layer at the same time.
[0087] Step S2.2.1: The actuator identifies the valid block source address through the data segment (victim segment) identifier and the valid bitmap;
[0088] Step S2.2.2: The executor obtains the valid block target address from the pre-allocated address;
[0089] Step S2.2.3: The executor asynchronously submits a data migration request and continues processing the next valid block source address;
[0090] Step S2.2.4: The scheduler merges consecutive migration requests using a delayed submission strategy. The migration address is converted into a physical flash page address through direct L2P mapping of the split flash translation layer sFTL and submitted to the data transfer engine. After the data transfer is completed, the scheduler notifies the executor.
[0091] Step S2.2.5: After the executor submits all migration requests, it waits for the migration requests to be completed.
[0092] Step S2.3: While migrating the data, the executor performs lazy updates on the metadata corresponding to the data blocks. That is, the modifications to these metadata are temporarily stored in a designated area in the CSD memory and then sent back to the host after the data migration is completed.
[0093] Step S2.4: After the executor completes the data migration, it packages the execution status and the metadata modified during the execution process stored in memory, and then sends it back to the garbage collection and unloading module in the host.
[0094] Step S3: The garbage collection offloading module receives dirty metadata from the compute storage device (CSD) and synchronizes it with the file system metadata in memory.
[0095] Step S3 specifically includes:
[0096] Step S3.1: The garbage collection unloading module receives the GC completion information packet sent back by CSD, unpacks it to obtain the metadata modified during the GC process;
[0097] Step S3.2: The garbage collection unloading module performs batch synchronization of dirty metadata with file system metadata. If the metadata is cached in memory, it directly modifies the memory; otherwise, it reads the metadata from storage into memory first and then modifies it.
[0098] Step S3.2.1: Modify the metadata of the log header segment to record the file index block number corresponding to the newly allocated data block and its offset in the index block;
[0099] Step S3.2.3: Update the cache of the mapping (extent) from consecutive file logical blocks to consecutive file system logical blocks, and delete the extent cache based on the old address;
[0100] Step S3.2.2: Modify the block number corresponding to the logical block in the file index block, changing it from the original address to the new address after data migration.
[0101] The present invention also provides a cooperative file system garbage collection system based on computing storage devices. The cooperative file system garbage collection system based on computing storage devices can be implemented by executing the process steps of the cooperative file system garbage collection method based on computing storage devices. That is, those skilled in the art can understand the cooperative file system garbage collection method based on computing storage devices as a preferred embodiment of the cooperative file system garbage collection system based on computing storage devices.
[0102] Reference Figure 1 As shown, the system specifically includes: a garbage collection unloading module, a garbage collection execution module, and a separate flash memory conversion layer;
[0103] The garbage collection unloading module includes a request preparation module and a metadata synchronization module. This module is used to prepare garbage collection requests, unload garbage collection requests, and perform metadata synchronization after the request is completed.
[0104] The garbage collection execution module includes a request sending and receiving module, a data migration module, and a migration scheduling module. This module is used to perform the specific execution of garbage collection on the computing storage device.
[0105] The split flash translation layer divides the physical storage space into an in-place update (IPU) area and an out-of-place update (OPU) area. It includes an IPU management module, an OPU management module, and a flash GC module. This module is used to manage the translation of logical and physical addresses of data blocks and device-level garbage collection.
[0106] In the garbage collection unloading module:
[0107] The request preparation module is used to prepare garbage collection requests and unload them into the garbage collection execution module. This module consists of four units: valid block identification, metadata acquisition, address pre-allocation, and data packaging.
[0108] Reference Figure 2 As shown, when the file system issues a garbage collection request, the request is handed over to the request preparation module.
[0109] In the request preparation module, the valid block identification unit retrieves a data segment validity bitmap from the file system metadata in memory based on the identifier of the data segment to be reclaimed. This bitmap identifies the offset of valid blocks within the data segment, thus confirming the logical address of the valid block. The metadata acquisition unit, based on these logical addresses, searches the reverse mapping table from logical block address to index block address in the memory metadata cache to obtain the identifier of the index block corresponding to the logical address. If the cache does not contain the required metadata, it is first read from storage into memory. The address pre-allocation unit, based on the number of valid blocks, modifies the data segment validity bitmap to allocate destination block addresses for migration in batches from the data header segment, while simultaneously invalidating the old logical addresses of the valid blocks. Finally, the data packaging unit packages the data collected by the first three units into a request, including the migration segment identifier, valid block bitmap, index block identifier, and pre-allocated address vector. This request is transmitted to the garbage collection execution module of the computing storage device via the kernel's I / O stack.
[0110] The metadata synchronization module is used to synchronize dirty metadata to the metadata cache in the host memory. This module consists of three units: data unpacking, metadata updating, and resource release.
[0111] After the garbage collection execution module completes the garbage collection request, it sends a metadata synchronization request back to the host. This request wakes up the metadata synchronization module through a series of interrupt handling functions and callback functions in the I / O stack.
[0112] In the metadata synchronization module, the data unpacking unit unpacks the metadata to be synchronized from the received data. This includes the reverse mapping table from the block address of the target segment to the index block address, and the index entries to be modified (including their index block identifier, the offset of the index entry within the block, and the modified index entry value). The metadata update unit first updates the reverse mapping table to the metadata of the log header data segment, then updates the dirty index entries to the index pages in the metadata, and invalidates the affected extent cache. Finally, the resource release unit releases the previously held resources, including the locks on the index pages and data pages involved, as well as the semaphores that blocked file system operations, thereby restoring the file system to normal operation.
[0113] In the garbage collection execution module:
[0114] The request sending and receiving module is responsible for receiving requests from the unloading module and sending a metadata synchronization request to the unloading module after the request is executed.
[0115] Reference Figure 3As shown, the request transceiver module first handles the reception of garbage collection requests. After receiving the NVMe instruction, it identifies whether the instruction is a garbage collection request by checking whether it contains a special identifier. If so, it sets the DMA controller so that the metadata carried by the request is transferred to a specific area of the controller memory and puts the request into the processing queue of the data migration module.
[0116] The data migration module retrieves requests from the processing queue and begins execution. First, it looks up the migration segment identifier, valid block bitmap, and allocation address vector in the request metadata, quickly calculating the source and destination addresses of the valid blocks to be migrated. Next, it sends a migration request to the migration scheduling module, which is executed asynchronously. Then, the data migration module updates the index reverse mapping table of the target data segment and obtains the index entries that need modification based on the index reverse mapping table of the source data segment, recording their modified values. After processing all valid data blocks to be migrated, the data migration module waits for all submitted migration requests to be executed asynchronously. Once a request is completed, the data migration module places a metadata synchronization request in its completion queue. The request sending and receiving module reads the completion queue and, upon receiving this request, constructs a metadata synchronization request containing the metadata that needs to be updated. This request is then sent to the metadata synchronization module within the garbage collection unloading module.
[0117] The migration scheduling module is responsible for interacting with the storage medium and executing the migration of specific data blocks from the source address to the destination address. This module maintains a time window to merge consecutive migration requests and improve migration efficiency. The migration scheduling module periodically merges consecutive migration requests from the time window and executes them, then moves the window forward to the next unexecuted request.
[0118] In the discrete flash memory conversion layer:
[0119] When formatting and anchoring the file system, the storage device identifies the partitioning of the upper-layer file system (F2FS in this embodiment) and divides the physical storage into two parts: the OPU area and the IPU area. The OPU area corresponds to the main data area of F2FS and only allows off-site updates, while the IPU area corresponds to the metadata area of F2FS and allows in-site updates.
[0120] The OPU management module is responsible for managing the OPU region. Within the OPU region, the logical-to-physical (L2P) mapping of storage blocks is direct, and there is no additional device-level garbage collection module. Since the file system only performs off-site updates to the OPU region and has a corresponding garbage collection mechanism, the device side of the OPU region does not require an additional log storage structure, thus significantly reducing write amplification and the maintenance costs of complex L2P mappings.
[0121] The IPU management module is responsible for managing the IPU region. Within the IPU region, L2P mapping is a non-direct page-level mapping that uses a round-robin strategy to allocate physical write addresses to new logical write addresses. Because a device-level log structure is maintained, in-place updates at the file system layer are permitted, allowing the file system to update the metadata region in-place, avoiding the overhead of maintaining a complex log structure for metadata.
[0122] The flash GC module is responsible for device-level garbage collection of the IPU area. Since the file system does not maintain a log structure for the IPU area, device-level log structure needs to be maintained through device garbage collection to enable support for in-place updates.
[0123] Reference Figure 4 As shown, when the split flash conversion layer receives a read / write request, it first determines whether the request is located in the OPU region based on the read / write address. If the read / write request is located in the OPU region, the physical address is obtained from the logical block address through direct L2P mapping, and then the specified data transfer operation is performed on that physical address. This process significantly simplifies the read / write request execution flow and improves read / write efficiency. If the request is located in the IPU region, the L2P mapping table of the IPU region is first searched to obtain the corresponding physical address through the logical block address. Secondly, the split flash conversion layer determines whether the remaining available physical space is lower than a preset threshold. If so, the execution of the read / write request is blocked, flash GC is triggered, and the flash GC module selects the blocks to be cleaned from the flash blocks of the IPU, performs valid data migration, and updates the L2P mapping table. If not, the read / write request continues to be executed, and the specified data transfer operation is performed on the specified physical address.
[0124] This invention provides a cooperative file system garbage collection method and system based on computing and storage devices, which can effectively improve the performance of garbage collection and reduce the tail latency of the storage system. At the same time, it can make full use of the hardware advantages of the host and storage devices to achieve efficient data migration, reduce the data processing pressure on the host, and reduce write amplification caused by garbage collection, and has wide application value.
[0125] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0126] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
Claims
1. A method for garbage collection in a cooperative file system based on computing storage devices, characterized in that, include: Step S1: Prepare GC requests through the garbage collection offloading module in the cooperative file system, construct data migration requests, and offload the data migration requests to the compute storage device CSD executor; Step S2: Perform data migration operations in the compute storage device (CSD) based on the split flash translation layer (sFTL) in the cooperative file system, and simultaneously complete garbage collection of the file system layer and the flash translation layer (FTL). Step S3: The garbage collection offloading module receives dirty metadata from the compute storage device (CSD) and synchronizes it with the file system metadata in memory.
2. The garbage collection method for a cooperative file system based on computing storage devices according to claim 1, characterized in that, Step S1 includes: Step S1.1: Identify the data segments to be migrated and initialize GC requests; Step S1.2: Determine valid data blocks and pre-allocate destination addresses for them; Step S1.3: Package the pre-allocated address, victim segment identifier, and valid bitmap into a request and offload it to the compute storage device (CSD).
3. The garbage collection method for a cooperative file system based on computing storage devices according to claim 1, characterized in that, Step S2 includes: Step S2.1: The executor in the computational storage device (CSD) receives the GC request, unpacks the incoming request, and obtains the pre-allocated address, victim segment identifier, and valid bitmap; Step S2.2: The executor migrates valid data blocks through the provided address, migrates physical flash pages based on the direct logical address to physical address L2P mapping of the storage block in the split flash translation layer sFTL, and completes garbage collection of the file system layer and the flash translation layer FTL layer at the same time. Step S2.2.1: The actuator identifies the valid block source address through the data segment identifier and the valid bitmap; Step S2.2.2: The executor obtains the valid block target address from the pre-allocated address; Step S2.2.3: The executor asynchronously submits a data migration request and continues processing the next valid block source address; Step S2.2.4: The scheduler merges consecutive migration requests using a delayed submission strategy. The migration address is converted to a physical flash page address through the direct on-block logical address to physical address L2P mapping of the split flash translation layer sFTL, and then submitted to the data transfer engine. After the data transfer is completed, the scheduler notifies the executor. Step S2.2.5: After the executor submits all migration requests, it waits for the migration requests to be completed. Step S2.3: While migrating the data, the executor performs lazy updates on the metadata corresponding to the data blocks. That is, the modifications to these metadata are temporarily stored in a designated area in the CSD memory and then sent back to the host after the data migration is completed. Step S2.4: After the executor completes the data migration, it packages the execution status and the metadata modified during the execution process stored in memory, and then sends it back to the garbage collection and unloading module in the host.
4. The garbage collection method for a cooperative file system based on computing storage devices according to claim 1, characterized in that, Step S3 includes: Step S3.1: The garbage collection unloading module receives the GC completion information packet sent back by CSD, unpacks it to obtain the metadata modified during the GC process; Step S3.2: The garbage collection unloading module performs batch synchronization of dirty metadata with file system metadata. If the metadata is cached in memory, it directly modifies the memory; otherwise, it reads the metadata from storage into memory first and then modifies it. Step S3.2.1: Modify the metadata of the log header segment to record the file index block number corresponding to the newly allocated data block and its offset in the index block; Step S3.2.3: Update the cache of the mapping from consecutive file logical blocks to consecutive file system logical blocks, and delete the extent cache based on the old address; Step S3.2.2: Modify the block number corresponding to the logical block in the file index block, changing it from the original address to the new address after data migration.
5. A cooperative file system garbage collection system based on computing storage devices, based on the cooperative file system garbage collection method based on computing storage devices as described in claims 1-4, characterized in that, Includes: a garbage collection offloading module, a garbage collection execution module, and a separate flash memory conversion layer; The garbage collection unloading module is used to prepare garbage collection requests, unload garbage collection requests, and perform metadata synchronization after the request is completed. Garbage collection execution module: Used to calculate the specific execution of garbage collection on storage devices; Split flash translation layer: Used to manage the translation of logical and physical addresses of data blocks and device-level garbage collection.
6. The cooperative file system garbage collection system based on computing storage devices according to claim 5, characterized in that, The garbage collection unloading module includes: a request preparation module; The request preparation module is used to prepare garbage collection requests and unload them to the garbage collection execution module. This module includes a valid block identification unit, a metadata acquisition unit, an address pre-allocation unit, and a data packaging unit. When the file system issues a garbage collection request, the request will be handed over to the request preparation module. In the request preparation module, the valid block identification unit obtains the data segment validity bitmap from the file system metadata in memory according to the identifier of the data segment to be reclaimed. The offset of the valid block in the data segment is identified through the bitmap, and the logical address of the valid block is confirmed. The metadata acquisition unit searches the reverse mapping table from logical block address to index block address in the metadata cache in memory based on the logical address, and obtains the identifier of the index block corresponding to the logical address. If the cache does not contain the required metadata, it will first read it from storage into memory. The address pre-allocation unit allocates destination block addresses available for migration in batches from the data header data segment based on the number of valid blocks by modifying the data segment validity bitmap, while pre-setting the old logical addresses of the valid blocks to invalid; The data packaging unit packages the data collected by the first three units into a request, including a migration segment identifier, a valid block bitmap, an index block identifier, and a pre-allocated address vector. This request is transmitted to the garbage collection execution module of the computing storage device through the kernel's I / O stack.
7. The cooperative file system garbage collection system based on computing storage devices according to claim 6, characterized in that, The garbage collection and unloading module also includes: a metadata synchronization module; The metadata synchronization module is used to synchronize dirty metadata to the metadata cache in the host memory. The module includes a data unpacking unit, a metadata update unit, and a resource release unit. After the garbage collection execution module completes the garbage collection request, it sends a metadata synchronization request back to the host. This request wakes up the metadata synchronization module through a series of interrupt handling functions and callback functions in the I / O stack. In the metadata synchronization module, the data unpacking unit unpacks the metadata that needs to be synchronized from the received data, including the reverse mapping table from the block address of the migration target segment to the index block address, and the index entries that need to be modified. The metadata update unit updates the reverse mapping table to the metadata of the log header data segment, updates the dirty index entries to the index pages in the metadata, and invalidates the cached segments involved. The resource release unit releases previously held resources, including locks on index pages and data pages, as well as semaphores that block file system operations, thereby restoring the file system to normal operation.
8. The cooperative file system garbage collection system based on computing storage devices according to claim 5, characterized in that, The garbage collection execution module includes: a request sending and receiving module, a data migration module, and a migration scheduling module; The request sending and receiving module is responsible for receiving requests from the uninstallation module and sending a metadata synchronization request to the uninstallation module after the request is executed. The request transceiver module first processes the reception of garbage collection requests. After receiving the NVMe instruction, it identifies whether the instruction is a garbage collection request by checking whether it contains a special identifier. If so, it sets the DMA controller so that the metadata carried by the request is transferred to a specific area of the controller memory and puts the request into the processing queue of the data migration module. The data migration module retrieves a request from the processing queue and begins execution. First, it searches for the migration segment identifier, valid block bitmap, and allocation address vector in the request metadata, and quickly calculates the source and target addresses of the valid blocks to be migrated. Second, it sends a migration request to the migration scheduling module, which is executed asynchronously. Subsequently, the data migration module updates the index reverse mapping table of the target data segment and obtains the index entries that need to be modified based on the index reverse mapping table of the source data segment, and then records their modified values. After all valid data blocks that need to be migrated have been processed, the data migration module waits for all submitted migration requests to be executed asynchronously. After the requests are completed, the data migration module puts a metadata synchronization request into its completion queue. The request sending and receiving module reads the completion queue and constructs a metadata synchronization request when it receives the request. This request contains the metadata that needs to be updated and is sent to the metadata synchronization module in the garbage collection unloading module. The migration scheduling module is responsible for interacting with the storage medium and executing the migration of specific data blocks from the source address to the destination address. This module maintains a time window to merge consecutive migration requests and improve migration efficiency. The migration scheduling module periodically merges consecutive migration requests from the time window and executes them, and then moves the window forward to the next unexecuted request.
9. The cooperative file system garbage collection system based on computing storage devices according to claim 5, characterized in that, The separate flash memory conversion layer divides the physical storage space into an on-site update IPU area and an off-site update OPU area, including an IPU management module, an OPU management module, and a flash memory GC module; The IPU management module is responsible for managing the IPU area. In the IPU area, the logical address to physical address L2P mapping of storage blocks is a non-direct page-level mapping. This mapping adopts a round-robin strategy to allocate physical write addresses for new logical write addresses, allowing the file system to update the metadata area in-situ. The OPU management module is responsible for managing the OPU region. In the OPU region, the logical address to physical address L2P mapping of storage blocks is a direct mapping, and there is no additional device-level garbage collection module. The device side of the OPU region does not require an additional log storage structure. The flash GC module is responsible for device-level garbage collection in the IPU area; it needs to maintain the log structure of the device layer through device garbage collection in order to support in-place updates.
10. The cooperative file system garbage collection system based on computing storage devices according to claim 9, characterized in that, When the discrete flash conversion layer receives a read / write request, it first determines whether the request is located in the OPU area based on the read / write address. If the read / write operation is located in the OPU area, the physical address is obtained from the logical block address through direct L2P mapping, and then the specified data transfer operation is performed on that physical address. If located in the IPU region, the L2P mapping table of the IPU region is searched first to obtain the corresponding physical address through the logical block address. Secondly, the separate flash conversion layer determines whether the remaining available physical space is lower than a preset threshold. If so, the execution of the read / write request is blocked, the flash GC is triggered, and the flash GC module selects the blocks to be cleaned from the flash blocks of the IPU, performs valid data migration, and updates the L2P mapping table. If not, the read / write request continues to be executed, and the specified data transmission operation is performed on the specified physical address.
