F2FS garbage collection optimization method based on ZNS SSD

Abstract

The invention relates to an F2FS garbage collection optimization method based on a ZNS SSD, and the method comprises the steps: designing a thread to monitor the data failure rate of a ZNS SSD partition according to the intensity of a write-in load, and adaptively adjusting the sampling period of the monitoring thread and a model formula for predicting the data failure rate of the partition, so as to predict the data failure rate of each partition of the ZNS SSD. And selecting a proper partition for recovery according to the partition data effective rate, the partition data failure rate and the current write-in load intensity. According to various characteristics of data to be migrated, the popularity of the data is defined, the data to be migrated are clustered, the popularity is divided, and the data to be migrated are classified and placed in special partitions. In addition, the migrated data is managed in a unified mode, redundant garbage collection can be remarkably reduced, and the system operation efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer storage technology, and in particular relates to an F2FS garbage collection optimization method based on ZNS SSD. Background Technology

[0002] With the commercial deployment of NVMe Zoned Namespace (ZNS) solid-state storage technology, the collaborative optimization between storage devices and file systems faces a systematic restructuring. ZNS SSDs, by exposing physically erased block-aligned partition zone interfaces, allow the host to directly control data layout, fundamentally changing the design premise of traditional garbage collection (GC) mechanisms. In 4KB random write scenarios, ZNS SSDs can reduce write amplification by approximately 58% compared to traditional FTL SSDs, providing a new hardware foundation for building high-performance persistent storage systems. ZNS SSDs require data to be written strictly sequentially within the physical zone, and erase operations can only be performed at the granularity of the entire zone (typically hundreds of MB to GB).

[0003] Flash-Friendly File System (F2FS) is a journal-structured file system specifically designed for flash memory. Utilizing its LFS (Log-Structured File System) mode, it enforces sequential data writing, a feature that makes it highly compatible with ZNS SSDs. Currently, F2FS has added basic support for ZNS SSDs. However, F2FS's garbage collection (GC) mechanism is not yet optimized for the zone characteristics of ZNS SSDs, resulting in a significant decrease in GC efficiency for traditional F2FS on ZNS SSDs. This is especially true in write-intensive workloads, where frequent GC can lead to severe performance degradation.

[0004] On the one hand, in write-intensive workloads, F2FS frequently triggers foreground GC, selecting zones with a high proportion of invalid data for reclamation (using a greedy algorithm). These zones often contain hot data; however, valid data within these zones, after migration, may become invalid again within a short period. This migrated data may be migrated again in subsequent GC cycles, resulting in a large amount of redundant data migration. Therefore, during F2FS GC, the overhead of reclamation zones should be carefully considered to further reduce the impact of GC. On the other hand, when F2FS performs data migration during GC, it treats all migrated valid data as cold data. In reality, the hotness of this data can vary significantly, leading to a mixture of hot and cold data in the same zone. This interleaving of hot and cold data within the same zone further exacerbates the frequency of GC triggering. Summary of the Invention

[0005] To address the problems existing in the background technology, this invention provides an F2FS garbage collection optimization method based on ZNS SSD, comprising:

[0006] S1: Enable threads to monitor partition data failure rate. Based on different load types, adaptively adjust the sampling period of the monitoring threads and the model formula for predicting partition data failure rate to obtain accurate partition data failure rate.

[0007] S2: When selecting partitions to be reclaimed, appropriate partitions are selected for reclamation based on the partition data availability rate, partition data failure rate, and current load intensity.

[0008] S3: Based on the various characteristics of the data to be migrated, define the data popularity, cluster and classify the data to be migrated according to popularity, and place them into special partitions.

[0009] S4: Manage the migrated data in a unified manner and adapt it to the entire garbage collection mechanism to reduce redundant garbage collection.

[0010] The present invention has at least the following beneficial effects

[0011] This invention significantly improves the overall performance of ZNS SSDs by comprehensively analyzing multiple features of data and optimizing the GC mechanism accordingly. By designing appropriate monitoring threads to collect key data features, the monitoring threads can be adaptively adjusted according to different loads. Feature-based garbage collection algorithms select suitable zones for collection, reducing redundant garbage collection. This invention clusters data based on multiple features, optimizes the data migration mechanism, and uniformly manages the migrated data to adapt to the entire GC mechanism. In summary, the garbage collection optimization method based on multiple data features proposed in this invention optimizes the GC mechanism by comprehensively considering multiple data features, effectively reducing data migration and zone erasure frequency, and improving system bandwidth. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0013] Figure 2 This is an overall framework diagram of an embodiment of the present invention;

[0014] Figure 3 This is a schematic diagram of a data failure rate monitor according to an embodiment of the present invention;

[0015] Figure 4 This is a flowchart of the recycling partitioning algorithm according to an embodiment of the present invention;

[0016] Figure 5 This is a flowchart of migration data management according to an embodiment of the present invention. Detailed Implementation

[0017] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0018] See Figure 1 Figure 2 The present invention provides an F2FS garbage collection optimization method based on ZNS SSD, comprising:

[0019] S1: Enable threads to monitor partition data failure rate. Based on different load types, adaptively adjust the sampling period of the monitoring threads and the model formula for predicting partition data failure rate to obtain accurate partition data failure rate.

[0020] S2: When selecting partitions to be reclaimed, appropriate partitions are selected for reclamation based on the partition data availability rate, partition data failure rate, and current load intensity.

[0021] S3: Based on the various characteristics of the data to be migrated, define the data popularity, cluster and classify the data to be migrated according to popularity, and place them into special partitions.

[0022] S4: Manage the migrated data in a unified manner and adapt it to the entire garbage collection mechanism to reduce redundant garbage collection.

[0023] Figure 3 This embodiment proposes a load-based data failure rate monitoring method, which specifically includes the following steps:

[0024] S11: When the garbage collection thread starts, the partition data failure rate ZIR monitoring thread will start synchronously and initialize the sliding window size 5T and sampling frequency T. The window size is set to 5T, and the step size for each slide is T (1 slide). Under typical load, with a window size of 5T, the prediction error (MSE) is negligible; each slide of 1 step ensures the real-time performance requirement for prediction.

[0025] S12: For subsequent changes in workload, adjust the ZIR thread sampling frequency T for different workloads. The calculation method is as follows:

[0026]

[0027] in Indicates the current , The system can achieve the highest k is the adjustment coefficient, which takes the value between 1 and 2; Indicates the lowest sampling frequency. Indicates the highest sampling frequency;

[0028] S13: After the ZIR monitoring thread starts, it calculates the current ZIR based on the amount of effective data in the partition at different times.

[0029]

[0030] in This represents the amount of valid data in a partition at a given moment. This represents the amount of valid data in the partition after time T, starting from time T1.

[0031] S14: Two models are used to predict ZIR: Exponentially Weighted Moving Average (EWMA) and linear regression. At low to medium loads, the data failure trend is stable, so linear regression is used for prediction.

[0032]

[0033] in It's a timestamp. For all historical values; This represents the average of timestamps within the window; under high load, data loss trends are unstable, so an exponentially weighted moving average (EWMA) is used to respond to short-term trends for forecasting.

[0034]

[0035] in This represents the value calculated by the window at the current time. average value; Represents the value calculated at the previous time step. ; It is a weighting factor. Data points closer to the present time have greater weight, making predictions more sensitive to recent changes.

[0036] Figure 4 The flowchart of the garbage collection partition selection algorithm based on multiple data features of the present invention is shown below, which specifically includes the following steps:

[0037] Step 101: Start the garbage collection partition selection algorithm;

[0038] Step 102: Initialize the partitions to be reclaimed and the array to store partition costs;

[0039] Step 103: Traverse each partition to be reclaimed;

[0040] Step 104: Obtain the data characteristics of each partition: effective data rate, data loss rate, and current garbage collection urgency. The effective data rate represents the proportion of valid data within the partition's total capacity.

[0041]

[0042] in Indicates the number of valid blocks within the partition. Indicates the total capacity of the partition; GC_Urgency indicates the garbage collection urgency.

[0043]

[0044] in Indicates the current state of the system , Indicates the maximum that the system can achieve. Data loss rate ZIR is obtained through S1;

[0045] Step 105: Calculate the reclamation cost for each partition according to the formula;

[0046] Step 106: Determine whether all partitions have been traversed. If yes, proceed to step 107; otherwise, proceed to step 103.

[0047] Step 107: Sort the zones in ascending order of cost, and select the first zone after sorting as the victim zone;

[0048] Step 108: Return victim_zone as the partition to be recycled;

[0049] Step 109: Process ends.

[0050] The clustering transfer method based on data features proposed in this invention includes the following specific steps:

[0051] S31: Record three data characteristics to define the popularity of data: data update frequency: the more frequently the data is updated, the higher the popularity; data modification time: the more recent the update time, the higher the popularity; data future failure rate: the higher the future failure rate of the data, the higher the popularity.

[0052] S32: Based on three data characteristics, use K-means to cluster the data and divide it into three categories: cold, warm, and hot data;

[0053] S33: Add a new partition type, GC Zone, specifically designed to store migrated data;

[0054] S34: When migrating data for garbage collection, determine the data type based on the clustering results and place it in GCZone.

[0055] Figure 5 The migration data management method proposed in this invention includes the following specific steps:

[0056] Step 201: Process begins;

[0057] Step 202: Determine whether to start data migration. If yes, proceed to step 204; otherwise, proceed to step 203.

[0058] Step 203: Determine if the number of GC Zones is less than or equal to 6. If yes, end the process; otherwise, proceed to step 205.

[0059] Step 204: Cluster all partitions using k-means;

[0060] Step 205: Determine if there are partitions of the same type in the GC Zone. If yes, proceed to step 208; otherwise, proceed to step 206.

[0061] Step 206: Remove the partition that was first marked as a GC Zone;

[0062] Step 207: Traverse all segments of the victim zone, compare the clustering results, and divide the data to be migrated into different heat levels;

[0063] Step 208: Remove duplicate partitions from the GC Zone. For example, if there are two identical hot partitions, remove one of them.

[0064] Step 209: Determine if the number of GC Zones is less than or equal to 6. If yes, end the process; otherwise, proceed to step 210.

[0065] Step 210: Among all GC Zones, prioritize removing the partition that has experienced the most GC cycles. If the number of GC cycles is the same, remove the partition with the lower ZIR.

[0066] Step 211: Determine if there are partitions of the same type in all GC Zones. If yes, proceed to step 213; otherwise, proceed to step 212.

[0067] Step 212: Select a partition from the free partitions to migrate valid data and mark the partition as a GC Zone;

[0068] Step 213: Select a partition with the same data type as the data being migrated for data migration;

[0069] Step 214: Process ends.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An F2FS garbage collection optimization method based on ZNS SSD, characterized in that, include: S1: Based on the load write intensity of F2FS, a load-based prediction method for ZNS SSD partition data failure rate is designed to predict the data failure rate of each partition of ZNS SSD. S2: Based on the partition data validity rate, partition data failure rate and the current F2FS write load intensity, a garbage collection partition selection method based on multiple data characteristics is proposed to select appropriate partitions for collection in order to reduce the amount of data migration. S3: For waste recycling data awaiting migration, a partitioning and data placement method based on the popularity of the migration data is proposed; S4: For the migrated data, a unified management method for the migrated data is proposed to adapt to the entire garbage collection mechanism and reduce redundant garbage collection.

2. The waste recycling optimization method based on multiple data features according to claim 1, characterized in that, Step S1 is a load-based method for predicting the data failure rate of ZNS SSD partitions, specifically including: S11: Construct a partitioned data failure rate ZIR monitoring thread, and initialize the sliding window size and sampling frequency T of the monitoring thread; S12: Adjust the ZIR thread sampling frequency T for different load intensities under varying workloads, including: Current IOPS refers to the current IOPS. max This is the highest level the system can achieve; k is the adjustment coefficient, ranging from 1 to 2; T low T represents the lowest sampling frequency. high Indicates the highest sampling frequency; S11: Construct a partitioned data failure rate ZIR monitoring thread, and initialize the sliding window size and sampling frequency T of the monitoring thread; S12: Adjust the ZIR thread sampling frequency T for different load intensities under varying workloads, including: S13: After the ZIR monitoring thread starts, it calculates the current ZIR based on the amount of effective data in the partition at different times, including: Among them, Valid_blks T1 Valid_blks represents the amount of valid data in a partition at a given time. T1+T This represents the amount of valid data in the partition after time T, starting from time T1. S14: Two models are used to predict ZIR: Exponentially Weighted Moving Average (EWMA) and linear regression. Under low to medium write loads, such as when write bandwidth is less than 70% of maximum bandwidth, linear regression is used for prediction because the data failure trend is stable. Under high write loads, such as when write bandwidth is greater than or equal to 70% of maximum bandwidth, EWMA is used to predict ZIR in response to short-term trends because the data failure trend is unstable.

3. The waste recycling optimization method based on multiple data features according to claim 1, characterized in that, Step S2, the garbage collection partition selection method based on multiple data features, specifically includes: S21: Calculate the garbage collection urgency (GC_Urgency) based on the current system's write load intensity, as well as the percentage of valid data blocks in the partition and the partition data failure rate. Design a garbage collection partition selection algorithm based on multiple data characteristics; S22: After garbage collection starts, iterate through all partitions to be collected, and calculate the collection cost for each partition based on the new partition collection algorithm designed. S23: By comparing the recycling costs of each partition to be recycled, select the partition with the lowest recycling cost for data migration.

4. The waste recycling optimization method based on multiple data features according to claim 1, characterized in that, Step S3, the partitioning and data placement method based on the popularity of migrated data, specifically includes: S31: Record three data characteristics of the data to be migrated to define the data's popularity: updated_count (the more frequently the data is updated, the higher the popularity); modified_time (the more recent the update time, the higher the popularity); and future failure rate (zir). pre The higher the future data loss rate, the higher the popularity. S32: Based on the three data features recorded in step S31, set the three centroids for the k-means algorithm. S33: For each piece of data to be migrated, calculate the distance between its data features and the three cluster centers, and assign the data to the cluster to which its nearest cluster center belongs; S34: For each cluster, calculate the average value of the data features in that cluster and use that average value as the new cluster center. S35: Repeat steps S33 and S34 until the cluster centers no longer change or the maximum number of iterations is reached. The data to be migrated is categorized into three types: cold data, warm data, and hot data. S36: Add a new partition type, GC Zone, specifically designed to store migrated data; S37: When migrating data for garbage collection, determine the data type based on the clustering results and place it in the GC Zone.

5. The waste recycling optimization method based on multiple data features according to claim 1, characterized in that, Step S4, the unified management method for migration data, specifically includes: S41: During garbage collection, based on the type of the migrated data, select a partition of the same type from the GC Zone that is dedicated to storing the migrated data for migration. If no Zone of this type exists, select a partition from the remaining free partitions and add it to the GC Zone. S42: When the number of GC Zones reaches 6, we prioritize removing partitions that have experienced more garbage collection rounds. If two partitions have experienced the same number of garbage collection rounds, we remove the partition with the lower ZIR. S43: During foreground garbage collection, we choose to skip the GC Zone because it stores migrated data, and we want to prevent it from becoming invalid again in a short time. These partitions are then collected uniformly during background garbage collection.

Images