QLC medium write amplification optimization method and device

Abstract

According to the QLC medium write amplification optimization method and device, by responding to a data write-in request, metadata of a RocksDB database is written into a metadata pool formed by TLC media, and user data is written into a data pool formed by QLC media. The service load monitoring module is used for collecting operation state data, and the intelligent load analysis module is used for judging whether the current state is in a non-service or low-service load state or not. In a low-load or non-business state, closing a garbage collection (GC) mechanism, and only executing Purge operation to recycle invalid data; and under a normal load, the GC trigger condition is controlled according to the storage space utilization rate and the data block life cycle set by the dynamic threshold configuration module. And obtaining the write-in data volume of the QLC medium through the SMART interface, and calculating the write magnification to evaluate the optimization effect. The method can be applied to electronic equipment such as a server, the write magnification of the QLC medium is effectively reduced, the service life of the QLC medium is prolonged, and the performance and stability of a storage system are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer storage systems, and in particular to a QLC media write amplification optimization method and device. Background Art

[0002] As an important technical direction in the field of solid-state storage, QLC (Quadruple-Level Cell) has been widely used in recent years with the continuous growth of storage density requirements. In storage systems, QLC media provides cost advantages for large-scale data storage through its high storage density characteristics, but its low erase and write durability has become a key factor restricting its performance and lifespan. In related technologies, storage systems usually use the collaborative operation of flash memory media and storage management software to build a complete storage management process from data writing, garbage collection (GC) to data migration. Specifically, the process covers key links such as data classification, storage path planning, and invalid data recovery. Among them, key-value storage systems such as RocksDB are core components of metadata management and are widely used in distributed storage architectures. They are stored on the same medium as DATA data to form a storage system that integrates multiple technologies.

[0003] However, the existing storage system methods directly adopt a hybrid storage strategy, which does not fully consider the differences in write modes between the RocksDB database and DATA data. This may lead to an increase in write amplification, or a large number of invalid write operations due to the frequent triggering of the GC mechanism under low load, thereby affecting the life of the QLC media and the overall performance of the system. Specifically, due to its four-level structure, the erase and write operations of QLC media are highly complex, and the erase granularity is much larger than the write granularity, resulting in a significant increase in the write amplification factor in scenarios with frequent data updates. In addition, traditional GC mechanisms are usually triggered based on fixed thresholds and lack the ability to dynamically perceive the business load status, so that data migration is still performed when there is no actual business demand, further increasing the write burden. These technical defects limit the deployment and optimization space of QLC media in high-performance storage systems in actual applications. Summary of the Invention

[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] The present invention proposes a QLC media write amplification optimization method, specifically relating to a write amplification control technology for QLC (Quadruple-Level Cell) media in solid-state storage devices. QLC media has the advantages of high storage density and low cost because each storage cell can store 4 bits of data, and is widely used in large-scale data storage systems. However, the erase and write life of QLC media is relatively low, and in scenarios where data is frequently updated, the write amplification phenomenon is particularly serious, resulting in decreased storage performance, reduced data reliability, and shortened media life. Therefore, how to effectively control the write amplification factor of QLC media has become a key issue in current storage system design.

[0006] Another object of the present invention is to provide a QLC media write amplification optimization device.

[0007] To achieve the above objectives, the present invention provides a QLC media write amplification optimization method, comprising:

[0008] In response to a data write request, the RocksDB database metadata is written to a metadata pool consisting of TLC media, and the user data is written to a data pool consisting of QLC media;

[0009] The service load monitoring module collects the operating status data of the storage system, wherein the operating status data includes I / O request rate, CPU usage rate and memory usage rate;

[0010] Based on the operating status data, the intelligent load analysis module determines whether the current state is no business or low business load state;

[0011] When it is determined that there is no business or low business load, the garbage collection GC mechanism is turned off through the GC state management module, and the Purge operation is performed to recycle invalid data;

[0012] When the business load is judged to be normal, the triggering conditions of the GC mechanism are controlled according to the thresholds set by the dynamic threshold configuration module. The thresholds include storage space usage and data block lifecycle.

[0013] The performance monitoring module obtains the amount of data written to the QLC media and calculates the write amplification factor based on the ratio of the amount of data written to the amount of data requested by the application to evaluate the optimization effect.

[0014] The QLC media write amplification optimization method according to the embodiment of the present invention may also have the following additional technical features:

[0015] In one embodiment of the present invention, writing metadata of the RocksDB database into a metadata pool formed by TLC media and writing user data into a data pool formed by QLC media includes:

[0016] Create multiple RocksDB partitions on TLC media;

[0017] Write OSS metadata to the corresponding RocksDB partition based on the data pool identifier, and write user data to the data pool of QLC media.

[0018] In one embodiment of the present invention, the number and capacity of RocksDB partitions on the TLC media are dynamically configured based on the number of data pools in the system.

[0019] In one embodiment of the present invention, determining whether the current state is a no-traffic or low-traffic load state by the intelligent load analysis module includes:

[0020] Get running status data;

[0021] Predict business load trends based on operational status data;

[0022] Based on the prediction results, determine whether the current state is no business or low business load.

[0023] In one embodiment of the present invention, the GC mechanism is dynamically started and stopped according to the judgment result of the intelligent load analysis module; and a Purge operation is performed to recycle invalid data when the GC mechanism is turned off.

[0024] To achieve the above objectives, the present invention further provides a QLC media write amplification optimization device, comprising:

[0025] A data classification writing module, configured to write RocksDB database metadata to a metadata pool composed of TLC media and user data to a data pool composed of QLC media in response to a data write request;

[0026] A service load monitoring module is used to collect operating status data of the storage system, including I / O request rate, CPU usage, and memory usage;

[0027] An intelligent load analysis module, configured to determine whether the current state is a no-business or low-business load state based on the operation status data;

[0028] The GC state management module is used to disable the garbage collection (GC) mechanism when there is no business or low business load, and only perform the Purge operation to recycle invalid data;

[0029] A dynamic threshold configuration module is used to set the trigger threshold of the GC mechanism, which includes storage space usage and data block life cycle;

[0030] The performance monitoring module is used to obtain the amount of data written to the QLC media and calculate the write amplification factor based on the ratio of the amount of data written to the amount of data requested by the application to evaluate the optimization effect.

[0031] The QLC media write amplification optimization method and device of the embodiments of the present invention significantly reduce the write amplification of the QLC media by storing the metadata of the RocksDB database separately from the user data (DATA), and combining it with a dynamic control strategy of the garbage collection (GC) mechanism in a non-business state, thereby improving the performance and reliability of the storage system and extending the service life of the QLC media.

[0032] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0034] Figure 1 is a flowchart of a QLC media write amplification optimization method according to an embodiment of the present invention;

[0035] Figure 2 This is a data interaction diagram of a QLC media write amplification optimization method according to an embodiment of the present invention;

[0036] Figure 3 2 is a structural diagram of a QLC media write amplification optimization device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0039] The following describes a QLC media write amplification optimization method and apparatus according to an embodiment of the present invention with reference to the accompanying drawings.

[0040] Figure 1 FIG. 1 is a flow chart of a QLC media write amplification optimization method according to an embodiment of the present invention. Figure 1 The C shown includes:

[0041] S1, in response to a data write request, writes the metadata of the RocksDB database to a metadata pool consisting of TLC media, and writes user data to a data pool consisting of QLC media;

[0042] S2, collecting operating status data of the storage system through the business load monitoring module, wherein the operating status data includes I / O request rate, CPU usage rate and memory usage rate;

[0043] S3, based on the operating status data, determining whether the current state is no business or low business load state through the intelligent load analysis module;

[0044] S4: When it is determined that there is no business or low business load, the garbage collection GC mechanism is turned off through the GC state management module, and a Purge operation is performed to recycle invalid data;

[0045] S5, when it is determined that the business load state is normal, controlling the triggering conditions of the GC mechanism according to the thresholds set by the dynamic threshold configuration module, wherein the thresholds include storage space usage and data block life cycle;

[0046] S6: The performance monitoring module obtains the amount of data written to the QLC medium and calculates the write amplification factor based on the ratio of the amount of data written to the amount of data requested by the application to evaluate the optimization effect.

[0047] In one embodiment of the present invention, the core of this separation storage strategy is to physically isolate different types of data to improve overall storage performance, reduce write amplification, and extend device life. The specific implementation includes the following key steps:

[0048] Create multiple RocksDB partitions on TLC media: Each piece of TLC (Triple-Level Cell) media is divided into multiple independent RocksDB partitions. Each partition is used to store metadata information for a specific Object Storage Service (OSS) instance. For example, on a 2TB NVMe TLC SSD, you can create multiple RocksDB partitions of equal or varying sizes, such as 20GB each, to carry metadata operations for corresponding data pools. This partitioning mechanism not only isolates metadata between different data pools, but also helps improve system manageability and stability.

[0049] OSS metadata is written to the corresponding RocksDB partition based on the data pool identifier: When the system receives a write request, it parses the data pool identifier (such as pool_id or ossid) in the request to determine the data pool to which it belongs, and writes the corresponding metadata information to the RocksDB partition associated with that data pool. For example, when ossid = 0, the system automatically writes metadata to the RocksDB partition corresponding to / dev / nvme0n1p3. This identifier-based routing mechanism ensures a clear and controllable metadata write path, avoiding resource contention and interference when writing to multiple data pools concurrently.

[0050] Writing user DATA data to a data pool composed of QLC media: The user's actual business data (DATA) is uniformly written to a data pool composed of QLC (Quad-Level Cell) media. QLC media has higher storage density and lower cost, making it suitable for large-scale sequential read and stable write scenarios. By centrally writing DATA data to the QLC data pool, its capacity advantages can be fully utilized while avoiding the life loss caused by frequent writes. In addition, combined with the separate writing of RocksDB and DATA data, the write pattern on the QLC media is more regular, thereby significantly reducing the write amplification factor.

[0051] Dynamically configure the number and capacity of RocksDB partitions to adapt to system changes: The system features dynamic resource allocation, intelligently adjusting the number and capacity of RocksDB partitions on TLC media based on the number of deployed data pools, workload characteristics, and available storage space. For example, when a new data pool is added, the system dynamically creates a new RocksDB partition on the TLC device and allocates space on demand. When a data pool is deleted or merged, the corresponding partition's space resources are reclaimed. This elastic configuration mechanism improves storage resource utilization and enhances system scalability and flexibility, making it suitable for dynamically changing business environments.

[0052] In one embodiment of the present invention, in order to further optimize the garbage collection (GC) mechanism and reduce invalid writes to QLC media, the present invention introduces an intelligent load analysis module based on operating status perception to determine the system's business load status in real time. The specific process is as follows:

[0053] Obtaining operating status data: The intelligent load analysis module continuously collects system operating status information, including but not limited to: I / O request rate (IOPS), CPU utilization, memory usage, network bandwidth usage, storage queue depth, data write / read throughput, etc.

[0054] These indicators form the basic data source for evaluating the system load status and help the system fully understand the current operating status.

[0055] Predicting business load trends based on operational status data: Using machine learning algorithms or time series prediction models (such as ARIMA and LSTM), the system models and analyzes historical and real-time data, identifying cyclical, sudden, or incremental business patterns and predicting future load trends. For example, the system can identify low business hours during the early morning hours and idle business hours during holidays, and make predictions based on these predictions to adjust the GC status in advance.

[0056] Determine whether the system is currently in a "no business" or "low business load" state based on the prediction results: The system uses the prediction model output and preset business load thresholds (such as IOPS <50, CPU utilization <10%, and memory utilization <20%) to comprehensively determine whether the system is currently in a "no business" or "low business load" state. For example, if IOPS remains below the set threshold for 10 consecutive minutes, the system is considered to be in a "no business" state; if the IOPS fluctuates briefly but is generally low, the system is considered to be in a "low business load" state. This refined state identification mechanism provides an accurate basis for subsequent GC control.

[0057] Dynamically control GC startup and shutdown to optimize QLC media write behavior: When the system determines that there is no business or low business load, the system will dynamically disable the garbage collection (GC) mechanism to avoid triggering unnecessary space reclamation operations during this period, thereby reducing the additional write burden on the QLC media. Under normal business load, the GC mechanism remains enabled and performs garbage collection according to the preset strategy to ensure efficient use of storage space.

[0058] Performing a lightweight purge with GC disabled: Even when GC is disabled, the system can still clean up some confirmed invalid data blocks through a lightweight purge. This low-overhead, non-intrusive purge process only processes data blocks that will not affect ongoing write operations. This frees up some space resources without triggering a full GC process, maintaining a certain level of space reclamation capability while avoiding additional pressure on the QLC media.

[0059] In summary, this solution achieves refined management of storage resources and performance optimization by writing RocksDB metadata and user data to TLC and QLC media, respectively, and combining intelligent load analysis with a dynamic GC control mechanism. This solution effectively reduces the write amplification of QLC media, improving storage efficiency and device lifespan, and is suitable for data management needs in high-concurrency, large-scale distributed storage environments.

[0060] In one embodiment of the present invention, a method for separating RocksDB database and DATA data storage is proposed. Specifically, the RocksDB database is stored in a metadata pool composed of TLC (Triple-Level Cell) media, while the DATA data is stored in a data pool composed of QLC media. This separate storage method optimizes data writing patterns and reduces the write amplification of QLC media in various block business scenarios.

[0061] To implement the aforementioned separation storage strategy, we first create multiple RocksDB partitions on TLC media. Each RocksDB partition stores metadata for a specific data pool, the OSS (Object Storage Server). For example, on each TLC disk, we can create three 20GB RocksDB partitions, each corresponding to the OSS metadata for a specific data pool. This allows RocksDB database write operations to be concentrated on the TLC media, while the QLC media primarily handles data write operations, thus avoiding write amplification caused by interference between writes of different data types.

[0062] During the data write process, the storage system automatically writes the RocksDB database data to the corresponding partition of the TLC media according to the data type, and writes the DATA data to the data pool of the QLC media. This separation storage strategy not only improves the efficiency of data writing, but also effectively reduces the write amplification factor of the QLC media. In the sequential write scenario, the write amplification factor of the separated RocksDB database can be controlled between 1.001-1.005, while the write amplification factor without separation may be as high as 1.023-1.678; in the random write scenario, the write amplification factor of the separated RocksDB database can be controlled between 1.001-1.019, while the write amplification factor without separation may be between 1.011-1.532.

[0063] The specific steps are as follows:

[0064] 1. Create a RocksDB partition on the TLC disk. For example, on the nvme0n1 disk of All Flash 1, run the command "sudo ceph-disk -v prepare --bluestore / dev / nvme0n1p1 / dev / nvme0n1p2 --block.db / dev / nvme0n1p3 --ossid 0." Here, / dev / nvme0n1p1 is the Ceph data partition, / dev / nvme0n1p2 is the Ceph block partition, and / dev / nvme0n1p3 is the RocksDB partition. 0 refers to the OSS ID of the specific data pool. This separates the RocksDB database from the DATA data, laying the foundation for subsequent write tests.

[0065] 2. Create a 2TB LUN and mount it. This node also acts as a client to connect to and write data to the mounted LUN. For example, using 8k random writes, the test data volume is 1TB. The vdbench reference script is as follows:

[0066] messagescan=no

[0067] hd=default,vdbench= / root / vdbench50406,user=root,shell=ssh

[0068] sd=sd1,lun= / dev / sdc,openflags=o_direct,threads=32

[0069] wd=wd1, sd=sd*, seekpct=100, rdpct=0, xfersize=8k

[0070] rd=rd1,wd=wd1,iorate=max,elapse=604800,maxdata=1TB,interval=1,warmup=0

[0071] seekpct controls the proportion of random I / O requests. A value of 100 indicates random writes, while a value of 0 indicates sequential writes. By adjusting the parameters in the vdbench script, you can simulate data write scenarios with different block sizes (such as 4K, 8K, 32K, 64K, 128K, 256K, 512K, and 1024K) and write modes (sequential and random) to test the storage system.

[0072] 3. Use the smartctl command to monitor the difference in Data Units Written (written sector count) of the QLC disk before and after data is written. The command is as follows:

[0073] smartctl-a / dev / nvme2n1|grep “Data Units Written”

[0074] smartctl-a / dev / nvme3n1|grep “Data Units Written”

[0075] smartctl-a / dev / nvme4n1|grep “Data Units Written”

[0076] nvme2n1, nvme3n1, and nvme4n1 are the three QLC disk devices in All Flash 1; in All Flash 2, the corresponding QLC disk devices are nvme2n1 and nvme3n1. By calculating the difference in written sector counts, we can determine the actual amount of data written to the QLC drive for 1TB, and thus calculate the write amplification factor.

[0077] Furthermore, the present invention performs dynamic adjustment of non-business GC.

[0078] It is understandable that the garbage collection (GC) mechanism is an important function in the storage system, which is used to reclaim invalid space in the storage device. However, in traditional storage systems, the GC mechanism usually works according to a fixed strategy without considering the current business load. This results in GC still being frequently triggered when there is no business or low business load, generating a large number of unnecessary write operations, thereby exacerbating the write amplification phenomenon of QLC media. In order to solve this problem, the present invention proposes a mechanism for dynamic adjustment of GC when there is no business.

[0079] In one embodiment of the present invention, a real-time data collection module has been developed to monitor the storage system's service load. This module collects various service-related metrics, such as I / O request rates, CPU usage, and memory usage. This data is used to determine the current service load.

[0080] In one embodiment of the present invention, an intelligent load analysis algorithm is designed to accurately identify no or low traffic load conditions. This algorithm combines machine learning and statistical analysis techniques to analyze collected traffic load data in real time. By establishing a traffic load model, it can predict traffic load trends and, based on this, determine whether a state of no traffic is currently in effect.

[0081] In one embodiment of the present invention, a GC state management module is introduced to control the start and stop states of the GC mechanism. Under normal traffic load, the GC mechanism operates normally according to preset policies to ensure efficient storage device space utilization. However, when the intelligent load analysis algorithm determines that the current state is no traffic or low traffic load, the GC state management module automatically disables the GC mechanism to avoid unnecessary write operations.

[0082] In one embodiment of the present invention, a dynamic GC threshold configuration mechanism is implemented to adapt to diverse storage environments and business requirements. This mechanism allows administrators to set GC trigger thresholds based on actual conditions, such as storage space usage and data block lifecycles. When relevant storage system metrics reach or exceed these thresholds, the GC mechanism is reactivated to ensure normal operation of the storage device.

[0083] Specifically, Figure 2 This is the overall data interaction diagram of the QLC media write amplification optimization method of the present invention, as shown in FIG. Figure 2 As shown:

[0084] 1) Data write request and type judgment:

[0085] The process begins with receiving a data write request. The system determines the type of data being written based on the request content:

[0086] If it is a RocksDB database operation (such as metadata update, index write, etc.), it will be directed to "Writing to RocksDB partition of TLC media";

[0087] If it is a DATA data operation (ie, actual data writing by the user), it will be directed to the "data pool for writing to QLC media".

[0088] This judgment mechanism enables automatic routing of different types of data. TLC media is used to carry frequently written RocksDB operations, while QLC media is used to process large amounts of DATA data, thereby physically separating the write load and reducing write amplification.

[0089] 2) Sequential write optimization processing:

[0090] Regardless of the data type, sequential write optimization is performed before writing to the target media. The system utilizes the Log Structured Merge (LSM Tree) feature to convert random writes into batched sequential writes, improving I / O efficiency, reducing disk fragmentation, and optimizing storage space utilization. This phase ensures the efficiency and stability of the write process.

[0091] 3) Write pattern recognition and separate storage execution:

[0092] After completing the sequential optimization, the system further identifies the current write mode:

[0093] If it is a sequential write, a separate storage strategy is used to directly write to the corresponding media;

[0094] If it is random write, the same separation storage strategy is adopted, and the access characteristics are recorded to provide a basis for subsequent GC adjustments.

[0095] This phase ensures that data in different write modes can be properly processed and provides feedback information for the dynamic garbage collection mechanism.

[0096] 4) No business GC dynamic adjustment mechanism:

[0097] After completing the write path decision, the system enters the non-business GC dynamic adjustment phase. Traditional GC mechanisms have a fixed trigger frequency, which easily generates invalid writes during idle time, exacerbating QLC wear. To this end, this invention introduces the following intelligent mechanisms:

[0098] The real-time data acquisition module monitors indicators such as I / O request rate, CPU usage, and memory usage;

[0099] The intelligent load analysis algorithm determines whether the current state is no business or low load based on the collected data;

[0100] The GC status management module decides whether to enable or suspend GC operations based on the analysis results;

[0101] The dynamic threshold configuration mechanism allows administrators to set GC trigger conditions for flexible control.

[0102] This mechanism effectively avoids unnecessary garbage collection in a non-business state and reduces the additional write burden on the QLC media.

[0103] 5) Writing completion and effect evaluation:

[0104] Finally, after all write operations are completed, the system confirms that the data has been correctly written to the disk. Monitoring tools (such as smartctl) are used to count changes in the QLC device's written sectors and calculate the actual write amplification factor. The effectiveness of this method is verified by comparing the write amplification performance in separate and non-separate scenarios.

[0105] The entire process integrates multiple technical means such as data classification, sequence optimization, write mode adaptation, and intelligent GC control, significantly improving the write efficiency and media life of the storage system, and is suitable for high-concurrency, large-scale distributed storage environments.

[0106] The beneficial effects of the present invention are as follows:

[0107] Significantly reduces QLC write amplification: By separating the storage locations of the RocksDB database and DATA data, the write amplification of QLC media in various block business scenarios can be effectively reduced. When the data block size of the write is between 4-128K, separating the RocksDB database can reduce QLC write amplification by 0.02T-0.04T; when the data block size of the write is between 256-1024K, it can reduce QLC write amplification by 0.3T-0.6T. For both sequential write and random write scenarios, the write amplification of the separated RocksDB database is significantly lower than that of the non-separated case, thereby effectively controlling the write amplification phenomenon of QLC media.

[0108] Improving storage system performance and reliability: Separating storage strategies and disabling non-business GC can optimize the storage system's data writing process, reduce unnecessary write operations, and improve storage system performance and reliability. In both sequential and random write scenarios, the throughput and latency of a separated RocksDB database are superior to those of a non-separated one. For example, when the block size is 4KB, the throughput of a separated RocksDB database is approximately 9.00-9.10 IOPS / K and the latency is approximately 3.54-3.71ms; without separation, the throughput is approximately 8.62-9.00 IOPS / K and the latency is approximately 3.54-3.71ms. Separating RocksDB also demonstrates superior performance in random write scenarios.

[0109] Extending QLC media lifespan: Reducing write amplification reduces the actual write volume to the QLC media, thus helping to extend the media's lifespan. This is crucial for storage systems using QLC media, reducing maintenance costs and replacement frequency, and improving the overall cost-effectiveness of the storage system.

[0110] Optimizing storage system architecture: The technical solution of this invention improves the storage system's adaptability to QLC media through a rational data storage strategy and system configuration optimization. This not only solves the QLC write amplification problem in existing storage systems, but also provides new ideas and methods for the development of future storage technologies, promoting the further evolution of storage system architecture.

[0111] In summary, the present invention separates the RocksDB database from the DATA data, leveraging the high performance of TLC media to store RocksDB metadata and QLC media to store DATA data. This separate storage strategy leverages the advantages of both media types, optimizing data write patterns and reducing write amplification on QLC media. The key to implementation lies in properly partitioning the storage media, creating RocksDB partitions, and ensuring that data is accurately written to designated storage locations.

[0112] When using non-bare disks, disabling GC during non-business conditions can avoid write amplification issues caused by unnecessary data migration and write operations during the GC process. This requires precise control of the storage system's GC mechanism and real-time monitoring of system status to ensure that purge operations are only performed when there is no actual business demand. This optimization of the GC mechanism can effectively reduce the write amplification of QLC media without affecting normal system functionality.

[0113] According to an embodiment of the present invention, a QLC media write amplification optimization method is provided, which achieves separate data storage by creating RocksDB partitions on TLC media. This method is innovative in reducing QLC write amplification, effectively distinguishing the present invention from existing technologies by monitoring system logs in real time and adjusting garbage collection policies to avoid unnecessary data migration and write operations.

[0114] In order to implement the above embodiment, Figure 3 As shown, this embodiment also provides a QLC media write amplification optimization device, including:

[0115] A data classification writing module 100 is configured to write RocksDB database metadata into a metadata pool composed of TLC media and user data into a data pool composed of QLC media in response to a data write request;

[0116] The service load monitoring module 200 is used to collect the operation status data of the storage system, wherein the operation status data includes I / O request rate, CPU usage rate and memory usage rate;

[0117] An intelligent load analysis module 300 is configured to determine whether the system is currently in a no-service or low-service load state based on the operating status data;

[0118] The GC state management module 400 is used to disable the garbage collection (GC) mechanism when there is no business or low business load, and only perform the Purge operation to recycle invalid data;

[0119] A dynamic threshold configuration module is used to set the trigger threshold of the GC mechanism, which includes storage space usage and data block life cycle;

[0120] The performance monitoring module 500 is used to obtain the amount of data written to the QLC medium and calculate the write amplification factor based on the ratio of the amount of data written to the amount of data written requested by the application program to evaluate the optimization effect.

[0121] Furthermore, the data classification writing module 100 includes:

[0122] RocksDB partition creation unit, used to create multiple RocksDB partitions on TLC media;

[0123] The data write control unit is used to write OSS metadata to the corresponding RocksDB partition according to the data pool identifier, and write user data to the data pool of QLC media.

[0124] Furthermore, the RocksDB partition creation unit dynamically configures the number and capacity of RocksDB partitions on the TLC medium according to the number of data pools in the system.

[0125] Furthermore, the intelligent load analysis module 300 includes:

[0126] Data acquisition interface, used to obtain operating status data;

[0127] Load forecasting model, used to predict business load trends based on operating status data;

[0128] The state judgment unit is used to judge whether the current state is no business or low business load based on the prediction result.

[0129] Furthermore, the GC status management module 400 includes:

[0130] GC start-stop control unit, used to dynamically start and stop the GC mechanism based on the judgment results of the intelligent load analysis module;

[0131] The Purge execution unit is used to perform Purge operations to reclaim invalid data when the GC mechanism is disabled.

[0132] The QLC media write amplification optimization device provided by an embodiment of the present invention achieves separate data storage by creating RocksDB partitions on TLC media. This method is innovative in reducing QLC write amplification, effectively distinguishing the present invention from existing technologies by monitoring system logs in real time and adjusting GC policies to avoid unnecessary data migration and write operations.

[0133] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0134] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

Claims

1. A QLC media write amplification optimization method, characterized in that: The following steps are involved: In response to a data write request, the RocksDB database metadata is written to a metadata pool consisting of TLC media, and the user data is written to a data pool consisting of QLC media; The service load monitoring module collects the operating status data of the storage system, wherein the operating status data includes I / O request rate, CPU usage rate and memory usage rate; Based on the operating status data, the intelligent load analysis module determines whether the current state is no business or low business load state; When it is determined that there is no business or low business load, the garbage collection GC mechanism is turned off through the GC state management module, and the Purge operation is performed to recycle invalid data; When the business load is judged to be normal, the triggering conditions of the GC mechanism are controlled according to the thresholds set by the dynamic threshold configuration module. The thresholds include storage space usage and data block lifecycle. The performance monitoring module obtains the amount of data written to the QLC media and calculates the write amplification factor based on the ratio of the amount of data written to the amount of data requested by the application to evaluate the optimization effect.

2. The method according to claim 1, wherein Writing metadata of the RocksDB database into a metadata pool composed of TLC media and writing user data into a data pool composed of QLC media includes: Create multiple RocksDB partitions on TLC media; Write OSS metadata to the corresponding RocksDB partition based on the data pool identifier, and write user data to the data pool of QLC media.

3. The method according to claim 2, wherein The number and capacity of RocksDB partitions on TLC media are dynamically configured based on the number of data pools in the system.

4. The method according to claim 1, wherein The determining whether the current state is a no-service or low-service load state by the intelligent load analysis module includes: Get running status data; Predict business load trends based on operational status data; Based on the prediction results, determine whether the current state is no business or low business load.

5. The method according to claim 1, wherein The GC mechanism is dynamically started and stopped based on the judgment results of the intelligent load analysis module; when the GC mechanism is turned off, a Purge operation is performed to recycle invalid data.

6. A QLC media write amplification optimization device, characterized in that: include: A data classification writing module, configured to write RocksDB database metadata to a metadata pool composed of TLC media and user data to a data pool composed of QLC media in response to a data write request; A service load monitoring module is used to collect operating status data of the storage system, including I / O request rate, CPU usage, and memory usage; An intelligent load analysis module, configured to determine whether the current state is a no-business or low-business load state based on the operation status data; The GC state management module is used to disable the garbage collection (GC) mechanism when there is no business or low business load, and only perform the Purge operation to recycle invalid data; A dynamic threshold configuration module is used to set the trigger threshold of the GC mechanism, which includes storage space usage and data block life cycle; The performance monitoring module is used to obtain the amount of data written to the QLC media and calculate the write amplification factor based on the ratio of the amount of data written to the amount of data requested by the application to evaluate the optimization effect.

7. The device according to claim 6, characterized in that The data classification writing module includes: RocksDB partition creation unit, used to create multiple RocksDB partitions on TLC media; The data write control unit is used to write OSS metadata to the corresponding RocksDB partition according to the data pool identifier, and write user data to the data pool of QLC media.

8. The device according to claim 7, wherein The RocksDB partition creation unit dynamically configures the number and capacity of RocksDB partitions on the TLC medium according to the number of data pools in the system.

9. The device according to claim 6, wherein The intelligent load analysis module includes: Data acquisition interface, used to obtain operating status data; Load forecasting model, used to predict business load trends based on operating status data; The state judgment unit is used to judge whether the current state is no business or low business load based on the prediction result.

10. The device according to claim 6, wherein The GC status management module includes: GC start-stop control unit, used to dynamically start and stop the GC mechanism based on the judgment results of the intelligent load analysis module; The Purge execution unit is used to perform Purge operations to reclaim invalid data when the GC mechanism is disabled.

Images