Garbage collection method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a garbage collection method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of storage, and the method comprises the steps: determining invalid data proportions corresponding to a plurality of flash memory blocks in a key-value solid-state drive through a garbage collection engine in the key-value solid-state drive, determining the input / output load state of the key-value solid-state drive, the key-value solid-state drive being deployed on a storage device in a distributed storage system; and under the condition that the input / output load state is an idle state, performing garbage collection on the plurality of flash memory blocks through the garbage collection engine and the invalid data proportion. Therefore, the problems that the system complexity is increased and the storage performance is reduced due to the fact that a distributed storage system depends on a multi-layer software stack in the aspect of full IO stack GC in the prior art can be solved, and the technical effects of reducing the system complexity and improving the storage performance are achieved.

Description

Technical Field

[0001] This application relates to the field of storage technology, and more particularly to a waste recycling method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the rapid development of data centers and cloud computing, distributed storage systems are facing increasing challenges. Traditional storage systems suffer from degraded storage performance and increased system complexity due to their complex software stacks and inefficient metadata management. In particular, traditional systems require multiple layers of software stack support for full I / O stack garbage collection (GC), resulting in additional overhead and performance bottlenecks.

[0003] The problem that distributed storage systems rely on multiple software stacks for full IO stack garbage collection, leading to increased system complexity and decreased storage performance, has not yet been effectively solved. Summary of the Invention

[0004] This application provides a garbage collection method, apparatus, electronic device, storage medium, and program product to at least solve the problem in the related art that distributed storage systems rely on multiple software stacks for full IO stack GC, leading to increased system complexity and decreased storage performance.

[0005] This application provides a garbage collection method, comprising: determining the proportion of invalid data corresponding to multiple flash memory blocks in the key-value solid-state drive through a garbage collection engine in the key-value solid-state drive, and determining the input / output load state of the key-value solid-state drive, wherein the key-value solid-state drive is deployed on a storage device in a distributed storage system; and performing garbage collection on the multiple flash memory blocks through the garbage collection engine and the proportion of invalid data when the input / output load state is idle.

[0006] This application also provides a garbage collection device, comprising: a determination module, configured to determine the proportion of invalid data corresponding to multiple flash memory blocks in the key-value solid-state drive through a garbage collection engine in the key-value solid-state drive, and to determine the input / output load state of the key-value solid-state drive, wherein the key-value solid-state drive is deployed on a storage device in a distributed storage system; and a garbage collection module, configured to perform garbage collection on the multiple flash memory blocks through the garbage collection engine and the proportion of invalid data when the input / output load state is idle.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described garbage collection methods.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described garbage collection methods.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described garbage collection methods.

[0010] This application utilizes a garbage collection engine within a key-value solid-state drive (SSD) to determine the proportion of invalid data corresponding to multiple flash memory blocks in the SSD and to determine the input / output load state of the SSD, wherein the SSD is deployed on a storage device in a distributed storage system. When the input / output load state is idle, garbage collection is performed on the multiple flash memory blocks using the garbage collection engine and the determined proportion of invalid data. Therefore, this addresses the technical problem in related technologies where distributed storage systems rely on multiple software stacks for full IO stack GC, leading to increased system complexity and decreased storage performance. By using a key-value SSD in the storage device of the distributed storage system, garbage collection of multiple flash memory blocks is achieved through the garbage collection engine within the SSD, thereby reducing system complexity and improving storage performance. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a hardware structure block diagram of a computer terminal for a garbage collection method according to an embodiment of this application;

[0013] Figure 2 This is a flowchart of a waste recycling method according to an embodiment of this application;

[0014] Figure 3 This is a diagram of a traditional distributed storage architecture;

[0015] Figure 4 This is a diagram of a KV-SSD-based storage system architecture according to an embodiment of this application;

[0016] Figure 5 This is a structural block diagram of a waste recycling device according to an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0018] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0019] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] The specific application environment architecture or specific hardware architecture on which the garbage collection method depends is described here.

[0021] The methods and embodiments provided in this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a garbage collection method according to an embodiment of this application. Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0022] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the garbage collection method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0023] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0024] Figure 2 This is a flowchart of a garbage collection method according to an embodiment of this application, applied to a storage device in a distributed storage system. Figure 2 As shown, the process includes the following steps:

[0025] Step S202: Determine the proportion of invalid data corresponding to each of the multiple flash blocks in the key-value solid-state drive through the garbage collection engine in the key-value solid-state drive, and determine the input / output load state of the key-value solid-state drive, wherein the key-value solid-state drive is deployed on the storage device in the distributed storage system;

[0026] Among them, key-value solid state drives (KV-SSDs) are also known as key-value solid state drives.

[0027] Step S204: When the input / output load state is idle, garbage collection is performed on the plurality of flash memory blocks using the garbage collection engine and the invalid data ratio.

[0028] Through the above steps, the garbage collection engine in the key-value solid-state drive (SSD) determines the proportion of invalid data corresponding to each of the multiple flash memory blocks in the SSD, and determines the input / output load state of the SSD, wherein the key-value SSD is deployed on a storage device in a distributed storage system. When the input / output load state is idle, garbage collection is performed on the multiple flash memory blocks using the garbage collection engine and the proportion of invalid data. Therefore, this solves the technical problem in related technologies where distributed storage systems rely on multiple software stacks for full IO stack GC, leading to increased system complexity and decreased storage performance. By using a key-value SSD in the storage device of the distributed storage system, garbage collection of multiple flash memory blocks can be achieved through the garbage collection engine in the key-value SSD, thereby reducing system complexity and improving storage performance.

[0029] The embodiments of this application provide a waste recycling method, and the method is described in detail in conjunction with the execution flow of the waste recycling method.

[0030] In an exemplary embodiment, garbage collection of the plurality of flash memory blocks is performed using the garbage collection engine and the invalid data ratio, including: determining a target flash memory block among the plurality of flash memory blocks whose invalid data ratio is higher than a preset ratio, and performing garbage collection on the target flash memory block using the garbage collection engine, wherein the target flash memory block after garbage collection is marked as overwhelmable.

[0031] Furthermore, garbage collection of the target flash memory block is performed by the garbage collection engine, including: determining the key access frequency of valid data in the target flash memory block; determining the flash memory block storing cold data among the plurality of flash memory blocks based on the key access frequency, wherein the key access frequency of the cold data is lower than a first threshold, and the proportion of cold data to valid data in the flash memory block storing cold data is higher than a second threshold; migrating the valid data in the target flash memory block from the flash memory block storing cold data to other flash memory blocks among the plurality of flash memory blocks to achieve garbage collection of the target flash memory block.

[0032] In a KV-SSD storage system, the default ratio is 30%. This means that when the proportion of invalid data in a flash block exceeds 30%, the block will be considered a target flash block and will need to be garbage collected.

[0033] Before garbage collection, the system further analyzes the access frequency of data key-value pairs in flash memory block A. Assuming the proportion of cold data to valid data in block A is higher than a second threshold, block A is considered a block with a high proportion of cold data and suitable for garbage collection. Block B (located on the same or a different storage device) is then designated as a relatively free block to receive valid data from block A. For example, 30 cold data key-value pairs in block A are migrated to block B, freeing up space in block A, while valid hot data from block A is migrated to other hot data blocks to maintain the continuity and efficiency of hot data distribution. After garbage collection, block A is marked as "overwritable," meaning it can be written to with new data without causing additional write amplification burden on the system. By centrally migrating cold data to relatively free blocks while preserving the continuous distribution of hot data, the system can significantly optimize data distribution, reduce the total number of data migrations, and improve the access speed of hot data.

[0034] In summary, the above technical process, through a refined data management strategy, effectively improves the storage efficiency and performance of KV-SSD-based distributed storage systems, while also helping to extend the lifespan of the devices. This strategy is particularly important in large-scale data storage and processing scenarios, significantly improving the overall reliability and cost-effectiveness of the system.

[0035] In one exemplary embodiment, determining the proportion of invalid data corresponding to each of the multiple flash memory blocks in the key-value solid-state drive through a garbage collection engine in the key-value solid-state drive includes: determining a first amount of data marked as invalid data in each flash memory block through the garbage collection engine, and determining a second amount of data in all data in each flash memory block; and determining the ratio of the first amount of data to the second amount of data as the proportion of invalid data.

[0036] In one exemplary embodiment, determining the input / output load status of the key-value solid-state drive includes: collecting input / output metrics of the key-value solid-state drive, wherein the input / output metrics include: input / output request queue length, input / output throughput, and cache hit rate; and determining the input / output load status through the input / output metrics.

[0037] Furthermore, determining the input / output load state through the input / output metrics includes: determining the magnitude relationship between each of the multiple metrics in the input / output metrics and the corresponding metric threshold; and determining that the input / output load state is idle when the magnitude relationship determines that all of the multiple metrics are less than the respective metric thresholds.

[0038] Determining the input / output (I / O) load status of a key-value solid-state drive (KV-SSD) is to optimize the timing of garbage collection (GC) operations and ensure that the GC process does not affect normal I / O performance.

[0039] Specifically, the KV-SSD firmware continuously monitors the following information to obtain input and output metrics:

[0040] 1) I / O request queue: This is a buffer that receives read and write commands from the host. The number and type (read / write) of requests in the queue indicate the current I / O load status.

[0041] 2) I / O operation throughput and latency data are key indicators of device activity. High throughput and / or long latency indicate that the device is handling a large number of requests and is under heavy load.

[0042] 3) KV-SSDs may include a processor (CPU) and cache, and the utilization of these components is also an important factor in assessing the overall load. High CPU utilization may mean that the device is performing computationally intensive tasks (such as encryption, compression, or data processing). Low cache hit rate may indicate that a large amount of data demand cannot be met from the cache, increasing the burden on the main NAND storage.

[0043] 4) Current I / O patterns, such as the ratio of random I / O to sequential I / O, and the ratio of small I / O to large I / O. Different types of I / O operations have different impacts on device load; random small I / O operations generally have a greater impact on performance.

[0044] All the monitored data are combined to form a quantitative or categorized I / O load status. When the device enters an I / O idle state, i.e., the request queue length is low, throughput and latency are within reasonable ranges, and CPU and cache utilization are not high, the firmware will consider it an appropriate time to perform GC operations.

[0045] The GC strategy in the KV-SSD firmware is adaptive, dynamically adjusting the GC initiation and execution strategy based on the current I / O load. If the device is detected to be experiencing high load, GC operations may be postponed until the load decreases. Conversely, if the load is low and persists for a period of time, GC operations will be initiated to free up more storage space and optimize the use of NAND cells.

[0046] For example: Suppose at a specific moment, KV-SSD monitors the following state:

[0047] There are 5 requests waiting to be processed in the I / O request queue (historical average is 20); the current I / O throughput is 100MB / s (up to 200MB / s when the device is fully loaded); the CPU utilization is 15% (historical average is 50%); the cache hit rate is 90% (historical average is 75%); most of the ongoing I / O operations are sequential reads, with no write or update operations.

[0048] Based on the above metrics, the KV-SSD firmware determines that the current I / O load is idle and automatically triggers the GC process to clean up invalid data and reorganize storage space, thereby improving storage efficiency and device performance while avoiding impact on currently low-load business operations. In this way, the firmware can dynamically adapt to the real-time needs of the storage system, ensuring that GC operations are performed at the most appropriate time, without interfering with normal data access, and effectively optimizing storage resources.

[0049] In an exemplary embodiment, before determining the proportion of invalid data corresponding to each of the multiple flash blocks in the key-value solid-state drive by the garbage collection engine in the key-value solid-state drive, and before determining the input / output load state of the key-value solid-state drive, the method further includes: storing the received data into the multiple flash blocks using a preset method, wherein the preset method includes one of the following: naming method and hash method; wherein storing the data using the naming method includes: splitting the data according to a preset length, and taking the remainder of the ratio of the offset of the split data to the preset length to obtain a data identifier for the data; adding the scenario corresponding to the data before the data identifier, so as to store the data using the added data identifier; wherein storing the data using the hash method includes: splitting the data according to a preset length, and calculating the hash value of the split data using a hash algorithm, using the obtained hash value as the data identifier for the data; recording the correspondence between the sequence number of the split data and the data identifier, and storing the data using the data identifier.

[0050] It should be noted that the data to be stored and the corresponding metadata received by the key-value solid-state drive can both be stored using the naming and hashing methods described above.

[0051] In summary, by employing naming or hashing methods to store data, and utilizing intelligent garbage collection strategies based on the proportion of invalid data and I / O load status, KV-SSD-based distributed storage systems can provide more efficient, reliable, and flexible data storage and access services. This not only optimizes the utilization of storage resources but also significantly improves system performance and stability, especially in high-concurrency, high-volume cloud computing and data center environments, helping to address key challenges in data storage and access.

[0052] To better understand the process of the above-mentioned waste recycling method, the implementation flow of the above-mentioned waste recycling method will be described below in conjunction with optional embodiments, but this is not intended to limit the technical solution of the embodiments of this application.

[0053] With the rapid development of data centers and cloud computing, distributed storage systems face increasing challenges. Traditional storage systems, due to their complex software stacks and inefficient metadata management, suffer from degraded storage performance and increased system complexity. Particularly in full-IO stack garbage collection (GC), traditional systems require multi-layered software stack support, leading to additional overhead and performance bottlenecks. Simultaneously, traditional storage systems also suffer from inefficient and unreliable metadata management. Therefore, a new storage system is urgently needed to address these issues.

[0054] In recent years, key-value solid-state drives (KV-SSDs), as a new type of storage device, have gradually become a hot research topic in storage systems due to their high performance, low latency, and efficient metadata management capabilities. However, how to fully utilize the advantages of KV-SSDs to achieve full IO stack GC optimization and simplified metadata management remains an urgent problem to be solved.

[0055] Existing distributed storage systems primarily rely on traditional hard disk storage or solid-state drive (SSD) storage. While these storage devices offer advantages in storage capacity and cost, they suffer from significant shortcomings in performance and latency. Particularly in full I / O stack garbage collection (GC), traditional storage systems require multiple layers of software stacks, leading to increased system complexity and decreased performance. Specifically, for example... Figure 3 As shown, distributed storage systems expand storage capacity and improve performance by distributing data across multiple independent devices. However, traditional distributed storage systems typically rely on a multi-layered software stack, including file systems and key-value database software, when implementing key-value storage. This not only increases system complexity but also introduces additional resource overhead and performance bottlenecks.

[0056] Meanwhile, existing metadata management systems also suffer from inefficiency and poor reliability. Traditional metadata management systems typically rely on multiple components, including metadata servers, metadata storage, and metadata caching, leading to increased system complexity and decreased performance.

[0057] To address the aforementioned shortcomings, this application proposes an optional embodiment of a full-IO-stack GC-optimized distributed storage system based on key-value solid-state drives (KV-SSDs) (allowing the execution of the aforementioned garbage collection method).

[0058] This system fully leverages the advantages of KV-SSD in full-stack garbage collection (GC) and metadata management, achieving a significant improvement in storage performance and a reduction in system complexity. By implementing a full-stack GC mechanism directly on the KV-SSD, the additional overhead and performance bottlenecks caused by multi-layered software stacks in traditional storage systems are avoided. Furthermore, combined with the efficient metadata management capabilities of KV-SSD, the optional embodiments of this application further simplify the metadata system and improve system reliability and scalability.

[0059] Key-Value SSDs (KV-SSDs) are a new type of storage medium that provides a key-value interface directly on the SSD hardware (NVMe 2.0 and later versions), thus simplifying the storage system's software stack. Compared to traditional SSDs, KV-SSDs offer the following advantages: 1) Full I / O stack GC optimization: KV-SSDs implement garbage collection mechanisms at the hardware level, avoiding the GC overhead caused by multi-layered software stacks in traditional storage systems. 2) Efficient metadata management: KV-SSDs have built-in efficient metadata management capabilities, significantly reducing the complexity and resource consumption of the metadata system.

[0060] Optionally, in an optional embodiment of this application, a full I / O stack GC mechanism is implemented directly on the KV-SSD, avoiding the additional overhead and performance bottlenecks caused by multi-layered software stacks in traditional storage systems. This directly implemented GC mechanism can significantly reduce GC latency and overhead, improving the performance and reliability of the storage system.

[0061] The core of implementing a full IO stack GC mechanism on KV-SSDs lies in sinking garbage collection operations to the device firmware layer, eliminating the multi-layered redundancy of traditional storage stacks through native hardware support for the KV interface. The specific implementation includes the following key technical points:

[0062] 1) Hardware-level GC triggering mechanism: The KV-SSD firmware monitors the proportion of invalid data and I / O load status of NAND blocks (equivalent to flash memory blocks in the above embodiment) in real time, and automatically triggers GC during device idle periods. Compared with traditional SSDs that rely on the host file system to trigger GC, this reduces the overhead of data copying from kernel mode to user mode and context switching.

[0063] 2) Hot and Cold Data Separation and Recycling: KV-SSD prioritizes the recycling of cold data blocks and preserves the continuous distribution of hot data by analyzing key access frequency. Compared to the traditional SSD's global flash translation layer (FTL), which requires a full disk scan of metadata, this solution reduces the effective data migration during the GC process by up to 60%.

[0064] Optionally, the optional embodiments of this application combine the efficient metadata management capabilities of KV-SSD to further simplify the metadata system. Traditional metadata management systems typically rely on multiple components, including metadata servers, metadata storage, and metadata caches, while the optional embodiments of this application can directly store and manage metadata on KV-SSD, reducing system complexity and overhead.

[0065] Optionally, in an optional embodiment of this application, the LSM tree and FTL are optimized collaboratively. By deeply integrating the compaction process of the LSM tree with the block reclamation algorithm of the FTL, KV-SSD directly cleans up invalid key-value pairs at the physical layer, avoiding the three-layer write amplification of document database → file system → FTL in traditional systems. Experiments show that the write amplification factor of this scheme can be reduced to 1.5 times, while the traditional architecture typically reaches 3-5 times.

[0066] Specifically, Figure 4 This is a storage system architecture diagram based on KV-SSD according to an embodiment of this application, compared to Figure 3 , Figure 4 By using a KV-SSD hardware KV interface, the original system's pure software database KV interface is taken over for metadata management. This avoids the huge CPU and latency overhead caused by a complex database software stack. Consequently, due to the reduction in software layers and the opening of the database black box, data migration and replication throughout the entire software stack become visible. Through the aforementioned GC mechanism and hot / cold data separation and reclamation mechanism, the goal of significantly reducing GC data migration is achieved, thereby improving disk lifespan and performance.

[0067] It's also worth noting that KV-SSDs have a built-in garbage collection engine that monitors data usage on the storage medium and automatically reclaims invalid data blocks. When the proportion of invalid data blocks on the storage medium reaches a certain level, the garbage collection engine initiates the reclamation process, marking the invalid data blocks as overwriteable and releasing the corresponding storage space. Because the garbage collection process is implemented at the KV-SSD hardware level, it does not require multiple layers of software stack transformation and processing, thus significantly improving the efficiency and performance of garbage collection.

[0068] Furthermore, optional embodiments of this application also provide a method for generating key-value pairs based on KV-SSD. In optional embodiments of this application, data is stored on the KV-SSD in the form of key-value pairs. The key is the data name (dataID), and the value is the actual data content. To ensure that the key and value meet the KV-SSD interface specifications, optional embodiments of this application employ the following two commonly used key-value pair generation methods:

[0069] Naming convention: Data is divided into segments of a fixed length, such as 4MB. The data ID is the remainder of the data's offset divided by 4MB. In practical applications, a marker can be added before the ID to further distinguish different data based on the specific scenario. For example, in file or object scenarios, the Inode number + offset % 4MB can be used as the data ID; in block scenarios, the LUN ID + offset % 4MB can be used as the data ID.

[0070] Hash method: This method divides data into fixed-length blocks, such as 4MB, and uses a hash algorithm (such as MD5 or SHA) to calculate the hash value, which is then used as the data ID. It's important to note that this method also requires calculating the data block's sequence number (data index) and recording the mapping between data IDs and data indices. The advantage of this method is that data blocks with identical content will have the same calculated data ID, thus providing deduplication capabilities. However, the disadvantage is the significant amount of computation required and the additional need to maintain the mapping between data indices and data IDs.

[0071] Therefore, in an optional embodiment of this application, metadata can be stored on a KV-SSD in the form of key-value pairs. The key is the name or identifier of the metadata (such as file ID, object ID, etc.), and the value is the specific content of the metadata (such as file size, modification time, permission information, etc.). Because KV-SSD provides an efficient key-value interface and indexing mechanism, we can quickly retrieve and access metadata without maintaining complex data structures and index relationships. Furthermore, KV-SSD has a built-in data consistency verification and recovery mechanism to ensure the reliability and integrity of the metadata.

[0072] Therefore, in the optional embodiments of this application, the garbage collection method based on KV-SSD can collect both data and metadata.

[0073] In summary, the optional embodiments of this application implement a full IO stack GC mechanism directly on the KV-SSD, avoiding the additional overhead and performance bottlenecks caused by multi-layered software stacks in traditional storage systems. Combined with the efficient metadata management capabilities of KV-SSD, the metadata system is further simplified, reducing system complexity and overhead.

[0074] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0075] This embodiment also provides a waste recycling device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0076] Figure 5 This is a structural block diagram of a waste recycling device according to an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0077] The determination module 52 is used to determine the proportion of invalid data corresponding to each of the multiple flash blocks in the key-value solid-state drive through the garbage collection engine in the key-value solid-state drive, and to determine the input and output load status of the key-value solid-state drive, wherein the key-value solid-state drive is deployed on a storage device in a distributed storage system.

[0078] The garbage collection module 54 is used to perform garbage collection on the plurality of flash memory blocks by means of the garbage collection engine and the invalid data ratio when the input / output load state is idle.

[0079] The aforementioned device determines the proportion of invalid data corresponding to multiple flash memory blocks in the key-value solid-state drive (SSD) through its garbage collection engine, and also determines the input / output load state of the SSD, which is deployed on a storage device in a distributed storage system. When the input / output load state is idle, garbage collection is performed on the multiple flash memory blocks using the garbage collection engine and the determined proportion of invalid data. Therefore, this addresses the technical problem in related technologies where distributed storage systems rely on multiple software stacks for full I / O stack garbage collection, leading to increased system complexity and decreased storage performance. By using a key-value solid-state drive in the storage device of the distributed storage system, garbage collection of multiple flash memory blocks can be achieved through the garbage collection engine within the key-value solid-state drive, thereby reducing system complexity and improving storage performance.

[0080] In an exemplary embodiment, the garbage collection module is further configured to: determine a target flash block in the plurality of flash blocks where the proportion of invalid data is higher than a preset proportion, and perform garbage collection on the target flash block through the garbage collection engine, wherein the target flash block after garbage collection is marked as overwhelmable.

[0081] In one exemplary embodiment, the garbage collection module is further configured to: determine the key access frequency of valid data in a target flash memory block; determine a flash memory block storing cold data among the plurality of flash memory blocks based on the key access frequency, wherein the key access frequency of the cold data is lower than a first threshold, and the proportion of cold data in the flash memory block storing cold data to valid data is higher than a second threshold; migrate the valid data in the target flash memory block from the flash memory block storing cold data to other flash memory blocks among the plurality of flash memory blocks to achieve garbage collection of the target flash memory block.

[0082] In one exemplary embodiment, the determining module is further configured to: determine a first amount of data marked as invalid data in each flash memory block by means of the garbage collection engine, and determine a second amount of data in all data in each flash memory block; and determine the ratio of the first amount of data to the second amount of data as the invalid data ratio.

[0083] In an exemplary embodiment, the determining module is further configured to: collect input / output metrics of the key-value solid-state drive, wherein the input / output metrics include: input / output request queue length, input / output throughput, and cache hit rate; and determine the input / output load status through the input / output metrics.

[0084] In an exemplary embodiment, the determining module is further configured to: determine the magnitude relationship between each of the plurality of indicators in the input / output indicators and the indicator threshold corresponding to each indicator; and determine that the input / output load state is an idle state when the magnitude relationship determines that all of the plurality of indicators are less than the indicator thresholds corresponding to the plurality of indicators respectively.

[0085] In one exemplary embodiment, the apparatus further includes a storage module, configured to, before determining the proportion of invalid data corresponding to each of the plurality of flash memory blocks in the key-value solid-state drive by the garbage collection engine in the key-value solid-state drive, and before determining the input / output load state of the key-value solid-state drive, store the received data into the plurality of flash memory blocks using a preset method, wherein the preset method includes one of the following: a naming method and a hash method; wherein storing the data using the naming method includes: dividing the data according to a preset length, and taking the remainder of the ratio of the offset of the divided data to the preset length to obtain a data identifier for the data; adding the scene corresponding to the data before the data identifier, so as to store the data using the added data identifier; wherein storing the data using the hash method includes: dividing the data according to a preset length, and calculating the hash value of the divided data using a hash algorithm, using the obtained hash value as the data identifier for the data; recording the correspondence between the sequence number of the split data and the data identifier, and storing the data using the data identifier.

[0086] For a description of the features in the embodiment corresponding to the waste recycling device, please refer to the relevant description of the embodiment corresponding to the waste recycling method, which will not be repeated here.

[0087] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described waste recycling method embodiments.

[0088] Embodiments of this application also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the above-described garbage collection method embodiments at runtime.

[0089] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0090] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described garbage collection method embodiments.

[0091] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described garbage collection method embodiments.

[0092] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0093] The above provides a detailed description of a waste recycling method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A waste recycling method, characterized in that, include: The proportion of invalid data corresponding to each of the multiple flash blocks in the key-value solid-state drive is determined by the garbage collection engine in the key-value solid-state drive, and the input / output load status of the key-value solid-state drive is determined, wherein the key-value solid-state drive is deployed on a storage device in a distributed storage system. When the input / output load state is idle, garbage collection is performed on the plurality of flash memory blocks using the garbage collection engine and the invalid data ratio.

2. The waste recycling method according to claim 1, characterized in that, Garbage collection of the plurality of flash memory blocks is performed using the garbage collection engine and the invalid data ratio, including: A target flash block with an invalid data ratio higher than a preset ratio is identified among the plurality of flash blocks, and garbage collection is performed on the target flash block by the garbage collection engine. The target flash block after garbage collection is marked as overwriteable.

3. The waste recycling method according to claim 2, characterized in that, Garbage collection of the target flash memory block is performed using the garbage collection engine, including: Determine the key access frequency of valid data in the target flash memory block; The flash memory blocks storing cold data in the plurality of flash memory blocks are determined by the key access frequency, wherein the key access frequency of the cold data is lower than a first threshold, and the proportion of cold data to valid data in the flash memory blocks storing cold data is higher than a second threshold. The valid data in the target flash block of the flash memory block storing cold data is migrated to other flash memory blocks in the plurality of flash memory blocks to achieve garbage collection of the target flash memory block.

4. The waste recycling method according to claim 1, characterized in that, The proportion of invalid data corresponding to each of the multiple flash blocks in the key-value solid-state drive is determined by the garbage collection engine in the key-value solid-state drive, including: The garbage collection engine determines a first amount of data marked as invalid in each flash memory block and a second amount of data in all data within each flash memory block. The ratio of the first data volume to the second data volume is determined as the invalid data ratio.

5. The waste recycling method according to claim 1, characterized in that, Determining the input / output load state of the key-value solid-state driver includes: The input / output metrics of the key-value solid-state drive are collected, wherein the input / output metrics include: input / output request queue length, input / output throughput, and cache hit rate; The input / output load status is determined by the input / output metrics.

6. The waste recycling method according to claim 4, characterized in that, Determining the input / output load status using the input / output metrics includes: Determine the magnitude relationship between each of the multiple indicators in the input and output indicators and the corresponding indicator threshold of each indicator; If, based on the magnitude relationship, it is determined that all of the multiple indicators are less than the respective threshold values ​​of the multiple indicators, the input / output load state is determined to be an idle state.

7. The waste recycling method according to claim 1, characterized in that, Before determining the proportion of invalid data corresponding to each of the multiple flash blocks in the key-value solid-state drive through the garbage collection engine in the key-value solid-state drive, and before determining the input / output load state of the key-value solid-state drive, the method further includes: The received data is stored in the plurality of flash memory blocks using a preset method, wherein the preset method includes one of the following: naming method or hash method; Storing the data by naming includes: The data is divided into segments according to a preset length, and the remainder of the ratio of the offset of the segmented data to the preset length is taken to obtain the data identifier of the data. Add the scenario corresponding to the data before the data identifier, so that the data can be stored using the added data identifier; Storing the data using a hash method includes: The data is divided into segments according to a preset length, and the segmented data is calculated using a hash algorithm. The resulting hash value is used as the data identifier of the data. Record the correspondence between the sequence number of the split data and the data identifier, and store the data through the data identifier.

8. A waste recycling device, characterized in that, include: The determination module is used to determine the proportion of invalid data corresponding to multiple flash blocks in the key-value solid-state drive through the garbage collection engine in the key-value solid-state drive, and to determine the input and output load status of the key-value solid-state drive, wherein the key-value solid-state drive is deployed on a storage device in a distributed storage system. The garbage collection module is used to perform garbage collection on the plurality of flash memory blocks by means of the garbage collection engine and the invalid data ratio when the input / output load state is idle.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the garbage collection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the garbage collection method as described in any one of claims 1 to 7.

Images