Method and system for garbage collection and / or compression for key value engine

Abstract

Methods and systems for garbage collection and compression for key value engines in data storage and communication systems are described herein. The method includes determining a disk capacity usage of the key value engine, and adjusting a garbage collection percentage threshold and a number of garbage collection threads based on whether the disk capacity usage of the key value engine satisfies and / or exceeds a predetermined disk capacity usage threshold. The method may further include performing a periodic compression process to reduce data handling during garbage collection occurrences by merging the one or more layers into a last layer of the one or more expired pages of the one or more applications to integrate the one or more expired pages on a log structure merging (LSM) tree.

Description

Technical Field

[0001] Embodiments described herein generally relate to non-volatile storage systems in data storage and communication systems. More specifically, embodiments described herein relate to methods and systems for optimizing garbage collection and / or compression of a key-value engine in a data storage and communication system. Background Art

[0002] Non-volatile storage, such as block-level storage, is a common form of storage used to store data on a storage area network (e.g., a server), in a cloud-based storage environment, at a data center, in a server system, etc. Cloud-based storage devices or storage appliances are common products provided by cloud service providers (CSPs). In some cloud-based storage devices or storage appliances, a key-value engine can be used to meet the persistence requirements of cloud services. Although a key-value engine can be used as a storage backend for cloud services, previous key-value engines may introduce overhead for ingesting data, such as write amplification factor (WAF) and tail latency. Summary of the invention

[0003] The methods and systems described herein are intended to optimize garbage collection and / or compaction processes in a key-value engine system, which overcome one or more of the above-mentioned deficiencies of previous key-value engines. In some embodiments, as described herein, the key-value engine is designed, programmed, or otherwise configured to provide one or more of the following features: awareness of application-side data deletion, including expired pages, such as communication and / or collaboration between the key-value engine and the file system; flexible and diverse garbage collection processes; periodic compaction capabilities; addressing read amplification issues; and / or identifying and separating hot data and cold data, such as including one or more of compression, rewriting, garbage collection, etc. Thus, as described herein, the key-value engine is configured to actively reclaim storage space without blocking foreground client threads, such as by having capacity awareness and being able to dynamically adjust the parallelism level of the garbage collection thread, such as dynamically adjusting the garbage collection thread and the garbage percentage threshold, and providing awareness or indication of application-side data deletion, including expired pages at the key-value store, such as the underlying database storing (in memory) the keys, values, etc. of the key-value engine (such as, PageStore) to the file system. Thus, a key-value engine system including key-value separation is provided that is more convenient for garbage collection, for example, by reducing unnecessary data movement during garbage collection and compaction.

[0004] In an exemplary embodiment, a method for garbage collection and compaction of a key-value engine in a data storage and communication system is provided. The method includes determining disk capacity usage of the key-value engine, and adjusting a garbage collection percentage threshold and a number of garbage collection threads based on whether the disk capacity usage of the key-value engine meets and / or exceeds a predetermined disk capacity usage threshold.

[0005] In an embodiment, the method may also include performing a periodic compaction process to consolidate one or more expired pages on a log structured merge (LSM) tree by merging one or more layers into a last layer of one or more expired pages of one or more applications, thereby reducing data disposal during garbage collection.

[0006] In another embodiment, an active garbage collection system for a key-value engine is provided. The active garbage collection system includes an adaptive garbage collection module for implementing a flexible and adaptive garbage collection strategy, wherein the adaptive garbage collection module is configured to determine a disk capacity usage of the key-value engine, and adjust a garbage collection percentage threshold and a number of garbage collection threads based on whether the disk capacity usage of the key-value engine meets and / or exceeds a predetermined disk capacity usage threshold.

[0007] In yet another embodiment, a non-volatile storage device is provided, on which an executable component is stored. The non-volatile storage device on which the executable component is stored, when executed by a processor, causes the processor to: determine the disk capacity usage of the key-value engine; and adjust the garbage collection percentage threshold and the number of garbage collection threads based on whether the disk capacity usage of the key-value engine meets and / or exceeds a predetermined disk capacity usage threshold.

[0008] Therefore, the methods and systems discussed herein may have one or more of the following advantages:

[0009] Accurate garbage collection management - In some embodiments, the key-value engine can be designed, programmed, or otherwise configured to consider garbage rates at the entry level rather than the space level, for example, to select blob files with the highest garbage ratio / percentage to address the problem of accurate garbage collection scheduling. By designing, programming, or otherwise configuring the key-value engine to consider garbage rates at the entry level, blob files with the highest garbage space for garbage collection can be selected to achieve more efficient garbage collection operations. In addition, the key-value engine can be enhanced by having garbage collection awareness, for example, by being aware of application-side data deletions communicated to the file system, including expired pages, for example, via a kernel file interface, an application programming interface, etc., to reduce unnecessary data movement. In addition, a periodic compaction process can be used to clean up expired pages and further reduce unnecessary data movement during the garbage collection process.

[0010] Adaptive Garbage Collection - In some embodiments, the key-value engine can be designed, programmed, or otherwise configured so that the garbage collection process is flexible, diverse, concurrent, and aware of the physical capacity of the storage device. For example, in some embodiments, the key-value engine can be designed, programmed, or otherwise configured to consider a wider range of garbage collection strategies, such as improving concurrency when in high water levels or areas, and optimizing space reclamation. In this way, the overall garbage collection performance of the key-value engine can be significantly enhanced compared to previous key-value engines, whose garbage collection strategies result in premature garbage collection, which can cause additional write amplification issues, insufficient concurrency during high water levels, and / or slow space reclamation.

[0011] Periodic compaction management - In some embodiments, the key-value engine can be designed, programmed, or otherwise configured to provide periodic compaction functionality, for example, for cleaning up expired pages. In some embodiments, periodic compaction can include merging multiple layers of the same page (or expired page) in a log structure merge (LSM) tree into the last layer so that the upper layers can perform corresponding functions during the compaction process. By doing so, the amount of data affected by the garbage collection process can be reduced. In addition, the key-value engine can be designed, programmed, or otherwise configured to prioritize the operations of garbage collection and / or compaction, for example, a higher priority process will be initiated before a lower priority process.

[0012] Garbage Collection Read Amplification Optimization - In some embodiments, the key-value engine can be designed, programmed, or otherwise configured so as to reduce the read amplification that occurs during garbage collection itself. For example, in some embodiments, redundant key sets can be stored at the end of a sorted string table (SST).

[0013] Optimization of hot and cold data - In some embodiments, the key-value engine can be designed, programmed, or otherwise configured to identify and separate hot and cold data to reduce write amplification, especially for mixed data scenarios where when hot data becomes invalid, rewriting cold data becomes necessary to reclaim space, which can result in additional WAF. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings illustrate various embodiments of the systems, methods and embodiments of various other aspects of the present disclosure. It will be appreciated by any person of ordinary skill in the art that the element boundaries (e.g., boxes, box groups or other shapes) shown in the figures represent an example of boundaries. In some examples, an element may be designed as multiple elements, or multiple elements may be designed as one element. In some examples, an element shown as an internal component of an element may be implemented as an external component in another element, and vice versa. Non-restrictive and non-exhaustive descriptions are described with reference to the following drawings. The components in the figures are not necessarily drawn to scale, but emphasize the explanation principle. In the detailed description below, the embodiments are described only as illustrations, because various changes and modifications are apparent to those skilled in the art according to the detailed description below.

[0015] Figure 1 is a schematic diagram of an exemplary cloud-based storage system arranged in accordance with at least some embodiments described herein.

[0016] Figure 2 are exemplary embodiments of a key-value store and file system arranged in accordance with at least some embodiments described herein.

[0017] Figure 3 is a schematic representation of an active garbage collection module arranged in accordance with at least some embodiments described herein.

[0018] Figure 4A and Figure 4B is a representation of a separate index for a key-value engine arranged in accordance with at least some embodiments described herein.

[0019] Figure 4C are illustrative examples for determining and / or calculating a junk ratio or junk rate for a blob file in accordance with at least some embodiments described herein.

[0020] Figure 5A and Figure 5B is an illustrative example of operation or process flow of an adaptive garbage collection manager in accordance with at least some embodiments described herein.

[0021] Fig. 6A and Figure 6B is a diagram of an exemplary embodiment of compression operations of a periodic compression manager in accordance with at least some embodiments described herein.

[0022] Figure 7 is an exemplary embodiment of a process flow for scheduling periodic compression in accordance with at least some embodiments described herein.

[0023] Figure 8is a schematic block diagram of an exemplary computer system that may be used to implement the apparatus, arranged in accordance with at least some embodiments described herein. DETAILED DESCRIPTION

[0024] In the following detailed description, specific embodiments of the present disclosure are described herein with reference to the accompanying drawings that form a part of the description. In this description and in the accompanying drawings, similar reference numerals represent elements that can perform the same, similar or equivalent functions, unless the context otherwise dictates. In addition, unless otherwise specified, the description of each consecutive figure may refer to the features of one or more figures from the previous figures to provide a clearer context and a more substantial explanation of the current exemplary embodiment. Nevertheless, the exemplary embodiments described in the detailed description, the accompanying drawings, and the claims are not intended to be restrictive. Other embodiments may be utilized, and other changes may be made without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that, as shown in the overall description and the accompanying drawings herein, the various aspects of the present disclosure may be arranged, replaced, combined, separated, and designed in a variety of different configurations, all of which are clearly contemplated herein.

[0025] It should be understood that the disclosed embodiments are merely examples of the present disclosure, which may be embodied in various forms. Well-known functions or configurations are not described in detail to avoid obscuring the present disclosure in unnecessary detail. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but only as a basis for the claims, and as a representative basis for teaching those skilled in the art to adopt the present disclosure in various ways in virtually any appropriately detailed structure.

[0026] In addition, the present disclosure may be described herein in terms of functional block components and various processing steps. It should be understood that such functional blocks may be implemented by any number of hardware and / or software components configured to perform the specified functions.

[0027] The scope of the present disclosure should be determined by the appended claims and their legal equivalents, rather than by the examples given herein. For example, the steps recited in any method claim may be performed in any order and are not limited to the order presented in the claims. Furthermore, unless specifically described herein as "critical" or "essential", no element is essential to the practice of the present disclosure.

[0028] As described herein, a "key-value engine" may refer to a network, an algorithm, a program, software, hardware, firmware, or any combination thereof that is configured to or otherwise provides a key-value store for meeting the consistency requirements of a cloud service provider. The key-value engine may be based on a log structure merge (LSM) tree architecture, in which key values ​​are separated to reduce write amplification and / or resource contention. The key-value engine may be configured to store data in a key-value store (KVSe) in memory. In some embodiments, the LSM tree may be used to achieve high-speed writes by buffering all updates in an in-memory structure (e.g., MemTable). When the MemTable is full, the data may be flushed to a permanent storage device in a sorted string table (SST), where the SST is immutable. In some embodiments, in order to optimize read operations and reduce space usage, compression may be used to merge several SSTs into one to reduce overlap, for example, due to overlapping key ranges in multiple SSTs.

[0029] As described herein, "blob file" is a term and may refer to an object storage solution for a cloud service, which may be a binary file used to store data outside of an LSM tree, e.g., storing values ​​in a separate key-value system. An SST entry may include a key or pointer to one or more blob files, e.g., a blob file may be shared between one or more SST entries.

[0030] As described herein, a "module" may refer to a network, an algorithm, a program, software, hardware, or any combination thereof that is configured or otherwise provided to perform the recited function.

[0031] Although the key-value separation process can be used to reduce unnecessary data movement during compaction and garbage collection, previous key-value engines may introduce overhead for ingesting data, such as large amounts of unnecessary data movement during garbage collection and / or compaction. For example, previous key-value engines may lack access to internal application information, such as application-side data deletion, which may lead to inefficient space usage. For example, the key-value engine may not be configured to communicate and / or cooperate with the file system.

[0032] The methods and systems described herein are intended to optimize the garbage collection and / or compression process in a key-value engine system, which overcomes one or more of the above-mentioned deficiencies of previous key-value engines. In some embodiments, as described herein, the key-value engine is designed, programmed, or otherwise configured to provide one or more of the following features: perception of application-side data deletion, including expired pages, such as communication and / or collaboration between the key-value engine and the file system; flexible and diverse garbage collection processes; periodic compression functions; solving read amplification problems; and / or identifying and isolating hot data and cold data, for example, including compression, rewriting, garbage collection, etc. In this way, as described herein, the key-value engine is configured to actively reclaim storage space without hindering foreground client threads, for example, by having capacity awareness and being able to dynamically adjust the parallelism level of the garbage collection thread, for example, dynamically adjusting the garbage collection thread and the garbage percentage threshold, and providing the file system with perception or indication of application-side data deletion (including expired pages at PageStore). Therefore, a key-value engine system including key-value separation is provided, which is more convenient for garbage collection, for example, by reducing unnecessary data movement during garbage collection and compression.

[0033] First, it should be understood that garbage management in the key-value engine discussed in this article may include one or more of the following: 1) compaction, which is a process used to clean up garbage (e.g., unused memory space or data) on the page storage side; 2) automatic compaction, which is mainly used to maintain the LSM form while also handling garbage cleaning on the page storage side; 3) periodic compaction, which is used to deal with any residual garbage that may not have been identified for garbage collection during automatic compaction; and 4) garbage collection, which is responsible for cleaning up garbage data within the key-value engine.

[0034] Figure 1 is a schematic diagram of an exemplary cloud-based storage system 100 , arranged in accordance with at least some embodiments described herein.

[0035] System 100 may include terminal devices 110, 120, 130 and 140, network 160 and (multiple) servers 150. It should be understood that Figure 1 Only illustrative numbers of terminal devices, networks, and servers are shown. The embodiments described herein are not limited to the number of terminal devices, networks, and / or servers described. That is, the number of terminal devices, networks, and / or servers described herein is only for descriptive purposes and is not intended to be limiting.

[0036] According to at least some exemplary embodiments, terminal devices 110, 120, 130, and 140 may be various electronic devices and / or digital or virtual devices. Various electronic devices may include, but are not limited to, mobile devices such as smartphones, tablet computers, e-book readers, laptop computers, desktop computers, and / or any other suitable electronic devices or hosts for browsers or applications.

[0037] According to at least some exemplary embodiments, the network 160 may be a medium for providing a communication link between the terminal devices 110, 120, 130, 140 and the server 150. The network 160 may be the Internet, a local area network (LAN), a wide area network (WAN), a local interconnect network (LIN), a cloud, etc. The network 160 may be implemented by various types of connections, such as a wired communication link, a wireless communication link, an optical fiber cable, etc.

[0038] According to at least some exemplary embodiments, the server 150 may be a server for providing various services, such as a server for providing cloud services (including, for example, cloud storage, retrieval services, big data analysis, machine learning, data mining, etc.) to terminal users using one or more of the terminal devices 110, 120, 130, and 140. The server 150 may be implemented in a server system including a distributed server cluster that includes a plurality of servers 150 or a single server. In some embodiments, one or more server systems may communicate with each other via a network 160. The server 150 may also include a controller for controlling hardware and / or storing software and firmware and providing the functionality of the server.

[0039] In some embodiments, server(s) 150 may include storage devices, such as non-volatile memory (NVM) solid-state drives (SSDs), such as flash memory, or communicate with storage devices using various communication interfaces that allow for rapid storage of data on and / or retrieval of data from the storage devices.

[0040] The end user may use one or more of the end devices 110, 120, 130, and 140 to interact with the server(s) 150 via the network 160. In some embodiments, various applications, such as social media applications, social network applications, shopping applications, game applications, metadata management, etc., may be installed on the end devices 110, 120, 130, and 140. In some embodiments, the end user may request big data analysis or data mining of data on storage devices connected to the server(s) 150 and / or data residing or provided to the cloud computing system to support data analysis, learning models, etc.

[0041] It should be understood that the software application or service according to the embodiments described herein and / or according to the services provided by the cloud service provider may be executed by the server 150 and / or the terminal devices 110, 120, 130, and 140 (which may be referred to as user devices herein). Therefore, the apparatus for the software application and / or service may be arranged in the server 150 and / or the terminal devices 110, 120, 130, and 140.

[0042] It should also be understood that, in the case where services are not performed remotely, the system 100 may not include the network 160 , but may include only the terminal devices 110 , 120 , 130 , and 140 , and / or the server 150 .

[0043] It should also be understood that the terminal devices 110, 120, 130 and 140 and / or the server 150 may each include one or more processors, firmware and a storage device storing one or more programs. The terminal devices 110, 120, 130 and 140, and / or the server 150 may each further include an Ethernet connector, a wireless fidelity receiver, etc. When one or more programs are executed by one or more processors, one or more processors may be caused to perform the (multiple) methods described in any embodiment described herein. In addition, it should be understood that a computer-readable non-volatile medium may be provided according to the embodiments described herein. The computer-readable medium stores a computer program. When executed by a processor, the computer program is used to perform the (multiple) methods described in any embodiment described herein.

[0044] Figure 2 A schematic diagram of a key-value store and file system according to one embodiment is shown. The key-value store and file system 200 includes a key-value engine 202. The key-value engine 202 may include one or more of a log engine 204, a hash engine 206, a sort engine 208, and a garbage collection manager module 210. The key-value store and file system 200 may also include a collaboration layer 212 and a file system 214. The key-value store and file system 200 may interact with a kernel space 216, which includes one or more disks 218. The key-value store and file system 200 may also interact with an application 220.

[0045] The key-value store and file system 200 can be used for data storage in cloud applications, for example, to provide data persistence required by cloud services. The key-value engine 202 can be configured to provide key-value storage, for example, as part of the storage backend of a cloud service. Non-limiting examples of cloud services using the key-value engine 202 include Internet-based shopping, social media, metadata management, etc. The file system 214 can be a dedicated user-level append-only file system that is configured to provide storage dedicated to facilitating the operation of the key-value engine 202.

[0046] The log engine 204 can be configured to allow concurrent write of multiple log files, thereby reducing the number of compression and garbage collection operations. The log written by the log engine 204 can be configured so that the disposal of the log does not need strong sorting. The log engine 204 can be configured to improve the throughput performance problem in log writing and improve recovery speed by reducing the synchronous write overhead of the log from multiple input / output (I / O) operations to a single I / O operation, using lock-free queue aggregation to write to control delay and improve throughput, and / or providing an asynchronous interface to enhance the thread model. The key value engine 202 and the file system 214 can be integrated and configured to cooperate with each other, and the log engine 204 can be used to store a pre-write log (WAL) with a predefined structure, and the predefined structure has a defined actual file size. The file size defined for WAL can then cause the need for less I / O operations, thereby enhancing performance, while alleviating the potential trade-off about data consistency.

[0047] The hash engine 206 can be configured to handle point queries within the key-value engine 202. In particular, the hash engine 206 is configured to reduce tail latency in point queries. The hash engine 206 includes a separation of data and index components, and maintains the index in a cache memory, such as by compressing the index and / or caching partial data. The partial data can be selected using, for example, a least recently used strategy.

[0048] The sorting engine 208 can be configured to perform range scan operations while reducing write amplification factors and / or read / write latency associated with such operations. The sorting engine 208 is configured to use a partitioned log structure merge (LSM) tree. Classification of I / O streams and scheduling of tasks can further be performed by the sorting engine 208.

[0049] The garbage collection manager module 210 may be configured to perform garbage collection and / or compression operations in the key-value store and file system 200. The garbage collection manager module 210 may be configured to reduce unnecessary data movement during garbage collection and / or compression operations in the key-value store and file system 200. The garbage collection manager module 210 may perform garbage collection and / or compression operations based on awareness of application-side data deletions, such as page expiration. The garbage collection and compression performed by the garbage collection manager module 210 may be configured to arrange data to support other modules, such as the sorting engine 208. The garbage collection manager module 210 may be configured to coordinate the preservation of data during garbage collection and / or compression operations.

[0050] The collaboration layer 212 can be configured to facilitate collaboration between the key-value engine 202 and the file system 214. The collaboration layer 212 can also facilitate efficient compression and / or garbage collection operations in the key-value engine 202 based on the collaboration between the key-value engine 202 and the file system 214. The collaboration can reduce write amplification problems caused by compression and / or garbage collection operations. In an embodiment, the collaboration layer 212 can expose region usage information from the key-value engine 202 to the file system 214.

[0051] The file system 214 can be configured to separate data from the log and use a log-structured append-only write as the write model. In an embodiment, the file system can also provide pre-allocated data space, where synchronous writes occur only for the persistence of data, and in one embodiment, metadata persistence is not required. In one embodiment, data persistence and global log persistence for different files can be performed separately. These aspects of the file system can allow the file system to avoid some metadata persistence operations, such as those caused by a single data write persistence operation.

[0052] The file system 214 can be configured to support general files and instant files. General files and instant files can be written sequentially, and both can be read sequentially or randomly. General files can be optimized to achieve continuous low latency in continuous or random reads. General files can be used to write data in batches without flushing the data to disk after each write, such as SST files. Storage space is allocated in large units, where a non-limiting example of the unit size is 1MB each. Large allocation units can reduce the metadata size of general files so that during normal file system operations, the metadata of all general files can be kept in memory. By keeping the metadata in memory, no further I / O is required for the read operation of the general file to access the metadata regardless of the read offset. This can reduce the read tail latency of the general file. Instant files can be optimized for fast incremental synchronous writes while having good sequential and random read performance near the tail. Instant files can be used to write data that needs to be frequently refreshed to disk to obtain instant persistence, such as pre-write log files of key-value systems. Each individually written data and metadata can be bundled together for instant files. The bundled data and metadata can be written to a journal file shared by all instant files. Bundling data and writing to a journal file can improve the speed of incremental writes and sync operations. This approach is structured to support sequential reads, but can make tradeoffs with respect to random reads. Since live files are expected to be mostly read sequentially, with random reads mostly concentrated near the tail, the most recently written data for each live file being actively written can be cached to improve read performance.

[0053] The file system 214 may include a user space I / O scheduler to assign I / O priorities to different I / O types. The I / O scheduler marks the foreground I / O as a high priority, while the background I / O is marked as a low priority. In addition, the key-value engine 202 may include a scheduler to schedule its background tasks to ensure that each I / O issued by the upper layer application has a consistent I / O amplification. Through this common design of I / O scheduling in both the key-value engine 202 and the file system 214, since both I / O amplification and I / O latency are consistent, tail latency can remain stable and low. In addition, reading common files from the file system does not require I / O for metadata, and using a large space for common files can ensure that most read operations require a single I / O.

[0054] Kernel space 216 may contain disks 218. Disks 218 may include one or more storage media, such as solid-state drives (SSDs). In an embodiment, at least some of disks 218 are zoned storage (ZNS) SSDs.

[0055] Application 220 may be any suitable application that utilizes key-value store and file system 200, such as online shopping, social media, metadata management applications, etc. Application 220 may interface with key-value store and file system 200 via any suitable application programming interface (API). In one embodiment, the API may be specific to a particular type of file, such as having properties of a file as a generic file or an instant file determined by the API through which the file has been received.

[0056] Figure 3 is a key-value engine (e.g., a system for optimizing garbage collection and / or compaction processes in a key-value engine system) arranged in accordance with at least some embodiments described herein. Figure 2202 in ). In some embodiments, the active garbage collection module 300 can be configured to actively reclaim storage space without hindering foreground client threads, such as CPU utilization, which is achieved by both capacity awareness and the ability to dynamically adjust the parallelism of garbage collection threads. In addition, the active garbage collection module 300 can be configured to have awareness of application-side data deletions, so that the active garbage collection module 300 can coordinate garbage collection and / or compression with any underlying file system, for example, by transmitting garbage collection information (at the key-value engine level) and / or other monitoring information to the file system via an interface (such as, an application programming interface (API), a kernel file interface, etc.) to minimize unnecessary data movement. In order to perform the above operations, the active garbage collection module 300 may include one or more modules of an accurate garbage collection manager module 310, an adaptive garbage collection manager module 320, a periodic compression manager module 330, a garbage collection read amplification optimizer module 340 and / or a hot / cold data separator module 350. The accurate garbage collection manager module 310, the adaptive garbage collection manager module 320, the periodic compaction manager module 330, the garbage collection read amplification optimizer module 340 and / or the hot / cold data separator module 350 can be configured to coordinate with other modules to optimize the garbage collection and / or compaction process, or can operate independently of the other remaining modules, for example, being able to operate simultaneously as background operations.

[0057] The accurate garbage collection manager module 310 can be designed, programmed or otherwise configured to provide more accurate information about the amount of effective garbage for garbage collection, for example, by being configured as a communication or collaboration interface between a key-value engine and a file system, so that the file system and / or the key-value engine is notified or aware of the internal garbage collection and / or compression process. For example, in some embodiments, the accurate garbage collection manager module 310 can be configured to accurately measure the amount of garbage within a blob file, for example, by considering the actual data size of the key-value pair. In this way, the key-value engine can include the occurrence of garbage collection to enable the selection of files based on a more informed and accurate selection of garbage collection, thereby improving overall disk space utilization and garbage collection efficiency. In addition, the accurate garbage collection manager module 310 can be configured to consider the garbage rate at the entry level rather than the space level. By improving the garbage collection calculation at the entry level, it can be ensured that the blob file with the highest garbage space is selected, thereby achieving a more efficient garbage collection operation or strategy, for example, reducing tail latency, especially when garbage collection occurs on both the key-value engine and the file system, for example, to reduce unnecessary data movement of expired or deleted files / data.

[0058] Figure 4A , Figure 4B , Figure 4C An exemplary embodiment of a key-value engine with an accurate garbage collection manager as discussed herein is shown.

[0059] Figure 4A is an illustrative example of a previous key-value engine having a separate index for key-value pairs 405, 410, wherein the file number of each key-value pair is stored in the key 405 for the LSM tree entry, e.g., the key, file number, offset, and size of the entry for the corresponding blob file (e.g., value 410 in the SST). Although such indexes in previous key-value engines allow for additional key-value pairs, such previous key-value separate indexing techniques do not include data about the size of the separated key-value data, such as information, e.g., may only include the number of data entries that each SST entry depends on in various blob files, without including the size of the value data itself. Although the garbage ratio can be calculated based on the number of entries and files selected for garbage collection to occur, such determination may result in inaccurate file selection, incomplete capacity release, and increased CPU usage for garbage collection. Therefore, when a previous key-value engine is configured to determine and / or select files to be garbage collected, the previous key-value engine may rely solely on the count of expired entries to determine the files to be garbage collected. Since the size of data within a blob file can vary significantly, relying solely on entry counts for statistics may be inaccurate. Thus, previous key-value engines may ignore files that have a relatively low number of expired entries but contain a large amount of junk data, such as junk data. Although junk data has a lower priority in the garbage collection scheme, it may stay on disk for an extended period of time, resulting in wasted disk space.

[0060] Figure 4Bis an illustrative embodiment of the operation of the accurate garbage collection module 310, wherein the key-value engine is designed, programmed, or otherwise configured to store key-value pairs 415, 420 in an LSM tree, such as in an SST in each level of the LSM tree. In some embodiments of key-value separation, the link between the key and the data can be stored in a KeySST file, while the data content itself can reside in other files, so that compression may only require triggering the key to relocate less data. In some embodiments, the LSM tree entry key 415 may include the key and file number of the corresponding blob file, for example, for each key-value pair value 420, and the LSM tree may also include the value size of the blob file, for example, in an index or delta block 425. By including the value size of the blob file, the key-value engine can be designed, programmed, or otherwise configured so that the garbage ratio or garbage rate of each blob file can be calculated, for example, based on the blob file size and the blob file available size, as discussed further below, to better inform and / or more accurately select the blob file for garbage collection to occur. That is, by configuring the key-value engine to store the file number together with the value size of the blob file in the LSM tree, the key-value engine is configured to have a more accurate file selection process, a more complete capacity release process, and reduced CPU usage during the garbage collection process by being able to select the blob file with the highest amount of garbage data. In some embodiments, the key-value engine can be designed, programmed, or otherwise configured so that the incremental block is configured to store only the value size without storing the key, thereby reducing the space occupied by any additional key. Thus, in some embodiments, the key-value engine can be configured to store a bitmap indicating whether each key-value pair is separated (together with some auxiliary information), so that various operations such as seeking (Seek) can be supported without explicitly storing the key when the key is not required. Therefore, since the data in the incremental block is only used during compression, for example, this means that the incremental block is only accessed once during the life cycle of the SST, the storage cost of the incremental block can be minimized and / or the read cost can be kept low.

[0061] Figure 4CIt is an illustrative embodiment of determining and / or calculating the garbage ratio or garbage rate of a blob file by a key-value engine according to some embodiments. As discussed above, the SST of the LSM tree can be used to store the data size of the blob file (together with the file number and key). In this way, the key-value engine can be configured to use the data size to determine and / or calculate the garbage ratio or garbage rate of the blob file, for example, during compression. In one embodiment, the key-value engine can be configured to determine the total real-time data size of the blob file based on the SST entry, for example, depending on the sum of the real-time data sizes of all SST entries of a particular blob file to determine the effective data size of the blob file. Since the total used size of the blob file is known, the amount of garbage data can be determined by the difference between the total used size of the blob file (e.g., the remaining size) and the sum of the real-time data sizes from the SST entries. Therefore, the garbage ratio or garbage rate can be determined for each blob file.

[0062] For example, returning a reference Figure 4C In an exemplary embodiment during compression, the key-value engine may be configured to perform a full scan of all SSTs 430, 435, 440 to build a dependency graph of the SSTs 430, 435, 440 and the corresponding blob files 445, 450, 455. After compression, the SSTs 430, 435, 440 may be compressed into SSTs 432, 433 that depend on the blob files 445, 450, 455. In this way, the garbage ratio or garbage rate in the blob file may be determined based on the sum of the live data size and the total used size of the blob file. For example, blob1 445 may have a total used size of 2500, where the SST entry 433 that depends on blob1 445 has a live data size of 800, and other SSTs that depend on blob1 445 have a live data size of 500. Thus, blob1 445 has a live data size of 1300 and a total used size of 2500, so that junk data may be determined as the difference between the total used size and the live data size, and wherein the junk rate of blob1 445 is 48%.

[0063] In this way, a key-value engine with an accurate garbage collection manager module 310 can be designed, programmed, or otherwise configured to provide more accurate information about the amount of effective garbage for garbage collection, for example, by being configured to determine the garbage ratio or garbage rate at the entry level rather than at the space level to improve the overall disk space utilization and garbage collection efficiency of garbage collection occurrences, for example, performing garbage collection occurrences, particularly when garbage collection occurs on both the key-value engine and the file system, for example, reclaiming memory independently at both the key-value engine and the file system.

[0064] The adaptive garbage collection manager module 320 can be designed, programmed or otherwise configured to make garbage collection decisions based on multiple real-time factors, such as, actual disk capacity usage of the user, capacity of the user (including LSM amplification), disk utilization (LSM amplification and blob file garbage rate), disk utilization that triggers garbage collection, disk utilization that triggers write stop, disk over-sizing, guaranteed performance utilization, key-value engine garbage ratio, LSM and PageStore garbage ratio, maximum space available to the user, garbage collection goals to reduce write traffic, etc. In some embodiments, the adaptive garbage collection manager module 320 can be configured to be aware of capacity usage, for example, the size of all files scanned using the system interface in the current directory of the instance. In some embodiments, the adaptive garbage collection manager module 320 can be configured to dynamically adjust garbage collection thresholds, bandwidth and concurrency based on capacity availability / usage without affecting other tasks such as background compaction. For example, in some embodiments, the disk capacity may be determined, and the garbage collection percentage threshold and the number of garbage collection threads may be adjusted according to predetermined policies, such as actual capacity, LSM enlargement, blob file garbage ratio or garbage rate, garbage collection trigger, trigger to stop writing, disk over-sizing, guaranteed performance utilization, key-value engine garbage ratio, LSM and PageStore garbage ratio, maximum space, etc. In some embodiments, to prevent an excessive number of garbage collection threads from affecting other tasks such as background compaction, the thread pool model may be optimized by separating garbage collection threads and allocating them as needed (e.g., based on disk capacity usage).

[0065] Figure 5A and Figure 5B is an illustrative example of the operation or process flow of the adaptive garbage collection manager module 320 for disk space utilization according to various embodiments. The adaptive collection manager module 320 can be designed, programmed or otherwise configured to manage the disk space utilization of the system based on different disk capacity usage conditions (e.g., watermark ranges), so that the garbage collector thread count and the garbage collection threshold can be dynamically adjusted based on whether the disk capacity usage conditions / disk space utilization meet and / or exceed predetermined thresholds.

[0066] Figure 5AAn exemplary operation of a process flow 500 for disk space utilization is shown. It should be understood that the process flow 500 or operation may include one or more operations, actions, or functions, as shown in one or more of blocks 505, 520, 530, 532, 534, 536, 540, and 545. For example, in one embodiment, when disk space utilization is below a first predetermined threshold, a garbage collection percentage threshold and the number of garbage collection threads may be adjusted, the number of garbage collection threads being set to a first thread value, such as one, and the garbage collection (GC) percentage threshold being dynamically set between a first GC percentage threshold (e.g., 50%) and a second GC percentage threshold (e.g., 90%) based on disk usage; and when disk space utilization is between the second predetermined threshold and a third predetermined threshold, the number of garbage collection threads and the garbage collection percentage threshold are dynamically adjusted, wherein the dynamic adjustment includes increasing or decreasing the number of garbage collection threads and / or the threshold of the garbage collection percentage based on whether disk space utilization and / or disk usage is increasing or decreasing, as discussed further below. These various operations, functions, or actions may correspond, for example, to software, firmware, program code, or program instructions executable by a processor that causes these functions to be performed. Although shown as discrete blocks, obvious modifications may be made, for example, two or more of the blocks may be reordered; further blocks may be added; and various blocks may be divided into additional blocks, combined into fewer blocks, or eliminated depending on the desired implementation. In addition, it should be understood that the determination of disk capacity usage may be continuously and / or periodically checked or determined, such as every minute, every two minutes, every five minutes, etc. It should be understood that operations including initialization, etc. may be performed prior to the process flow.

[0067] Process flow 500 may begin at block 505 .

[0068] At block 505 (Page Store Compression), the processor for the adaptive garbage collection manager module 320 may be designed, programmed, or otherwise configured to determine whether the page store should be compressed, for example, based on a maximum disk space allocation per box and a disk space utilization threshold and / or expired or deleted data or files. In some embodiments, the key-value engine may be configured such that the server load may be periodically checked after the garbage collection process of the key-value engine, such as a GetServerLoad command. The process flow may proceed to block 510.

[0069] At block 510 (Garbage Collection), a processor for the adaptive garbage collection manager module 320 may be designed, programmed, or otherwise configured to optimize disk space utilization, for example, by dynamically adjusting an adaptive garbage collection process. The dynamic adjustment of the adaptive garbage collection process may be based on various predetermined disk capacity usage (or disk space utilization), server load, etc. For example, in one embodiment, the dynamic adjustment of the adaptive garbage collection process may be based on blocks 520, 530, 540, 545, and may be based on a continuous or periodic determination of disk capacity usage.

[0070] At block 520 (disk space usage < yellow line), the processor for the adaptive garbage collection manager module 320 may be designed, programmed or otherwise configured to determine disk space usage. If the adaptive garbage collection manager module 320 determines that the disk space usage is below the yellow line, e.g., 75%, then no garbage collection is initiated, e.g., no disk space optimization is required.

[0071] At block 530, (disk space usage>=yellow line, and disk space usage<red line), the processor for the adaptive garbage collection manager module 320 may be designed, programmed, or otherwise configured to determine disk space usage, wherein if the disk space usage is greater than or equal to the yellow line threshold, such as 75%, and is lower than the red line threshold, such as 80%, then the adaptive garbage collection manager module 320 may also be configured to dynamically adjust the operation of the key-value engine based on server load, such as CPU and disk, to optimize disk space. For example, in one embodiment, the dynamic adjustment of the operation may be based on blocks 532, 534, 536.

[0072] At block 532 (Serverload < yellow line), the processor for the adaptive garbage collection manager module 320 may be designed, programmed, or otherwise configured to determine the server load, wherein if the server load is less than the yellow line threshold, such as 50%, full load operation is initiated.

[0073] In box 534 (Serverload>=yellow line and Serverload<red line), the processor for the adaptive garbage collection manager module 320 can be designed, programmed or otherwise configured to determine the server load, where if the server load is greater than or equal to the yellow line threshold, such as 50%, but less than the red line threshold, such as 80%, then the 50% load operation is initiated.

[0074] At block 536 (Server Load>Red Line), a processor for the adaptive garbage collection manager module 320 may be designed, programmed, or otherwise configured to determine the server load, wherein if the server load is greater than a red line threshold, such as 80%, single thread operation is initiated.

[0075] At block 540 (disk space usage>=redline), the processor for the adaptive garbage collection manager module 320 may be designed, programmed, or otherwise configured to determine disk space usage, wherein full load operation is initiated if the disk space usage is greater than or equal to a redline threshold, e.g., 80%.

[0076] In box 545 (disk space usage >= 85%), the processor for the adaptive garbage collection manager module 320 may be designed, programmed, or otherwise configured to determine disk space usage, wherein if the disk space usage is greater than or equal to an alert threshold, such as 85%, an alert operation is initiated, for example, a user or controller of a cloud-based service is alerted that the disk space usage is high, or a warning is generated at the PageStore.

[0077] In this way, the adaptive garbage collection manager module 320 can be designed, programmed or otherwise configured to make garbage collection optimization decisions based on multiple real-time factors, such as disk capacity usage, capacity reported by the PageStore LSM tree, so that the adaptive garbage collection manager module 320 can be configured to dynamically adjust garbage collection thresholds, bandwidth and / or concurrency based on capacity availability / usage, for example, without affecting other tasks, such as background compaction.

[0078] Figure 5B Another example operation of the process flow 550 for disk space utilization is shown. It should be understood that the process flow 550 or operation may include one or more operations, actions or functions, as shown in one or more of blocks 555, 556, 560, 561, 562, 563, 570, 571, 572, 573, 580, 581, 582, 583, 590, 591, 592, 593 and 595. These various operations, functions or actions may correspond to software, firmware, program code or program instructions executable by a processor, which causes these functions to be performed. Although shown as discrete blocks, obvious modifications may be made, for example, two or more blocks in the block may be reordered; further blocks may be added; and various blocks may be divided into additional blocks, combined into fewer blocks or eliminated depending on the desired implementation. Although predetermined thresholds for low water and high water or water level levels of regions are provided, it should be understood that such disclosure is not intended to be limiting. Instead, a predetermined watermark level is provided as an exemplary embodiment, where different thresholds may be used based on the desired results of disk space utilization. In addition, it should be understood that the determination of disk capacity usage may be continuously and / or periodically checked or determined, such as every minute, every two minutes, every five minutes, etc. It should be understood that operations including initialization, etc. may be performed prior to the process flow.

[0079] Process flow 550 may begin at block 555 .

[0080] At block 555 (Page Store Compression), the processor for the adaptive garbage collection manager module 320 may be designed, programmed, or otherwise configured to determine whether any page in the page store should be compressed, for example, based on a maximum disk space allocation per box and a disk space utilization threshold and / or expired or deleted data or files. In some embodiments, the key-value engine may be configured such that the server load may be periodically checked after the garbage collection process of the key-value engine, such as a GetServerLoad command. The process flow may proceed to block 556.

[0081] At block 556 (Garbage Collection), the processor for the adaptive garbage collection manager module 320 may be designed, programmed, or otherwise configured to optimize disk space utilization, for example, by dynamically adjusting the adaptive garbage collection process. The dynamic adjustment of the adaptive garbage collection process may be based on various predetermined disk capacity usage, server load, etc. For example, in one embodiment, the dynamic adjustment of the adaptive garbage collection process may be based on blocks 560, 570, 580, 590, 595, and may be based on continuous or periodic determination of disk capacity usage. In an exemplary embodiment, the adaptive garbage collection manager module 320 may include one or more predetermined disk space utilization thresholds for dynamically setting garbage collection threads and garbage collection thresholds, which may include a low water mark threshold for low water marks or areas, such as for disk space utilization below 90%, and a high water mark threshold for high water marks or areas, such as above 90%, such as 92% to 95%. The low water mark thresholds may include a blue line threshold of 30%, a yellow line threshold of 60%, and a red line threshold of 90%. The high watermark thresholds for disk space utilization between 92% and 95% may include a blue line threshold of 92%, a yellow line threshold of 94%, and a red line threshold of 95%. Although predetermined disk space utilization thresholds are described herein, this disclosure is not intended to be limiting, as other thresholds may be set based on optimization strategies. Thus, the processor for the adaptive garbage collection manager module 320 may be designed, programmed, or otherwise configured to optimize disk space usage, and to set garbage collection threads and garbage collection thresholds based on predetermined disk space utilization thresholds and subsequent operations of the adaptive garbage collection manager module 320.

[0082] At block 560 (disk usage < low water mark redline), if the adaptive garbage collection manager module 320 determines that the disk space utilization is at a low water mark threshold, e.g., below a redline threshold of the low water mark threshold, e.g., 90%, the disk space usage is considered to be in a low water mark region and is adaptively controlled based on the low water mark threshold. The adaptive collection manager module 320 may be configured to set the garbage collection thread to a first thread value, e.g., 1, and the garbage collection threshold may be dynamically set between a first GC percentage threshold and a second GC percentage threshold, e.g., 50% to 90%, based on disk usage at a low water mark level (e.g., DiskUsage). For example, in one embodiment, at 561, if the adaptive garbage collection manager module 320 determines that the disk usage is less than 30%, the garbage collection thread may remain at 1, and the garbage threshold may be set to or greater than 90%. At 562, if the adaptive garbage collection manager module 320 determines that the disk usage is less than 60%, the garbage collection thread may remain at 1, and the garbage threshold may be set to or greater than 70%. At 563, if the adaptive garbage collection manager module 320 determines that disk usage is less than 90%, then the garbage collection thread may remain at 1 and the garbage threshold may be set to be equal to or greater than 50%.

[0083] If the adaptive garbage collection manager module 320 determines that the disk space utilization is at a high water mark threshold, such as the second predetermined threshold and the third predetermined threshold, such as between 90% and 95%, then the disk space usage is considered to be in a high water mark area, and the adaptive garbage collection manager module 320 is configured to dynamically adjust the number of garbage collection threads and the garbage collection threshold based on the high water mark threshold, wherein the dynamic adjustment includes increasing or decreasing the number of garbage collection threads and the garbage collection percentage threshold based on whether the disk space utilization and / or disk usage increases or decreases, as discussed below with respect to boxes 570, 580, 590, and 595.

[0084] At 570 (disk usage < high water mark blue line), if the adaptive garbage collection manager module 320 determines that the disk space usage is less than the high water mark threshold blue line, such as 92% (but greater than the red line low water mark threshold), the adaptive garbage collection manager module 320 can be configured to set the garbage collection threshold to 10%, and the garbage collection threads can be dynamically adjusted between 2 and 8 threads based on the increase or decrease in disk usage, as discussed herein with respect to blocks 571, 572, 573. For example, in one embodiment, at 571, if disk usage increases within a predetermined time frame, such as a period of 15 seconds, 30 seconds, 60 seconds, 90 seconds, the garbage collection thread count can be increased by 1 and can be limited to a maximum of 8 threads. At 572, if disk usage decreases within the same predetermined time frame, the garbage collection thread count can be decreased by 1 and can be limited to a minimum of 2 threads. At 573, if disk usage drops to 89%, eg, below the low water mark threshold redline-[delta], the garbage collection thread count may be set back to 1.

[0085] At 580 (disk usage < high water mark yellow line), if the adaptive garbage collection manager module 320 determines that the disk space usage is less than the yellow line of the high water mark threshold, such as 94% (but greater than the blue line of the high water mark threshold), the adaptive garbage collection manager module 320 can be designed, programmed or otherwise configured to set the garbage collection threshold to 8%, while the garbage collection thread count can dynamically change between 10 and 25 based on the increase or decrease in disk usage, as discussed herein with respect to 581, 582, and 583. In one embodiment, at 581, if disk usage increases within a predetermined time frame, the garbage collection thread count can be increased by 2 and can be limited to a maximum of 25. At 582, if disk usage decreases within the same predetermined time frame, the garbage collection thread count can be decreased by 2 and can be limited to a minimum of 10. At 583, if disk usage drops to 91%, for example, below the blue line of the high water mark threshold - [increment], the garbage collection thread count can be set to 2.

[0086] At 590 (disk usage < high water mark red line), if it is determined that the disk space usage is less than the high water mark threshold red line, such as 95% (but greater than the high water mark threshold yellow line), the adaptive garbage collection manager module 320 can be designed, programmed or otherwise configured to set the garbage collection threshold to 2%, while the garbage collection thread count can dynamically change between 25 and 40 based on the increase or decrease in disk usage, as discussed herein with respect to 591, 592, and 593. At 591, in one embodiment, if disk usage increases within a predetermined time frame, the garbage collection thread count can be increased by 3 and can be limited to a maximum of 40. At 592, if disk usage decreases within the same predetermined time frame, the garbage collection thread count can be decreased by 3 and can be limited to a minimum of 25. At 593, if disk usage drops to 93%, for example, below the low water mark yellow line - [increment], the garbage collection thread count can be set to 10.

[0087] It should be understood that at blocks 570, 580, 590, the predetermined time frames may be the same or different.

[0088] In box 595 (disk space usage>=high water mark red line), the processor for the adaptive garbage collection manager module 320 can be designed, programmed or otherwise configured to determine disk space usage, wherein if the disk space usage is greater than or equal to the red line of the high water mark threshold, such as 95%, a warning operation is initiated, for example, a user or controller of a cloud-based service is alerted to high disk space usage, and / or a warning is generated at the PageStore layer, and write flow control can be initiated to manage operations.

[0089] Thus, the adaptive garbage collection manager module 320 can be designed, programmed, or otherwise configured to optimize disk space utilization by making garbage collection decisions based on multiple real-time factors (such as, disk capacity usage, capacity reported by PageStore LSM trees, etc.), such that the adaptive garbage collection manager module 320 can be configured to dynamically adjust garbage collection thresholds, bandwidth, and / or concurrency based on capacity availability / usage without affecting other tasks, such as background compaction. Thus, as discussed herein, dynamic adjustments can help optimize key-value engine performance based on disk space utilization levels, for example, by improving thread pool and CPU usage, for example, for garbage collection.

[0090] The periodic compaction manager module 330 can be designed, programmed or otherwise configured to address low garbage collection efficiency and delayed space release issues in a production environment caused by compaction, for example, by assigning priorities to garbage collection and / or compaction processes. For example, a key-value engine can combine three types of background operations, including but not limited to flushing, background garbage collection, and compaction. A flush may occur when a memTable is full or a write-ahead log (WAL) reaches its capacity, and a flush operation may be triggered by a key-value engine with a "flush operation" priority of "high", for example, a flush may preempt any low thread pool. When the garbage ratio of a particular blob file (e.g., garbage data on a blob file to the total size of the blob file) reaches a threshold, garbage collection of the key-value engine can be triggered so that a dedicated garbage collection thread can be dedicated to garbage collection operations. Compacting can include automatic compaction and mark compaction, where automatic compaction can occur to adjust the form of an LSM tree for a key-value engine, for example, to balance the LSM tree, and mark compaction can occur when there is no automatic compaction, so that the SST of a key-value engine that has been marked will be selected for mark compaction, which may have a low priority.

[0091] A problem in the compaction process of previous key-value engines is the delayed push-down of compaction, which can cause space expansion. The delayed push-down of compaction can become more pronounced in key-value engines that utilize key-value separation techniques, which can adversely affect the efficiency of garbage collection. In addition, previous key-value engines lack the ability to identify space expansion caused by garbage data in cold SSTs at the lowest level.

[0092] In some embodiments, to overcome the deficiencies discussed above, a periodic compression manager module 330 may be provided that includes a periodic compression scheduler module 332 and a periodic compression executor module 334. In some embodiments, the periodic compression manager module 330 may be designed, programmed, or otherwise configured to function at a column family (CF) level (e.g., in a PageStore), thereby giving each column family the flexibility to enable or disable it based on individual requirements (e.g., design requirements) during a compression operation.

[0093] Fig. 6A , Figure 6BAn exemplary embodiment of a compression operation performed by a periodic compression manager module 330 according to various embodiments is schematically illustrated. As discussed herein, a periodic compression manager module 330 is provided to address the problems of low garbage collection efficiency and delayed space release in a production environment caused by compression by including a periodic compression operation provided in conjunction with an existing automatic compression process for a key-value engine. Thus, the periodic compression manager module 330 is capable of reducing unnecessary data movement of application data that has expired but not identified by a key-value engine side page during garbage collection, for example, via communication with PageStore to clear PageStore garbage, by consolidating one or more expired pages (of the same page) of one or more applications (by merging one or more layers of one or more expired pages into the last layer) to reduce data disposal, for example, when communicating with a file system via an interface.

[0094] For example, in an embodiment, Fig. 6A Schematically shown, the compression operation 600 may include a periodic compression operation 605, an automatic compression operation 610, and a garbage collection operation 620. The periodic compression operation 605 may be initiated to scan the LSM tree 615 and the corresponding SST file from the top to the bottom of the LSM tree. Since the periodic compression operation 605 is used to scan the LSM tree from top to bottom, the periodic compression manager module 330 is configured to identify SST files that may need to be compressed, for example, SST files including expired pages and / or blob files with a garbage ratio or garbage rate higher than a predetermined threshold (e.g., the ratio between valid user data, garbage data including PageStore garbage and key-value engine garbage, and / or free space). The automatic compression operation 610 can be automatically initiated to maintain the LSM tree shape and handle garbage cleaning on the PageStore side. The automatic compression operation 610 can be triggered by external conditions, such as external writes or markings. The garbage collection operation 620 can be initiated to clean up garbage data within the key-value engine, for example, when the garbage ratio of the blob file reaches a predetermined threshold. In some embodiments, after the periodic compaction operation 605, the automatic compaction operation 610, and / or the garbage collection operation 620 have been performed, the periodic compaction manager module 330 may also be configured to periodically check the server load and / or clear the PageStore garbage data.

[0095] Figure 6BAn exemplary embodiment of an operation of identifying an SST file and a corresponding blob file is shown, which SST file and the corresponding blob file can be marked for compression, such as a key-value engine application. In one embodiment, the periodic compression manager module 330 can be configured to determine the garbage ratio of the blob file based on the data in the SST of the LSM tree, for example, garbage data to real-time data. For example, in one embodiment, if the recyclable LSN (log sequence number) of segment 101 exceeds 1100, then such LSN indicates that all data within segment 101 of the SST is considered to be garbage data. By comparing the segment IDs and aggregating the results, the exact size of the garbage data in the entire SST can be determined. Note that the term "size" used herein may not refer to the size of the garbage data within the SST itself, but rather to the size within the corresponding blob file to accurately reflect the total amount of garbage data. This design ensures that the entire database is scanned within a given time frame. Exemplary embodiments of design requirements may include one or more of the following: actual capacity of the user, for example, greater than 70%; user capacity (including LSM amplification), for example, greater than 80%; disk utilization (LSM amplification and blob file garbage ratio), for example, greater than 90%; frequency of scanning all SSTs, for example, daily level; API for the upper layer to determine whether compression is needed; minimizing overhead to avoid I / O spikes; and supporting real-time identification of upper-layer garbage data in scenarios with continuous data writing.

[0096] It should be understood that the triggering mechanism for periodic compaction may be different from automatic compaction and garbage collection. For example, while automatic compaction and garbage collection may rely on external conditions, such as external writes or markings, periodic compaction needs to support internal active scanning to discover expired SST files and initiate periodic compaction (e.g., time-based incremental scans). In this way, the periodic compaction manager module 330 may include a periodic compaction scheduler module 332, which may be designed, programmed, or otherwise configured to collaborate with business logic to understand spatial amplification and to minimize scheduling overhead, for example, to coordinate the timing of periodic compaction based on workload characteristics and the priority of operations. In some embodiments, the periodic compaction scheduler module 332 may be designed, programmed, or otherwise configured to be implemented as a scheduling task. For example, in an embodiment where periodic compaction has been enabled for a column family, the LSM tree SST file may be scanned from top to bottom. When a suitable SST file is found, for example, for compression, the column family may be added to a pending compaction queue. In this way, the periodic compaction scheduler module 332 can be configured to schedule periodic compaction independently of external events by actively scanning the LSM tree to identify eligible SST files, thereby ensuring efficient management of storage space.

[0097] Figure 7700 are exemplary operations for scheduling periodic compression by the periodic compression scheduler module 332 according to one embodiment.

[0098] It should be understood that the process flow 700 or operation may include one or more operations, actions, or functions, as shown in one or more of blocks 705, 710, 715, 720, 725, 730, 735, and 740. These various operations, functions, or actions may correspond, for example, to software, firmware, program code, or program instructions executable by a processor that causes these functions to be performed. Although shown as discrete blocks, obvious modifications may be made, for example, two or more blocks in the blocks may be reordered; further blocks may be added; and various blocks may be divided into additional blocks, combined into fewer blocks, or eliminated depending on the desired implementation. It should be understood that operations including initialization, etc. may be performed prior to the process flow.

[0099] Process flow 700 may begin at block 705. A processor of the periodic compaction scheduler module 332 of the key-value engine may be designed, programmed, or otherwise configured to initiate an active scan of the LSM tree to identify any eligible SST files for compaction, as discussed herein. Process flow 700 may proceed to block 710.

[0100] At 710 , the periodic compaction scheduler module 332 may be configured to actively scan the LSM tree from top to bottom to identify any column families that have any expired SST files. Process flow may proceed to block 715 .

[0101] At 715 , the periodic compaction scheduler module 332 may be configured to actively scan the SST files of each of the column families to determine whether any SST files in the column family include outdated data and / or junk data that needs to be compacted. Process flow may proceed to block 720 .

[0102] At 720, the periodic compression scheduler module 332 can be configured to determine whether any SST files are marked for compression and create a thread hold for the marked SST files. If no SST files are marked for compression, and the last check time (e.g., the actual creation time of the file) and the thread hold are greater than a certain time (e.g., "now"), the process flow can return to box 715 for rescanning SST files to find files that may need to be compressed. If the SST file has been marked for compression, and the last check time and the thread hold are less than the time "now", the process flow can proceed to box 725.

[0103] At 725 , the periodic compression scheduler module 332 can be configured to scan the count of SST files marked for compression. Process flow can proceed to block 730 .

[0104] At 730, the periodic compression scheduler module 332 may be configured to determine whether a threshold for automatic compression or flagging compression has been triggered. If it is determined at the periodic compression checker that periodic compression should be triggered, e.g., via flagging periodic compression, then the periodic compression is scheduled for execution, e.g., by the periodic compression executor module 334. If the periodic compression checker is not scheduled for execution, then at 730, process flow may proceed to 740.

[0105] At 740, the periodic compression scheduler module 332 can be configured to determine whether automatic compression should be initiated or performed. In one embodiment, if the scan count is less than 10 minutes / (thread_hold)*(SST_total), the automatic compression process can be initiated. If the scan count is not less than 10 minutes / (thread_hold)*(SST_total), the process can return to block 715 for rescanning the SST file.

[0106] The periodic compression executor module 334 may be designed, programmed or otherwise configured to extend the functionality of the MarkCompaction module by also being configured to accommodate periodic compression operations. Thus, the periodic compression executor module 334 is configured to examine SST files that have been marked for compression and perform compression operations thereon, for example, merging multiple data layers by removing duplicate entries and old, expired data.

[0107] In one embodiment, the periodic compression executor module 334 can be configured to check each layer in turn from top to bottom to identify any expired SSTs that meet the criteria, because the higher the layer, the higher the priority should be, for example, more consistent with the process of pushing files from the top down. The periodic compression executor module 334 can be implemented together with the periodic compression scheduler module 332, for example, in box 730, to implement the idea of ​​time-based incremental scanning. In one embodiment, in the periodic compression checker box 730, the property of LastCheckTime can be checked as follows: For the last layer of the file, the actual creation time is used as the LastCheckTime. However, for files in other layers, LastCheckTime is determined based on the file range, where the intersection between the file range and the input file range is determined, and then the earliest creation time is calculated. Based on the LastCheckTime, the file can be compressed, for example, if it is an old or expired page.

[0108] It should be understood that in some embodiments, two types of files may be considered during periodic compaction: (i) upper layer files Li (i=0~N-1) and (ii) lower layer files LN. Thus, when pushing files down from upper layer Li files, adjacent LSNs may be merged to simplify the compaction process. For lower layer LN files, an active compaction method may be used to detect expired pages, for example, using mvpage of CompactionFilter.

[0109] In addition, in some embodiments, a time-to-live (TTL) parameter can be used to specify the maximum amount of time a piece of data should be retained or considered valid. When a TTL is set for a particular piece of data, the key-value store can be configured to automatically remove or invalidate the data after the specified period has passed to help manage data expiration and ensure that obsolete or unnecessary data is automatically removed from the system, thereby freeing up storage space and improving system performance. In some embodiments, factors to be considered in configuring TTL may include, but are not limited to, compression bandwidth monitoring and overhead assessment, which includes the overall bandwidth of push-down TTL compression and the overall bandwidth of low-end TTL compression; and compression effect monitoring and benefit assessment, which includes the overall data deletion amount of push-down TTL compression and the overall data deletion amount of low-end TTL compression.

[0110] In view of the above, a periodic compression manager module 330 can be provided to overcome the shortcomings of previous key-value engine designs, for example, by alleviating inefficiencies associated with garbage collection and addressing space-related challenges by providing a more adaptive and flexible compression solution at the column family level that can communicate with the file system to reduce unwanted / unnecessary data movement.

[0111] The garbage collection read amplification optimizer module 340 can be designed, programmed or otherwise configured to reduce the read amplification caused by the garbage collection process itself. In scenarios involving key-value engines using key-value separation technology, read amplification may be more significant and may have an adverse effect on the efficiency of garbage collection.

[0112] It should be understood that the garbage collection process can be divided into three specific steps: (i) reading keys from the blob file on the disk (reading phase), (ii) checking the validity of each key (validity checking phase), and (iii) writing the key-value pairs that are still valid to a new file (writing phase). However, previous garbage collection mechanisms may encounter significant efficiency issues. In previous garbage collection mechanisms, the system may be characterized by a high garbage ratio and a low capacity threshold for a large amount of written data. Since the previous garbage collection mechanism is triggered when both conditions are met, for example, when the garbage ratio is extremely high and the proportion of valid data is low, the previous garbage collection mechanism suffers from severe read amplification and hinders the overall garbage collection efficiency. For example, since the entire SST needs to be scanned during garbage collection to check the validity of all keys, the previous garbage collection mechanism may be inefficient.

[0113] In this way, the garbage collection read amplification optimizer module 340 can be designed, programmed or otherwise configured to store redundant key sets at the end of the SST. Therefore, during garbage collection, only the key set at the end of the SST needs to be read. For invalid keys, the corresponding values ​​can be skipped, and for valid keys, the corresponding values ​​can be retrieved. By configuring the garbage collection read amplification optimizer module 340 to store the key set at the end of the SST, read amplification can be minimized by enabling only valid key values ​​to be retrieved. In some embodiments, the BlockBasedTable format of the SST can be used, which ensures that each key-value pair occupies a dedicated data block, and the key information stored in the index block is a complete representation of the key in the data block. Therefore, the index block contains comprehensive key information for the SST.

[0114] The hot / cold data separator module 350 can be designed, programmed, or otherwise configured to reduce write amplification caused by mixed data (e.g., hot data and cold data). It should be understood that one of the main problems with mixed hot and cold data is that when hot data becomes invalid, cold data needs to be rewritten to reclaim space, resulting in additional write amplification. In this way, the hot / cold data separator module 350 can be configured to automatically identify the hotness and coldness of values ​​and ensure that values ​​with different hotness levels are moved to different blob files during refresh / compression / garbage collection / key-value separation to reduce garbage collection migration write amplification. To handle such tasks, the hot / cold data separator module 350 may include a hot data and cold data identifier module 351, a refresh optimizer module 352, a compression optimizer module 353, an optimization strategy module 354, and a compression optimizer module 355.

[0115] Cold data and hot data identifier module 351 can be designed, programmed or otherwise configured to identify cold data and hot data. In some embodiments, since PageStore does not have a rewrite function, but provides external read and write operations for pages, multiple versions of each page can be stored with a key organized as PageID+LSN (log sequence number), so the basic granularity for identifying hot pages and cold pages can be defined as PageID. In some embodiments, hot pages can include repeated writes to the same PageID, thereby generating an increasing LSN value. When LSN advances, during key SST compression, the compression filter can compare hot pages with (e.g., *LSN) to determine whether the data has become invalid. When the write frequency of some PageIDs is much higher than that of other PageIDs, these pages may be regarded as hot pages.

[0116] In some embodiments, an application programming interface (API) may be provided to provide application-level awareness of data heat and coldness by introducing an interface similar to CompactionFilter. In this way, the hot / cold data separator module 350 may be designed, programmed, or otherwise configured to implement hot and cold separations, for example, for rewriting data or moving data to different blob files. In other embodiments, the key-value engine may be designed, programmed, or otherwise configured to collect PageID information based on refresh and compaction contexts, thereby using frequency to distinguish between hot and cold data.

[0117] The refresh optimizer module 352 can be designed, programmed or otherwise configured to, during refresh time, identify pages that appear multiple times in the memTable, and treat pages that exceed a certain threshold K as hot pages, while pages that appear relatively less often can be identified as cold pages. Although heat classification has been discussed herein at hot data and cold data, such disclosure is not intended to be limiting. Instead, heat classification can be extended to more than just cold data and hot data. For example, during refresh, supporting multiple output streams can allow for finer granularity, classifying pages as warm, hot, cold, etc.

[0118] The compression optimizer module 353 can be designed, programmed or otherwise configured to optimize compression, for example, by implementing periodic KeySST compression, utilizing information from both upper and lower layers. In addition, the compression optimizer module 353 can be designed, programmed or otherwise configured to avoid overwriting to prevent delayed deletion propagation.

[0119] The optimization strategy module 354 can be designed, programmed or otherwise configured to optimize the garbage collection pickup strategy. It should be understood that since hot blob files and cold blob files can be distinguished during the refresh phase, and hot blob files can have a higher garbage rate, the correlation between the garbage rate and the blob file type can be established. In addition, in the case of blob files whose garbage rate reaches the garbage collection threshold, blob files of the same category can be merged. This is because achieving the same garbage rate may require a shorter hot blob file waiting time, while cold blob files may require a longer time. In addition, the average service life of cold blob files is expected to be significantly longer than that of hot blob files.

[0120] In some embodiments, cold pages can be identified within hot blob files by identifying pages that have undergone multiple copy operations during garbage collection, because such pages may become cold, for example, using statistics from a refresh window, for example, performing a reverse lookup on the refresh window. In some embodiments, the cold-hot conversion can be performed by removing hot key-value pairs, so there is no need to perform a reverse lookup on the refresh window.

[0121] The compression optimizer module 355 can be designed, programmed or otherwise configured to apply a compression algorithm to hot data or cold data on the upper layer and keep the hot data unchanged in the key-value engine. That is, since hot data has a relatively short life and a small workload, there is no need to use a heavy compression algorithm on hot data. However, for warm data and cold data, the warm data and cold data can benefit from compression to reduce storage space usage.

[0122] In some embodiments, for example, in a small write load scenario, where it may be challenging to distinguish between hot and cold data, all data may be considered hot due to its frequent use and short lifespan, and no compression algorithms may be applied to the data pages. This may help maintain efficient handling of data with minimal overhead.

[0123] In some embodiments, for example, a large write Zipf load scenario where a large amount of data is being written and there is a Zipf distribution indicating that some data is accessed more frequently than other data, such data can be identified as hot data. This hot data is then exempted from compression to maintain its accessibility and effectively reduce computational overhead. In addition, separating the storage of hot and cold data helps minimize garbage collection write amplification, thereby improving overall system performance.

[0124] Figure 8 An electronic device (e.g., a server, a host device, or a processor) adapted to implement at least some of the embodiments described herein. Figure 1800 of an exemplary computer system 800 of a terminal device shown in FIG. Figure 8 The computer system shown is for illustration only and is not intended to limit the functionality and applicability of the embodiments described herein.

[0125] As depicted, computer system 800 may include a central processing unit (CPU) 805. CPU 805 may perform various operations and processes based on programs stored in read-only memory (ROM) 810 or programs loaded from storage device 840 to random access memory (RAM) 815. RAM 815 may also store various data and programs required for the operation of system 800. CPU 805, ROM 810, and RAM 815 may be connected to each other via bus 820. Input / output (I / O) interface 825 may also be connected to bus 820.

[0126] The components connected to the I / O interface 825 may also include an input device 830 including a keyboard, a mouse, a digital pen, a drawing tablet, etc.; an output device 835 including a display such as a liquid crystal display (LCD), a speaker, etc.; a storage device 840 including a hard disk, etc.; and a communication device 845 including a network interface card such as a LAN card, a modem, etc. The communication device 845 can perform communication processing via a network such as the Internet, a WAN, a LAN, a LIN, a cloud, etc. In one embodiment, a drive 850 may also be connected to the I / O interface 825. A removable medium 855 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. may be installed on the drive 850 as desired, so that a computer program read from the removable medium 855 may be installed in the storage device 840.

[0127] It should be understood that reference FIG. 5A to FIG. 7 The processes described in the flowcharts and / or the processes described in other figures may be implemented as computer software programs or implemented in hardware. A computer program product may include a computer program stored in a computer-readable non-volatile medium. The computer program includes program code for executing the methods shown in the flowcharts and / or GUIs. In this embodiment, the computer program may be downloaded and installed from a network via a communication device 845, and / or may be installed from a removable medium 855. When the computer program is executed by a central processing unit (CPU) 805, the computer program may implement the above-mentioned functions specified in the methods in the embodiments disclosed herein.

[0128] Therefore, the methods and systems discussed herein may have one or more of the following advantages:

[0129] Accurate garbage collection management - In some embodiments, the key-value engine can be designed, programmed, or otherwise configured to consider garbage rates at the entry level rather than the space level, for example, to select blob files with the highest garbage ratio / percentage to address the problem of accurate garbage collection scheduling. By designing, programming, or otherwise configuring the key-value engine to consider garbage rates at the entry level, blob files with the highest garbage space for garbage collection can be selected to achieve more efficient garbage collection operations. In addition, the key-value engine can be enhanced by having garbage collection awareness, for example, by being aware of application-side data deletions communicated to the file system, including expired pages, for example, via a kernel file interface, an application programming interface, etc., to reduce unnecessary data movement. In addition, a periodic compaction process can be used to clean up expired pages and also reduce unnecessary data movement during the garbage collection process.

[0130] Adaptive Garbage Collection - In some embodiments, the key-value engine can be designed, programmed, or otherwise configured so that the garbage collection process is flexible, diverse, concurrent, and aware of the physical capacity of the storage device. For example, in some embodiments, the key-value engine can be designed, programmed, or otherwise configured to consider a wider range of garbage collection strategies, such as improving concurrency when in high water levels or areas, and optimizing space reclamation. In this way, the overall garbage collection performance of the key-value engine can be significantly enhanced compared to previous key-value engines, whose garbage collection strategies result in premature garbage collection, which can cause additional write amplification issues, insufficient concurrency during high water levels, and / or slow space reclamation.

[0131] Periodic compaction management - In some embodiments, the key-value engine can be designed, programmed, or otherwise configured to provide periodic compaction functionality, for example, for cleaning up expired pages. In some embodiments, periodic compaction can include merging multiple layers of the same page (or expired page) in a log structure merge (LSM) tree into the last layer so that the upper layers can perform corresponding functions during the compaction process. By doing so, the amount of data affected by the garbage collection process can be reduced. In addition, the key-value engine can be designed, programmed, or otherwise configured to prioritize the operations of garbage collection and / or compaction, for example, a higher priority process will be initiated before a lower priority process.

[0132] Garbage Collection Read Amplification Optimization - In some embodiments, the key-value engine can be designed, programmed, or otherwise configured so as to reduce the read amplification that occurs during garbage collection itself. For example, in some embodiments, redundant key sets can be stored at the end of a sorted string table (SST).

[0133] Optimization of hot and cold data - In some embodiments, the key-value engine can be designed, programmed, or otherwise configured to identify and separate hot and cold data to reduce write amplification, especially for mixed data scenarios where when hot data becomes invalid, rewriting cold data becomes necessary to reclaim space, which can result in additional WAF.

[0134] It should be understood that the disclosed and other solutions, examples, embodiments, modules and functional operations described in this document can be implemented in digital electronic circuit devices (circuitry), or in computer software, firmware or hardware, including the structures disclosed in this document and their structural equivalents, or in a combination of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a computer-readable medium, for execution by a data processing device or controlling the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing device" includes all devices, equipment and machines for processing data, such as a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0135] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any manner, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions). A computer program may be deployed to execute on one or more computers located at one site or distributed at multiple sites and interconnected by a communications network.

[0136] The processes and logic flows described in this document may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and the apparatus may also be implemented as, special purpose logic circuit devices, such as field programmable gate arrays, application specific integrated circuits, and the like.

[0137] For example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operably coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as erasable programmable read-only memory, electrically erasable programmable read-only memory, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and compact optical disk read-only memory and digital video optical disk read-only memory disks. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit device.

[0138] It should be understood that different features, variations and multiple different embodiments have been shown and described in various details. The content sometimes described in this application according to specific embodiments is only for illustrative purposes, and is not intended to limit or imply that the content envisioned is only one specific embodiment or multiple specific embodiments. It should be understood that the present disclosure is not limited to any single specific embodiment or the variants listed. Those skilled in the art will think of many modifications, variations and other embodiments, and these modifications, variations, variations and embodiments are intended to and are actually covered by the present disclosure. In fact, the scope of the present disclosure is intended to be determined by the appropriate legal interpretation and explanation of the present disclosure, including equivalent forms understood by those skilled in the art relying on the complete disclosure present at the time of submission.

[0139] aspect:

[0140] It should be understood that any of the aspects may be combined with each other.

[0141] Aspect 1. A method for garbage collection of a key-value engine in a data storage and communication system, the method comprising: determining the disk capacity usage of the key-value engine; adjusting the garbage collection percentage threshold and the number of garbage collection threads based on whether the disk capacity usage of the key-value engine meets and / or exceeds a predetermined disk capacity usage threshold.

[0142] Aspect 2. The method according to Aspect 1 also includes: performing a periodic compaction process to consolidate the one or more expired pages on a log structured merge (LSM) tree by merging one or more layers into the last layer of one or more expired pages of one or more applications to reduce data disposal during the garbage collection.

[0143] Aspect 3. A method according to aspect 2, wherein the performing of the periodic compression process occurs together with an automatic compression process.

[0144] Aspect 4. A method according to any one of aspects 2 to 3, wherein the execution of the periodic compression process includes: coordinating the timing of executing the periodic compression process and reclaiming memory during the execution of the timed compression process.

[0145] Aspect 5. The method according to any one of Aspects 1 to 4 further includes: determining a garbage volume for one or more blob files in the key-value engine, wherein the determining of the garbage volume includes: storing the value size of the one or more blob files in an incremental block in a sorted string table (SST) of the key-value engine; summing the sizes of all SST entries depending on the one or more blob files to determine the valid data size of the one or more blob files; determining any remaining size of the one or more blob files as the garbage volume; and performing a garbage collection process on the selected blob files.

[0146] Aspect 6. The method according to Aspect 5 further includes: storing a redundant key set for the one or more blob files at the end of the SST, wherein only the redundant key set is read during garbage compaction.

[0147] Aspect 7. The method according to any one of Aspects 5 to 6 further includes: determining the hotness and / or coldness of the one or more blob files, wherein the determining of the hotness and / or coldness includes one or more of the following items: using an application programming interface with application-level awareness to identify the hotness and / or coldness of the one or more blob files; using the frequency of page writes to distinguish the hotness and / or coldness of the one or more blob files; or identifying pages that appear multiple times in the memTable, wherein the hotness and / or coldness of the one or more blob files is based on a threshold.

[0148] Aspect 8. A method according to any one of Aspects 1 to 7, wherein the adjustment of the garbage collection percentage threshold and the number of garbage collection threads includes one or more of the following items: wherein when the disk space utilization is lower than a first predetermined threshold, the number of garbage collection threads is set to a first thread value, and the garbage collection (GC) percentage threshold is dynamically set between a first GC percentage threshold and a second GC percentage threshold based on disk usage; wherein when the disk space utilization is between a second predetermined threshold and a third predetermined threshold, the number of garbage collection threads and the garbage collection percentage threshold are dynamically adjusted, wherein the dynamic adjustment includes: increasing or decreasing the number of garbage collection threads and / or the garbage collection percentage threshold based on whether the disk space utilization and / or the disk usage increases or decreases.

[0149] Aspect 9. An active garbage collection system for a key-value engine, the active garbage collection system comprising: an adaptive garbage collection module for implementing a flexible and adaptive garbage collection strategy, wherein the adaptive garbage collection module is configured to determine the disk capacity usage of the key-value engine, and adjust the garbage collection percentage threshold and the number of garbage collection threads based on whether the disk capacity usage of the key-value engine meets and / or exceeds a predetermined disk capacity usage threshold.

[0150] Aspect 10. The active garbage collection system according to Aspect 9 further comprises a periodic compression module, wherein the periodic compression module is used to implement periodic compression to optimize garbage collection efficiency and space recovery.

[0151] Aspect 11. The active garbage collection system according to any one of Aspects 9 to 10 further includes one or more of the following items: an accurate garbage collection module for providing accurate information for garbage collection; a garbage collection read amplification optimizer module for reducing read amplification during garbage collection; or a hot data and cold data separator module for facilitating the identification and separation of hot data and cold data to minimize write amplification.

[0152] Aspect 12. The active garbage collection system according to aspect 10, wherein the periodic compaction module comprises: a periodic compaction scheduler module for scheduling periodic compaction operations and a periodic compaction executor for executing the scheduled compaction operations.

[0153] Aspect 13. An active garbage collection system according to any one of Aspects 11 to 12, wherein the hot data and cold data separator module includes one or more of the following items: a hot data and cold data identifier module, for classifying data as hot data or cold data based on usage patterns; a refresh optimizer module, for optimizing data refresh to reduce write amplification based on whether the data is hot data or cold data; a compression optimizer module, for optimizing the compression process based on the hot data or the cold data to improve space recovery; a garbage collection optimization strategy module, for providing diversified garbage collection strategies based on the hot data or the cold data; and a compression optimizer module, for optimizing data compression technology based on the hot data or the cold data to reduce storage overhead.

[0154] Aspect 14. An active garbage collection system according to any one of Aspects 9 to 13, wherein the adaptive garbage collection module is configured to determine the disk capacity usage of the key-value engine and adjust the garbage collection percentage threshold and the number of garbage collection threads based on a predetermined disk capacity usage threshold.

[0155] Aspect 15. An active garbage collection system according to Aspect 14, wherein the garbage collection percentage threshold and the number of garbage collection threads include one or more of the following items: wherein when the disk space utilization is lower than a first predetermined threshold, the number of garbage collection threads is set to a first thread value, and the garbage collection GC percentage threshold is dynamically set between a first GC percentage threshold and a second GC percentage threshold based on disk usage; wherein when the disk space utilization is between a second predetermined threshold and a third predetermined threshold, the number of garbage collection threads and the garbage collection percentage threshold are dynamically adjusted, wherein the dynamic adjustment includes: increasing or decreasing the number of garbage collection threads and / or the garbage collection percentage threshold based on whether the disk space utilization and / or the disk usage increases or decreases.

[0156] Aspect 16. A non-volatile storage device having an executable component stored thereon, which, when executed by a processor, causes the processor to: determine the disk capacity usage of the key-value engine; and adjust the garbage collection percentage threshold and the number of garbage collection threads based on whether the disk capacity usage of the key-value engine meets and / or exceeds a predetermined disk capacity usage threshold.

[0157] Aspect 17. A non-volatile storage device according to Aspect 16, wherein the processor is further configured to perform a periodic compaction process to consolidate the one or more expired pages on a log structured merge (LSM) tree by merging one or more layers into the last layer of one or more expired pages of one or more applications to reduce data disposal during garbage collection.

[0158] Aspect 18. The non-volatile storage device according to Aspect 17, wherein the performing of the periodic compression process comprises: coordinating the timing of performing the periodic compression process and reclaiming memory during the performing of the periodic compression process.

[0159] Aspect 19. A non-volatile storage device according to any one of Aspects 16 to 18, wherein the processor is further configured to: determine a garbage volume for one or more blob files in the key-value engine, wherein the determining of the garbage volume comprises: storing the value size of the one or more blob files in an incremental block in a sorted string table SST of the key-value engine; summing the sizes of all SST entries depending on the one or more blob files to determine the valid data size of the one or more blob files; determining any remaining size of the one or more blob files as the garbage volume; and performing a garbage collection process on the selected blob files.

[0160] Aspect 20. A non-volatile storage device according to Aspect 19, wherein the processor is further configured to: determine the hotness and / or coldness of the one or more blob files, wherein the determining of the hotness and / or the coldness includes one or more of the following items: using an application programming interface with application-level awareness to identify the hotness and / or the coldness of the one or more blob files; using the frequency of page writes to distinguish the hotness and / or the coldness of the one or more blob files; or identifying pages that appear multiple times in the memTable, wherein the hotness and / or the coldness of the one or more blob files is based on a threshold.

[0161] Aspect 21. A non-volatile storage device according to any one of Aspects 18 to 20, wherein the processor is further configured to: store a redundant key set for the one or more blob files at the end of the SST, wherein only the redundant key set is read during garbage compaction.

[0162] Aspect 22. A non-volatile storage device according to any one of Aspects 16 to 21, wherein the adjustment of the garbage collection percentage threshold and the number of garbage collection threads includes one or more of the following items: wherein when the disk space utilization is lower than a first predetermined threshold, the number of garbage collection threads is set to a first thread value, and the garbage collection (GC) percentage threshold is dynamically set between a first GC percentage threshold and a second GC percentage threshold based on disk usage; wherein when the disk space utilization is between a second predetermined threshold and a third predetermined threshold, the number of garbage collection threads and the garbage collection percentage threshold are dynamically adjusted, wherein the dynamic adjustment includes: increasing or decreasing the number of garbage collection threads and / or the garbage collection percentage threshold based on whether the disk space utilization and / or the disk usage increases or decreases.

[0163] The terms used in this specification are intended to describe particular embodiments and are not intended to be limiting. Unless expressly indicated otherwise, the terms "a", "an", and "the" include plural forms. When the terms "comprises" and / or "comprising" are used in this specification, the presence of stated features, integers, steps, operations, elements, and / or components is specified, but the presence or addition of one or more other features, integers, steps, operations, elements, and / or components is not excluded.

[0164] With respect to the foregoing description, it should be understood that changes in detail may be made without departing from the scope of the present disclosure, especially in the construction materials adopted and the shape, size and arrangement of parts. This specification and the described embodiments are exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A method for garbage collection of a key-value engine in a data storage and communication system, the method comprising: Determining disk capacity usage of the key-value engine; Based on whether the disk capacity usage of the key-value engine meets and / or exceeds a predetermined disk capacity usage threshold, a garbage collection percentage threshold and the number of garbage collection threads are adjusted.

2. The method according to claim 1, further comprising: A periodic compaction process is performed to consolidate the one or more expired pages on a log-structured merge LSM tree by merging one or more layers into a last layer of one or more expired pages of one or more applications to reduce data handling during the garbage collection.

3. The method of claim 2, wherein said performing said periodic compression process occurs in conjunction with an automatic compression process.

4. The method of claim 2, wherein performing the periodic compression process comprises: Coordinate the timing of performing the periodic compaction process and reclaim memory during the performing the timed compaction process.

5. The method according to claim 1, further comprising: determining a garbage volume for one or more blob files in the key-value engine, Wherein determining the volume of the garbage comprises: Storing the value size of the one or more blob files in a delta block in a sorted string table SST of the key-value engine; summing the sizes of all SST entries dependent on the one or more blob files to determine an effective data size of the one or more blob files; determining any remaining size of the one or more blob files as the trash volume; and The garbage collection process for the selected blob files occurs.

6. The method according to claim 5, further comprising: A redundant set of keys for the one or more blob files is stored at an end of the SST, wherein only the redundant set of keys is read during garbage compaction.

7. The method according to claim 5, further comprising: determining the hotness and / or coldness of the one or more blob files, Wherein the determining of the heat and / or the cold comprises one or more of the following: identifying the hotness and / or the coldness of the one or more blob files using an application programming interface with application-level awareness; Using the frequency of page writes to distinguish the hotness and / or coldness of the one or more blob files; or Pages that appear multiple times in a memTable are identified, wherein the hotness and / or the coldness of the one or more blob files is based on a threshold.

8. The method of claim 1, wherein the adjusting the garbage collection percentage threshold and the number of garbage collection threads comprises one or more of: wherein when the disk space utilization is lower than a first predetermined threshold, the number of garbage collection threads is set to a first thread value, and the garbage collection GC percentage threshold is dynamically set between a first GC percentage threshold and a second GC percentage threshold based on disk usage; When the disk space utilization is between the second predetermined threshold and the third predetermined threshold, the number of garbage collection threads and the garbage collection percentage threshold are dynamically adjusted, wherein the dynamic adjustment includes: The number of garbage collection threads and / or the garbage collection percentage threshold are increased or decreased based on whether the disk space utilization and / or the disk usage increases or decreases.

9. An active garbage collection system for a key-value engine, the active garbage collection system comprising: An adaptive garbage collection module is used to implement a flexible and adaptive garbage collection strategy, wherein the adaptive garbage collection module is configured to determine the disk capacity usage of the key-value engine, and adjust the garbage collection percentage threshold and the number of garbage collection threads based on whether the disk capacity usage of the key-value engine meets and / or exceeds a predetermined disk capacity usage threshold. 10 . A non-volatile storage device having an executable component stored thereon, wherein when the executable component is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 8.

Images