Storage systems and data storage methods

Abstract

This invention discloses a storage system and data storage method, relating to the field of storage technology. The method includes constructing a user interface for key-value operations, an independent key storage layer, and a value storage layer. The key storage layer is configured to store keys and corresponding value reference information, while the value storage layer is configured to independently store the value data corresponding to the value reference information. When a data write operation request is received through the user interface, the value storage layer allocates corresponding value reference information for the value data to be written, first records the value write log of the value data, and then persistently stores the value data. In the key storage layer, the value reference information and its corresponding key are first recorded in the key write log, and then persistently stores the value reference information and its corresponding key. This invention can solve the problem of key-value coupling affecting storage performance and cost in related technologies, achieving complete KV separation and improving storage system performance and resource utilization.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and in particular to a storage system and a data storage method. Background Technology

[0002] Key-value (KV) is a data organization method in which the key and its corresponding value are stored and managed as a whole. This tight coupling of KV can seriously affect the performance and resource utilization of the storage system in scenarios with large-scale data and large values.

[0003] Therefore, achieving complete separation of key and value storage (KV) and improving storage system performance and resource utilization is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] This invention provides a storage system and data storage method that can achieve complete KV separation, effectively improving storage system performance and resource utilization.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a data storage method, comprising: Construct a unique user interface for key-value operations with the user client, and construct independent key storage layer and value storage layer; wherein, the key storage layer is configured to store keys and value reference information corresponding to the keys, and the value storage layer is configured to independently store value data corresponding to the value reference information; When a data write operation request is received through the user operation interface, the target value reference information is allocated for the target value data to be written in the value storage layer. The value of the target value data is first recorded and written to the log, and then the target value data is persisted and stored. In the key storage layer, the target value reference information and the key of the corresponding target key are first recorded and written to the log, and then the target value reference information and the corresponding target key are persisted and stored.

[0006] Another aspect of the present invention provides a storage system, including a user interface, a storage device, and a processor; the storage device includes a program storage area, a key storage layer, and a value storage layer; wherein, the program storage area is used to store computer programs; and the processor is used to implement the steps of the above data storage method when executing the computer program.

[0007] The advantages of the technical solution provided by this invention are that it provides a unified key-value operation interface, shields the underlying implementation of key-value separation, and ensures that users can use the existing key-value storage method without modifying the code. By constructing independent key storage layers and value storage layers, where the key storage layer only stores keys and lightweight value reference information, and no longer contains the value data itself, while the value data itself is stored independently in the value storage layer, the key storage layer and value storage layer are independent of each other, achieving complete separation of keys and values. When the background merging operation processes the data in the key storage layer, the objects read and written are only key and value reference information, and there is no longer any movement or rewriting of value data. This fundamentally eliminates the repeated reading and writing of value data caused by the merging operation, and completely eliminates the disk I / O overhead caused by the repeated reading and writing of value data in the background merging operation. This directly brings about the technical effects of significantly reducing the write amplification factor, greatly reducing disk input and output overhead, effectively improving write throughput, reducing the number of solid-state drive erase and write cycles, and extending the service life. Based on the complete separation of keys and values, the write operation adopts the following sequence: first, record the value in the log and then persist the value data; second, record the key in the log and then persist the key-value reference information. This ensures that if any step fails, data rollback can be performed through the corresponding write log, avoiding inconsistencies between keys and values. Therefore, the atomicity and consistency of data writing are guaranteed under the key-value separation architecture. Furthermore, this invention also provides a corresponding storage system for the data storage method, which has corresponding advantages. Attached Figure Description

[0008] To more clearly illustrate the technical solutions of the present invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 A flowchart illustrating a data storage method provided by the present invention; Figure 2 This invention provides a schematic diagram of the data writing process. Figure 3 This is a schematic diagram of the data reading process provided by the present invention; Figure 4 This is a schematic diagram of the data deletion process provided by the present invention; Figure 5 This is a structural framework diagram of an exemplary embodiment of the data storage device provided by the present invention; Figure 6 A structural diagram of an exemplary embodiment of the storage system provided by the present invention; Figure 7This is a schematic diagram of the data storage process of the storage system provided by the present invention. Detailed Implementation

[0010] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. In this specification and the aforementioned drawings, the terms "first," "second," "third," "fourth," etc., are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. The term "exemplary" means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior to or better than other embodiments.

[0011] In a key-value (KV) storage system architecture, data persistence involves the complete combination of key-value pairs, such as the SSTable (Sorted String Table) file in an LSM-tree (Log-Structured Merge-Tree) or the leaf nodes of a B+ tree. Taking an LSM-tree-based storage engine as an example, both the in-memory data structure (Mem Table) and the disk-based SSTable (Sorted String Table) store complete keys and values. This design requires processing the complete key-value pairs during data writing; during data reading, the complete key-value pair record must be located using the key, and then the value is extracted. To maintain data order and clean up invalid data, the LSM-tree frequently performs compaction operations in the background. This process involves reading, merging, and rewriting multiple SSTable files, including the repeated movement and rewriting of a large amount of value data. This not only consumes significant CPU resources and disk I / O bandwidth, reducing the overall write throughput of the system, but also accelerates the wear and tear of the storage media, leading to increased storage costs. Secondly, because the key and value are coupled, the cache used to accelerate queries is easily occupied by large values, which leads to a decrease in the cache hit rate of the key and thus affects read performance.

[0012] To address the issues arising from key-value coupling, the related technology RocksDB (an embedded key-value storage engine) separates the storage of large values ​​(blobs) from the keys. The keys and pointers to blobs are retained in the original LSM-tree structure, while the large values ​​are stored separately in the Blob DB (the plugin name for storing large binary objects) within the blob file. The overall workflow is as follows: write operations are recorded in the WAL (Write-Ahead Log) to ensure persistence; then, key-value pairs are written to an ordered structure (MemTable) in memory. When the MemTable reaches a preset capacity threshold, its state transitions to a read-only immutable memory table (Immutable MemTable), triggering a flush operation to generate an SSTable stored on disk. A background thread periodically performs compaction, merging multiple SSTable files to delete expired data and merge duplicate keys, thereby maintaining data order and efficient use of storage space.

[0013] However, this method only separates large values ​​exceeding a threshold; smaller values ​​remain bound to the key, failing to achieve complete key-value separation. Furthermore, the lifecycle of a blob file heavily depends on the corresponding SSTable in the LSM-tree, requiring garbage collection (GC) coordination at the SSTable level, increasing system complexity and overhead. Additionally, since the key and value still share the same storage medium, differentiated cost optimization based on their different access characteristics and capacity requirements is impossible, ultimately failing to meet user needs in terms of storage resource utilization.

[0014] Therefore, this invention first constructs a unified entry point for the storage system to interact with the external environment, namely, a user interface for key-value operations with the user terminal. Within the storage system, it constructs independent key and value storage layers. The key storage layer is specifically used to store keys and their corresponding value references, while the value storage layer is specifically used to independently store the value data corresponding to the value references. In practical applications, the key storage layer can, for example, adopt a log-structured merged tree architecture, including components such as a key memory table, an immutable key memory table, an ordered key string table, and a key write-ahead log. The value storage layer can include components such as a value memory cache, a value persistent storage pool, a value metadata index, and a value write-ahead log. When a data write operation request is received through the user interface, the value storage layer first allocates corresponding target value reference information for the target value data to be written. Subsequently, the target value data is first recorded in the value write log, that is, the target value data and its metadata information are written to the value write-ahead log file for persistence, and then the target value data itself is persisted, that is, the target value data is written to the persistent storage pool of the value storage layer. In the key storage layer, the target value reference information and its corresponding target key are first recorded in the key write log. This means the mapping between the target key and the target value reference information is persisted in the key write-ahead log file. Then, the target value reference information and its corresponding target key are persisted in the key memory table. This data storage method achieves complete separation of keys and values, solving the technical problems in related key-value storage systems where tight coupling between keys and values ​​leads to repetitive reading and writing of large amounts of value data during background merging operations, high disk I / O overhead, and limited write throughput.

[0015] The specific application environment architecture or hardware architecture upon which the data storage method depends is described herein. The following examples illustrate some possible application scenarios related to the technical solution of this invention, such as log storage systems, and may include the following: Data center business systems generate massive amounts of log data daily. Each log record contains a log identifier as a key and information such as log content, timestamp, log level, and source address as a value. The value size of log data varies significantly; some logs contain detailed error stack information, resulting in large value sizes, while others are only brief status records, leading to small value sizes. Furthermore, log data exhibits a clear temporal distribution, with recent logs accessed more frequently and historical logs accessed less frequently. This invention's storage system is deployed in this scenario. The storage system constructs independent key storage layers and value storage layers. The key storage layer uses solid-state drives (SSDs) as the storage medium to ensure low latency for key queries, while the value storage layer uses a hybrid storage pool of SSDs, hard disk drives (HDDs), and object storage. The mapping between log identifiers and value reference information is stored in the key storage layer; log content and equivalent data are stored in the value storage layer.

[0016] When a business system writes a new log record, the storage system first performs a hash calculation on the log content at the value storage layer to generate a data fingerprint. If the fingerprint already exists in the fingerprint index, it means that the same log content has been stored before, and the system directly reuses the existing value reference information; if it does not exist, the system allocates new value reference information and stores the log content in the value storage layer. The value storage layer dynamically selects the storage medium based on the size of the log content and the expected access frequency: logs that are accessed frequently in the near future are stored on solid-state drives (SSDs); logs that are large in size but accessed less frequently are stored on hard disk drives (HDDs); and historical archived logs are stored in object storage.

[0017] When a business system needs to query logs within a certain time period, it can initiate a batch read request through the user interface, simultaneously querying multiple log identifiers. The key storage layer queries the value reference information corresponding to these log identifiers in parallel, and the value storage layer allocates an independent read thread for each value reference, concurrently reading log content from different storage media. Finally, the log records are assembled and returned to the business system.

[0018] When the system restarts due to a power outage, the storage system replays the key-write-ahead log and value-write-ahead log respectively, reconstructs the memory table state and metadata index before the failure, and traverses all key storage layer data to recalculate the reference count of each value reference information to ensure that the data before and after the failure is consistent and the reference count is accurate.

[0019] Through the above implementation methods, the log storage system effectively reduces storage costs, improves write throughput and batch read performance while ensuring data reliability, and meets the comprehensive requirements of business systems for high performance, low cost and high reliability of log storage.

[0020] It should be noted that the above application scenarios are only shown to facilitate understanding of the ideas and principles of the present invention, and the embodiments of the present invention are not limited in any way. On the contrary, the embodiments of the present invention can be applied to any applicable scenario. After introducing the technical solution of the present invention, various non-limiting embodiments of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Please refer to [link to previous text] first. Figure 1 According to a data storage method provided by the present invention, it can be implemented as a computer program product, installed and running in the processor of a service node or log storage server of a distributed key-value storage system, for implementing separate storage and efficient access of key-value data. In some embodiments of the method, the method includes the following steps: S101: Construct a unique user interface for key-value operations with the user end, and construct independent key storage layers and value storage layers.

[0021] The user interface serves as the unified entry point for the storage system to interact with external systems. It receives various key-value operation requests and responses from users, including write, read, delete, and scan operations. It shields the underlying key and value storage layers from their separate implementation logic, allowing users to maintain compatibility with existing key-value storage usage without modifying any code. The key storage layer is an independent storage level configured to store keys and their corresponding value references, but does not contain any value data itself. The value reference information is used to associate the keys in the key storage layer with the value data in the value storage layer. It is a lightweight, globally unique identifier, such as a combination of a universally unique identifier and a timestamp, used to uniquely identify a value and locate it within the value storage layer. The value storage layer is also an independent storage level configured to independently store the value data corresponding to the value reference information. It is dedicated to storing value data that corresponds one-to-one with the value reference information, and is not data-coupled with the key storage layer. It can be managed and adapted to storage media independently. By implementing a key-value storage layer, complete KV separation is achieved across all scenarios, overcoming the limitation of only separating large values ​​in related technologies. All values ​​and keys are stored independently, adapting to mixed large / small value scenarios.

[0022] In key-value coupled storage architectures (such as storage engines based on log-structured merge trees), background compaction operations need to read complete key-value pairs stored in multiple ordered string table (SSTable) files. After merging and sorting, the valid key-value pairs are rewritten to new SSTable files. During this process, value data, as a component of the key-value pairs, is repeatedly read and written, generating significant data migration overhead, i.e., write amplification. When keys and values ​​are completely separated through independent key and value storage layers, the ordered string table (SSTable) in the key storage layer only stores keys and lightweight value reference information (such as value identifiers), no longer containing the value data itself. Therefore, when the background compaction operation processes data in the key storage layer, the objects read and written are only key and value reference information, completely eliminating the migration of value data. This means that the disk I / O overhead of repeatedly reading and writing value data caused by compaction is completely eliminated.

[0023] S102: When a data write operation request is received through the user operation interface, the target value reference information is allocated for the target value data to be written in the value storage layer. The value of the target value data is first recorded and written to the log, and then the target value data is persisted and stored. In the key storage layer, the target value reference information and the key of the corresponding target key are first recorded and written to the log, and then the target value reference information and the corresponding target key are persisted and stored.

[0024] The target value reference information is the value reference information of the data to be written corresponding to the currently received data write operation request. It is generated during allocation. The value data to be written is defined as the target value data, and the key of the data to be written is defined as the target key. The target value reference information is used to uniquely identify the target value data. The value write log records the allocation of target value reference information and the persistence storage process of target value data in the value storage layer, which is used for data recovery and consistency verification in case of failure. The key write log records the association and persistence storage process of target keys and corresponding target value reference information in the key storage layer, which works with the value write log to ensure the consistency of key-value association.

[0025] In this embodiment, the user client sends a data write operation request to the storage system through the user operation interface. When the storage system receives the data write operation request through the user operation interface, it first allocates globally unique target value reference information for the target value data to be written in the value storage layer. The allocation process and related target value data information are recorded in the value write log. After the log recording is completed, the target value data is persistently stored in the corresponding storage medium of the value storage layer. After the above operations are completed in the value storage layer, the target value reference information and its corresponding target key association are first recorded in the key write log in the key storage layer. After the log recording is completed, the association is persistently stored in the key storage layer. This realizes the operation logic of writing the log first and then storing the data, ensuring that if any step fails, the data can be rolled back through the log, avoiding the problem of key-value inconsistency.

[0026] The technical solution provided in this embodiment offers a unified key-value operation interface, shielding the underlying implementation of key-value separation and ensuring that users can use existing key-value storage methods without modifying their code. By constructing independent key and value storage layers, with the key storage layer storing only keys and lightweight value reference information (excluding the value data itself), and the value data stored independently in the value storage layer, the key and value storage layers are completely independent, achieving complete separation of keys and values. When the background merging operation processes data in the key storage layer, the objects read and written are only key and value reference information, completely eliminating the need to move and rewrite value data. This fundamentally eliminates the repeated reading and writing of value data caused by the merging operation, completely eliminating the disk I / O overhead caused by repeated reading and writing of value data in the background merging operation. This directly results in a significant reduction in write amplification factor, a substantial reduction in disk input / output overhead, an effective increase in write throughput, a reduction in solid-state drive erase / write cycles, and an extension of lifespan. Based on the complete separation of keys and values, the write operation adopts the following order: first record the value written to the log and then persist the value data; first record the key written to the log and then persist the correspondence between key and value reference information. This ensures that if any step fails, the data can be rolled back through the corresponding write log, avoiding inconsistencies between keys and values. Thus, the atomicity and consistency of data writing are guaranteed under the key-value separation architecture.

[0027] Based on the above embodiments, the value memory cache area of ​​the value storage layer is optimized to achieve cache data filtering and writing based on access frequency. This specifically solves the technical problem in traditional key-value stores where memory caches store data indiscriminately, resulting in large-volume, low-frequency access data occupying cache resources and high-frequency data having a low cache hit rate. This optimization may include the following: In this embodiment, the value storage layer also includes a value memory cache area, which counts the access frequency of each value data stored in the value storage layer and selects candidate value data with an access frequency greater than or equal to a preset access frequency; and writes the value data of each candidate value data whose storage space occupancy value is within a preset data scale range to the value memory cache area.

[0028] Candidate value data refers to value data in the value storage layer whose access frequency is greater than or equal to a preset access frequency, and is eligible for memory caching. The preset access frequency is a threshold value set for the access frequency of the value data; it is the standard for determining whether value data is frequently accessed and can be flexibly configured according to business needs. Storage space usage refers to the amount of storage space occupied by the value data in the storage medium, i.e., the data volume. The preset data size range is a critical range for the cached data volume set in the value memory cache area; only value data within this range is allowed to be written to the cache, preventing large data volumes from occupying too many cache resources.

[0029] In this embodiment, the access frequency of each value data in the value storage layer is continuously counted, and the number of times each value data is accessed within a preset time period is recorded; such as Figure 2 As shown, based on the statistical results, value data with an access frequency greater than or equal to a preset access frequency is selected as candidate value data. This data is high-frequency access data and is eligible to be written to the memory cache. For all the selected candidate value data, it is further determined whether its storage space usage is within a preset data size range: if the storage space usage of the candidate value data is within the preset data size range, it is written to the value memory cache area of ​​the value storage layer to provide fast access support for subsequent read operations; if the storage space usage of the candidate value data exceeds the preset data size range, even if it is high-frequency access data, it is not written to the value memory cache area to avoid it occupying too many cache resources; for example, the value memory cache only stores value data with high access frequency (e.g., read more than 3 times in the last hour) and moderate size (e.g., less than 1MB). The value storage layer uses the LRU (Least Recently Used) strategy to manage the value memory cache area. When the cache area reaches the storage limit, the least recently used cache data is deleted to free up space for new candidate value data that meets the conditions.

[0030] In this embodiment, based on dual filtering of access frequency and data size, only frequently accessed and moderately sized value data is written to the memory cache, avoiding the occupation of cache resources by large-volume, low-frequency data and significantly improving the cache hit rate of the value memory cache area. By using LRU cache management, it is ensured that the latest and most frequently used value data is always stored in the cache area, further improving cache utilization efficiency and reducing read operation latency. The cache filtering and management logic is automatically executed in the background of the value storage layer without user intervention and without affecting the normal operation of the key storage layer, ensuring the overall operating efficiency of the storage system. The preset access frequency and preset data size range can be flexibly configured according to the user's business needs to adapt to different application scenarios, improving the scenario adaptability of the storage system.

[0031] Based on the above embodiments, in order to avoid the problem of low storage resource utilization and high storage costs caused by the duplicate storage of identical value data in the value storage layer, this embodiment adds an index library and data fingerprint calculation logic to the value storage layer to achieve global deduplication of value data, which may include the following: In this embodiment, the value storage layer also includes an index library. The target value data is hashed in the value storage layer, and the hash calculation result is used as a unique identifier for the target value data content corresponding to the target data fingerprint. The index library is queried according to the target data fingerprint. If the target data fingerprint does not exist in the index library, the corresponding target value reference information is allocated for the target value data to be written in the value storage layer. While persistently storing the target value data, a new index record is generated using the target data fingerprint as the index key and the target value reference information and the metadata information of the target value data as the index value, identifying the mapping relationship between the target data fingerprint and the target value reference information, and then written to the index library.

[0032] The index library is a dedicated index structure in the value storage layer used to store the mapping relationship between data fingerprints, value reference information, and value data metadata information. It supports fast querying and insertion based on data fingerprints. Hash calculation is the process of converting target value data of arbitrary length into a fixed-length hash value, used to generate a data fingerprint that uniquely identifies the value data content. The target data fingerprint is a fixed-length string obtained through hash calculation; it is a unique identifier for the value data content. Value data with the same content generates completely identical data fingerprints, while different content generates different fingerprints. Metadata information describes the attributes of the value data, including the storage medium, physical storage location, creation time, expiration time, and data size. A new index record uses the target data fingerprint as the key (defined as the index key for easy differentiation) and the target value reference information and target value metadata information as the value (also defined as the index value for easy differentiation). The generated index entries, used to record the mapping relationship among the three, are written to the index library to establish the association between data fingerprints and value data.

[0033] In this embodiment, when a data write operation is performed, the value storage layer uses a hash algorithm (such as SHA-256) to calculate the value data to be written, generating a data fingerprint that uniquely identifies the value content. The value storage layer maintains a distributed index, which stores data fingerprints as keys and corresponding value reference information and the physical storage location of the value data as values. Before writing the value data, such as... Figure 2 As shown, the system first uses the calculated data fingerprint as the query condition to search the index. If the search result indicates that the fingerprint already exists, it means that the same value data has already been stored. The system will not write the value data again, but will directly obtain the existing value identifier for subsequent processing, thus achieving deduplication at the storage level. If the search result indicates that the fingerprint does not exist, the current value data is determined to be new and unique. The system assigns a new value identifier to it and writes the value data to the persistent storage pool. At the same time, a new record is added to the fingerprint index, establishing a mapping relationship between the data fingerprint and the value identifier. Taking the value reference information as Value ID as an example, the value storage layer achieves value deduplication through hash fingerprint, assigns Value ID and writes it to the Value WAL and storage medium. The key storage layer writes (Key, ValueID) and Key WAL, and asynchronously flushes to disk to generate an SSTable containing only Key and ValueID.

[0034] As shown above, this embodiment achieves global deduplication of value data based on data fingerprints during writing. Regardless of how many times the same value data is written, only one copy is stored in the value storage layer, completely solving the problem of duplicate value data consuming large amounts of storage resources in traditional storage. Data fingerprints are generated through hash calculations, possessing uniqueness and fixed length. Querying and comparison based on data fingerprints are extremely efficient, without adding extra system overhead due to deduplication logic, ensuring the execution efficiency of data write operations. The introduction of the index library enables precise association between data fingerprints, value reference information, and metadata information. When reusing value reference information of duplicate data, existing value data can be quickly located, ensuring the consistency of key-value associations during duplicate data writing. The deduplication logic is executed independently in the value storage layer, completely transparent to the key storage layer and user interface. The user does not need to be aware of the deduplication process or modify any writing logic, improving the usability of the storage system. Combined with the key-value separation architecture, the deduplication of value data, along with the lightweight storage of the key storage layer, further improves the resource utilization of the entire storage system, maximizing the optimization effect of storage costs.

[0035] Based on the above embodiments, this embodiment further optimizes the persistent storage process of value data to achieve differentiated storage medium allocation based on the characteristics of value data, which may include the following: Based on the access frequency and physical attributes of the target value data, determine the matching target storage medium and persistently store the target value data to the target storage medium; use the target value reference information as the metadata key, and use the physical storage information, creation time, and expiration time of the target storage medium as the metadata value, and store the metadata key and metadata value as the metadata index information of the target value data.

[0036] Access frequency refers to the number of times value data is accessed by a user through read operations. It serves as an indicator of the access frequency of value data and can be categorized into high-frequency, medium-frequency, and low-frequency levels. Physical attributes are inherent properties of the value data itself, including data size, data type, and update frequency, and are the basis for determining value data storage needs. The target storage medium is matched based on the access frequency and physical attributes of the value data, representing the most suitable persistent storage medium, including solid-state drives (SSDs), hard disk drives (HDDs), and object storage. The metadata key uses value reference information as the key, and its corresponding value is defined as the metadata value. The metadata key is used to locate the metadata value of the corresponding value data in the metadata index. The metadata value is a set of information including the physical storage information of the target storage medium, the creation time of the value data, and the expiration time; it is the content of the metadata index. Physical storage information describes the specific storage location of the value data in the target storage medium, and may include storage medium type, file path, offset, and length.

[0037] In this embodiment, when persistently storing target value data, the access frequency and physical attributes (such as data size and update frequency) of the target value data are first analyzed. Based on preset media matching rules, a matching target storage medium is determined. For example, high-frequency access, small size, and high update frequency value data is matched with a solid-state drive (SSD), while low-frequency access, large size, and low update frequency value data is matched with a hard disk drive (HDD) or object storage. The target value data is then persistently stored in the matched target storage medium. For instance, the persistent storage pool of the Value storage layer includes SSDs, HDDs, and object storage nodes. Storage media are allocated based on the Value size (<1MB / 1MB-100MB / >100MB) and access frequency (>10 times / hour / 1-10 times / hour / <1 time / hour). Smaller values ​​are stored on SSDs, and larger values ​​are stored on HDDs. The target value reference information assigned to the target value data is used as the metadata key, and the physical storage information of the target storage medium where the target value data is located, the creation time of the target value data, and the expiration time are used as the metadata value. The key-value pair composed of the metadata key and the metadata value is used as the metadata index information of the target value data and stored in the metadata index of the value storage layer to complete the metadata record of the value data.

[0038] In this embodiment, differentiated storage media allocation is achieved based on the access frequency and physical attributes of value data. High-performance solid-state drives (SSDs) are used to store high-frequency, core value data, while low-cost hard disk drives (HDDs) and object storage are used to store low-frequency, large-volume value data. This significantly reduces storage hardware costs while ensuring core data access performance. Comprehensive metadata indexing information, using value reference information as keys, includes physical storage information, creation time, and expiration time of value data, enabling accurate and rapid location of value data and improving the data reading and management efficiency of the value storage layer. The metadata indexing information, combined with the data fingerprint index, allows the value storage layer to achieve both value data deduplication and accurate value data location, doubly optimizing the management efficiency and resource utilization of the value storage layer. Media matching rules can be flexibly configured according to the user's actual business needs, adapting to the storage requirements of different industries and scenarios, improving the scenario adaptability and flexibility of the storage system. The allocation of differentiated storage media is executed independently in the value storage layer, decoupled from the key storage layer operations, and does not affect the performance of the key storage layer, ensuring a balance between performance and cost for the entire storage system.

[0039] Furthermore, based on the above embodiments, this embodiment also adds a data read recovery method when the metadata index fails. If the Value metadata index is not found, such as if the metadata is corrupted, the value reference relationship and the correspondence between the value can be reconstructed through value writing logs such as Value WAL to ensure data reliability. This solves the technical problem in traditional key-value stores where value data cannot be located and read due to metadata corruption or loss, thus causing data access failures. This may include the following: When a data read operation request is received through the user operation interface, the system queries the corresponding target read value reference information from the key storage layer based on the target read key of the data to be read; using the target read value reference information as the target metadata key, it queries the corresponding target metadata value from the value storage layer, and reads the target read value data from the value storage layer based on the physical storage information in the target metadata value; if the target metadata value cannot be found in the value storage layer, it reads the most recent target value write log from the value write log file based on the target read value reference information, and reads the target metadata from the target value write log; using the target read value reference information as the new metadata key, and the target metadata as the corresponding new metadata value, the new metadata key and the new metadata value are stored as metadata key-value pairs for the target read value data.

[0040] The target metadata key refers to the metadata query key used in data read operations, which uses the target read value reference information as the key to retrieve the corresponding metadata value in the metadata index of the value storage layer. The value corresponding to the target metadata key is defined as the target metadata value. The target metadata value and the target metadata key form a key-value pair. The target metadata value is the metadata value in the metadata index that matches the target metadata key, containing information such as the physical storage information and creation time of the target read value data. The most recent target value write log is a value write log file that records the most recent allocation of the target read value data's value reference information and the log entry for the persistent storage process, containing the metadata information of the value data. The target metadata is extracted from the most recent target value write log, containing metadata information such as the physical storage information, creation time, and expiration time of the target read value data. The new metadata key uses the target read value reference information as the new metadata key, which works with the recovered target metadata to rebuild the metadata index. The new metadata value uses the target metadata extracted from the value write log as the new metadata value. The metadata key-value pair, consisting of the new metadata key and the new metadata value, is used to rebuild the failed metadata index information and restore the ability to locate the value data.

[0041] In this embodiment, when the user interface receives a data read operation request, after obtaining the target read value reference information, the target read value reference information is used as the target metadata key to query the metadata index of the value storage layer, attempting to obtain the corresponding target metadata value. If the target metadata value is found, the target read value data is read from the corresponding target storage medium based on the physical storage information therein, completing the normal read operation. If the target metadata value cannot be found in the metadata index of the value storage layer, it indicates that the metadata index is invalid, and the data read compensation logic is immediately triggered: using the target read value reference information as the query basis, the value is written from the value storage layer. The system reads the most recent target value write log from the log file and extracts the target metadata (including physical storage information, creation time, expiration time, etc.) from this log entry. Using the target value reference information as the new metadata key and the target metadata extracted from the value write log as the corresponding new metadata value, the system re-stores the metadata key-value pair composed of the new metadata key and the new metadata value into the metadata index of the value storage layer, thus completing the reconstruction of the invalid metadata index information. Based on the physical storage information in the reconstructed metadata key-value pair, the system reads the target value data from the corresponding storage medium, completing the read operation.

[0042] As shown above, this embodiment solves the problem of unreadable value data due to metadata corruption or loss in traditional storage by restoring metadata information through value write logs and rebuilding the metadata index, significantly improving the accessibility of value data and the data reliability of the storage system. Data recovery triggering and execution are fully automatic, requiring no user intervention. Read operation requests and return results from the user interface remain unchanged, improving the usability and stability of the storage system. Data recovery is only triggered when the metadata index fails; normal read operations continue to execute according to the original process, without adding extra overhead and ensuring the overall read performance of the storage system. Extracting metadata from the most recent value write log ensures that the recovered metadata information is up-to-date and accurate, guaranteeing the precision of value data location. The rebuilt metadata key-value pairs are rewritten to the metadata index, achieving self-repair of the metadata index and preventing subsequent read operations from encountering metadata index failure again, thus improving the self-maintenance capability of the storage system.

[0043] Based on the above embodiments, this application also provides a method for processing data read operations. Data reading requires achieving fast mapping from key to value, while reducing I / O overhead through caching optimization, as shown in the appendix. Figure 3 As shown, during a Get operation, the Key storage layer queries value reference information such as the Value ID. The Value storage layer, based on the Value ID, first checks the cache and then reads from the persistent storage, returning the Value, which may include the following: When a data read operation request is received through the user operation interface, the system queries the corresponding target read value reference information from the key storage layer based on the target read key of the data to be read. Based on the target read value reference information, the system first reads the corresponding target read value data from the value memory cache of the value storage layer. If the value memory cache does not contain the target read value data, the system then reads the corresponding target read value data from the storage medium of the value storage layer based on the target read value reference information. The system then assembles the target read key and the corresponding target read value data into a target key-value pair and returns the target key-value pair to the user through the user operation interface.

[0044] In this system, a data read operation request is sent by the user client to the storage system through the user interface to read the value data corresponding to a specified key. The target read key is the key specified in the data read operation request, the key whose corresponding value data needs to be read. The target read value reference information is the value reference information associated with the target read key in the key storage layer, which can be used to locate the corresponding value data in the value storage layer. The value memory cache is a memory-based cache area in the value storage layer that stores frequently accessed target value data, used for fast response to read operation requests and reducing disk I / O access. The storage medium is the physical medium used in the value storage layer for persistently storing value data, including solid-state drives, hard disk drives, object storage, etc. The target key-value pair is a complete key-value pair assembled from the target read key and its corresponding target read value data in the value storage layer, and is the final return result of the data read operation.

[0045] In this embodiment, when the user operation interface receives a data read operation request sent by the user terminal, it parses the read operation request, obtains the target read key of the data to be read, and performs a precise query in the key storage layer based on the target read key to obtain the corresponding target read value reference information. Using the target read value reference information as the query basis, it first reads the corresponding target read value data from the value memory cache of the value storage layer. If the target read value data is not stored in the value memory cache, it then reads the corresponding target read value data from the persistent storage medium of the value storage layer based on the target read value reference information. The queried target read key and the corresponding target read value data are assembled to form a complete target key-value pair, and the target key-value pair is returned to the user terminal that initiated the request through the user operation interface to complete a single data read operation.

[0046] As shown above, this embodiment effectively improves the efficiency of reading frequently accessed and moderately sized data by combining an optimized value memory cache with cache-priority reading, significantly enhancing the read performance of the storage system. Frequently accessed value data can be directly read from the memory cache, greatly reducing the number of accesses to the disk storage medium, lowering data read operation latency, and improving read performance. The key-value separation architecture allows queries in the key storage layer and reads in the value storage layer to be executed independently, avoiding the cumbersome process of locating the complete key-value pair before extracting the value data in traditional coupled storage, thus improving the execution efficiency of a single read operation. The value memory cache only stores value data and does not conflict with the key storage layer cache, avoiding the problem of large value data occupying the cache and causing a low key cache hit rate in the traditional model, thereby improving the overall utilization of cache resources.

[0047] Based on the above embodiments, this embodiment also implements concurrent processing of multiple data read operation requests, which can solve the technical problems of serial execution of batch read operations, low efficiency of multi-key queries, and read performance bottlenecks caused by the single query method of data at the same level in key-value storage. It may include the following: When multiple data read operation requests are received through the user operation interface; if at least two data read operation requests have target read keys located in the key memory table or immutable key memory table, the corresponding target read value reference information is queried in parallel; if at least two data read operation requests have target read keys located in different ordered string tables, the corresponding target read value reference information is read concurrently from each ordered string table using multi-threading; if at least two data read operation requests have target read keys located in the same ordered string table, the corresponding target read value reference information is located in the same ordered string table using binary search; the value storage layer allocates an independent reading thread for each target read value reference information, and concurrently reads the corresponding target read value data from the value storage layer; each target read key and the corresponding target read value data are assembled into a key-value pair set, and the key-value pair set is returned to the user end through the user operation interface.

[0048] In this embodiment, the key storage layer includes a Key MemTable, an Immutable Key MemTable, and an Ordered String Table (SSTable). The key storage layer only stores key-value references, resulting in a significantly smaller data volume compared to traditional key-value stores. Therefore, the Key MemTable, Immutable Key MemTable, and SSTable offer faster query speeds, and the Block Cache can cache more keys, effectively improving the hit rate. The Key MemTable is an ordered data structure built in memory within the key storage layer, used to store keys and their corresponding value references written in real-time, supporting efficient insertion and query operations. The Immutable Key MemTable is updated to a read-only memory table structure when the remaining storage space reaches a preset threshold, avoiding concurrent conflicts between write and disk flush operations, and awaiting subsequent disk flushing to generate the Ordered String Table. The Ordered String Table is a disk file generated from the Immutable Key MemTable after flushing. It stores data in key order, containing only key-value references, and supports efficient range queries and binary search. Binary search is an efficient search algorithm for ordered string tables with sorted storage. It quickly locates the value reference information corresponding to the target read key by continuously narrowing the search range. Each read thread is allocated separately for each target read value reference and is an independent execution thread used to read the corresponding value data in the value storage layer, supporting multi-threaded concurrent reads. The key-value pair set is a collection assembled from the target read keys and their respective target read value data corresponding to multiple data read operation requests; it is the final return result of multiple data read operations.

[0049] In this embodiment, when the user operation interface simultaneously receives multiple data read operation requests sent by the user terminal, the concurrent query process for multi-key-value reference information is as follows: If at least two of the multiple read operation requests have target read keys located in the key memory table or immutable key memory table, then these target read keys are queried in parallel, and the corresponding target read value reference information is obtained simultaneously; if at least two of the target read keys are located in different ordered string tables, then the corresponding target read value reference information is read simultaneously from each ordered string table using a multi-threaded concurrent approach; if at least two of the target read keys are located in the same ordered string table, then the target read value reference information corresponding to each target read key is quickly located in that ordered string table using a binary search algorithm. The concurrent reading process for multi-value data is as follows: the value storage layer allocates an independent reading thread for each target read value reference information obtained from the query, and reads the corresponding target read value data simultaneously from the memory cache or persistent storage medium of the value storage layer using a multi-threaded concurrent approach. Finally, each target read key and its corresponding target read value data are assembled into key-value pairs. All key-value pairs are integrated into a key-value pair set, which is then returned to the user terminal at once through the user operation interface to complete the multi-data read operation.

[0050] As shown above, this embodiment employs differentiated concurrent query methods for batch read requests and keys in different storage locations. It utilizes parallel queries directly in the memory table, multi-threaded concurrent reading of different ordered string tables, and binary search within the same ordered string table to maximize the query efficiency of multi-key-value reference information. Multiple values ​​can be read in parallel from the value storage layer based on value reference information, avoiding the serial overhead of sequential reading in traditional bound storage, resulting in a 2-3 times performance improvement in batch reads. The binary search algorithm significantly improves the efficiency of multi-key queries within the same ordered string table, reducing query time complexity compared to sequential search and further reducing the overall latency of multi-read operations. Thread concurrency processing is executed entirely at the underlying level, maintaining a unified request reception and result return logic at the user interface, completely transparent to the user and without increasing user operating costs.

[0051] Based on the above embodiments, this embodiment also provides a data deletion operation and a corresponding value data garbage collection method. This can avoid data residue from deletion operations in key-value storage and the waste of storage resources or data loss caused by unreasonable garbage collection timing, reducing the difficulty of value data lifecycle management. The data deletion process needs to handle the reference relationship between Key and Value simultaneously to avoid Value data leakage. Figure 4 As shown, it may include the following: In this embodiment, the target value reference information includes an identifier and a timestamp. For example, the target value reference information can be a combination of UUID (Universally Unique Identifier) ​​and timestamp to ensure global uniqueness and facilitate the filtering of expired data by time window during subsequent GC. When a data deletion request is received through the user interface, a target deletion marker is added to the key to be deleted in the key storage layer, and the reference count of the value reference information corresponding to the key to be deleted is decremented by one in the value storage layer. When the remaining storage space of the key memory table in the key storage layer is less than a preset storage threshold, the state of the key memory table is updated to read-only to serve as an immutable key memory table. After the immutable key memory table is written to disk, an ordered string table containing only key and value reference information is generated. During the merging of the ordered string tables, all version records of keys carrying the target deletion marker are not written to the newly generated ordered string table file, and the reference count of the value reference information corresponding to the key carrying the target deletion marker is decremented by one simultaneously. The value storage layer sends the waiting-to-be-deleted value data with a reference count of zero to the recycling queue, and sets a retention time attribute for each waiting-to-be-deleted value data so that it will be recycled when the retention time of each waiting-to-be-deleted value data exceeds a preset time threshold.

[0052] The data deletion request is sent by the client to the storage system through the user interface to delete the key-value data corresponding to a specified key. The key to be deleted is the key specified in the data deletion request, the key whose corresponding key-value data needs to be deleted. The target deletion marker is a read-only marker added to the key to be deleted in the key storage layer to indicate that the key is in a pending deletion state. This uses marker-based deletion instead of direct physical deletion to reduce immediate write overhead, similar to tombstone marking. The value reference information to be deleted is the value reference information associated with the key to be deleted in the key storage layer, serving as the basis for locating the corresponding value data to be deleted in the value storage layer. The reference count is a counter set for each value reference in the value storage layer, used to count the number of times the value reference is associated with a key in the key storage layer, and is the core basis for determining whether value data needs to be reclaimed. The preset storage threshold is the storage space threshold set for the key memory table. When the remaining storage space is less than this value, the key memory table is triggered to update its state to an immutable key memory table. The reclamation queue is a queue in the value storage layer used to store value data to be deleted with a reference count of zero, for unified management and subsequent garbage collection of this type of value data. The retention time attribute is a time-limited attribute set for each data value to be deleted in the recycling queue. It limits the retention period of the data value to prevent data loss due to accidental deletion or system failure. The preset time threshold is a critical value set for the retention time of the data value to be deleted. When the retention time exceeds this value, garbage collection of the corresponding data value is triggered.

[0053] In this embodiment, when the user interface receives a data deletion request, it parses the key to be deleted and adds a tombstone marker to the key storage layer to indicate that it is in a state to be deleted. Simultaneously, in the value storage layer, the reference count of the corresponding value reference information is updated, decrementing the reference count of the value reference information corresponding to the key to be deleted by one, and updating the association status of the value data. Marked deletion is used instead of physical deletion to avoid immediately modifying the key's SSTable; keys marked for deletion are only cleaned up during compaction, reducing write overhead. The reference count of the value is jointly maintained by the compaction and delete operations. When compaction deletes an expired key, the corresponding VR reference count is decremented by one synchronously; the delete operation directly decrements the reference count of the value reference information by one. When the remaining storage space of the key memory table in the key storage layer is less than a preset storage threshold, the state of the key memory table is updated to read-only, becoming an immutable key memory table. After the immutable key memory table is asynchronously flushed to disk, an ordered string table containing only key and value reference information is generated. During the background merging of the ordered string tables in the key storage layer, all version records of keys carrying target deletion markers are not written to the newly generated ordered string table file, achieving physical cleanup of the keys to be deleted. Simultaneously, the reference count of the value reference information corresponding to these keys carrying target deletion markers is decremented by one to ensure that the reference count is consistent with the actual key association state. The value storage layer monitors the reference count of each value reference information in real time. Values ​​with a reference count of 0 enter the garbage collection queue to ensure that values ​​without keys can be reclaimed in a timely manner. At the same time, a retention time attribute is set for each value data to be deleted in the collection queue, recording the time it entered the queue. For example, a 24-hour GC retention time can be set to avoid the inability to recover values ​​due to accidental deletion or system failure, thus improving data security. When the retention time of the value data to be deleted exceeds a preset time threshold, a garbage collection operation is triggered, completely deleting the value data from the persistent storage medium of the value storage layer and releasing the corresponding storage resources. Furthermore, ensuring the order of writing Value WAL before Value and Key WAL before Key, if the Value is written successfully but the Key fails to be written, the GC module will reclaim the unreferenced Value.

[0054] As shown above, this embodiment employs a key deletion method combining mark-and-delete and background merge cleanup, avoiding immediate physical modifications to the key storage layer data, significantly reducing write overhead and improving the execution efficiency of deletion operations. Based on reference counting to manage the lifecycle of value data, both key deletion and merge cleanup synchronously update the reference count, ensuring that the reference count accurately reflects the actual referenced state of the value data, thus preventing value data leakage and residue at the source. The recycling queue, combined with a time-retention garbage collection mechanism, not only enables timely reclamation of unreferenced value data, releasing storage resources, but also prevents value data loss due to accidental deletion, system failures, or other unforeseen circumstances, improving the security of data deletion. Garbage collection operations are executed independently in the value storage layer, decoupled from key storage layer operations, and do not affect normal data processing in the key storage layer, ensuring the overall operational stability of the storage system. When merging ordered string tables, only key and value reference information is processed, without the participation of value data. Garbage collection only targets useless data in the value storage layer; the combination of these two methods further reduces disk I / O overhead and improves the utilization rate of storage resources.

[0055] Furthermore, this embodiment also provides a recovery method after a storage system failure, avoiding problems such as inconsistent key-value associations, unrecoverable data corruption, and garbage collection anomalies caused by distorted reference counts that are prone to occur during failure recovery in key-value stores. This may include the following: In response to fault recovery operations, the system reads keys from the key storage layer and writes them to the log file. Based on this log file, it reconstructs the key and value reference information data in the key memory table before the fault. If an immutable key memory table exists that has not been flushed to disk, it re-triggers the flush operation to generate an ordered string table, verifies the integrity of each layer's ordered string table, deletes the corrupted target ordered string table, and restores the target ordered string table using backup data. The system also reads values ​​from the value storage layer and writes them to the log file. Based on this log file, it reconstructs the metadata index information, verifies the data integrity of each value data stored in the value storage layer, and restores the corrupted value data using backup data or redundant copies. Finally, it iterates through and counts the occurrence counts of each value reference information in each ordered string table of the key storage layer and the key memory table, and corrects the reference count of the corresponding value data based on the statistical results.

[0056] Fault recovery operations are a series of operations performed to restore normal system operation and ensure data consistency and integrity after a storage system crash (such as a power outage or abnormal process exit). The key write log file is a persistent key write log collection in the key storage layer, recording the association and persistence process of all key-value reference information. It serves as the basis for rebuilding key storage layer data during fault recovery. Unflushed immutable key memory tables refer to immutable key memory tables that have not yet undergone asynchronous flushing during system failure; their data exists only in memory and is not persisted to disk. The flushing operation persists the key-value reference information in the immutable key memory table to disk, generating an ordered string table. Integrity verification is a validity check performed on the data files (ordered string tables and value data) in the storage layer to determine if the data is corrupted, lost, or tampered with. Backup data is redundant data pre-stored by the storage system for the ordered string tables in the key storage layer and the value data in the value storage layer, used for recovery in case of data corruption. The value write log file is a persistent collection of value write logs in the value storage layer, recording the allocation and persistence process of all value data. It serves as the basis for rebuilding the value storage layer's metadata during fault recovery. Metadata index information is the index information used to locate value data in the value storage layer, containing the association between value reference information and the physical storage location and creation time of the value data. Redundant copies are multiple identical copies of value data created in the value storage layer, stored on different storage media or nodes, and serve as another basis for recovery in case of data corruption.

[0057] In this embodiment, the key storage layer recovery process is as follows: All keys are read from the disk media of the key storage layer and written to the log file. Based on the contents recorded in the log file, all key and value reference information data in the key memory table before the failure is reconstructed in the order of operations, restoring the real-time data state of the key storage layer. If an immutable key memory table that has not been flushed to disk is detected, the flushing operation is triggered again to persist its data to the disk and generate an ordered string table. Integrity checks are performed on all ordered string tables at each level of the key storage layer. If a corrupted target ordered string table is found, the table is directly deleted, and it is fully restored using pre-stored backup data to ensure the integrity of the key storage layer data. The value storage layer recovery process is as follows: All values ​​are read from the disk media of the value storage layer and written to the log file. Based on the contents recorded in the log file, the metadata index information of the value storage layer is reconstructed, restoring the mapping relationship between value reference information and value data. Data integrity checks are performed on all persistently stored value data in the value storage layer. If corrupted value data is found, it is restored using backup data or redundant copies of the value data to ensure the accessibility of the value data. Traverse all ordered string tables and restored key memory tables in the key storage layer, count the number of times each value reference information appears, and this number is the actual number of times the value data is referenced; compare the actual number of references obtained with the current reference count in the value storage layer, and use the actual number of references as the standard to correct the reference count of the corresponding value data to ensure the accuracy of the reference count.

[0058] Taking the Key WAL (Key Write Log) as an example, the key storage layer uses MemTable and SSTable as its data structures. For easy distinction, the SSTable in the key storage layer can be defined as Key SSTable. The value WAL (Value Write Log) and the value reference information are represented by Value IDs. The key storage layer reconstructs the MemTable and SSTable by replaying the Key WAL, while the value storage layer reconstructs the metadata index by replaying the Value WAL and recalculates the Value ID reference count by traversing the Key data. When the storage system crashes (e.g., power outage, abnormal process exit), the process of restoring key and value data consistency through the WAL log includes: Key storage layer recovery: reading the Key WAL log and reconstructing the (Key, Value ID) data in the Key MemTable before the crash; if there is a Key Immutable MemTable that has not been flushed to disk, re-triggering the flush operation to generate a Key SSTable; verifying the integrity of each Level Key SSTable, deleting corrupted SSTables, and restoring them through backups. Value storage layer recovery involves: reading the Value WAL log, rebuilding the Value metadata index (i.e., the mapping between Value IDs and their physical locations); verifying the data integrity in the Value persistent storage pool, and recovering corrupted Value data through backups or redundant copies (if multiple copies are enabled); traversing all Key SSTables and Key MemTables, counting the occurrences of each Value ID, recalculating the reference count for each Value ID, and correcting the reference count in the Value metadata index. Reference counting manages the Value lifecycle, and combined with WAL log replay, enables self-healing from failures, balancing efficiency and security.

[0059] As shown above, this embodiment reconstructs the core data of the key storage layer and value storage layer based on key write logs and value write logs respectively, ensuring that the association between keys and values ​​can be accurately restored after fault recovery and ensuring key-value consistency after fault recovery. For immutable key memory tables that have not been flushed to disk, a disk flush is retried, and damaged data files are restored through backups or redundant copies, achieving complete recovery of data in both the key and value storage layers and minimizing data loss caused by faults. After fault recovery, the reference count of value data is recalculated and corrected to ensure that the reference count is consistent with the actual referenced state, avoiding subsequent garbage collection anomalies caused by distorted reference counts, such as accidental deletion of useful data or residual useless data. The recovery operations of the key and value storage layers are executed independently and can be processed concurrently through multi-threading, significantly shortening the overall fault recovery time and improving the high availability of the storage system. The entire fault recovery process requires no user intervention; the system automatically completes data reconstruction, verification, and correction. After recovery, normal services can be provided directly through the user interface, reducing the manpower cost of system maintenance.

[0060] It should be noted that there is no strict order of execution between the steps in this invention. As long as they conform to the logical order, these steps can be executed simultaneously or in a certain preset order. Figures 1-4 This is just an illustrative example and does not mean that this is the only possible execution order.

[0061] This invention also provides a corresponding apparatus for the data storage method, further enhancing the practicality of the method. The apparatus can be described from both a functional module perspective and a hardware perspective. The data storage apparatus provided by this invention is described below. This apparatus is used to implement the data storage method provided by this invention. In this embodiment, the data storage apparatus may include or be divided into one or more program modules. These one or more program modules are stored in a storage medium and executed by one or more processors to complete the data storage method disclosed in Embodiment 1. The program module referred to in this embodiment is a series of computer program instruction segments capable of performing a specific function, which is more suitable than the program itself for describing the execution process of the data storage apparatus in the storage medium. The following description will specifically introduce the functions of each program module in this embodiment. The data storage apparatus described below and the data storage method described above can be referred to in correspondence.

[0062] From the perspective of functional modules, see Figure 5 , Figure 5 This is a structural diagram of the data storage device provided in this embodiment under one specific implementation. The device may include: The structure deployment module 501 is used to build a unique user operation interface for key-value operations with the user terminal, and to build independent key storage layer and value storage layer; wherein, the key storage layer is configured to store keys and value reference information corresponding to the keys, and the value storage layer is configured to independently store value data corresponding to the value reference information.

[0063] The data writing module 502 is used to, when a data write operation request is received through the user operation interface, allocate corresponding target value reference information for the target value data to be written in the value storage layer, first record the value of the target value data in the log, and then persist the target value data; in the key storage layer, first record the target value reference information and the key of the corresponding target key in the log, and then persist the target value reference information and the corresponding target key.

[0064] For example, in some embodiments of this example, the device may further include a data reading module, which is used to, when a data read operation request is received through a user operation interface, query the corresponding target read value reference information from the key storage layer according to the target read key of the data to be read; read the corresponding target read value data from the value memory cache of the value storage layer according to the target read value reference information; if the value memory cache does not contain the target read value data, then read the corresponding target read value data from the storage medium of the value storage layer according to the target read value reference information; assemble the target read key and the corresponding target read value data into a target key-value pair, and return the target key-value pair to the user terminal through the user operation interface.

[0065] For example, in some other embodiments of this example, the data reading module can also be used when multiple data read operation requests are received through the user operation interface; if at least two data read operation requests have target read keys located in a key memory table or an immutable key memory table, then the corresponding target read value reference information is queried in parallel; if at least two data read operation requests have target read keys located in different ordered string tables, the corresponding target read value reference information is read concurrently from each ordered string table through multi-threading; if at least two data read operation requests have target read keys located in the same ordered string table, then the corresponding target read value reference information is located in the same ordered string table through binary search; the value storage layer allocates an independent reading thread for each target read value reference information, and concurrently reads the corresponding target read value data from the value storage layer; assembles each target read key and the corresponding target read value data into a key-value pair set, and returns the key-value pair set to the user end through the user operation interface.

[0066] For example, in some other embodiments of this example, the apparatus may further include a data deletion module. This module is used to add a target deletion mark to the key to be deleted in the key storage layer and decrement the reference count of the value reference information corresponding to the key to be deleted in the value storage layer when a data deletion operation request is received through the user operation interface. When the remaining storage space of the key memory table in the key storage layer is less than a preset storage threshold, the state of the key memory table is updated to read-only to serve as an immutable key memory table. When the immutable key memory table generates an ordered string table containing only key and value reference information after being written to the disk, during the merging of the ordered string tables, all version records of the key carrying the target deletion mark are not written to the newly generated ordered string table file, and the reference count of the value reference information corresponding to the key carrying the target deletion mark is decremented by one. The value storage layer sends the waiting-to-be-deleted value data with a reference count of zero to the recycling queue, and sets a retention time attribute for each waiting-to-be-deleted value data so that it can be recycled when the retention time of each waiting-to-be-deleted value data exceeds a preset time threshold.

[0067] For example, in some other embodiments of this embodiment, the apparatus may further include a data recovery module, which is used to, in response to a fault recovery operation, read a key write log file from the key storage layer and reconstruct the key and value reference information data in the key memory table before the fault based on the key write log file; if there is an immutable key memory table that has not been flushed to disk, the flush operation is triggered again to generate an ordered string table, and the integrity of each layer of ordered string tables is verified, the damaged target ordered string table is deleted, and the target ordered string table is restored through backup data; read a value write log file from the value storage layer, reconstruct metadata index information based on the value write log file, verify the data integrity of each value data stored in the value storage layer, and restore the damaged value data through backup data or redundant copies; traverse and count the occurrence times of each ordered string table in the key storage layer and each value reference information in the key memory table, and correct the reference count of the corresponding value data based on the statistical results.

[0068] For example, in some other embodiments of this embodiment, the above-mentioned data writing module 502 can also be used as follows: the value storage layer further includes an index library; the target value data is hashed in the value storage layer, and the hash calculation result is used as a unique identifier of the target data content corresponding to the target data fingerprint; the index library is queried according to the target data fingerprint; if the target data fingerprint does not exist in the index library, the corresponding target value reference information is allocated for the target value data to be written in the value storage layer; while persistently storing the target value data, a new index record is generated using the target data fingerprint as the index key and the target value reference information and the metadata information of the target value data as the index value, identifying the mapping relationship between the target data fingerprint and the target value reference information, and written to the index library.

[0069] As an exemplary implementation of the above embodiments, the write data module 502 can be further used to: determine the matching target storage medium based on the access frequency and physical attributes of the target value data, and persistently store the target value data to the target storage medium; use the target value reference information as the metadata key, and use the physical storage information, creation time, and expiration time of the target storage medium in the target storage medium as the metadata value, and store the metadata key and metadata value as the metadata index information of the target value data.

[0070] As another exemplary implementation of the above embodiments, the data reading module can be further configured to: when a data read operation request is received through the user operation interface, query the corresponding target read value reference information from the key storage layer according to the target read key of the data to be read; use the target read value reference information as the target metadata key to query the corresponding target metadata value in the value storage layer, and read the target read value data from the value storage layer according to the physical storage information in the target metadata value; if the target metadata value cannot be found in the value storage layer, read the most recent target value write log from the value write log file according to the target read value reference information, and read the target metadata from the target value write log; use the target read value reference information as the new metadata key, use the target metadata as the corresponding new metadata value, and store the new metadata key and the new metadata value as metadata key-value pairs for the target read value data.

[0071] As another exemplary implementation of the above embodiments, the write data module 502 may be further used as follows: the value storage layer also includes a value memory cache area, and further includes: statistically analyzing the access frequency of each value data stored in the value storage layer, and selecting candidate value data with a access frequency greater than or equal to a preset access frequency; writing the value data in each candidate value data whose storage space occupancy value is within a preset data scale range to the value memory cache area.

[0072] The data storage device mentioned above is described from the perspective of a functional module. Furthermore, the present invention also provides an electronic device, described from a hardware perspective. This electronic device includes a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described data storage method embodiments.

[0073] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data storage method embodiments when it is run.

[0074] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0075] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data storage method embodiments.

[0076] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data storage method embodiments.

[0077] Finally, the present invention also provides a data storage system, see [link to relevant documentation]. Figure 6 The system may include a user interface 601, a storage device 602, and a processor 603. The storage device 602 includes a program storage area, a key storage layer, and a value storage layer for storing computer programs. The storage device is a hardware device with multiple storage layers and multiple interfaces. The program storage area may, for example, use a high-speed read-only storage medium. The program code includes all logic such as interface management, key-value separation storage, log recording, data reading and writing, deletion and recycling, fault recovery, and data deduplication. The key storage layer may, for example, be a hybrid storage layer composed of memory and solid-state drives (SSDs). The memory portion constructs a key memory table and an immutable key memory table, while the SSD portion stores a key write log file and an ordered string table, used only for storing key and value reference information. The value storage layer may be a hybrid storage layer composed of memory, SSDs, hard disk drives (HDDs), and object storage. The memory portion constructs a value memory cache and an index library, while the SSD / HDD / object storage portion stores value write log files, value data, and metadata index information, specifically used for storing value data and related metadata. The processor 603 is used to implement the steps of the data storage method described in any of the above embodiments when executing the computer program.

[0078] For example, such as Figure 7As shown, the storage system may include an interface layer, storage devices, and a processor. The storage devices have a key storage layer and a value storage layer. The processor has a compaction manager and a garbage collection (GC) module. The key storage layer includes a Key MemTable (memory table), a Key Immutable MemTable (immutable memory table), a Key SSTable (ordered string table), and a Key WAL. The value storage layer includes a value memory cache, a value metadata index, a value persistent storage pool, and a value WAL.

[0079] User operation interfaces can be implemented, for example, through APIs (Application Programming Interfaces). The storage system has an interface layer that provides a unified key-value operation interface (Put (write operation), Get (read operation), Delete (delete operation), Scan (scan operation)).

[0080] The key storage layer stores key and value references. Each value reference is a globally unique identifier, such as a UUID, used to locate the value within the key storage layer; an LSM-tree structure can be used. The keyMemTable is an in-memory skip list structure that stores key and value references written in real-time. The keyImmutable MemTable becomes read-only when the key memtable is full, awaiting disk flushing. The key SSTable is an ordered file generated by flushing the keyImmutable Memtable, containing only key and value references; its file size is much smaller than a traditional SSTable. The key WAL (Write-Ahead Log) serves as the key write log, recording the write logs of key and value references, used to recover key memtable data in case of system crashes. The value storage layer stores value data and related metadata. The value memory cache is an LRU (Least Recently Used) based cache, storing frequently accessed value data to reduce disk I / O. The Value persistent storage pool supports differentiated storage media (SSD (Solid State Drive) / HDD (Hard Disk Drive) / object storage), allocating storage media based on the access frequency, size, and other characteristics of the Value. For example, small Values ​​are stored on SSDs, and large Values ​​are stored on HDDs. Each index in the Value metadata index uses a Value Reference as the key, storing metadata such as the physical location of the Value (e.g., media type, file path, offset, length), creation time, and expiration time, supporting rapid Value location. The Value WAL is used as a log to record the write, update, and deletion logs of Values, ensuring the persistence and consistency of Value data. The Compaction Manager implements compaction operations in the Key storage layer, such as merging Key SSTables, deleting expired Keys, and synchronously updating the reference count in the Value storage layer to prevent Value data leakage. The garbage collection module, based on a reference counting and time window mechanism, periodically scans the Value storage layer for Value data with a reference count of 0 (no key pointing to it) that has exceeded its retention time, releasing storage resources.

[0081] As can be seen from the above, compared with related technologies that do not completely separate KV storage systems, this embodiment improves write performance by reducing the write amplification factor from 10-20 to 2-3, reducing disk I / O by 70%+, increasing write throughput by 2-3 times, reducing SSD erase / write cycles by 80%, and extending lifespan by 5 times. In terms of read performance, the key cache hit rate is improved by 50%-60%, value read latency is reduced by 60%-70%, batch read performance is improved by 2-3 times through parallel processing, and value deduplication improves resource utilization by 50%-70%. Differentiated media deployment (SSD+HDD+object storage) reduces hardware costs by 40%-60% compared to an all-SSD solution. It is compatible with traditional KV APIs, allowing applications to migrate without code modification, reducing labor costs by 90%. Dual WAL and multiple replicas ensure no data loss, shortening fault recovery time to 1 / 4 of the traditional mode. It supports PB-level data scales and millions of QPS, with dynamic node expansion without downtime, adapting to various scenarios such as log storage and financial transactions.

[0082] The foregoing has provided a detailed description of a storage system and data storage method provided by the present invention. The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Whether the units and algorithm steps of the various examples described in the disclosed embodiments are executed by electronic hardware or computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, and such implementations should not be considered beyond the scope of the present invention. Several improvements and modifications can be made to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A data storage method, characterized in that, include: Construct a unique user interface for key-value operations with the user client, and construct independent key storage layer and value storage layer; wherein, the key storage layer is configured to store keys and value reference information corresponding to the keys, and the value storage layer is configured to independently store value data corresponding to the value reference information; When a data write operation request is received through the user operation interface, the target value reference information is allocated for the target value data to be written in the value storage layer. The value write log of the target value data is recorded first, and then the target value data is persisted. In the key storage layer, the key write log of the target value reference information and its corresponding target key is recorded first, and then the target value reference information and its corresponding target key are persisted.

2. The data storage method according to claim 1, characterized in that, Also includes: When a data read operation request is received through the user operation interface, the corresponding target read value reference information is queried from the key storage layer according to the target read key of the data to be read; Based on the target read value reference information, the corresponding target read value data is first read from the value memory cache of the value storage layer. If the value memory cache does not contain the target read value data, then the corresponding target read value data is read from the storage medium of the value storage layer based on the target read value reference information. The target read key and the corresponding target read value data are assembled into a target key-value pair, and the target key-value pair is returned to the user terminal through the user operation interface.

3. The data storage method according to claim 1, characterized in that, Also includes: When multiple data read operation requests are received through the user operation interface; If at least two data read operation requests have target read keys located in the key memory table or immutable key memory table, then the corresponding target read value reference information is queried in parallel. If at least two data read operation requests have target read keys located in different ordered string tables, the corresponding target read value reference information is read from each ordered string table concurrently using multiple threads. If at least two data read operation requests have target read keys located in the same ordered string table, then the corresponding target read value reference information is located in the same ordered string table using binary search. The value storage layer allocates an independent reading thread for each target read value reference information and reads the corresponding target read value data concurrently from the value storage layer; Each target read key and its corresponding target read value data are assembled into a key-value pair set, and the key-value pair set is returned to the user terminal through the user operation interface.

4. The data storage method according to claim 1, characterized in that, The target value reference information includes an identifier and a timestamp, and also includes: When a data deletion operation request is received through the user operation interface, a target deletion mark is added to the key to be deleted in the key storage layer, and the reference count of the value reference information corresponding to the key to be deleted is decremented by one in the value storage layer. When the remaining storage space of the key memory table in the key storage layer is less than the preset storage threshold, the state of the key memory table is updated to read-only state to serve as an immutable key memory table. When the immutable key memory table generates an ordered string table containing only key and value reference information after being written to the disk, during the merging of each ordered string table, all version records of keys carrying target deletion marks are not written to the newly generated ordered string table file, and the reference count of the value reference information corresponding to the key carrying the target deletion mark is decremented by one. The value storage layer sends the value data waiting to be deleted with a reference count of zero to the recycling queue, and sets a retention time attribute for each value data waiting to be deleted so that it can be recycled when the retention time of each value data waiting to be deleted exceeds a preset time threshold.

5. The data storage method according to claim 1, characterized in that, Also includes: In response to the fault recovery operation, the key is read from the key storage layer and written to the log file. The key and value reference information data in the key memory table before the fault is reconstructed based on the key write log file. If there is an immutable key memory table that has not been flushed to disk, the flush operation is triggered again to generate an ordered string table. The integrity of each layer of ordered string tables is verified, the damaged target ordered string table is deleted, and the target ordered string table is restored through backup data. Values ​​are read from the value storage layer and written to the log file. Metadata index information is reconstructed based on the values ​​written to the log file. The data integrity of each value data stored in the value storage layer is verified, and the damaged value data is restored through backup data or redundant copies. The occurrence count of each value reference information in each ordered string table and key memory table of the key storage layer is traversed and counted, and the reference count of the corresponding value data is corrected based on the statistical results.

6. The data storage method according to any one of claims 1 to 5, characterized in that, The value storage layer also includes an index library, and further includes: The target value data is hashed in the value storage layer, and the hash calculation result is used as a unique identifier of the target data fingerprint corresponding to the target value data content. If the target data fingerprint is not found in the index database, then the target value reference information is allocated for the target value data to be written in the value storage layer. While the target value data is persistently stored, a new index record is generated using the target data fingerprint as the index key and the target value reference information and the metadata information of the target value data as the index value. This new index record identifies the mapping relationship between the target data fingerprint and the target value reference information and is written to the index database.

7. The data storage method according to claim 6, characterized in that, Persistent storage of the target value data includes: Based on the access frequency and physical properties of the target value data, a matching target storage medium is determined, and the target value data is persistently stored in the target storage medium. The target value reference information is used as the metadata key, and the physical storage information, creation time, and expiration time of the target storage medium are used as the metadata value. The metadata key and the metadata value are stored as the metadata index information of the target value data.

8. The data storage method according to claim 7, characterized in that, Also includes: When a data read operation request is received through the user operation interface, the corresponding target read value reference information is queried from the key storage layer according to the target read key of the data to be read; Using the target read value reference information as the target metadata key, the corresponding target metadata value is queried in the value storage layer, and the target read value data is read from the value storage layer according to the physical storage information in the target metadata value; If the target metadata value cannot be found in the value storage layer, the most recent target value write log is read from the value write log file according to the target read value reference information, and the target metadata is read from the target value write log. The target read value reference information is used as the new metadata key, and the target metadata is used as the corresponding new metadata value. The new metadata key and the new metadata value are stored as metadata key-value pairs of the target read value data.

9. The data storage method according to claim 6, characterized in that, The value storage layer also includes a value memory cache, and further includes: The access frequency of each value data stored in the value storage layer is statistically analyzed, and candidate value data with an access frequency greater than or equal to a preset frequency is selected from them; Values ​​whose storage space occupancy is within a preset data size range are written to the value memory cache.

10. A storage system, characterized in that, This includes the user interface, storage devices, and processor; The storage device includes a program storage area, a key storage layer, and a value storage layer; wherein, the program storage area is used to store computer programs; The processor is configured to implement the data storage method as described in any one of claims 1 to 9 when executing the computer program.

Images