Data redundancy method and device for distributed key value storage based on block data copy
By adopting a distributed key-value storage method based on block data replicas, the LBA space of block devices is directly managed. By using distributed write caching and NVMe-oF technology, the problems of index space amplification and low write efficiency in existing technologies are solved, thereby improving system performance and maintainability.
Patent Information
- Application Number
- CN202511560380.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-29
- Publication Date
- 2026-02-03
AI Technical Summary
Existing RPC-based key-value copy mechanisms suffer from problems such as increased index space, redundant calculations in background tasks, and inefficient foreground writes.
A distributed key-value storage method based on block data replicas is adopted. The primary replica key-value storage instance is determined by hash value, and the block device LBA space is directly managed. Distributed write caching technology and NVMe-oF technology are used for data redundancy, bypassing the traditional file system to achieve load balancing and efficient writing.
It reduces CPU overhead from index space amplification and redundant calculations, improves system performance and maintainability, optimizes write path efficiency and resource utilization, and reduces latency.
Smart Images
Figure CN121455412A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of computer storage, and more particularly relates to a data redundancy method and device for distributed key-value storage based on block data replication. BACKGROUND
[0002] Key-value storage is an important modern data storage form, and its core idea is to store data in the form of key-value pairs in a key-value index, supporting fast retrieval of corresponding values through keys. The specific implementation of key-value index is diverse, such as Log-Structured Merge Tree (LSM-tree), B+Tree, Hash Index, and Skip List, etc. Each implementation has its own advantages and disadvantages, and is suitable for different application scenarios.
[0003] Key-value index essentially realizes the conversion of key-value requests to underlying file system or block device requests. In modern computer systems, file system and block device are two core abstractions for persistent data storage. Block device provides low-level access capabilities to storage hardware, while file system is a high-level abstraction built on block device, aiming to provide users and application programs with more intuitive and structured data organization methods. Block device takes fixed-size logical blocks (such as 512 bytes or 4KB) as the minimum read-write unit, and provides linear address space through logical block address (LBA). This abstraction hides the complexity of hardware and provides a stable interface for the upper system. However, directly operating block device is too complex for application programs. To solve this problem, file system is introduced on top of block device. File system provides users with a simple and unified read-write interface and data view through organizing files and directories, managing metadata, coordinating data block allocation, etc.
[0004] To support large-scale applications, key-value storage is usually deployed in a distributed architecture, with data dispersedly stored on multiple nodes to form a distributed key-value storage system. To ensure the availability of system services and the reliability of data, distributed key-value storage system usually stores the same data on different nodes through replica-based data redundancy mechanism. In this way, when a node is down or data is lost, other nodes can still provide services and ensure data is not lost.
[0005] The existing copy-based data redundancy mechanism is based on key-value copy. Specifically, after a client receives a write key-value request from a user, the client first sends the request to a master key-value storage instance through remote procedure call (RPC), and then the master key-value storage instance copies the key-value request into k-1 copies (k is a preset replication factor) and distributes the copies to k-1 slave key-value storage instances through RPC. Each key-value storage instance maintains its own key-value index structure and independently writes to disk.
[0006] However, the existing technology has the problem of overhead of the RPC-based key-value copy mechanism, including index space amplification, repeated background task calculation, and inefficient foreground write. SUMMARY
[0007] To solve the problem of the overhead of the RPC-based key-value copy mechanism in the prior art, including index space amplification, repeated background task calculation, and inefficient foreground write, the present application provides a data redundancy method and device for distributed key-value storage based on block data copy.
[0008] To achieve the above-mentioned purpose, in a first aspect, the present application provides a data redundancy method for distributed key-value storage based on block data copy, comprising: S100, receiving a key-value request from a user, calculating a hash value of a key corresponding to the key-value request, determining a corresponding master key-value storage instance according to the hash value, and forwarding the key-value request to the corresponding master key-value storage instance; S200, calling a key-value index interface through the key-value storage instance, and converting the key-value request into a block device read-write request; S300, submitting the block device read-write request to a block device driver engine, determining whether the block device read-write request is a write request or a read request, generating a copy and writing the copy to a local and remote copy disk if the request is a write request, and reading corresponding block data from the local or remote copy disk if the request is a read request.
[0009] Optionally, step S100 specifically comprises: S110, receiving a key-value request generated by a user through a key-value interface; S120, calculating a hash value of a key corresponding to the key-value request using a hash function; S130, performing modulo operation on the calculated hash value, determining a master key-value storage instance corresponding to the key-value request according to the modulo result, encapsulating the key-value request into an RPC message through RPC technology, and forwarding the key-value request to an RPC server of a node where the corresponding master key-value storage instance is located.
[0010] Optionally, step S200 specifically comprises: S210, parsing the RPC message through the RPC server, extracting the target key value, and calling the key value index interface in the storage instance of the target key value; S220, submitting the key value request to the key value index through the key value index interface, and converting the key value request into a read-write request for the block device through the key value index; Wherein, the key value index adopts B+ Tree, learning index, LSM-tree, Trie, Radix Tree, HashTable, append write log index or hybrid index; the key value index adopts a way of directly managing LBA space of the block device.
[0011] Optionally, when the key value index adopts LSM-tree, step S220 specifically comprises: S221, when the block device read-write request is a write request, the key value request is first written into a write-ahead log (WAL) and then written into a Memtable located in memory; when the Memtable reaches a set threshold, its content is flushed as an SSTable and is persisted; S222, when the block device read-write request is a read request, it is checked whether the key value pair of the key exists in the Memtable, and if it exists, the result is directly returned; if it does not exist, each layer of SSTable is further searched; Wherein, in the foreground key value persistent writing process in S221, a distributed write cache technology is used to write the key value data into a local memory buffer, and at the same time, an RDMA technology is used to synchronously copy the key value data into a pre-configured memory buffer of a remote node.
[0012] Optionally, it further comprises: In the background, the key value index adopts a merging operation to recycle invalid data and keep the data in order; in the merging process, the generated new SSTable calls the interface of the block device driver engine to make it generate a block data copy.
[0013] Optionally, step S300 specifically comprises: S310, submitting the block device read-write request to the block device driver engine, and judging whether the block device read-write request is a write request or a read request; if it is a write request, go to S320; if it is a read request, go to S330; S320, the block device driver engine copies and sends the write request to all configured replica block devices according to a pre-configured replica strategy; S330, if it is a read request, the block device driver engine detects the current access load of all block device nodes that have a copy of the key value, and selects the lightest copy node as the data source based on the load balancing algorithm, and sends a read request to it.
[0014] Optionally, for read and write of the remote copy disk, NVMe-oF technology is used as the underlying remote block device access protocol.
[0015] In a second aspect, the application further provides a data redundancy device based on block data copy distributed key value storage, for executing the data redundancy method based on block data copy distributed key value storage as any one of the first aspect, comprising: A first module for receiving a user's key value request, calculating the hash value of the corresponding key of the key value request, determining the corresponding primary copy key value storage instance according to the hash value, and forwarding the key value request to the corresponding primary copy key value storage instance. A second module for calling a key value index interface through the key value storage instance, and converting the key value request into a block device read-write request. A third module for submitting the block device read-write request to a block device driver engine, judging whether the block device read-write request is a write request or a read request; if it is a write request, generating a copy and writing it on the local and remote copy disks; if it is a read request, reading the corresponding block data from the local or remote copy disk.
[0016] Through the above technical scheme conceived by the application, compared with the prior art, the following beneficial effects can be achieved: 1. The application provides a data redundancy method based on block data copy distributed key value storage, which replaces the original data redundancy scheme based on key value copy with block data copy to realize data redundancy, and adopts a block data copy based manner to avoid the index space amplification problem caused by maintaining redundant key value copies in the remote key value index, reduces the additional CPU overhead caused by repeated calculation, effectively optimizes the performance loss introduced by the multi-layer software stack in the foreground writing process, and improves the system performance and maintainability.
[0017] 2, The application provides a data redundancy method based on block data copy of distributed key value storage, wherein a distributed key value storage system adopts a mode of directly managing LBA space of a block device in a key value index, data is organized by bypassing file semantics of a traditional file system, thereby reducing additional delay caused by file system processing, meanwhile, the mode supports remote nodes to directly identify block data copies, and synchronization of file system metadata is not needed, and system performance and maintainability are further improved, a distributed write cache design is introduced in the key value index, a problem of mismatch between key value request granularity and block device sector size is effectively solved, performance reduction caused by directly exposing a block interface to a key value request is avoided, and efficiency and resource utilization of a write path are improved. BRIEF DESCRIPTION OF DRAWINGS
[0018] Figure 1 is the overall flowchart of the data redundancy method based on block data copy of distributed key value storage provided by the application.
[0019] Figure 2 is the overall architecture diagram of the distributed key value storage in the data redundancy method based on block data copy of distributed key value storage provided by the application. DETAILED DESCRIPTION
[0020] In order to make the object, technical scheme and advantages of the application more clear, the application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.
[0021] The content involved in the above embodiments will be described below in combination with a preferred embodiment.
[0022] As shown in Figure 1 , the application provides a data redundancy method based on block data copy of distributed key value storage, comprising: S100, receiving a key value request of a user, calculating a hash value of a key corresponding to the key value request, determining a corresponding primary copy key value storage instance according to the hash value, and forwarding the key value request to the corresponding primary copy key value storage instance; S200, calling a key value index interface through the key value storage instance, and converting the key value request into a block device read-write request; S300, submitting the block device read-write request to a block device driver engine, judging whether the block device read-write request is a write request or a read request, if it is a write request, generating a copy and writing it on a local and remote copy disk, and if it is a read request, reading corresponding block data from the local or remote copy disk.
[0023] To address the shortcomings of existing technologies, this invention provides a data redundancy method for distributed key-value storage based on block data replicas. It improves the structure of the key-value index engine itself by removing the underlying file system and adding a new NVMe-oF driver layer for handling NVMe-oF requests and load balancing. The original key-value replica-based data redundancy scheme is replaced with block data replicas for data redundancy. Figure 2 As shown, the block device LBA space is directly managed in the key-value index to support remote nodes in quickly identifying block data replicas. The conversion efficiency of key-value-based replicas is much lower than that of block data-based replicas. Implementing replicas at the block layer eliminates the need to maintain key-value replicas on multiple nodes' key-value indexes, thus avoiding the aforementioned problems of key-value index space amplification and redundant computation within the index. Writing replicas directly at the block layer, compared to sending key-value replicas via RPC, bypasses the processing of the RPC, key-value index, and file system software stack, thereby solving the performance limitations of the foreground write operations. Furthermore, the use of distributed write caching technology avoids performance degradation caused by key-value write requests directly calling the block device interface.
[0024] The key-value store client receives key-value requests from users and forwards them to the designated primary replica key-value store instance using a consistent hashing algorithm based on the key in the request, thus resolving consistency conflicts among multiple clients. After being forwarded to the designated key-value store instance, the key-value request is processed to calculate the key-value index, transforming the request into a read / write request for the block device. Based on the replica configuration and the calculation results of algorithms such as load balancing, read / write operations are performed on the local or remote replica disk device.
[0025] Optionally, step S100 specifically includes: S110, Receive key-value requests generated by users through the key-value interface; S120. Calculate the hash value of the key corresponding to the key-value request using a hash function; S130. Perform modulo operation on the calculated hash value, and determine the primary replica key-value storage instance corresponding to the key-value request based on the modulo result; encapsulate the key-value request into an RPC message using RPC technology, and forward it to the RPC server of the node where the corresponding primary replica key-value storage instance is located.
[0026] Users send key-value requests to the key-value client for processing by calling the key-value interface provided by the key-value storage client. Commonly used hash functions include, but are not limited to: MD5, SHA-1, MurmurHash, and CityHash.
[0027] In different embodiments, the consistent hashing algorithm can adopt different algorithms according to different application scenarios; in the present embodiment, MurmurHash is taken as an example to illustrate that the hash function of Murmurhash is used to calculate the hash value of the key of the request. According to the consistent hashing algorithm, it is calculated that the key value should be sent to which primary replica key value storage instance. The consistent hashing algorithm is an algorithm for determining the forwarding node and its service program in a distributed scenario, which is used to solve the consistency conflict problem of multiple client requests, and at the same time, load balancing is realized as much as possible. A simple implementation of the consistent hashing algorithm is to take the modulus of the hash value to the total number of service programs to be forwarded, and then forward to the node and its service program with the corresponding number according to the result of the modulus operation.
[0028] For example, the primary replica key value instance is calculated in the form of modulus operation. It is assumed that there are currently 3 physical nodes, and 2 key value storage instances run on each node, and a key value index is maintained on each key value storage instance. The key value storage instances are numbered in advance, and the numbering starts from 0: the storage instance number of the first physical node is 0, 1; the key value storage instance number of the second physical node is 2, 3, and so on. The hash value obtained by hashing the user key value is 217, and then 1 is obtained by taking the modulus of 217 to 6, so it is sent to the key value storage instance numbered 1, that is, the second key value storage instance of the first physical node.
[0029] Optionally, step S200 specifically comprises: S210, parsing the RPC message through the RPC server, extracting the target key value, and calling the key value index interface in the storage instance of the target key value; S220, submitting the key value request to the key value index through the key value index interface, and converting the key value request into a read-write request to the block device through the key value index; The key value index adopts B+ Tree, learning index, LSM-tree, Trie, Radix Tree, HashTable, append write log index or hybrid index; the key value index adopts a direct management of block device LBA space.
[0030] Further, the key value index can adopt different algorithms according to different application scenarios, for example, B+ Tree or learning index is used in the read frequent scenario, and LSM-tree is used in the write frequent scenario.
[0031] After the key-value index interface is called, the key-value index module will be processed according to its internal logic. The process is to convert the key-value request into an access request to the block device and complete the submission. Different types of key-value indexes may have different specific processing logic. The internal organization of the key-value index is designed to directly maintain the mapping relationship between the key value and the block device LBA (logical block address).
[0032] The traditional key-value index implementation usually chooses to organize the persistent data through the file system for ease of deployment and sharing of storage resources with other applications, that is, to map the key value to the file, and to manage the underlying block device space uniformly by the file system. However, this design method introduces the processing overhead of the file system itself on the one hand, affecting the performance; on the other hand, when using a block data copy-based data redundancy mechanism, since the copies are generated at the block layer bypassing the file system, the file system cannot identify these copy data, thereby causing the key-value index structure built based on the file system to also be unable to identify and recover these data.
[0033] In order to solve the above problems and improve the identifiability and effectiveness of the copy data, especially in the scenario where the remote node needs to recover the index through the copy, the system adjusts the organization method of the key-value index, and instead directly records the mapping between the key value and the block device LBA address. By implementing a set of block device address space management algorithm, the mapping between the key value and the block device LBA address is realized. Specifically, when the key-value index is an LSM-tree, the WAL write is allocated address space through append write, the metadata is allocated address space through a ring log queue, and the SSTable is allocated address space through an idle list. This method avoids dependence on the file system, so that the remote node can identify the copy and recover the complete key-value index without the intervention of the file system.
[0034] Optionally, when the key-value index adopts an LSM-tree, step S220 specifically includes: S221, when the block device read-write request is a write request, the key-value request is first written to a write-ahead log (WAL), and then written to a Memtable located in the memory; when the Memtable reaches a set threshold, its content is flushed to an SSTable and is persisted; S222, when the block device read-write request is a read request, it is checked whether the key-value pair of the key exists in the Memtable, and if it exists, the result is directly returned; if it does not exist, the SSTable of each layer is further searched; In the foreground key-value persistent write process in S221, a distributed write cache technology is used to write the key-value data to the local memory buffer, and at the same time, the RDMA technology is used to synchronize and copy the key-value data to the memory buffer of the remote node pre-configured.
[0035] In this embodiment, taking LSM-tree with key-value index as an example, when a key-value write request is submitted, it is first written into a write-ahead log (WAL), and then written into a MemTable located in memory. When the MemTable reaches a set threshold, its content is flushed into an SSTable file and persisted.
[0036] Specifically, for writing a WAL, it is first written into a distributed write cache, and then a block device driver engine interface is called after the write cache is full of block device sector size. For writing an SSTable, because it is already a large block of data accumulated by a Memtable, a block device driver engine interface is directly called. For a read operation, it is first checked whether the key-value pair of the key exists in the Memtable, and if it exists, the result is directly returned; if it does not exist, each layer of SSTable is further searched.
[0037] In the whole process, the write request is gradually converted into a file system write operation, and finally converted into a write request to the block device by the file system and written to the physical disk.
[0038] Further, in the key-value persistence write process in the foreground of key-value index processing (for LSM-tree, the process of WAL writing), a distributed write cache technology is introduced. There is a granularity mismatch problem between the data size of the key value itself (statistically about 100 bytes in the actual production environment of Meta, on average) and the sector size of the block device (usually 4 KB). Since the sector size of the block device is the minimum write unit, if each key-value write request is individually called to persist by the block interface, performance and space efficiency problems will be caused. If writing is performed according to sector alignment, a large amount of storage space will be wasted. If the key value itself is directly written, the "read-modify-write" mechanism must be used to complete the operation, which will introduce additional delay and cause the overall write performance to decline. Therefore, when the key-value write request needs to be written to the disk, the system will first write it into the buffer maintained inside the key-value index. Only when the accumulated data in the buffer reaches the block device sector size, the block interface is called to complete batch writing, so as to improve the storage efficiency while reducing the performance loss caused by write amplification. At the same time, in order to ensure the high reliability of the distributed key-value storage system, the write cache is designed in a distributed manner. That is, the key-value data is not only written into the local memory buffer, but also synchronized and replicated to the pre-configured memory buffer of the remote node through the RDMA technology, so as to enhance the disaster recovery capability and data consistency guarantee of the system.
[0039] The advantage of using the distributed write cache technology in the foreground persistence write of the key-value index is that, first, the technology can avoid the problem of separately calling the block device interface for each key-value write request, thereby avoiding the performance decline caused by the mismatch between the key-value data granularity and the block device sector size. Second, since the write cache is based on the memory medium, it has faster read-write speed compared to the traditional persistent storage device, effectively improving the foreground write performance. In addition, the distributed write cache reduces the risk of single point failure by replicating the write data to the memory buffer of multiple nodes. Even if a node power failure or failure occurs, other nodes can still retain cache data, enhancing the system's disaster recovery capability and data consistency guarantee.
[0040] Optionally, further comprising: In the background, the key-value index uses the merging operation to recover invalid data and keep the data in order; in the merging process, the generated new SSTable calls the interface of the block device driver engine to make it generate a block data copy.
[0041] Some internal operations of the key-value index may cause the write amplification phenomenon, that is, one user write request may trigger multiple write operations to the block device. For example, the LSM-tree index relies on the compaction operation to recover invalid data and keep the data in order. In the compaction process, the data in the old SSTable may be rewritten to the new SSTable, causing write amplification. These additional write operations also need to follow the data redundancy mechanism: by calling the interface of the block device driver engine, the driver engine generates a data block copy and writes it to the local and remote copy disks, respectively.
[0042] Optionally, step S300 specifically comprises: S310, submitting the block device read-write request to the block device driver engine, and judging whether the block device read-write request is a write request or a read request; if it is a write request, go to S320; if it is a read request, go to S330; S320, the block device driver engine copies and sends the write request to all configured copy block devices according to the pre-configured copy strategy; S330, if it is a read request, the block device driver engine detects the current access load of all block device nodes that store the key-value copy, and selects the lightest load copy node as the data source based on the load balancing algorithm, and sends a read request to it.
[0043] In this embodiment, in order to realize efficient read and write to the remote copy disk, NVMe over Fabrics (NVMe-oF) technology is used as the underlying remote block device access protocol. NVMe-oF is a protocol standard specially used for accessing remote NVMe storage devices, and its design goal is to retain the high performance characteristics of the local NVMe protocol in a network environment, while expanding its ability to be applicable to distributed storage systems. By supporting multiple underlying network structures such as RDMA, TCP or FC, NVMe-oF can realize cross-node, low-latency, high-bandwidth block device access. In traditional systems, remote block device access usually relies on iSCSI, NFS or file system-based shared storage solutions. Although these solutions realize remote data access functions, they have significant limitations in protocol processing, transmission efficiency and concurrency capabilities due to their early design not being aimed at modern high-speed storage hardware, and often become a system performance bottleneck. Especially in the distributed key-value storage scenario where high concurrency and low latency are required, the traditional access method is difficult to fully utilize the performance of the underlying hardware. In contrast, NVMe-oF has obvious advantages in several aspects. First, it adopts a lightweight protocol stack and an efficient command submission / completion queue mechanism, continuing the high-performance design of local NVMe, and minimizing the overhead of data in the transmission process through mechanisms such as zero-copy and user-mode access. Second, NVMe-oF supports multi-channel concurrent access, fully utilizes multi-core processor resources, and improves system throughput. Therefore, in the system design, the block device driver engine accesses the remote copy disk through NVMe-oF channel, so that the write request or read request converted by the key-value index can be efficiently transmitted to the remote node in a block-level manner, thereby ensuring consistency and reliability while realizing the low-latency and high-performance requirements of cross-node data copy access.
[0044] The advantage of using NVMe-oF as the underlying remote block device access method is that, first, it adopts a lightweight protocol stack and an efficient command submission / completion queue mechanism, continuing the high-performance design of local NVMe, and minimizing the overhead of data in the transmission process through mechanisms such as zero-copy and user-mode access. Second, NVMe-oF supports multi-channel concurrent access, fully utilizes multi-core processor resources, and improves system throughput.
[0045] The data redundancy scheme originally based on key-value copies is replaced by block data copy in the embodiment of the application to realize data redundancy, the block data copy based manner avoids the index space amplification problem caused by maintaining redundant key-value copies in the remote key-value index, reduces the additional CPU overhead caused by repeated calculation, and effectively optimizes the performance loss introduced by the multi-layer software stack in the foreground writing process. The distributed key-value storage system directly manages the LBA space of the block device in the key-value index, realizes bypassing the file semantics of the traditional file system to organize data, thereby reducing the additional delay caused by the file system processing. At the same time, this manner supports the remote node to directly identify the block data copy without relying on file system metadata synchronization, further improving the system performance and maintainability. The overhead problem of the key-value copy mechanism based on RPC in the prior art is solved, including index space amplification, repeated calculation of background tasks, and inefficient foreground writing. The beneficial effects of reducing performance loss, reducing delay, and improving system performance and maintainability are realized.
[0046] Embodiment two The application also provides a block data copy based distributed key-value storage data redundancy device for executing the block data copy based distributed key-value storage data redundancy method in any one of the embodiments. The first module is configured to receive a key-value request of a user, calculate a hash value of a key corresponding to the key-value request, determine a corresponding primary copy key-value storage instance according to the hash value, and forward the key-value request to the corresponding primary copy key-value storage instance. The second module is configured to call a key-value index interface through the key-value storage instance, and convert the key-value request into a block device read-write request. The third module is configured to submit the block device read-write request to a block device driver engine, determine whether the block device read-write request is a write request or a read request, generate a copy and write the copy on a local and remote copy disk if the block device read-write request is the write request, and read corresponding block data from the local or remote copy disk if the block device read-write request is the read request.
[0047] The block data copy based distributed key-value storage data redundancy device provided in the embodiment of the application is used to execute the block data copy based distributed key-value storage data redundancy method, and has the same or similar beneficial effects.
[0048] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the application, and is not used to limit the application, and any modification, equivalent replacement and improvement made within the spirit and principle of the application should be included in the protection scope of the application.
Claims
1. A method for data redundancy based on chunk data duplication of distributed key-value storage, characterized in that, Comprise: S100, receiving a user's key value request, calculating the hash value of the key corresponding to the key value request, determining the corresponding master replica key value storage instance according to the hash value, and forwarding the key value request to the corresponding master replica key value storage instance; S200, calling the key value index interface through the key value storage instance, and converting the key value request into a block device read-write request; S300, submitting the block device read-write request to the block device drive engine, and judging whether the block device read-write request is a write request or a read request; If it is a write request, a copy is generated and written on the local and remote copy disk; If it is a read request, the corresponding block data is read from the local or remote copy disk.
2. The method of claim 1, wherein, Step S100 specifically comprises: S110, receiving a key value request generated by a user through a key value interface; S120, using a hash function to calculate the hash value of the key corresponding to the key value request; S130, taking the modulus of the calculated hash value, determining the master replica key value storage instance corresponding to the key value request according to the modulus result, and encapsulating the key value request into an RPC message through RPC technology and forwarding it to the RPC server of the node where the corresponding master replica key value storage instance is located.
3. The method of claim 2, wherein, Step S200 specifically comprises: S210, parsing the RPC message through the RPC server, extracting the target key value, and calling the key value index interface in the storage instance of the target key value; S220, submitting the key value request to the key value index through the key value index interface, and converting the key value request into a read-write request to the block device through the key value index; Wherein, the key value index adopts B+ Tree, learning index, LSM-tree, Trie, Radix Tree, HashTable, append write log index or hybrid index; the key value index adopts the way of directly managing the LBA space of the block device.
4. The method of claim 3, wherein, When the key value index adopts LSM-tree, step S220 specifically comprises: S221, when the block device read-write request is a write request, the key value request is first written into a write-ahead log WAL, and then written into a Memtable located in the memory; when the Memtable reaches a set threshold, its content is flushed as an SSTable, and is persisted; S222, when the block device read-write request is a read request, it is checked whether the key value pair of the key exists in the Memtable, and if it exists, the result is directly returned; if it does not exist, each layer of SSTable is further searched; Wherein, in the foreground key value persistent writing process in S221, distributed write cache technology is used to write key value data into a local memory buffer, and at the same time, RDMA technology is used to synchronize and copy the key value data to a pre-configured memory buffer in a remote node.
5. The method of claim 4, wherein, Further comprising: In the background, the key value index adopts a merging operation to recycle invalid data and keep the data in order; In the merging process, the new SSTable generated is called through the interface of the block device drive engine to make it generate a block data copy.
6. The method of claim 1, wherein, Step S300 specifically comprises: S310, submit the block device read-write request to the block device drive engine, judge whether the block device read-write request is a write request or a read request; if it is a write request, go to S320; if it is a read request, go to S330; S320, the block device drive engine copies and sends the write request to all configured replica block devices according to the pre-configured replica strategy; S330, if it is a read request, the block device drive engine detects the current access load of all block device nodes that store the key value copy, and selects the lightest replica node as the data source based on the load balancing algorithm, and sends a read request to it.
7. The method of claim 6, wherein, For read and write of remote replica disks, NVMe-oF technology is used as the underlying remote block device access protocol.
8. An apparatus for data redundancy of a block data replica based distributed key- value store, configured to perform the method for data redundancy of a block data replica based distributed key-value store according to any one of claims 1 to 7, characterized in that, Comprise: A first module for receiving a user's key value request, calculating the hash value of the key corresponding to the key value request, determining the corresponding primary replica key value storage instance according to the hash value, and forwarding the key value request to the corresponding primary replica key value storage instance; A second module for calling a key value index interface through the key value storage instance to convert the key value request into a block device read-write request; A third module for submitting the block device read-write request to the block device drive engine, judging whether the block device read-write request is a write request or a read request; If it is a write request, generate a copy and write it to the local and remote replica disks; If it is a read request, read the corresponding block data from the local or remote replica disk.