Distributed key-value store system based on microkernel

Abstract

This invention provides a microkernel-based distributed key-value storage system, comprising: a node routing module deployed on the client, which constructs a sharding mapping relationship based on node load and achieves decentralized data sharding location based on key hashing; a network communication module, which establishes a lightweight communication connection between the client and the server; a data storage module, which adopts a layered structure of a sharding index layer and a sharding data layer, and achieves efficient concurrent access through an asymmetric storage method combining scalable hash tables and logs; and a data consensus module, which constructs a consistency group composed of master and slave nodes, and maintains data consistency within shards and migration switching under dynamic node changes through member management, cross-node capability control, and replica synchronization mechanisms. This invention can achieve high-concurrency access, low-overhead consistency maintenance, and efficient dynamic expansion capabilities while ensuring data security isolation and consistency.

Description

Technical Field

[0001] This invention relates to the field of distributed storage systems, and more specifically, to a microkernel-based distributed key-value storage system. Background Technology

[0002] Distributed key-value (KV) storage systems are distributed system infrastructures characterized by simple data models, unified access interfaces, and easy horizontal scaling. They are widely used in critical scenarios such as metadata management, configuration services, and caching systems. Compared to traditional relational databases, distributed key-value storage systems place greater emphasis on system availability and horizontal scalability, enabling efficient processing of high-concurrency requests by sharding massive amounts of data across different computing nodes.

[0003] Consistent hashing is an algorithm for efficient data sharding and routing in distributed systems. It divides the entire hash key-value space into several hash shards and maps these shards to several nodes, thus achieving data mapping to target nodes by hashing data items across this space. The core advantage of consistent hashing lies in the stability of its mapping: when a node joins or leaves the system, only the data within the affected interval on the hash ring needs to be re-migrated, avoiding the global data redistribution problem caused by traditional modulo hashing. Although consistent hashing has significant advantages in solving migration decisions for dynamically changing nodes, in engineering practice, how to further maintain the balance of data distribution and improve the effectiveness of migration decisions remains a key challenge in designing high-performance distributed storage systems. Some existing routing implementations often rely on centralized mapping tables, failing to fully utilize hardware and software features or server-side computing power to achieve decentralized autonomous mapping.

[0004] Microkernel architecture is an important paradigm in operating system design. Its design principle is to minimize kernel functionality, retaining only core functions such as inter-process communication, basic memory management and scheduling, and capacity allocation, while moving service components such as the file system, device drivers, and network protocol stack to run in user space. Compared to monolithic kernel architectures that integrate all critical functions into kernel space, microkernels significantly enhance system security, verifiability, and fault tolerance through physical isolation between components.

[0005] Most existing distributed key-value storage systems are designed based on monolithic kernel environments. Patent document CN102110146A discloses a metadata management method for a distributed file system based on key-value storage. This existing technology is mainly geared towards monolithic kernel environments and relies on centralized metadata management logic. When facing large-scale, high-concurrency distributed requests, the central node is prone to becoming a performance bottleneck. Furthermore, distributed storage systems implemented based on monolithic kernels cannot be directly applied to microkernels. The application practices of microkernel architectures in existing distributed key-value storage systems are still lacking. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the purpose of this invention is to provide a microkernel-based distributed key-value storage system.

[0007] A microkernel-based distributed key-value storage system provided by the present invention includes: The node routing module, deployed on the client, is used to build a sharding mapping relationship based on the load capacity of each storage node, and to locate data shards and target nodes based on the hash value of the key. The data storage module, deployed on the server-side storage node, is used to organize and store key-value data in units of shards; The network communication module is used to establish communication connections between the client and each storage node, and to transmit data based on a preset communication protocol; The data consensus module is deployed on the server-side storage node to maintain the consistency of multiple replicas of data within each shard and to perform data migration and master-slave switching when the node status changes dynamically.

[0008] Preferably, the method for constructing the fragment mapping relationship includes the following steps: Step S1: Obtain the configuration information of all active storage nodes in the cluster. The configuration information includes at least the node identifier, communication protocol version, IP address, port, and storage capacity information representing the node's load capacity. Step S2: Normalize the storage capacity of each storage node and calculate the weight of each node. ; Step S3: Based on the preset N binary bits Each fragment identifier, based on the weight Calculate the number of shards that should be allocated to each node. Then, the remaining unallocated fragments after calculation are allocated sequentially according to the decimal part of each node's weight in descending order, until all fragments are allocated. Step S4: Construct a sharding mapping table, wherein the sharding mapping table has a length equal to A fixed-length array, where each item in the array corresponds to a slice, and records the master node identifier and node running status corresponding to the slice; Step S5: Perform a hash calculation on the key to be operated on, take the high N bits of its hash value as the shard index, and locate the target storage node by querying the shard mapping table.

[0009] Preferably, the node routing module is further configured to support dynamic sharding reallocation, including: In response to the addition of a new storage node to the system, the client obtains the updated node list, recalculates the weight of each node and the target number of shards, identifies the node that holds more shards than its target value and marks its excess shards as pending migration, allocates the shards to the new storage node and updates the shard mapping table, and marks the corresponding shards as being in the process of migration. After the data migration is completed, the migration status mark is removed. In response to a storage node voluntarily leaving the system, the server's data consensus module selects a new master node for each shard it manages and returns the update result to the client.

[0010] Preferably, the network communication module includes: The protocol verification unit is used to verify the protocol version and magic number of the handshake message when establishing a communication connection. An access credential allocation unit is used to allocate access credentials containing a read key and a write key to the client after the connection is established; The data transmission unit is used to provide a data interaction interface based on the access credentials to support write, read, update and delete operations of key-value data.

[0011] Preferably, the data storage module includes a sharded index layer and a sharded data layer, wherein: The sharding index layer is used to locate the target shard based on the hash value prefix of the key; The sharded data layer adopts an asymmetric storage structure, wherein the primary shard uses a storage method that combines a scalable hash table with a lightweight log, and the replica shard uses a full log structure for storage. The lightweight log of the primary shard is used to record data operation information and to achieve concurrent appending through atomic operations; The replica shard recovers data by replaying its full log.

[0012] Preferably, the data storage module processes write requests for the primary shard through the following steps: B1: Receive a write request initiated by the client, the write request containing a key, a value, and a globally incrementing version number; B2: Calculate the hash value of the key, locate the target shard based on its high N bits, and locate the target data bucket based on its low bits in the directory of the scalable hash table; B3: Acquire the bucket-level lock of the target data bucket through atomic comparison and swap operations; B4: While holding the bucket-level lock, perform a quick match using the fingerprint table within the bucket. Check the existence of the key and its version number using the fingerprint table. If the version number in the request is not greater than the current version number, release the lock and return. Otherwise, if there is an empty slot in the bucket, perform an insertion or update operation and synchronously update the fingerprint at the corresponding position in the fingerprint table. If the bucket is full, trigger the bucket splitting process, and after the split is completed, relocate and perform an insertion or update operation. B5: After completing the scalable hash table update, the write request is encapsulated as a log entry and appended to the lightweight log corresponding to the shard through an atomic operation; B6: Release the bucket-level lock and return a write success response.

[0013] Preferably, the data consensus module includes: The group member management submodule is used to build a consistent group consisting of one master node and at least one slave node, and maintain the consistency of the group structure through member logs. The cross-node capability control submodule is used to allocate cross-node access capabilities to the secondary nodes within the consistency group. The cross-node access capabilities are used to control the access permissions of the secondary nodes to the log data of the primary node. The replica synchronization submodule is used to achieve data synchronization between the master node and the slave node through incremental log retrieval.

[0014] Preferably, the group member management submodule maintains the consistent group state through member logs, wherein: The member log records the term number, master node identifier, list of slave nodes, and change type; When inconsistencies are detected in the logs of members within a group, the logs of the master node shall be used for overwriting and synchronization. When a storage node joins or leaves, causing a change in group members, the group member management submodule updates the member logs according to different scenarios and executes the corresponding master-slave switchover and data migration process.

[0015] Preferably, the cross-node access capability generated by the cross-node capability control submodule includes a shard identifier, a node identifier, and a signature field generated based on the master node's local key; When a secondary node accesses the resources of the primary node, it must carry the cross-node access capability. After receiving the request, the primary node performs a validity check on the cross-node access capability and allows access only after the check passes.

[0016] Preferably, the replica synchronization submodule achieves data synchronization through the following steps: E1: On the master node side, maintain a log cursor for each slave node to indicate the position of the lightweight log that the slave node has synchronized; E2: Receive log pull requests periodically initiated by the secondary node, the requests carrying the identification information of the last record in the local log and the corresponding cross-node access capability; E3: Perform a validity check on the cross-node access capability, and after the check passes, perform a consistency check on the identifier information and the log cursor. E4: If the consistency check passes, then traverse the subsequent lightweight log entries starting from the log cursor position. For write operations, query the latest value corresponding to the expandable hash table, and encapsulate the key, value, and operation type into a complete log entry; for delete operations, directly encapsulate it into a log entry. E5: Batch return a set of encapsulated log entries to the secondary node; E6: The secondary node receives log entries and appends them to the local full log; E7: If the consistency check fails, the secondary node will be triggered to perform a full resynchronization process.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention eliminates centralized performance bottlenecks and achieves efficient read and write operations through client-side decentralized routing and bucket-level concurrency control of the main shard scalable hash table; it also ensures fine-grained data security isolation based on microkernel protection domains and cross-node capability control mechanisms.

[0018] 2. This invention achieves multi-replica data consistency within a shard with low computational and communication overhead by using small-scale consistency groups and an incremental pull synchronization mechanism based on lightweight logs.

[0019] 3. This invention uses a weight-based shard redistribution and log lazy migration strategy to migrate only the affected shards and continue to provide services when nodes change dynamically, which significantly shortens the service interruption window and improves system availability and scalability. Attached Figure Description

[0020] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a block diagram of the microkernel-based distributed key-value storage system according to an embodiment of the present invention; Figure 2 This is a diagram of the communication protocol message format of the network communication module according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the combination of the server-side data storage module and the data consensus module in an embodiment of the present invention; Figure 4 This is a schematic diagram of the key-value data writing process according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the key-value data reading process according to an embodiment of the present invention. Detailed Implementation

[0021] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0022] Before introducing the embodiments of the present invention, the terms used in this document will be explained.

[0023] The Genode operating system framework is a component-based operating system development architecture that follows the principle of least privilege. Its core employs a microkernel design and manages system resources through a recursive, hierarchical tree structure. In this invention, Genode provides a highly secure execution environment for distributed storage services. Its physical isolation mechanism between components and external user-space services ensure that storage nodes run within independent protection domains. This enables flexible scheduling of underlying heterogeneous memory resources while leveraging lightweight inter-process communication mechanisms to guarantee high reliability and scalability in a distributed environment.

[0024] This invention employs a consistent hashing algorithm to achieve decentralized data routing. It uses MurmurHash3, a non-cryptographic hashing algorithm, to generate highly uniformly distributed 64-bit feature values, reducing the probability of hash collisions and improving the distribution balance of storage nodes on the logical ring. Based on a microkernel-based capability security model, this invention achieves fine-grained resource access control and isolation in a distributed environment through explicit authorization and verification of cross-node access capabilities. Combining the modularity of Genode, data storage, communication, and consistency mechanisms are divided into independent components running in different protection domains, thereby improving module decoupling and scalability while ensuring system security.

[0025] Figure 1 This invention illustrates the modular composition of a microkernel-based distributed key-value storage system according to an embodiment of the present invention. The system mainly consists of four modules, including a node routing module, a network communication module, a data storage module, and a data consensus module.

[0026] The node routing module, deployed on the client side, enables dynamic data distribution and load balancing in a multi-storage-node environment. Its core function is to map backend physical storage nodes to several hash shards and locate the corresponding storage node based on the hash prefix of the key to be processed, thus ensuring uniform data distribution and system stability during expansion. The network communication module handles communication between the client and storage nodes, implementing data transmission based on a specific communication protocol. Its functions include a protocol verification unit, an access credential allocation unit, and a data transmission unit. The data storage module, deployed on the server side, performs efficient concurrent read and write access to key-value data through an extensible hash table and interacts with the client via the network communication module. The data consensus module, also deployed on the server side, maintains data consistency across multiple replicas on the server side, improving data reconstruction performance.

[0027] In one embodiment of the present invention, the client performs the following steps to initialize and construct the sharding mapping structure and perform key-value data routing operations in the node routing module: Step S1: Obtain the configuration information of all active storage nodes in the current cluster through the registration management node, including node ID, communication protocol version, IP address, port and total load capacity of the node; preferably, in this embodiment, the total storage space size is used as the load capacity indicator; the registration management node is only used for simple service discovery and does not participate in the real-time scheduling of read and write requests; Step S2: The client normalizes the storage space of all nodes and calculates the weight of each node. The calculation formula is: ,in This represents the storage space size of node i. M represents the storage space size of node j, and M represents the total number of nodes in the system.

[0028] Step S3: The shard ID is determined by N binary bits (in this embodiment, N=4, resulting in 16 shard IDs). Based on the weights... Calculate the number of shards that should be allocated to each node. ,Right now .like Less than Then, the remaining unallocated fragments are processed according to... The decimal parts are distributed in descending order to ensure that all equal ,in, This indicates the number of shards that node j should be allocated, and M represents the total number of nodes in the system.

[0029] Step S4: Construct a sharding mapping table, whose data structure has a length of [length missing]. A fixed-length array, each item of which corresponds to an N-bit prefix number. The array elements are the master node ID and node running status corresponding to the shard (0 indicates normal and 1 indicates migration status). Step S5: Use the MurmurHash3 algorithm to perform hash calculation on the key to obtain a 64-bit hash value hash(key). Take the high N bits as the shard index, and the target node can be located in the shard mapping table by looking up the table.

[0030] In one embodiment of the present invention, when a service node actively comes online, the client performs a sharding reallocation operation by executing the following steps, specifically including two scenarios: 1) Service nodes actively join the system: S1: After a new node goes online, it registers its own information with the registration and management node, and the client obtains the updated list of nodes and their storage space information; S2: The client recalculates the weights of all nodes according to the initialization phase method. and the number of target fragments ; S3: The client compares the current sharding mapping table with the target allocation result, identifies nodes whose number of shards exceeds their target value, and adds their excess shards to the migration queue. S4: Select shards from the queue to be migrated in order and assign them to new nodes until the number of new nodes reaches the target number of shards. At the same time, update the shard mapping table and mark the corresponding shards as "migration in progress". S5: The client distributes the updated sharding mapping relationship to the relevant nodes, and the original master node and the new node perform data synchronization according to the data migration mechanism; S6: Once the corresponding shard migration is complete, the client removes the "Migration in Progress" marker and completes the shard reallocation.

[0031] 2) Service nodes voluntarily leave the system: The server automatically invokes its internal data consensus module for each shard it manages to select a new node as the new master node for that shard and returns the update result to the client. No additional action is required from the client.

[0032] In one embodiment of the present invention, the network communication module realizes reliable data exchange between the client and the storage node through a preset communication protocol, including key-value data reading, writing, updating, and deleting operations. The message format of the communication protocol is as follows: Figure 2 As shown. Specifically, the network communication module will perform the following actions: A1: The client initiates a connection request and sends a handshake message containing the protocol version identifier and magic number. After the server completes the verification, a communication connection is established. A2: After the connection is established, before any data is actually written, the server assigns access credentials to the client for subsequent data access control. Specifically, the server generates an access token associated with the current session and returns it to the client. This access token includes a read key and a write key. A3: The client stores the aforementioned identifier and key as the unique security credentials for subsequent read and write operations. The client then uses these access credentials to initiate data operation requests through the data transmission interface.

[0033] A4: After receiving the request, the server verifies the access token, performs the corresponding data operation after the verification is successful, and returns the result.

[0034] Figure 3 This is a schematic diagram illustrating the combination of a server-side data storage module and a data consensus module in a distributed key-value storage system according to an embodiment of the present invention. The data storage module is mainly deployed on the server side and includes a data storage structure and a write processing submodule. The data storage structure adopts a hierarchical index and asymmetric storage organization method, dividing the internal structure of the storage node into a two-level structure of a sharded index layer and a sharded data layer. Different data organization forms are adopted for primary shards and replica shards, and an adapted log structure and locking mechanism are designed to achieve high-performance, high-concurrency data storage based on data consistency.

[0035] The sharding index layer, located at the top of the data storage module, routes data to the corresponding shard based on the hash prefix of the key value. Specifically, the system performs a hash function hash(key) on the input key and takes its prefix N bits as the shard identifier (in this embodiment). ), dividing the entire bond space into Each logical shard. Each storage node internally maintains a shard of size [size missing]. The sharding index table contains a sharding descriptor for each entry, which includes a sharding type identifier (primary shard or replica shard), sharding status information, and a pointer to its corresponding data structure.

[0036] Furthermore, the data storage module employs an asymmetric data organization method for different types of shards. For primary shards, a combined structure of an extensible hash table and a lightweight log is used; for replica shards, a full log structure is used. For primary shards, the core data structure is an extensible hash table (EH) for efficient storage of key-value pairs. This extensible hash table consists of a directory table and data buckets. The directory table is an array of pointers with a size of [missing information]. (GD stands for Global Depth). Each directory entry points to a data bucket; each data bucket contains 8 key-value data slots for storing specific key-value pairs. A fingerprint table is added within the bucket. For each 64-bit hash (key), bits 9-16 are taken as the fingerprint and stored in the fingerprint table in the order of key-value pairs stored within the bucket. During queries, the fingerprint table can be used for fast matching first, improving the response speed of queries within the bucket.

[0037] Specifically, each data bucket contains the following fields: a key-value pair array, a fingerprint table, a valid bitmap, a local depth field, and a bucket-level lock. The valid bitmap identifies whether each slot is occupied, using a bit representation (8 bits in size, 1 for present, 0 for empty). The bucket-level lock controls concurrent access, implemented using atomic variables; for example, it uses 1 byte to represent the lock state, where 0 indicates unlocked and 1 indicates locked, using the CAS (Compare-And-Swap) primitive for locking and releasing operations. The fingerprint table sequentially stores 8 bits of the key hash value from each data slot as a fingerprint for fast intra-bucket lookup and matching.

[0038] Specifically, the primary shard also maintains a lightweight log structure to support the synchronization of primary shard data to replica nodes. This lightweight log contains the following fields: operation type (1 byte, 0 for write, 1 for delete), key name, timestamp, and a pointer to the next node. It does not record read operations or actual value data. This log uses atomic tail pointers and a CAS mechanism to implement concurrent appending.

[0039] For replica shards, the data storage module no longer maintains a scalable hash table. Instead, it maintains a full log, recording the complete key-value pair data and operation type (1 byte, 0 for write, 1 for delete) for that shard. This log is used to rebuild a scalable hash table when the shard is promoted to a primary shard. The full log is maintained by a single thread in the server-side replica synchronization submodule. By replaying the full log in sequence as needed, the complete key-value data storage can be restored.

[0040] The write processing submodule is responsible for handling client write requests and coordinating local data updates on the master node. The specific execution process is as follows: B1: The client sends a write request to the master node. The request includes the key, value, and a globally incrementing version number. B2: The master node calculates hash(key), takes the high 4 bits (corresponding to 16 first-level shards) to locate the target shard in the first-level shard index, and locates the target data bucket in the shard's expandable hash table directory based on the low bits of hash(key). B3: Acquire the bucket-level lock of the target data bucket (implemented using atomic CAS, 0 indicates no lock, 1 indicates lock), and spin to retry if locking fails; B4: After acquiring the bucket lock, perform the following checks and updates: B4.1: First, a quick match is performed using the fingerprint table in the bucket to check if there is an entry that matches the fingerprint of the target key. If it exists, the specific key-value pair is further compared and its version number is read. If it does not exist, it is directly determined that the key does not exist in the bucket. B4.2: If the requested version is less than or equal to the current version, it is determined to be an old version request, the bucket lock is released and the request is returned; if the requested version is greater than the current version, it is determined to be a new version request, and the value data is updated. B4.3: If the key does not exist and there is a free slot in the bucket, the insertion operation is performed directly. While writing the corresponding key-value pair, the fingerprint at the corresponding position in the fingerprint table is updated synchronously, and the new version number is written. B4.4: If the key does not exist and the bucket is full, the bucket splitting process is triggered: B4.4.1: First, acquire the shared lock (read lock) of the directory; if the local depth of the current bucket is equal to the global depth, release the shared lock, acquire the exclusive lock (write lock), and then check the local depth again. If expansion is still needed, double the directory capacity (increase the global depth by one). B4.4.2: Based on the local depth of the current bucket, add one bit of hash low bit as the judgment criterion, redistribute the data in the original bucket to the two buckets according to the newly added hash bit; at the same time, split the fingerprint table and allocate the fingerprints of the corresponding key-value pairs to the fingerprint tables of the two new buckets according to the same rules. B4.4.3: Update the relevant pointers in the directory and release the directory lock; B4.4.4: After splitting, relocate the target bucket and perform insert or update operations; B5: After completing the hash table update, obtain the lightweight log tail pointer while maintaining the bucket lock, and encapsulate the current write request as a log item and append it to the lightweight log corresponding to the shard through a CAS operation; B6: Release the bucket-level lock and return a write success response.

[0041] The server-side data consensus module mainly consists of a group member management submodule, a cross-node capability control submodule, and a replica synchronization submodule. These submodules work together to ensure the consistency and availability of data within shards.

[0042] The group member management submodule is responsible for the initialization and basic structure maintenance of the consistency groups corresponding to the shards. Specifically, during the system initialization phase, the client maps keys to the corresponding shards based on the hash ring and directly selects the target node as the leader, while selecting the two nodes with adjacent node_ids on the hash ring as followers. The three nodes together form a fixed-size group. The leader allocates cross-node access capability (Cap) to this shard, which is used to authorize the followers to access its log data.

[0043] Furthermore, the group member management submodule is responsible for handling the dynamic adjustment of nodes within the consistency group and maintaining structural consistency within the group through the Member Log. Each shard corresponds to a Member Log, and each record in the log contains the following fields: term number, Leader node ID, Follower node list, and change type. During initialization, all nodes have a term of 1, with the master node being the Leader and the remaining nodes being Followers. All members within the group must maintain consistency in their Member Logs; when inconsistencies are detected, synchronization is performed based on the Leader node's log.

[0044] When dynamic changes occur in storage nodes during system operation, leading to changes in group members, the group member management submodule executes the following process to ensure master-slave switching and data consistency migration, specifically including the following two scenarios: 1) When a node joins the system, the shard master node migrates from node A to the new node B using lazy migration: C1: The client notifies the original master node A to migrate the target shard to the new node B, and marks the shard as being in the "migration" state in the hash ring; C2: A adds B to the current group as a candidate follower, updates the Member Log (increments the term by 1), synchronizes it to other follower nodes in the group, and authorizes Cap to B; C3: During the migration, all client write requests will still be handled by A, and A will complete the EH update and lightweight log append according to the existing write process; C4: B continuously pulls lightweight logs from A through the replica synchronization submodule and appends them locally to form full logs. The EH structure is not built immediately during this stage. C5: The EH construction process is triggered when either of the following conditions is met: 1) The client initiates a read request for this shard; 2) A detects that B's logs have been caught up (i.e., the log cursor has reached the end). After triggering, B performs log replay based on its local full logs and constructs the complete EH structure according to the scalable hash rules; C6: Once B completes the EH construction and keeps the log in sync, it sends a ready notification to A. During this period, all write requests sent to A are immediately forwarded to B. Then, the Member Log is updated, and B is officially switched to Leader. A leaves the replica group. C7: A synchronizes the latest Member Log to the nodes in the group and notifies the client to update the hash ring shard status, completing the migration process.

[0045] It should be noted that this embodiment adopts a lazy migration mechanism in this case. During the migration process, the hash table structure is not directly transmitted. Instead, data synchronization is performed based on logs, and the index structure is built as needed when the conditions are met, thereby reducing migration overhead and ensuring that the system can continue to provide services.

[0046] 2) When a node leaves the system, the shard master node should migrate from node A to another node B as soon as possible: D1: A queries the current load status of the two secondary nodes B and C in the group, selects the node ID with the lightest load and the current shard ID and returns it to the client as the new master node of the shard, and marks the shard as "migration in progress" in the hash ring; D2: A changes node B's mark to a candidate node and simultaneously instructs B to immediately trigger log replay and begin building EH; D3: A accesses the registration management node, selects the node D with the largest free space (excluding Followers) as the new secondary node, replaces the original node B, updates the Member Log (increases the term by 1) and synchronizes it to other Followers, and instructs D to start pulling the full log from A. D4: During the migration, all write requests are still handled by A and continuously synchronized to all Followers, including B, through the log synchronization mechanism; D5: Once B completes the EH construction and the logs remain synchronized, it sends a ready notification to A. D6: When A detects B's readiness information, it switches B to Leader, transfers D's log synchronization pointer to B, updates the Member Log and synchronizes it to other Followers, then leaves the replica group and notifies the client to update the hash ring state.

[0047] In addition, when a secondary node in a consistency group needs to leave the system, it will first send an offline notification to its primary node. Upon receiving the notification, the primary node will select the node with the most available idle resources from the registration management node as the new secondary node based on the current system load, add it to the consistency group, update the Member Log and synchronize it with other Followers, and issue a cross-node Cap to it. The new secondary node will then pull the log from the primary node through the replica synchronization submodule to complete the data synchronization. Once the new secondary node has caught up with the log, it will officially participate in the group's replica service.

[0048] The cross-node capability control submodule is used to establish access authorization relationships between nodes. Specifically, for each consistency group, the master node grants cross-node capabilities (Cap) to each slave node to control the slave node's access permissions to the master node's logs.

[0049] Furthermore, the Cap is organized using the following data structure: it includes a Shard ID, a Node ID, and a signature field. The signature field is generated by the master node using a local key and is used to prevent forged requests. When a slave node accesses resources on the master node, it must carry this Cap. Upon receiving the request, the master node performs the following verification: if the Cap does not exist or the signature is invalid, the request is discarded; otherwise, access to the lightweight log of the corresponding shard is allowed. This mechanism eliminates the need for complex authentication processes, completing the process solely through local verification, thus ensuring both efficiency and security.

[0050] The replica synchronization submodule is responsible for propagating updates from the master node to the two slave nodes within the group. The specific execution process is as follows: E1: Each slave node maintains a corresponding log cursor on the master node side to indicate the position of the lightweight log that the slave node has synchronized; E2: The secondary node periodically sends log fetch requests to the primary node (the period is 10ms in this embodiment, which can be dynamically adjusted to 50ms when the system load is high). The request carries the key and version information of the last record in the local log and the corresponding Cap. E3: After receiving the request, the master node first performs a validity check on Cap; if the check fails, the request is discarded directly; otherwise, a consistency check is performed based on the log identifier information carried in the request and the current log cursor state. E4: If the verification passes, the master node will start from the log cursor position corresponding to the slave node and sequentially traverse the subsequent lightweight log entries, performing the following processing on each traversed log entry: E4.1: Extract the key and operation type from the log item. If the operation type is deletion, it is directly encapsulated into a log item. E4.2: If the operation type is write, query the latest value corresponding to the key in the sharded scalable hash table; E4.3: Re-encapsulate the key, value, and operation type into a complete log item; E5: The master node packages the above-encapsulated N log items into a batch and returns them to the slave node (in this embodiment, N=Min(16, number of remaining log items)). E6: After receiving this batch of data, the secondary node appends these log entries to the local full log. E7: If the consistency check fails, the secondary node will trigger a full resynchronization process.

[0051] The following is combined Figure 4 The specific steps for writing key-value data are as follows: F1: The client calls the node routing module, performs MurmurHash3 calculation on the key to obtain hash(key), takes the high N bits (N=4 in this embodiment) to locate the shard ID, and looks up the corresponding master node ID in the shard mapping table; F2: The client establishes a connection with the target master node through the network communication module, performs version verification and handshake process, and calls the sendTryAlloc operation to obtain the access credentials required for writing (including blockID, version and read / write key). F3: The client encapsulates (operation, key, value, version) into a write request message and sends it to the target master node via the setKeyValue operation; F4: After receiving a request, the master node first verifies the access credentials. If the verification is successful, it updates the local scalable hash table and lightweight log through the write processing submodule of the data storage module. This log serves as the sole data source for replica synchronization. F5: The master node returns a write success response to the client, and the client completes a write operation.

[0052] F6: The secondary node calls the data consensus module to periodically pull data from the primary node to the local full log, completing the synchronization of data across multiple replicas.

[0053] The following is combined Figure 5 The specific steps for reading key-value data are as follows: To avoid read-write tearing, an optimistic read mechanism is adopted during key-value data reading. The core logic is as follows: before reading data, the version number corresponding to the key-value data is read first. After reading the data, the version number is read again. If the version numbers read twice are the same, it is determined that the data has not been modified during the reading process, thus avoiding read-write tearing.

[0054] H1: The client calls the node routing module to perform MurmurHash3 calculation on the key to obtain hash(key), takes the high N bits (N=4 in this embodiment) to locate the shard ID, and looks up the corresponding master node ID in the shard mapping table; H2: The client sends a read request message (containing the key and access credentials) to node A through the network communication module. H3: After receiving the request, node A first verifies the access credentials. If the verification passes, it determines whether the shard is in a migration state (the default is a normal service state; if A wants to actively go offline, it is in the process of migration): H3.1: If not in a migration state, calculate hash(key) to obtain the shard ID, and perform a lookup operation in the corresponding scalable hash table (EH) using the first-level index: First, locate the target data bucket based on hash(key), and perform a fast match using the fingerprint table within the bucket; if no fingerprint is found, the data does not exist, and a response is returned; if a fingerprint is found, quickly locate the slot of the key-value pair within the bucket, and determine whether the key data in the slot matches the key requested by the client; if they do not match, the data does not exist, and a response is returned; if they match, further read the key-value pair version number (denoted as v1), then read the key-value pair data, and after reading, read the version number of the key-value pair again (denoted as v2); if v1 equals v2, the data read is confirmed to be valid, and the key-value pair data is returned, completing one read operation for the client; if v1 does not equal v2, the optimistic read process is re-executed; H3.2: If in the migration state, further check the status flag of candidate node B (whether EH construction has started). H4: If it is detected that B has not yet started building EH (read request arrives, triggering condition is met), node A triggers B to enter the building phase. B immediately synchronizes the full log and performs a replay operation, building a complete EH structure according to the scalable hash rule. At the same time, node A completes the query in the local EH and returns the result, and the client completes a read operation. H5: If it is detected that B has started building EH but has not yet completed it, node A directly completes the query in the local EH and returns the result, and the client completes a read operation; H6: Once it is detected that B has completed the EH construction, node A updates the Member Log, officially switches B to Leader and removes it from the replica group. Subsequent client requests are then directly routed to node B.

[0055] In summary, this invention achieves low-overhead data consistency maintenance and high-concurrency write performance by constructing an asymmetric storage model based on a scalable hash table and log structure, combined with lightweight Cap permission control and a small-scale replica group mechanism. Furthermore, by adding a fingerprint table within the buckets of the scalable hash table to accelerate in-bucket queries and employing an optimistic read mechanism during data reading to avoid read-write tearing, the system's query efficiency and data reading accuracy are further improved. Finally, by introducing a log-based lazy migration strategy, full data redistribution is avoided during dynamic node changes, significantly reducing migration costs and improving the overall availability and scalability of the system.

[0056] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A microkernel-based distributed key-value storage system, characterized in that, include: The node routing module, deployed on the client, is used to build a sharding mapping relationship based on the load capacity of each storage node, and to locate data shards and target nodes based on the hash value of the key. The data storage module, deployed on the server-side storage node, is used to organize and store key-value data in units of shards; The network communication module is used to establish communication connections between the client and each storage node, and to transmit data based on a preset communication protocol; The data consensus module is deployed on the server-side storage node to maintain the consistency of multiple replicas of data within each shard and to perform data migration and master-slave switching when the node status changes dynamically.

2. The microkernel-based distributed key-value storage system according to claim 1, characterized in that, The method for constructing the fragmentation mapping relationship includes the following steps: Step S1: Obtain the configuration information of all active storage nodes in the cluster. The configuration information includes at least the node identifier, communication protocol version, IP address, port, and storage capacity information representing the node's load capacity. Step S2: Normalize the storage capacity of each storage node and calculate the weight Wi of each node; Step S3: Based on the preset N binary bits Each fragment identifier, based on the weight Calculate the number of shards that should be allocated to each node. The remaining unallocated fragments after calculation are allocated sequentially according to the fractional part of each node's weight in descending order, until all fragments are allocated. Step S4: Construct a sharding mapping table, wherein the sharding mapping table has a length equal to A fixed-length array, where each item in the array corresponds to a slice, and records the master node identifier and node running status corresponding to the slice; Step S5: Perform a hash calculation on the key to be operated on, take the high N bits of its hash value as the shard index, and locate the target storage node by querying the shard mapping table.

3. The microkernel-based distributed key-value storage system according to claim 1, characterized in that, The node routing module is also used to support dynamic fragmentation reallocation, including: In response to the addition of a new storage node to the system, the client obtains the updated node list, recalculates the weight of each node and the target number of shards, identifies the node that holds more shards than its target value and marks its excess shards as pending migration, allocates the shards to the new storage node and updates the shard mapping table, and marks the corresponding shards as being in the process of migration. After the data migration is completed, the migration status mark is removed. In response to a storage node voluntarily leaving the system, the server's data consensus module selects a new master node for each shard it manages and returns the update result to the client.

4. The microkernel-based distributed key-value storage system according to claim 1, characterized in that, The network communication module includes: The protocol verification unit is used to verify the protocol version and magic number of the handshake message when establishing a communication connection. An access credential allocation unit is used to allocate access credentials containing a read key and a write key to the client after the connection is established; The data transmission unit is used to provide a data interaction interface based on the access credentials to support write, read, update and delete operations of key-value data.

5. The microkernel-based distributed key-value storage system according to claim 1, characterized in that, The data storage module includes a sharded index layer and a sharded data layer, wherein: The sharding index layer is used to locate the target shard based on the hash value prefix of the key; The sharded data layer adopts an asymmetric storage structure, wherein the primary shard uses a storage method that combines a scalable hash table with a lightweight log, and the replica shard uses a full log structure for storage. The lightweight log of the primary shard is used to record data operation information and to achieve concurrent appending through atomic operations; The replica shard recovers data by replaying its full log.

6. The microkernel-based distributed key-value storage system according to claim 5, characterized in that, The data storage module processes write requests for the primary shard through the following steps: B1: Receive a write request initiated by the client, the write request containing a key, a value, and a globally incrementing version number; B2: Calculate the hash value of the key, locate the target shard based on its high N bits, and locate the target data bucket based on its low bits in the directory of the scalable hash table; B3: Acquire the bucket-level lock of the target data bucket through atomic comparison and swap operations; B4: While holding the bucket-level lock, perform a quick match using the fingerprint table within the bucket to check if the key exists and its version number. If the version number in the request is not greater than the current version number, release the lock and return; otherwise, if there is an empty slot in the bucket, perform an insertion or update operation and synchronously update the fingerprint at the corresponding position in the fingerprint table; if the bucket is full, trigger the bucket splitting process, and after the split is completed, reposition and perform an insertion or update operation. B5: After completing the scalable hash table update, the write request is encapsulated as a log entry and appended to the lightweight log corresponding to the shard through an atomic operation; B6: Release the bucket-level lock and return a write success response.

7. The microkernel-based distributed key-value storage system according to claim 1, characterized in that, The data consensus module includes: The group member management submodule is used to build a consistent group consisting of one master node and at least one slave node, and maintain the consistency of the group structure through member logs. The cross-node capability control submodule is used to allocate cross-node access capabilities to the secondary nodes within the consistency group. The cross-node access capabilities are used to control the access permissions of the secondary nodes to the log data of the primary node. The replica synchronization submodule is used to achieve data synchronization between the master node and the slave node through incremental log retrieval.

8. The microkernel-based distributed key-value storage system according to claim 7, characterized in that, The group member management submodule maintains the consistent group state through member logs, wherein: The member log records the term number, master node identifier, list of slave nodes, and change type; When inconsistencies are detected in the logs of members within a group, the logs of the master node shall be used for overwriting and synchronization. When a storage node joins or leaves, causing a change in group members, the group member management submodule updates the member logs according to different scenarios and executes the corresponding master-slave switchover and data migration process.

9. The microkernel-based distributed key-value storage system according to claim 7, characterized in that, The cross-node access capabilities generated by the cross-node capability control submodule include a shard identifier, a node identifier, and a signature field generated based on the master node's local key. When a secondary node accesses the resources of the primary node, it must carry the cross-node access capability. After receiving the request, the primary node performs a validity check on the cross-node access capability and allows access only after the check passes.

10. The microkernel-based distributed key-value storage system according to claim 7, characterized in that, The replica synchronization submodule achieves data synchronization through the following steps: E1: On the master node side, maintain a log cursor for each slave node to indicate the position of the lightweight log that the slave node has synchronized; E2: Receive log pull requests periodically initiated by the secondary node, the requests carrying the identification information of the last record in the local log and the corresponding cross-node access capability; E3: Perform a validity check on the cross-node access capability, and after the check passes, perform a consistency check on the identifier information and the log cursor. E4: If the consistency check passes, then traverse the subsequent lightweight log entries starting from the log cursor position. For write operations, query the latest value corresponding to the expandable hash table, and encapsulate the key, value, and operation type into a complete log entry; for delete operations, directly encapsulate it into a log entry. E5: Batch return a set of encapsulated log entries to the secondary node; E6: The secondary node receives log entries and appends them to the local full log; E7: If the consistency check fails, the secondary node will be triggered to perform a full resynchronization process.

Images