A storage and query method and system combining erasure coding and sharding technology

Abstract

A storage and query method combining erasure coding and sharding technology, including: sharding behavior, establishing a correspondence between nodes and shards, randomly dividing n nodes in the network into different shards, with c nodes in each shard. After the sharding, the system is divided into n / c shards. After the nodes are sharded, each node will save the mapping between the node and the shard ID; data storage behavior, a block is divided into (n-f) / c sub-blocks, and then, using Reed-Solomon encoding, the (n-f) / c sub-blocks are encoded into n / c coding blocks. Each node independently calculates the coding blocks it needs to store. Each node runs a consistent hashing algorithm locally, organizing the entire hash value space into a virtual ring. The coding blocks use the same function hashing algorithm to calculate hash values ​​as data keys, and determine the position of the coding blocks on the ring. From this position, the search is conducted clockwise along the ring. The first shard encountered is the shard where the data should be located. This solution divides the nodes in the system into n / c shards. A shard is a virtual group consisting of several nodes. Each n / c shard stores a coding block, which makes the total number of coding blocks smaller and thus reduces the computational complexity.

Description

Technical Field

[0001] The present invention relates to the field of blockchain, and in particular to a storage and query method and system combining erasure coding with sharding technology. Background Art

[0002] As a specialized shared database, blockchain can be widely used in finance, healthcare, the Internet of Things, information security, supply chains, and other scenarios. However, each node in the system must store a copy of the entire block to ensure data decentralization and consistency. This not only places a storage burden on nodes but also severely limits the system's storage scalability. Erasure codes are a commonly used data redundancy protection strategy in distributed storage systems. Therefore, research has begun to apply erasure codes to blockchains to reduce data storage redundancy. However, the introduction of erasure codes can reduce the data availability of blockchain systems. In a Byzantine environment, the assignment of code blocks to malicious nodes can lead to tampering or loss of code blocks. Furthermore, the distribution of code blocks across different nodes can cause problems such as data read delays.

[0003] Introducing multiple copies of code blocks can improve data reading efficiency. However, since both multiple copies and erasure codes are used for data redundancy, this can lead to excessive redundancy.

[0004] Despite extensive research efforts to address blockchain storage issues, current solutions still exhibit several limitations. Perard et al., Wilkinson et al., and Zhao et al. proposed blockchain storage frameworks based on erasure coding to reduce node storage resource requirements. However, it is difficult to estimate the number of system nodes to determine appropriate encoding parameters, and the availability of encoded blocks cannot be guaranteed.

[0005] Qi et al. proposed a sharded storage engine combining erasure coding and Byzantine fault tolerance: BFT-Store. This engine determines encoding parameters based on the Byzantine node ratio, ensuring data availability and eliminating the need for full replication in permissioned blockchain systems. Du et al. and Yin et al. inherited the assumptions of the Byzantine fault-tolerant protocol and further optimized the storage solution.

[0006] The above solution has the following defects:

[0007] Excessive redundant copies lead to high storage consumption, while too few copies affect data readability and availability. For example, BFT-Store uses both erasure coding and multiple replicas for data redundancy, resulting in excessive redundant copies. Some systems only store a single copy of the encoded block, increasing network latency for reading data and reducing blockchain availability.

[0008] The Chinese patent document with publication number CN109871366A discloses a blockchain sharding storage and query method based on erasure codes. Blocks are sharded and stored based on erasure codes. Under Byzantine fault tolerance (PBFT), consensus can be reached as long as no more than one-third of the nodes are malicious. So suppose there are n (n>3f) nodes in the blockchain system, of which f are malicious nodes. It is only necessary to divide block B into (nf) parts and f check blocks. Distribute these n blocks evenly to n nodes. Under Byzantine fault tolerance, it is assumed that in the worst case, there are f malicious nodes. Therefore, when reading the block, malicious blocks or unsent blocks will be received, and the total number is f. According to the characteristics of the erasure code, the correct block can be recovered from nf correct blocks. This method ensures that block data can be efficiently queried while saving storage space. However, this technical solution still has the following problems:

[0009] 1. High computational complexity. The computational complexity of erasure codes is affected by the total number of coding blocks. The computational complexity of the CN109871366A technical solution is high.

[0010] 2. Poor data availability. Although the CN109871366A technology takes Byzantine fault tolerance into account, it does not consider the possibility of honest nodes crashing, resulting in the loss of coded blocks, which reduces data availability under Byzantine fault tolerance. Since there are (nf) honest nodes in the system, if an honest node crashes, in the worst case, there are f malicious nodes, and it is impossible to recover (nf) coded blocks.

[0011] 3. Low read performance. CN109871366A technology has only one copy of the data per encoded block, resulting in low data readability. If the node storing block B locally is malicious, the node can only read block B through decoding recovery, which reduces read performance. The probability that a node can read data without decoding is 1-Pr(B) = f / n ≈ 1 / r (where r is (n-1) / f). Summary of the Invention

[0012] The technical problem to be solved by the present invention is how to reduce computational complexity and ensure data readability while achieving Byzantine fault tolerance.

[0013] The present invention solves the above technical problems through the following technical means: a storage and query method combining erasure coding and sharding technology, comprising the following steps:

[0014] Step S1, sharding behavior, specifically includes:

[0015] Step 1-1: Establish the corresponding relationship between nodes and shards. Randomly divide n nodes in the network into different shards, with c nodes in each shard. After sharding, the system is divided into n / c shards. The number of shards is expressed as Shard = {shard0, shard1, shard2, shardm} (m = n / c). The ID of each shard is composed of a hash of the public keys of the nodes in the shard. After the node is sharded, each node will save the mapping between the node and the shard ID, which is expressed as<ShardId,node0,node1,…,nodex> ;

[0016] Step S2: storing data, specifically including:

[0017] Step 2-1: Generate coded blocks from the block. After consensus is reached and the block is ready to be written to the node's local database, each node independently encodes the block locally. First, a block is split into (nf) / c sub-blocks. Then, RS encoding is used to encode the (nf) / c sub-blocks into n / c coded blocks.

[0018] Step 2-2: Each node independently calculates the encoded blocks it needs to store. Each node has the public keys of all other nodes and all shard IDs in the network. Each node runs the consistent hashing algorithm locally.

[0019] Step 2-3: The consistent hashing algorithm organizes the entire hash value space into a virtual ring. The value range of the entire hash value space is [0,2 h -1], h is the number of bytes of the shardId and the encoding block data key value, directly convert the shard ID to 2 h The modulo operation yields an integer. This integer is mapped onto the hash ring, and the integer value represents the position of a shard on the hash ring. The coded block uses the same hash function to calculate the hash value as the data key and determine the position of the coded block on the ring. From this position, the search proceeds clockwise along the ring. The first shard encountered is the shard where the data should be located.

[0020] Step 2-4: Determine which shard stores which encoding block;

[0021] Step S3: Read data to receive the client's request to read the local database.

[0022] As a further optimized technical solution, in step 1-1, the system's sharding method uses the VRF leader election algorithm to select the smallest public key node, and then runs the RandHound algorithm to randomly shard the nodes in the system, randomly dividing the n nodes in the network into different shards.

[0023] As a further optimized technical solution, in step 2-1, the coding block is locally encoded and represented as <chunk0,chunk1,…,chunk n / c >, in step 2-2, the hash algorithm uses chunkId hash as the data key and shardId as the storage device key.

[0024] As a further optimization solution, in steps 2-4, the node locally calculates each shard and obtains the corresponding stored encoding block, which is expressed as<ShardId,chunkId> , then, the nodes in the shard store the code block, according to the mapping relationship between shards and nodes<ShardId,node1,node2,…,nodex> , the corresponding shard of the node needs to store the hash-assigned encoding blocks.

[0025] As a further optimized technical solution, the step S3, reading data, specifically includes:

[0026] Step 3-1: The client sends a request to node A to read the coded block. If node A stores the coded block locally, node A queries the local database and returns the data.

[0027] Step 3-2: If the node does not store the encoding block locally, query the shard where the encoding block is located through chord routing;

[0028] Step 3-3: If node A does not receive return data from other shards, it initiates a broadcast decoding message. Nodes in other shards receive the decoding broadcast message from node A and return the encoded block. Node A decodes and recovers the data and returns it to the client.

[0029] As a further optimized technical solution, step 3-2 specifically includes: first determine whether the successor virtual shard node of the node holds the resource. If not, then search the node's routing table for the virtual shard node closest to the resource from far to near. Then, query all the nodes of the shard through the shard node table, and send the query message to these nodes in a group. Similarly, continuously query the closest virtual shard node and send group messages to all nodes in the shard. Finally, the query message is forwarded to the target node.

[0030] As a further optimization solution, the data query process in step 3-2 is optimized based on the chord greedy routing algorithm. The query algorithm includes the following steps:

[0031] Step 321. Node A checks whether the hash of the key falls between Node A and its immediate successor shard node. If so, the search is complete and the successor shard node of Node A is found. The query result is returned directly. Otherwise, the process proceeds to the next step.

[0032] In the routing table of node A in step 322, find the successor shard node of node A that is closest to hash(Key) and < hash(Key), and this virtual shard node is also the predecessor shard node closest to Key in the routing table. Query the shard node list, obtain all node IPs of this virtual shard node, and send query messages to these nodes in a group;

[0033] Repeat the query process in step 322 above until the shard storing the query metadata is located; [[ID=⑥]]

[0034] In step 324, if not found, do nothing, and the query node judges through the timeout mechanism. If found, the node storing the metadata and feedback relevant information to the query node, and node A verifies the encoded block hash value

[0035] The present invention also provides a storage and query system combining erasure code and sharding technology, which includes the following modules:

[0036] The shard division behavior module specifically includes the following units:

[0037] The unit for establishing the correspondence between nodes and shards is used to perform the following tasks: randomly divide n nodes in the network into different shards, with c nodes in each shard. After sharding, the system is divided into n / c shards, and the number of shards is expressed as Shard = {shard0, shard1, shard2, shardm} (m = n / c). The ID of each shard is composed of the combined hash of the public keys of the nodes in the shard. After the nodes are sharded, each node will save the mapping between the shard ID and the nodes, expressed as <ShardId, node0, node1,..., nodex>;

[0038] The data storage behavior module specifically includes the following units:

[0039] The local encoding unit is used to generate encoded blocks from the block. When the block is ready to be written into the local database of the node after consensus, each node independently encodes the block locally. First, a block is divided into (n - f) / c sub-blocks, and then, (n - f) / c sub-blocks are encoded into n / c encoded blocks using RS encoding;

[0040] The independent calculation unit is used for each node to independently calculate the encoded blocks it needs to store. Each node has the public keys of all other nodes and all shard IDs in the network, and each node runs the consistent hashing algorithm locally;

[0041] The location search unit, the consistent hashing algorithm organizes the entire hash value space into a virtual ring, and the value range of the entire hash value space is [0, 2 h-1], h is the number of bytes of the shardId and the encoding block data key value, directly convert the shard ID to 2 h The modulo operation yields an integer. This integer is mapped onto the hash ring, and the integer value represents the position of a shard on the hash ring. The coded block uses the same hash function to calculate the hash value as the data key and determine the position of the coded block on the ring. From this position, the search proceeds clockwise along the ring. The first shard encountered is the shard where the data should be located.

[0042] Storage unit, which determines which shard stores which encoding block;

[0043] The data reading behavior module is used to receive client requests to read the local database.

[0044] As a further optimized technical solution, in the unit of establishing the correspondence between nodes and shards, the system's sharding method uses the VRF leader election algorithm to select the node with the smallest public key, and then runs the RandHound algorithm to randomly shard the nodes in the system, randomly dividing n nodes in the network into different shards;

[0045] In the local coding unit, the coding block is locally encoded and represented as <chunk0,chunk1,…,chunk n / c >, in an independent computing unit, the hash algorithm uses the chunkId hash as the data key and the shardId as the storage device key;

[0046] In the storage unit, the node locally calculates each shard and obtains the corresponding stored encoding block, which is expressed as<ShardId,chunkId> , then, the nodes in the shard store the code block, according to the mapping relationship between shards and nodes<ShardId,node1,node2,…,nodex> , the corresponding shard of the node needs to store the hash-assigned encoding blocks.

[0047] As a further optimized technical solution, the data behavior reading module specifically includes the following units:

[0048] The request unit is used for the client to send a request to node A to read the code block. If node A stores the code block locally, node A queries the local database and returns the data.

[0049] Query unit, the query algorithm is as follows:

[0050] Step 321. Node A checks whether the hash of the key falls between Node A and its immediate successor shard node. If so, the search is complete and the successor shard node of Node A is found. The query result is returned directly. Otherwise, the process proceeds to the next step.

[0051] Step 322. In the routing table of node A, find the successor shard node of node A that is closest to hash(Key) and < hash(Key). This virtual shard node is also the predecessor shard node closest to Key in the routing table. Query the shard node list, obtain all node IPs of this virtual shard node, and send a query message to these nodes in a group.

[0052] Step 323. Repeat the query process in Step 322 above until the shard storing the queried metadata is located.

[0053] Step 324. If not found, do nothing. The query node judges through the timeout mechanism. If found, the node storing the metadata and the relevant information are fed back to the query node, and node A verifies the encoded block hash value.

[0054] Step 3-3: If node A does not receive the return data of other shards, it initiates a broadcast decoding message. The nodes of other shards return the encoded blocks when they receive the decoding broadcast message from node A. Node A decodes to recover the data and then returns it to the client.

[0055] The advantages of the present invention are as follows: Analyzing the application scenario of the current erasure code combined with blockchain storage, and combining with the actual requirements, a sharded storage optimization technology for ensuring data availability in the Byzantine environment is proposed. The main advantages are as follows:

[0056] A. Combining sharding technology to reduce computational complexity

[0057] The sharding in the prior art is the partitioning of blocks. Therefore, the sharding in the prior art is called an encoded block in this article. The sharding in this article refers to the sharding technology of the blockchain, which is a technology for forming virtual groups of nodes.

[0058] Initially, the original blockchain system was that nodes stored a complete block. Then, the erasure code was introduced in the prior art, encoding the complete block into n encoded blocks, and n nodes in the system each stored an encoded block. The advantage is that the encoded block is smaller than the complete block. Therefore, the prior art proposed that nodes only store one encoded block instead of a complete block to reduce storage consumption. Finally, this solution proposes to first divide the virtual group composed of nodes into n / c (c is a constant) shards, encode the complete block into n / c encoded blocks, and each of the n / c shards stores an encoded block. [[ID=,23]]

[0059] The technical solution of the present invention reduces the total number of encoded blocks from n in the prior art to n / c in this solution. The computational complexity of the erasure code is affected by the total number of encoded blocks, so the encoding computational complexity decreases.

[0060] B. Data availability.

[0061] Consider Byzantine fault tolerance and the possibility of honest node crashes. Assuming a worst-case scenario with f malicious nodes and f honest nodes crashing, (nf) / c coded blocks can still be collected from n coded blocks held by nodes in different shards. Due to the characteristics of erasure codes, the block can be recovered by recovering (nf) / c coded blocks. Therefore, this solution ensures data availability under Byzantine fault tolerance.

[0062] C. Data readability.

[0063] If the node that locally stores a block blockB is malicious, then a node can only read blockB by decoding and recovering, which reduces the reading performance. In patent document CN109871366A, the probability that a node can read data without decoding is 1-Pr(B)=f / n≈1 / r (where r is (n-1) / f). In this solution, each shard consists of c nodes, and the nodes in the same shard hold the same coding block. Therefore, a coding block is stored by c nodes, and a coding block has c data copies. Under the replica redundancy strategy, the probability that a node can read data without decoding is Assuming c=3, r=3, Pr(B) is greater than 1-1 / 27=26 / 27≈96%.

[0064] (2) A coding block allocation method based on hash value positioning is designed to ensure the randomness of coding block allocation.

[0065] (3) A coding block reading strategy was designed to improve data search efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 Schematic diagram of shard storage in an embodiment of the present invention;

[0067] Figure 2 Schematic diagram of coding block reading in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0069] Example 1

[0070] See Figure 1 As shown, the storage and query method combining erasure coding and sharding technology of the present invention includes the following steps:

[0071] Step S1, sharding behavior, specifically includes:

[0072] Step 1-1: Establish the corresponding relationship between nodes and shards. The system's sharding method uses the VRF leader election algorithm to select the smallest public key node, and then runs the RandHound algorithm to randomly shard the nodes in the system. The n nodes in the network are randomly divided into different shards, with c nodes in each shard. After sharding, the system is divided into n / c shards. The number of shards is expressed as Shard = {shard0, shard1, shard2, shardm} (m = n / c). The ID of each shard is composed of a hash of the public keys of the nodes in the shard. After the node is sharded, each node will save the mapping between the node and the shard ID, which is expressed as<ShardId,node0,node1,…,nodex> .

[0073] Step S2: storing data, specifically including:

[0074] Step 2-1: Generate a coded block from the block. After consensus is reached and the block is ready to be written to the node's local database, each node encodes the block independently locally. First, a block is divided into (nf) / c sub-blocks. Then, RS encoding is used to encode the (nf) / c sub-blocks into n / c coded blocks. The coded blocks are locally encoded and represented as <chunk0,chunk1,…,chunk n / c >

[0075] Step 2-2: Each node independently calculates the encoded blocks it needs to store. Since all nodes in the system are sorted by their public keys, the ordered sequence of public key addresses is consistent among all honest nodes. Therefore, each node has the public keys of all other nodes and all shard IDs in the network. Each node runs a consistent hashing algorithm locally, using the chunkId hash as the data key and the shardId as the storage device key.

[0076] Step 2-3: The consistent hashing algorithm organizes the entire hash value space into a virtual ring. The value range of the entire hash value space is [0,2 h -1], h is the number of bytes of the shardId and the encoding block data key value. Since the shard ID is the combined hash of all node IDs in the shard, the shard ID is directly converted to 2 h Modulo, the result must be a range from 0 to 2 hAn integer between -1, map this integer on the hash ring, and the value of the integer represents the position of a shard on the hash ring. The encoding block uses the same function hash algorithm to calculate the hash value as the data key, and determines the position of the encoding block on the ring. Starting from this position and looking clockwise along the ring, the first encountered shard is the shard to which the data should be located;

[0077] Step 2-4: In the above steps, determine which shard stores which encoding block. The nodes locally calculate the encoding blocks corresponding to each shard, expressed as <ShardId, chunkId>. Then, the nodes within the shard store the encoding block. According to the mapping relationship between the shard and the nodes <ShardId, node1, node2, …, nodex>, the corresponding shards of the nodes need to store the encoding blocks allocated by the hash;

[0078] Step S3, the data reading operation, refer to Figure 2 , the data reading operation includes the following steps:

[0079] Step 3-1: The client sends a request to read the encoding block to Node A. If Node A locally stores the encoding block, Node A queries the local database and returns the data;

[0080] Step 3-2: When the node does not locally store the encoding block, query the shard where the encoding block is located through the chord routing. First, determine whether the successor virtual shard node of this node holds the resource. If not, then find the virtual shard node closest to the resource held from far to near in the node's routing table. Then, query all the nodes of this shard through the shard node table, and group and send the query message to these nodes. And so on, continuously query the closest virtual shard node and group and send the message to all the nodes within this shard. Finally, the query message is forwarded to the target node. The above data query process is optimized according to the chord greedy routing algorithm. The optimized query algorithm is as follows:

[0081] Step 321. Node A checks whether the hash of the Key falls between Node A and its direct successor shard node. If so, end the search, and the successor shard node of Node A is the one found, and directly return the query result. Otherwise, go to the next step;

[0082] Step 322. In the routing table of Node A, find the successor shard node of Node A that is closest to hash(Key) and < hash(Key). This virtual shard node is also the predecessor shard node closest to Key in the routing table. Query the shard node list to obtain the IPs of all the nodes of this virtual shard node, and group and send the query message to these nodes;

[0083] Step 323: Repeat the query process in step 322 until the shard storing the query metadata is located.

[0084] Step 324. If not found, no action is taken and the query node determines the hash value of the coded block using a timeout mechanism. If found, the node storing the metadata feeds the relevant information back to the query node, and node A verifies the hash value of the coded block.

[0085] Step 3-3: If node A does not receive return data from other shards, it initiates a broadcast decoding message. Nodes in other shards receive the decoding broadcast message from node A and return the encoded block. Node A decodes and recovers the data and returns it to the client.

[0086] This scheme divides the nodes in the system into n / c shards. A shard is a virtual group of several nodes. Nodes in the same shard hold the same coding block. Assuming that each shard consists of c nodes, the encoding process is adjusted to divide the block into (nf) / c parts and f / c parts of parity blocks. The n / c coding blocks are then sent to different shards. This makes the total number of coding blocks smaller and thus reduces the computational complexity.

[0087] Example 2

[0088] This embodiment provides a storage and query system that combines erasure coding with sharding technology, including:

[0089] 1. Sharding behavior module, specifically including the following units:

[0090] Establish a corresponding relationship unit between nodes and shards to perform the following tasks: The system's sharding method uses the VRF leader election algorithm to select the smallest public key node, and then runs the RandHound algorithm to randomly shard the nodes in the system. The n nodes in the network are randomly divided into different shards, with c nodes in each shard. After sharding, the system is divided into n / c shards. The number of shards is expressed as Shard = {shard0, shard1, shard2, shardm} (m = n / c). The ID of each shard is composed of a hash of the public keys of the nodes in the shard. After the node is sharded, each node will save the mapping between the node and the shard ID, which is expressed as<ShardId,node0,node1,…,nodex> .

[0091] 2. Storage data behavior module, specifically including the following units:

[0092] The local encoding unit is used to generate a coded block from a block. When a block is ready to be written to the node local database after consensus, each node encodes the block independently locally. First, a block is divided into (nf) / c sub-blocks. Then, the (nf) / c sub-blocks are encoded into n / c coded blocks using RS encoding. The coded blocks are represented as <chunk0,chunk1,…,chunk n / c >

[0093] Independent computing units are used for each node to independently calculate the code blocks it needs to store. Since all nodes in the system are sorted by their public keys, the ordered sequence of public key addresses is consistent among all honest nodes. Therefore, each node has the public keys of all other nodes and all shard IDs in the network. Each node runs a consistent hashing algorithm locally, using the chunkId hash as the data key and the shardId as the storage device key.

[0094] Position search unit, the consistent hashing algorithm organizes the entire hash value space into a virtual ring, and the value range of the entire hash value space is [0,2 h -1], h is the number of bytes of the shardId and the encoding block data key value. Since the shard ID is the combined hash of all node IDs in the shard, the shard ID is directly converted to 2 h Modulo, the result must be a range from 0 to 2 h -1, and maps this integer to the hash ring. The integer value represents the position of a shard on the hash ring. The encoding block uses the same function hash algorithm to calculate the hash value as the data key and determine the position of the encoding block on the ring. From this position, the search is carried out clockwise along the ring. The first shard encountered is the shard where the data should be located.

[0095] The storage unit determines which shard stores which coding block through the tasks performed by the above units. The node locally calculates the coding block stored corresponding to each shard, which is expressed as<ShardId,chunkId> Then, the nodes in the shard store the code block according to the mapping relationship between shards and nodes.<ShardId,node1,node2,…,nodex> , the corresponding shard where the node is located needs to store the hash-assigned encoding blocks;

[0096] 3. Read data behavior module, specifically including the following units:

[0097] The request unit is used for the client to send a request to node A to read the code block. If node A stores the code block locally, node A queries the local database and returns the data.

[0098] Query unit. When the encoding block is not stored locally at the node, the shard where the encoding block is located is queried through the Chord routing. First, it is judged whether the successor virtual shard node of the node holds the resource. If not, the virtual shard node closest to the resource holder is searched in the node's routing table from far to near. Then, all nodes of the shard are queried through the shard node table, and the query message is sent to these nodes in a group. And so on, continuously query the closest virtual shard node and send messages to all nodes in the shard. Finally, the query message is forwarded to the target node. The above data query process is optimized according to the Chord greedy routing algorithm. The optimized query algorithm is as follows:

[0099] Step 321. Node A checks whether the hash of the Key falls between Node A and its direct successor shard node. If so, the search ends, and the successor shard node of Node A is the one sought, and the query result is directly returned. Otherwise, proceed to the next step;

[0100] Step 322. In the routing table of Node A, find the successor shard node of Node A that is closest to hash(Key) and < hash(Key). This virtual shard node is also the predecessor shard node closest to Key in the routing table. Query the shard node list to obtain the IPs of all nodes of this virtual shard node, and send the query message to these nodes in a group;

[0101] Step 323. Repeat the query process in Step 322 above until the shard storing the query metadata is located;

[0102] Step 324. If not found, do nothing. The query node makes a judgment through the timeout mechanism. If found, the node storing the metadata and the relevant information are fed back to the query node, and Node A verifies the encoding block hash value;

[0103] Step 3-3: If Node A does not receive the return data from other shards, a broadcast decoding message is initiated. The nodes of other shards return the encoding block when they receive the decoding broadcast message from Node A, and Node A decodes and restores the data and then returns it to the client.

[0104] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A storage and query method combining erasure coding and sharding technology, characterized in that: The steps include: Step S1, sharding behavior, specifically includes: Step 1-1: Establish the corresponding relationship between nodes and shards. Randomly divide n nodes in the network into different shards, with c nodes in each shard. After sharding, the system is divided into n / c shards. The number of shards is expressed as Shard={shard0, shard1, shard2, shardm}, m=n / c. The ID of each shard is composed of a hash of the public keys of the nodes in the shard. After the node is sharded, each node will save the mapping between the node and the shard ID, which is expressed as<ShardId,node0,node1,…,nodex> ; Step S2: storing data, specifically including: Step 2-1: Generate coded blocks from the block. After consensus is reached and the block is ready to be written to the node's local database, each node independently encodes the block locally. First, a block is split into (nf) / c sub-blocks. Then, RS encoding is used to encode the (nf) / c sub-blocks into n / c coded blocks. Step 2-2: Each node independently calculates the encoded blocks it needs to store. Each node has the public keys of all other nodes and all shard IDs in the network. Each node runs the consistent hashing algorithm locally. Step 2-3: The consistent hashing algorithm organizes the entire hash value space into a virtual ring. The value range of the entire hash value space is [0,2 h -1], h is the number of bytes of the shardId and the encoding block data key value, directly convert the shard ID to 2 h The modulo operation yields an integer. This integer is mapped onto the hash ring, and the integer value represents the position of a shard on the hash ring. The coded block uses the same hash function to calculate the hash value as the data key and determine the position of the coded block on the ring. From this position, the search proceeds clockwise along the ring. The first shard encountered is the shard where the data should be located. Step 2-4: Determine which shard stores which encoding block; Step S3: Read data to receive the client's request to read the local database.

2. The storage and query method combining erasure coding and sharding technology according to claim 1, characterized in that: In step 1-1, the system's sharding method uses the VRF leader election algorithm to select the node with the smallest public key, and then runs the RandHound algorithm to randomly shard the nodes in the system, randomly dividing the n nodes in the network into different shards.

3. The storage and query method combining erasure coding and sharding technology according to claim 1, characterized in that: In step 2-1, the coded block is locally encoded and represented as <chunk0,chunk1,…,chunk n / c >, in step 2-2, the hash algorithm uses chunkId hash as the data key and shardId as the storage device key.

4. The storage and query method combining erasure coding and sharding technology according to claim 3, characterized in that: In steps 2-4, the node locally calculates each shard and obtains the corresponding stored encoding block, which is expressed as<ShardId,chunkId> , then, the nodes in the shard store the code block, according to the mapping relationship between shards and nodes<ShardId,node1,node2,…,nodex> , the corresponding shard of the node needs to store the hash-assigned encoding blocks.

5. The storage and query method combining erasure coding and sharding technology according to claim 1, characterized in that: The step S3, reading data, specifically includes: Step 3-1: The client sends a request to node A to read the coded block. If node A stores the coded block locally, node A queries the local database and returns the data. Step 3-2: If the node does not store the encoding block locally, query the shard where the encoding block is located through chord routing; Step 3-3: If node A does not receive return data from other shards, it initiates a broadcast decoding message. Nodes in other shards receive the decoding broadcast message from node A and return the encoded block. Node A decodes and recovers the data and returns it to the client.

6. The storage and query method combining erasure coding and sharding technology according to claim 5, characterized in that: Step 3-2 specifically includes: First, determine whether the successor virtual shard node of this node holds this resource. If not, then search in the node's routing table from far to near to find the virtual shard node closest to the resource holder. Then, query all the nodes of this shard through the shard node table, and broadcast the query message to these nodes. And so on, continuously find the closest virtual shard node and broadcast the message to all the nodes within this shard. Finally, the query message is forwarded to the target node.

7. The storage and query method combining erasure coding and sharding technology according to claim 6, characterized in that: The data query process in Step 3-2 is optimized according to the chord greedy routing algorithm. The query algorithm includes the following steps: Step 321. Node A checks whether the hash of Key falls between Node A and its direct successor shard node. If so, end the search, and the successor shard node of Node A is the one sought, and directly return the query result. Otherwise, proceed to the next step; Step 322. In the routing table of Node A, find the successor shard node of Node A that is closest to hash(Key) and < hash(Key). This virtual shard node is also the predecessor shard node closest to Key in the routing table. Query the shard node list to obtain the IPs of all the nodes of this virtual shard node, and broadcast the query message to these nodes; Step 323. Repeat the query process in Step 322 above until the shard storing this query metadata is located; Step 324. If not found, do nothing, and the query node makes a judgment through the timeout mechanism. If found, the node storing this metadata and feedbacks the relevant information to the query node, and Node A verifies the hash value of the encoded block.

8. A storage and query system combining erasure coding and sharding technology, characterized in that: It includes the following modules: The shard division behavior module specifically includes the following units: The unit for establishing the correspondence between nodes and shards is used to perform the following tasks: Randomly divide n nodes in the network into different shards, with c nodes in each shard. After sharding, the system is divided into n / c shards. The number of shards is represented as Shard={shard0, shard1, shard2, shardm}, where m = n / c. The ID of each shard is composed of the combined hash of the public keys of the nodes within the shard. After the nodes are sharded, each node will save the mapping between the node and the shard ID, represented as <ShardId, node0, node1, …, nodex>; The data storage behavior module specifically includes the following units: The local encoding unit is used to generate encoded blocks from the block. When the block is ready to be written into the local database of the node after consensus, each node independently encodes the block locally. First, a block is divided into (n - f) / c sub-blocks, and then, (n - f) / c sub-blocks are encoded into n / c encoded blocks using RS encoding; The independent calculation unit is used for each node to independently calculate the encoded blocks it needs to store. Each node has the public keys of all other nodes and all the shard IDs in the network. Each node runs the consistent hashing algorithm locally; Position search unit, the consistent hashing algorithm organizes the entire hash value space into a virtual ring, and the value range of the entire hash value space is [0,2 h -1], h is the number of bytes of the shardId and the encoding block data key value, directly convert the shard ID to 2 h The modulo operation yields an integer. This integer is mapped onto the hash ring, and the integer value represents the position of a shard on the hash ring. The coded block uses the same hash function to calculate the hash value as the data key and determine the position of the coded block on the ring. From this position, the search proceeds clockwise along the ring. The first shard encountered is the shard where the data should be located. The storage unit determines which shard stores which encoded block; The data reading behavior module is used to receive the request from the client to read the local database.

9. A storage and query system combining erasure coding and sharding technology as claimed in claim 8, characterized in that: In the unit for establishing the correspondence between nodes and shards, the sharding method of the system uses the leader election algorithm of VRF to select the smallest public key node, and then runs the RandHound algorithm to randomly shard the nodes in the system, randomly dividing the n nodes in the network into different shards; In the local coding unit, the coding block is locally encoded and represented as <chunk0,chunk1,…,chunk n / c >, in an independent computing unit, the hash algorithm uses the chunkId hash as the data key and the shardId as the storage device key; In the storage unit, each node locally calculates that each shard gets the corresponding stored coding block, denoted as <ShardId,chunkId>. Then, the nodes within the shard store this coding block. According to the shard-node mapping relationship <ShardId,node1,node2,…,nodex>, the corresponding shards of the nodes need to store the coding blocks assigned by the hash.

10. A storage and query system combining erasure coding and sharding technology as claimed in claim 8, characterized in that: The data reading behavior module specifically includes the following units: The request unit is used for the client to send a request to node A to read the coding block. If node A locally stores this coding block, node A queries the local database and returns the data; The query unit, and the query algorithm is as follows: Step 321. Node A checks whether the hash of the Key falls between node A and its direct successor shard node. If so, the search ends, and the successor shard node of node A is the one found, and the query result is directly returned. Otherwise, proceed to the next step; Step 322. In the routing table of node A, find the successor shard node of node A that is closest to and < hash(Key) to the hash(Key). This virtual shard node is also the predecessor shard node closest to Key in the routing table. Query the shard node list to obtain the IPs of all nodes of this virtual shard node, and send query messages to these nodes in a group; Step 323. Repeat the query process in step 322 above until the shard storing the query metadata is located; Step 324. If not found, do nothing. The query node judges through the timeout mechanism. If found, the node storing this metadata and feedbacks the relevant information to the query node, and node A verifies the hash value of the coding block; Step 3-3: If node A does not receive the return data from other shards, it发起广播解码消息 (initiates a broadcast decoding message). The nodes of other shards return the coding block when they receive the decoding broadcast message from node A. Node A decodes and restores the data and then returns it to the client.

Images