Erasure code and fragmentation technology combined storage and query method and system

Abstract

The invention discloses a storage and query method combining erasure codes and a fragmentation technology. The storage and query method comprises the steps that fragmentation is divided, the corresponding relation between nodes and fragments is established, n nodes in a network are randomly divided into different fragments, each fragment has c nodes, a system is divided into n / c fragments after fragmentation, and after the nodes are fragmented, each node can store mapping of the nodes and fragment IDs; according to the data storage behavior, a block is divided into (n-f) / c sub-blocks, then RS coding is used for coding the (n-f) / c sub-blocks into n / c coding blocks, each node independently calculates the coding block needing to be stored, the nodes locally run a consistent Hash algorithm, the whole Hash value space is organized into a virtual circular ring, and the virtual circular ring is used for storing the data. The coding block uses the same function hash algorithm to calculate a hash value as a data key, the position of the coding block on the ring is determined, clockwise searching is carried out along the ring from the position, and the first encountered fragment is the fragment to which the data should be positioned. According to the scheme, the nodes in the system are divided into n / c fragments, each fragment is a virtual group composed of a plurality of nodes, and one coding block is stored in each of the n / c fragments, so that the total number of the coding blocks is smaller, and the calculation complexity is reduced accordingly.

Description

Technical Field

[0001] The present invention relates to the field of blockchain, and in particular to a storage and query method and system combining erasure code with sharding technology. Background Art

[0002] As a special shared database, blockchain can be widely used in finance, medical care, Internet of Things, information security, supply chain and other scenarios. However, each node in the system needs to store a copy of the entire block to ensure the decentralization and consistency of the data, which not only causes the storage burden of the node, but also seriously affects the storage scalability of the system. Erasure codes are currently a commonly used data redundancy protection strategy in distributed storage systems. Therefore, some studies have begun to apply erasure codes to blockchains to reduce data storage redundancy. However, the introduction of erasure codes will lead to a decrease in the data availability of blockchain systems, because in the Byzantine environment of blockchain systems, the coding blocks are assigned to malicious nodes, which will cause the coding blocks to be tampered with or lost. In addition, the distribution of coding blocks on different nodes will lead to problems such as data reading delays.

[0003] Introducing multiple copies of coding blocks can improve data reading efficiency. However, since both multiple copies and erasure codes are used for data redundancy, this will cause excessive redundancy.

[0004] Although the research community has made a lot of efforts to solve the blockchain storage problem, the current solutions still show one or more limitations. Perard et al., Wilkinson et al., and Zhao et al. proposed a blockchain storage framework based on erasure coding technology to reduce the node's demand for storage resources. However, it is impossible to estimate the number of system nodes to determine a suitable encoding parameter for encoding, and the availability of the encoded block cannot be guaranteed.

[0005] Qi et al. proposed a block sharding storage engine that combines erasure coding and Byzantine: BFT-Store. It determines the encoding parameters based on the proportion of Byzantine nodes to ensure data availability and break the full replication of the permissioned chain system. Du et al. and Yin et al. inherited the assumption of the Byzantine fault-tolerant protocol and further optimized the storage solution.

[0006] The above solution has the following defects:

[0007] Too many redundant copies lead to high storage consumption, or too few redundant copies affect data readability and availability. For example, BFT-Store introduces both erasure coding and multiple copies of data redundancy solutions, resulting in too many redundant copies. Some systems only store one copy of the coded block, which increases the network latency of reading data and reduces the availability of the blockchain.

[0008] The Chinese patent document with publication number CN109871366A discloses a blockchain sharding storage and query method based on erasure codes. Blocks are stored in shards based on erasure codes. Under Byzantine fault tolerance PBFT, consensus can be reached as long as no more than one-third of the malicious nodes are guaranteed. So suppose there are n (n>3f) nodes in the blockchain system, of which f are malicious nodes. It is only necessary to divide block B into (nf) parts and f check blocks. Distribute these n blocks evenly to n nodes. Under Byzantine fault tolerance. Assume that in the worst case, there are f malicious nodes. Therefore, when reading blocks, malicious blocks or unsent blocks will be received, and the total number is f. According to the characteristics of erasure codes. The correct block can be recovered from nf correct blocks. This method ensures that block data can be efficiently queried while saving storage space. However, the technical solution still has the following problems:

[0009] 1. High computational complexity. The computational complexity of erasure codes is affected by the total number of coding blocks, and the computational complexity of the CN109871366A technical solution is high.

[0010] 2. Poor data availability. Although the CN109871366A technology takes Byzantine fault tolerance into consideration, it does not take into account that honest nodes may crash, resulting in the loss of coded blocks, which reduces the availability of data under Byzantine fault tolerance. Since there are (nf) honest nodes in the system, if an honest node crashes, and there are f malicious nodes in the worst case, it is impossible to recover (nf) coded blocks and restore the blocks.

[0011] 3. Low reading performance. CN109871366A technology has only one copy of the data for each encoding block, and the data readability is low. If the node that locally stores a block blockB is malicious, a node can only read blockB by decoding and recovering, which reduces the reading performance. The probability that a node can read data without decoding is 1-Pr(B)=f / n≈1 / r (where r is (n-1) / f). Summary of the invention

[0012] The technical problem to be solved by the present invention is how to reduce the computational complexity and ensure data readability while achieving Byzantine fault tolerance.

[0013] The present invention solves the above technical problems through the following technical means: a storage and query method combining erasure coding and sharding technology, comprising the following steps:

[0014] Step S1, sharding behavior, specifically includes:

[0015] Step 1-1: Establish the correspondence between nodes and shards. Randomly divide n nodes in the network into different shards, with c nodes in each shard. After sharding, the system is divided into n / c shards. The number of shards is expressed as Shard = {shard0, shard1, shard2, shardm} (m = n / c). The ID of each shard is composed of the hash of the public key combination of the nodes in the shard. After the node is sharded, each node will save the mapping between the node and the shard ID, expressed as<ShardId,node0,node1,…,nodex> ;

[0016] Step S2: storing data, specifically including:

[0017] Step 2-1: Generate a coded block from the block. When the block is ready to be written into the node local database after consensus, each node encodes the block independently locally. First, a block is divided into (nf) / c sub-blocks, and then, the (nf) / c sub-blocks are encoded into n / c coded blocks using RS encoding;

[0018] Step 2-2: Each node independently calculates the encoding blocks it needs to store. Each node has the public keys of all other nodes and all shard IDs in the network. Each node runs the consistent hashing algorithm locally.

[0019] Step 2-3: The consistent hashing algorithm organizes the entire hash value space into a virtual ring. The value range of the entire hash value space is [0,2 h -1], h is the number of bytes of the shardId and the encoding block data key value, directly convert the shard ID to 2 h The modulo result is an integer. This integer is mapped on the hash ring. The value of the integer represents the position of a shard on the hash ring. The coding block uses the same function hash algorithm to calculate the hash value as the data key and determine the position of the coding block on the ring. From this position, search along the ring clockwise. The first shard encountered is the shard where the data should be located.

[0020] Step 2-4: Determine which shard stores which encoding block;

[0021] Step S3: Read data to receive the client's request to read the local database.

[0022] As a further optimized technical solution, in step 1-1, the system's sharding method uses the VRF leader election algorithm to select the smallest public key node, and then runs the RandHound algorithm to randomly shard the nodes in the system, randomly dividing the n nodes in the network into different shards.

[0023] As a further optimized technical solution, in step 2-1, the coding block is locally encoded and represented as <chunk0,chunk1,…,chunk n / c >, in step 2-2, the hash algorithm uses chunkId hash as the data key and shardId as the storage device key.

[0024] As a further optimization technical solution, in step 2-4, the node locally calculates each shard and obtains the corresponding stored encoding block, which is expressed as<ShardId,chunkId> Then, the nodes in the shard store the coded block according to the mapping relationship between shards and nodes.<ShardId,node1,node2,…,nodex> , the corresponding shard of the node needs to store the hash-assigned encoding blocks.

[0025] As a further optimized technical solution, the step S3, the data reading behavior specifically includes:

[0026] Step 3-1: The client sends a request to node A to read the coded block. If node A stores the coded block locally, node A queries the local database and returns the data.

[0027] Step 3-2: If the node does not store the encoding block locally, query the shard where the encoding block is located through the chord routing;

[0028] Step 3-3: If node A does not receive return data from other shards, it initiates a broadcast decoding message. Nodes in other shards receive the decoding broadcast message from node A and return the encoded block. Node A decodes and recovers the data and returns it to the client.

[0029] As a further optimized technical solution, step 3-2 specifically includes: first determine whether the successor virtual shard node of the node holds the resource. If not, then search the node's routing table for the virtual shard node closest to the resource from far to near. Then, query all the nodes of the shard through the shard node table, and send the query message to these nodes in a group. Similarly, continuously query the closest virtual shard node and send the message to all nodes in the shard. Finally, the query message is forwarded to the target node.

[0030] As a further optimized technical solution, the data query process in step 3-2 is optimized according to the chord greedy routing algorithm. The query algorithm includes the following steps:

[0031] Step 321. Node A checks whether the hash of the key falls between node A and its direct successor shard node. If so, the search is completed and the successor shard node of node A is found. The query result is directly returned. Otherwise, it goes to the next step.

[0032] In the routing table of node A in step 322, find the successor shard node of node A that is closest to hash(Key) and < hash(Key), and this virtual shard node is also the predecessor shard node closest to Key in the routing table. Query the shard node list, obtain the IPs of all nodes of this virtual shard node, and send query messages to these nodes in a group.

[0033] Repeat the query process in step 322 above until the shard storing the query metadata is located.

[0034] If not found, do nothing. The query node judges through the timeout mechanism. If found, the node storing the metadata and feedback relevant information to the query node, and node A verifies the encoded block hash value.

[0035] The present invention also provides a storage and query system combining erasure code and sharding technology, including the following modules:

[0036] The shard division behavior module specifically includes the following units:

[0037] The unit for establishing the correspondence between nodes and shards is used to perform the following tasks: randomly divide n nodes in the network into different shards, with c nodes in each shard. After sharding, the system is divided into n / c shards, and the number of shards is expressed as Shard = {shard0, shard1, shard2, shardm} (m = n / c). The ID of each shard is composed of the combined hash of the public keys of the nodes in the shard. After the nodes are sharded, each node will save the mapping between the shard ID and the nodes, expressed as <ShardId, node0, node1,..., nodex>.

[0038] The data storage behavior module specifically includes the following units:

[0039] The local encoding unit is used to generate encoded blocks for the block. When the block is ready to be written into the local database of the node after consensus, each node independently encodes the block locally. First, a block is divided into (n - f) / c sub-blocks, and then, (n - f) / c sub-blocks are encoded into n / c encoded blocks using RS encoding.

[0040] The independent calculation unit is used for each node to independently calculate the encoded blocks it needs to store. Each node has the public keys of all other nodes and all shard IDs in the network, and each node runs the consistent hashing algorithm locally.

[0041] The location search unit, the consistent hashing algorithm organizes the entire hash value space into a virtual ring, and the value range of the entire hash value space is [0, 2 h-1], h is the number of bytes of the shardId and the encoding block data key value, directly convert the shard ID to 2 h The modulo result is an integer. This integer is mapped on the hash ring. The value of the integer represents the position of a shard on the hash ring. The coding block uses the same function hash algorithm to calculate the hash value as the data key and determine the position of the coding block on the ring. From this position, search along the ring clockwise. The first shard encountered is the shard where the data should be located.

[0042] Storage unit, determines which shard stores which encoding block;

[0043] The data reading behavior module is used to receive client requests to read the local database.

[0044] As a further optimized technical solution, in the corresponding relationship unit between nodes and shards, the system's sharding method uses the VRF leader election algorithm to select the smallest public key node, and then runs the RandHound algorithm to randomly shard the nodes in the system, and randomly divides the n nodes in the network into different shards;

[0045] In the local coding unit, the coding block is locally encoded and represented as <chunk0,chunk1,…,chunk n / c >, in the independent computing unit, the hash algorithm uses the chunkId hash as the data key and the shardId as the storage device key;

[0046] In the storage unit, the node locally calculates each shard and obtains the corresponding stored encoding block, which is expressed as<ShardId,chunkId> Then, the nodes in the shard store the coded block according to the mapping relationship between shards and nodes.<ShardId,node1,node2,…,nodex> , the corresponding shard of the node needs to store the hash-assigned encoding blocks.

[0047] As a further optimized technical solution, the data reading behavior module specifically includes the following units:

[0048] A request unit, used for the client to send a request to node A to read the coded block. If node A stores the coded block locally, node A queries the local database and returns the data.

[0049] Query unit, the query algorithm is as follows:

[0050] Step 321. Node A checks whether the hash of the key falls between node A and its direct successor shard node. If so, the search is completed and the successor shard node of node A is found. The query result is directly returned. Otherwise, it goes to the next step.

[0051] Step 322. In the routing table of node A, find the successor shard node of node A that is closest to hash(Key) and < hash(Key). This virtual shard node is also the predecessor shard node closest to Key in the routing table. Query the shard node list, obtain all the node IPs of this virtual shard node, and send a query message to these nodes in a group.

[0052] Step 323. Repeat the query process in the above Step 322 until the shard storing the queried metadata is located.

[0053] Step 324. If not found, do nothing. The query node makes a judgment through the timeout mechanism. If found, the node storing the metadata feeds back the relevant information to the query node, and node A verifies the encoded block hash value.

[0054] Step 3-3: If node A does not receive the return data of other shards, it initiates a broadcast decoding message. The nodes of other shards return the encoded blocks when they receive the decoding broadcast message from node A. Node A decodes and restores the data and then returns it to the client.

[0055] The advantages of the present invention are as follows: Analyzing the application scenario of the current erasure code combined with blockchain storage, and in combination with the actual requirements, a shard storage optimization technology for ensuring data availability in the Byzantine environment is proposed. The main advantages are as follows:

[0056] A. Combining shard technology to reduce computational complexity

[0057] The shards in the prior art are the block divisions of the blockchain. Therefore, the shards in the prior art are called encoded blocks in this article. The shards in this article refer to the shard technology of the blockchain, which is a technology for forming virtual groups of nodes.

[0058] Initially, the original blockchain system had nodes storing a complete block. Then, erasure code was introduced in the prior art, encoding the complete block into n encoded blocks, and n nodes in the system each stored one encoded block. The advantage is that the encoded block is smaller than the complete block. Therefore, the prior art proposed that nodes only store one encoded block instead of a complete block to reduce storage consumption. Finally, this solution proposes to first divide the virtual group composed of nodes into n / c (c is a constant) shards, encode the complete block into n / c encoded blocks, and each of the n / c shards stores one encoded block.

[0059] The technical solution of the present invention reduces the total number of encoded blocks from n in the prior art to n / c in this solution. Since the computational complexity of the erasure code is affected by the total number of encoded blocks, the encoding computational complexity decreases.

[0060] B. Data availability.

[0061] Consider Byzantine fault tolerance and the possibility of honest nodes crashing. Assuming the worst case, there are f malicious nodes and f honest nodes crash, (nf) / c coded blocks can still be collected from n coded blocks held by nodes in different shards. According to the characteristics of erasure codes, the block can be recovered after (nf) / c coded blocks are recovered. Therefore, this solution ensures the availability of data under Byzantine fault tolerance.

[0062] C. Data readability.

[0063] If the node that locally stores a block blockB is malicious, then a node can only read blockB by decoding and recovering, which reduces the reading performance. In patent document CN109871366A, the probability that a node can read data without decoding is 1-Pr(B)=f / n≈1 / r (where r is (n-1) / f). In this scheme, each shard consists of c nodes, and the nodes in the same shard hold the same coding block. Therefore, a coding block is stored by c nodes, and a coding block has c data copies. Under the replica redundancy strategy, the probability that a node can read data without decoding is Assuming c=3, r=3, Pr(B) is greater than 1-1 / 27=26 / 27≈96%.

[0064] (2) A coding block allocation method based on hash value positioning is designed to ensure the randomness of coding block allocation.

[0065] (3) A coding block reading strategy was designed to improve data search efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is a schematic diagram of shard storage in an embodiment of the present invention;

[0067] Figure 2 It is a schematic diagram of coding block reading in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] Embodiment 1

[0070] See also Figure 1 As shown, the storage and query method combining erasure code and sharding technology of the present invention includes the following steps:

[0071] Step S1, sharding behavior, specifically includes:

[0072] Step 1-1: Establish the correspondence between nodes and shards. The sharding method of the system uses the VRF leader election algorithm to select the smallest public key node, and then runs the RandHound algorithm to randomly shard the nodes in the system, randomly dividing the n nodes in the network into different shards, with c nodes in each shard. After sharding, the system is divided into n / c shards, and the number of shards is expressed as Shard = {shard0, shard1, shard2, shardm} (m = n / c). The ID of each shard is composed of a hash of the public key combination of the nodes in the shard. After the node is sharded, each node will save the mapping between the node and the shard ID, expressed as<ShardId,node0,node1,…,nodex> .

[0073] Step S2: storing data, specifically including:

[0074] Step 2-1: Generate a coded block from the block. When the block is ready to be written into the node local database after consensus, each node encodes the block independently locally. First, a block is divided into (nf) / c sub-blocks. Then, RS encoding is used to encode the (nf) / c sub-blocks into n / c coded blocks. The coded blocks are locally encoded and represented as <chunk0,chunk1,…,chunk n / c >

[0075] Step 2-2: Each node independently calculates the encoding blocks it needs to store. Since all nodes in the system are sorted by their public keys, the ordered sequence of public key addresses is consistent among all honest nodes. Therefore, each node has the public keys of all other nodes and all shard IDs in the network. Each node runs a consistent hashing algorithm locally, and its hashing algorithm uses chunkId hash as the data key and shardId as the storage device key.

[0076] Step 2-3: The consistent hashing algorithm organizes the entire hash value space into a virtual ring. The value range of the entire hash value space is [0,2 h -1], h is the number of bytes of the shardId and the encoding block data key value. Since the shard ID is the combined hash of all node IDs in the shard, the shard ID is directly converted to 2 h Modulo, the result must be a value between 0 and 2 hAn integer between -1, map this integer on the hash ring, and the value of the integer represents the position of a shard on the hash ring. The encoding block uses the same function hash algorithm to calculate the hash value as the data key, and determines the position of the encoding block on the ring. Starting from this position and looking clockwise along the ring, the first encountered shard is the shard to which the data should be located;

[0077] Step 2-4: In the above steps, determine which shard stores which encoding block. The nodes locally calculate the encoding blocks corresponding to each shard, expressed as <ShardId,chunkId>. Then, the nodes within the shard store the encoding block. According to the shard-node mapping relationship <ShardId,node1,node2,…,nodex>, the corresponding shards of the nodes need to store the encoding blocks allocated by hashing;

[0078] Step S3, the data reading operation, refer to Figure 2 , the data reading operation includes the following steps:

[0079] Step 3-1: The client sends a request to read the encoding block to Node A. If Node A locally stores the encoding block, Node A queries the local database and returns the data;

[0080] Step 3-2: When the node does not locally store the encoding block, query the shard where the encoding block is located through the chord routing. First, judge whether the successor virtual shard node of this node holds the resource. If not, then find the virtual shard node closest to the resource held from far to near in the node's routing table. Then, query all the nodes of this shard through the shard node table, and group-send the query message to these nodes. And so on, continuously query the closest virtual shard node and group-send the message to all the nodes within this shard. Finally, the query message is forwarded to the target node. The above data query process is optimized according to the chord greedy routing algorithm. The optimized query algorithm is as follows:

[0081] Step 321. Node A checks whether the hash of the Key falls between Node A and its direct successor shard node. If so, end the search, and the successor shard node of Node A is the one sought, and directly return the query result. Otherwise, enter the next step;

[0082] Step 322. In the routing table of Node A, find the successor shard node of Node A that is closest to hash(Key) and < hash(Key). This virtual shard node is also the predecessor shard node closest to Key in the routing table. Query the shard node list to obtain the IPs of all the nodes of this virtual shard node, and group-send the query message to these nodes;

[0083] Step 323. Repeat the query process in step 322 until the shard storing the query metadata is located.

[0084] Step 324. If not found, no operation is performed, and the query node makes a judgment through the timeout mechanism. If found, the node storing the metadata feeds back the relevant information to the query node, and node A verifies the coded block hash value;

[0085] Step 3-3: If node A does not receive return data from other shards, it initiates a broadcast decoding message. Nodes in other shards receive the decoding broadcast message from node A and return the encoded block. Node A decodes and recovers the data and returns it to the client.

[0086] This scheme divides the nodes in the system into n / c shards. A shard is a virtual group composed of several nodes. Nodes in the same shard hold the same coding block. Assuming that each shard consists of c nodes, the encoding process is adjusted to divide the block into (nf) / c parts and f / c parts of check blocks. The n / c coding blocks are then sent to different shards, which makes the total number of coding blocks smaller and thus reduces the computational complexity.

[0087] Embodiment 2

[0088] This embodiment provides a storage and query system combining erasure coding and sharding technology, including:

[0089] 1. Sharding behavior module, including the following units:

[0090] Establish a corresponding relationship unit between nodes and shards to perform the following tasks: The system's sharding method uses the VRF leader election algorithm to select the smallest public key node, and then runs the RandHound algorithm to randomly shard the nodes in the system, randomly dividing the n nodes in the network into different shards, with c nodes in each shard. After sharding, the system is divided into n / c shards, and the number of shards is expressed as Shard = {shard0, shard1, shard2, shardm} (m = n / c). The ID of each shard is composed of a hash of the public key combination of the nodes in the shard. After the node is sharded, each node will save the mapping between the node and the shard ID, expressed as<ShardId,node0,node1,…,nodex> .

[0091] 2. Storage data behavior module, specifically including the following units:

[0092] The local encoding unit is used to generate a coded block from a block. When a block is ready to be written to the node local database after consensus, each node encodes the block locally independently. First, a block is divided into (nf) / c sub-blocks. Then, RS encoding is used to encode the (nf) / c sub-blocks into n / c coded blocks. The coded blocks are locally encoded and represented as <chunk0,chunk1,…,chunk n / c >

[0093] Independent computing unit, used for each node to independently calculate the coding blocks it needs to store. Since all nodes in the system are sorted by their public keys, the ordered sequence of public key addresses is consistent in all honest nodes, so each node has the public keys of all other nodes and all shard IDs in the network. Each node runs a consistent hashing algorithm locally, and its hashing algorithm uses chunkId hash as the data key and shardId as the storage device key;

[0094] Location search unit, the consistent hashing algorithm organizes the entire hash value space into a virtual ring, and the value range of the entire hash value space is [0,2 h -1], h is the number of bytes of the shardId and the encoding block data key value. Since the shard ID is the combined hash of all node IDs in the shard, the shard ID is directly converted to 2 h Modulo, the result must be a value between 0 and 2 h -1, and map this integer to the hash ring. The value of the integer represents the position of a shard on the hash ring. The coding block uses the same function hash algorithm to calculate the hash value as the data key and determine the position of the coding block on the ring. From this position, search along the ring clockwise. The first shard encountered is the shard where the data should be located.

[0095] The storage unit determines which shard stores which coding block through the tasks performed by the above units. The node locally calculates the coding block stored corresponding to each shard, which is expressed as<ShardId,chunkId> Then, the nodes in the shard store the coded block according to the mapping relationship between shards and nodes.<ShardId,node1,node2,…,nodex> , the corresponding shard where the node is located needs to store the hash-assigned encoding blocks;

[0096] 3. Read data behavior module, including the following units:

[0097] A request unit, used for the client to send a request to node A to read the coded block. If node A stores the coded block locally, node A queries the local database and returns the data.

[0098] Query unit: when the encoding block is not stored locally at a node, the shard where the encoding block is located is queried through the Chord routing. First, it is determined whether the successor virtual shard node of the node holds the resource. If not, the virtual shard node closest to the resource holder is searched in the node's routing table from far to near. Then, all nodes of the shard are queried through the shard node table, and the query message is sent to these nodes in a group. And so on, continuously querying the closest virtual shard node and sending messages to all nodes in the shard. Finally, the query message is forwarded to the target node. The above data query process is optimized according to the Chord greedy routing algorithm. The optimized query algorithm is as follows:

[0099] Step 321. Node A checks whether the hash of the Key falls between node A and its direct successor shard node. If so, the search ends, and the successor shard node of node A is the one sought, and the query result is directly returned. Otherwise, proceed to the next step;

[0100] In the routing table of node A, find the successor shard node of node A that is closest to hash(Key) and < hash(Key). This virtual shard node is also the predecessor shard node closest to Key in the routing table. Query the shard node list to obtain the IPs of all nodes of this virtual shard node, and send the query message to these nodes in a group;

[0101] Step 323. Repeat the query process in step 322 above until the shard storing the query metadata is located;

[0102] Step 324. If not found, do nothing. The query node judges through the timeout mechanism. If found, the node storing the metadata and the relevant information are fed back to the query node, and node A verifies the encoding block hash value;

[0103] Step 3-3: If node A does not receive the return data from other shards, it initiates a broadcast decoding message. The nodes of other shards return the encoding block when they receive the decoding broadcast message from node A, and node A decodes and restores the data and then returns it to the client.

[0104] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A storage and query method combining erasure coding and sharding technology, characterized in that: The steps include: Step S1, sharding behavior, specifically includes: Step 1-1: Establish the correspondence between nodes and shards. Randomly divide n nodes in the network into different shards, with c nodes in each shard. After sharding, the system is divided into n / c shards. The number of shards is expressed as Shard = {shard0, shard1, shard2, shardm} (m = n / c). The ID of each shard is composed of the hash of the public key combination of the nodes in the shard. After the node is sharded, each node will save the mapping between the node and the shard ID, expressed as<ShardId,node0,node1,…,nodex> ; Step S2: storing data, specifically including: Step 2-1: Generate a coded block from the block. When the block is ready to be written into the node local database after consensus, each node encodes the block independently locally. First, a block is divided into (nf) / c sub-blocks, and then, the (nf) / c sub-blocks are encoded into n / c coded blocks using RS encoding; Step 2-2: Each node independently calculates the encoding blocks it needs to store. Each node has the public keys of all other nodes and all shard IDs in the network. Each node runs the consistent hashing algorithm locally. Step 2-3: The consistent hashing algorithm organizes the entire hash value space into a virtual ring. The value range of the entire hash value space is [0,2 h -1], h is the number of bytes of the shardId and the encoding block data key value, directly convert the shard ID to 2 h The modulo result is an integer. This integer is mapped on the hash ring. The value of the integer represents the position of a shard on the hash ring. The coding block uses the same function hash algorithm to calculate the hash value as the data key and determine the position of the coding block on the ring. From this position, search along the ring clockwise. The first shard encountered is the shard where the data should be located. Step 2-4: Determine which shard stores which encoding block; Step S3: Read data to receive the client's request to read the local database.

2. The storage and query method combining erasure coding and sharding technology as claimed in claim 1, characterized in that: In step 1-1, the system's sharding method uses the VRF leader election algorithm to select the smallest public key node, and then runs the RandHound algorithm to randomly shard the nodes in the system, randomly dividing the n nodes in the network into different shards.

3. The storage and query method combining erasure coding and sharding technology as claimed in claim 1, characterized in that: In step 2-1, the coded block is locally encoded and represented as <chunk0,chunk1,…,chunk n / c >, in step 2-2, the hash algorithm uses chunkId hash as the data key and shardId as the storage device key.

4. The storage and query method combining erasure coding and sharding technology as claimed in claim 3, characterized in that: In steps 2-4, the node locally calculates each shard and obtains the corresponding stored encoding block, which is expressed as<ShardId,chunkId> Then, the nodes in the shard store the coded block according to the mapping relationship between shards and nodes.<ShardId,node1,node2,…,nodex> , the corresponding shard of the node needs to store the hash-assigned encoding blocks.

5. The storage and query method combining erasure coding and sharding technology as claimed in claim 1, characterized in that: The step S3, reading data behavior specifically includes: Step 3-1: The client sends a request to node A to read the coded block. If node A stores the coded block locally, node A queries the local database and returns the data. Step 3-2: If the node does not store the encoding block locally, query the shard where the encoding block is located through the chord routing; Step 3-3: If node A does not receive return data from other shards, it initiates a broadcast decoding message. Nodes in other shards receive the decoding broadcast message from node A and return the encoded block. Node A decodes and recovers the data and returns it to the client.

6. A storage and query method combining erasure coding and sharding technology as claimed in claim 5, characterized in that: Step 3-2 specifically includes: first, determine whether the successor virtual shard node of this node holds this resource. If not, then search in the node's routing table from far to near to find the virtual shard node closest to the resource holder. Then, query all the nodes of this shard through the shard node table, and send the query message to these nodes in a group. And so on, continuously find the closest virtual shard node and send the message to all the nodes in this shard. Finally, the query message is forwarded to the target node.

7. A storage and query method combining erasure coding and sharding technology as claimed in claim 6, characterized in that: The data query process in Step 3-2 is optimized according to the chord greedy routing algorithm. The query algorithm includes the following steps: Step 321. Node A checks whether the hash of the Key falls between Node A and its direct successor shard node. If so, end the search, and the successor shard node of Node A is the one found, and directly return the query result. Otherwise, go to the next step; Step 322. In the routing table of Node A, find the successor shard node of Node A that is closest to hash(Key) and < hash(Key). This virtual shard node is also the predecessor shard node closest to Key in the routing table. Query the shard node list to obtain the IPs of all the nodes of this virtual shard node, and send the query message to these nodes in a group; Step 323. Repeat the query process in the above Step 322 until the shard storing this query metadata is located; Step 324. If not found, do nothing, and let the query node judge through the timeout mechanism. If found, the node storing this metadata and feedback the relevant information to the query node, and Node A verifies the hash value of the encoded block.

8. A storage and query system combining erasure coding and sharding technology, characterized in that: It includes the following modules: The shard division behavior module specifically includes the following units: The unit for establishing the correspondence between nodes and shards is used to perform the following tasks: randomly divide n nodes in the network into different shards, with c nodes in each shard. After sharding, the system is divided into n / c shards. The number of shards is expressed as Shard = {shard0, shard1, shard2, shardm} (m = n / c). The ID of each shard is composed of the combined hash of the public keys of the nodes in the shard. After the nodes are sharded, each node will save the mapping between the node and the shard ID, expressed as <ShardId, node0, node1,..., nodex>; The data storage behavior module specifically includes the following units: The local encoding unit is used to generate encoded blocks from the block. When the block is ready to be written into the local database of the node after consensus, each node independently encodes the block locally. First, a block is divided into (n - f) / c sub-blocks, and then, (n - f) / c sub-blocks are encoded into n / c encoded blocks using RS encoding; The independent calculation unit is used for each node to independently calculate the encoded blocks it needs to store. Each node has the public keys of all other nodes and all the shard IDs in the network, and each node runs the consistent hashing algorithm locally; Location search unit, the consistent hashing algorithm organizes the entire hash value space into a virtual ring, and the value range of the entire hash value space is [0,2 h -1], h is the number of bytes of the shardId and the encoding block data key value, directly convert the shard ID to 2 h The modulo result is an integer. This integer is mapped on the hash ring. The value of the integer represents the position of a shard on the hash ring. The coding block uses the same function hash algorithm to calculate the hash value as the data key and determine the position of the coding block on the ring. From this position, search along the ring clockwise. The first shard encountered is the shard where the data should be located. The storage unit determines which shard stores which encoded block; The data reading behavior module is used to receive the request from the client to read the local database.

9. A storage and query system combining erasure coding and sharding technology as claimed in claim 8, characterized in that: In the unit for establishing the correspondence between nodes and shards, the sharding method of the system uses the leader election algorithm of VRF to select the smallest public key node, and then runs the RandHound algorithm to randomly shard the nodes in the system, randomly dividing the n nodes in the network into different shards; In the local coding unit, the coding block is locally encoded and represented as <chunk0,chunk1,…,chunk n / c >, in the independent computing unit, the hash algorithm uses the chunkId hash as the data key and the shardId as the storage device key; In the storage unit, each node locally calculates the encoding blocks corresponding to each shard, expressed as <ShardId,chunkId>. Then, the nodes within the shard store the encoding blocks. According to the mapping relationship between the shard and the nodes <ShardId,node1,node2,…,nodex>, the corresponding shards of the nodes need to store the encoding blocks allocated by the hash.

10. A storage and query system combining erasure coding and sharding technology as claimed in claim 8, characterized in that: The data reading behavior module specifically includes the following units: The request unit is used for the client to send a request to node A to read the encoding block. If node A locally stores the encoding block, node A queries the local database and returns the data; The query unit, and the query algorithm is as follows: Step 321. Node A checks whether the hash of the Key falls between node A and its direct successor shard node. If so, the search ends, and the successor shard node of node A is the one sought, and the query result is directly returned. Otherwise, proceed to the next step; Step 322. In the routing table of node A, find the successor shard node of node A that is closest to and < hash(Key), and this virtual shard node is also the predecessor shard node closest to Key in the routing table. Query the shard node list to obtain the IPs of all nodes of this virtual shard node, and send query messages to these nodes in a group; Step 323. Repeat the query process in step 322 above until the shard storing the query metadata is located; Step 324. If not found, do nothing. The query node judges through the timeout mechanism. If found, the node storing the metadata and the relevant information are fed back to the query node, and node A verifies the hash value of the encoding block; Step 3-3: If node A does not receive the return data from other shards, it initiates a broadcast decoding message. The nodes of other shards return the encoding block when they receive the decoding broadcast message from node A, and node A decodes and restores the data and then returns it to the client.

Images