An optimized structure and method for blockchain data grouping storage based on erasure code

Abstract

The present invention discloses a block chain data group storage optimization structure and method based on erasure code, the structure includes: node group setting module, encoding storage module, collection decoding module and synchronization recovery module; the method includes the following steps: (1) the block chain group setting module groups the nodes of the network, initializes the organization account book of each organization, and sets various values; (2) the encoding node uses the block encoding module to encode each block file, generates encoding segments, and allows each node in the organization to store the encoding segments; (3) when the block file needs to be restored, the block file is restored by the erasure code decoding algorithm; (4) when the local organization synchronizes the account book, a synchronization request is sent to other organizations, and the account book snapshot is compared. If the comparison results are different, the latest block file is obtained to synchronize the local account book. The present invention is used to reduce the redundant storage of nodes, optimize the storage of block chain data, and improve the storage efficiency of the block chain network.

Description

Technical Field

[0001] The present invention relates to the field of blockchain technology, and in particular to an erasure code-based blockchain data grouping storage optimization structure and method. Background Art

[0002] With the development of blockchain technology, blockchain has gradually moved from the original realization of digital currency to various fields such as financial institutions, technology companies and government departments. Blockchain is a decentralized distributed database technology that uses a combination of point-to-point transmission, cryptography, consensus algorithms and other technologies to achieve the advantages of decentralization, immutability, traceability and transparency. However, the traditional blockchain system uses full-copy redundant data storage, which causes problems such as high database storage overhead and poor scalability. Each node in the blockchain network obtains consistent blocks through a consensus algorithm, and these blocks are stored in the form of files by each blockchain node. As the number of blocks and block files continues to increase, the storage overhead of each node also continues to increase, which eventually makes the storage burden of nodes and the entire network serious.

[0003] There are currently popular solutions to the above problems: one is the light node solution. Compared with full nodes, this light node does not need to download all block data. It only needs to retain the block header data (its volume grows by 4MB per year), and locate the block Merkel Tree to be verified in the entire network through the block header and transaction conditions. A simple local verification can confirm the validity of the payment. However, light nodes cannot participate in transaction verification (unlike payment verification), and cannot participate in blockchain consensus. It can only appear in the public blockchain network as a "user client". If all users run such light nodes, the availability of the entire blockchain will be threatened and the security will be reduced. The other is the IPFS off-chain storage solution. Many blockchain storage models based on IPFS store data in IPFS, and then store the hash value of the corresponding IPFS content on the blockchain. However, the data in the IPFS system is open to everyone. As long as the content hash value is obtained, the data can be obtained, and it does not support data deletion, which has privacy and security issues and data leakage risks. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a blockchain data grouping storage optimization structure and method based on erasure code, which is used to reduce redundant storage of nodes, optimize blockchain data storage, and improve the storage efficiency of the blockchain network.

[0005] In order to solve the above technical problems, the present invention provides a blockchain data grouping storage optimization structure based on erasure codes, including: a node grouping setting module, a coding storage module, a collection and decoding module and a synchronization recovery module; the node grouping setting module divides the network into organizations, and then based on this, the coding storage module is used to encode and store data in groups. After the storage is successful, the decoding data can be collected and decoded through the collection and decoding module. Finally, considering the local node account book synchronization problem, the synchronization recovery module is used to restore the node data.

[0006] Preferably, in the node grouping setting module, the number of organizations in the blockchain network is set to n. o , divide several nodes into n o This module also sets each Org's node responsible for encoding and storing block files as L nodes, initializes the organization ledger GZ, and sets the number of nodes in the current organization n p , node number, number of encoded block files, number of unencoded block files, generation matrix G, check matrix H and other information are stored in GZ.

[0007] Preferably, in the coding storage module, the L node first divides each block file b into data segments m of the same size, encodes m using an erasure code encoding algorithm, multiplies m by a generator matrix G, and calculates a check segment c; m and c are combined to form a coding segment s; the L node uniformly stores the s corresponding to each b in each node according to a set cyclic shift storage coding segmentation algorithm; the blockchain b contains multiple blocks, and each block will be placed in b in sequence after generation until the maximum value of the b space size is exceeded, and then the next b will be generated, so that the latest u-th block file cannot be encoded, and the u-1th must be encoded; each node makes the blocks the same through a consensus mechanism, so the b recorded by each node is also the same; finally, when each node in the Org is stored in a coding segment of a certain b, the corresponding GZ will also update the coding storage information of a certain b, the L node will record the hash value of the original b, and each node will delete the original b.

[0008] Preferably, the cyclic shift storage coding segmentation algorithm is based on the input coding segment s and the number of nodes n p The parameters such as s are calculated by the algorithm and stored evenly in the nodes of the organization, and the numbers are not repeated, and the overhead of padding with 0 is not increased. Ultimately, the storage overhead of each node in the organization is uniform and no additional storage space is required.

[0009] Preferably, in the collection and decoding module, when the coded segment s needs to be restored to the original block file, the L node of the current organization will first collect the corresponding s from the organization. If s is insufficient, the L node needs to collect the missing s from other organizations across organizations. After the collection is successful, the L node restores the original block file according to the erasure code decoding algorithm in the decoding module, and compares the hash of the data block with the block file before encoding to see whether the recovery is successful.

[0010] Preferably, in the synchronization recovery module, when a node or organization needs to update its corresponding node ledger and organization ledger, the L node of the current organization initiates a request to synchronize and recover the organization ledger GZ; after receiving the synchronization recovery request, other organizations query their respective GZ ledger snapshots; the L node recovers the block file according to the collection decoding module, calculates its ledger snapshot, and then broadcasts it to other organizations; after receiving the ledger snapshot, other organizations compare it with the local organization ledger snapshot and return the comparison result to the L node; if the comparison result is inconsistent, the L node of the local organization initiates a request to synchronize the block file to the corresponding organization; the corresponding organization sends the corresponding missing block file or encoding segment to the L node.

[0011] Accordingly, a method for optimizing blockchain data grouping storage based on erasure coding includes the following steps:

[0012] (1) The blockchain grouping setting module groups the nodes of the network, organizes the number of organizations, initializes the organizational ledger of each organization, sets the encoding nodes, the number of nodes, the node number, the number of encoded block files, the number of unencoded block files, the generation matrix G, and the check matrix H;

[0013] (2) The encoding node uses the block encoding module to encode each block file, generates encoding segments, and uses the cyclic shift storage encoding segment algorithm to allow each node in the organization to store the encoding segments;

[0014] (3) When the block file needs to be restored, the collection decoding module is used to collect the required coding segments within or across organizations, and the block file is restored through the erasure code decoding algorithm;

[0015] (4) When the local organization synchronizes the ledger, it uses the synchronization recovery module to send synchronization requests to other organizations, compare the local ledger snapshot with the ledger snapshots of other organizations, and obtain the latest block file to synchronize the local ledger if the comparison results are different.

[0016] The beneficial effects of the present invention are as follows: the present invention divides the network into several organizations, each organization contains several nodes and maintains the same organizational ledger. Each node in the organization evenly stores the overhead of part of the organizational ledger, thereby reducing the data redundancy of the block nodes and reducing the data storage overhead; through the mechanism of erasure coding and grouped storage of ledgers, the data resource storage can be reduced while ensuring the availability and security of the block network. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a schematic diagram of an application scenario of blockchain data grouping storage based on erasure codes of the present invention.

[0018] Figure 2 This is a flow chart of the operation of the storage encoding module of the present invention.

[0019] Figure 3 The present invention is a flow chart of the collection and decoding module operation.

[0020] Figure 4 This is a flow chart of the operation of the synchronization recovery module of the present invention. DETAILED DESCRIPTION

[0021] like Figure 1 As shown, an optimized structure of blockchain data grouping storage based on erasure codes includes: a node grouping setting module, a coding storage module, a collection decoding module and a synchronization recovery module; the node grouping setting module divides the network into organizations, and based on this, the coding storage module is used to encode and store data in groups. After the storage is successful, the decoding data can be collected by the collection decoding module. Finally, considering the synchronization problem of local node ledgers, the synchronization recovery module is used to restore the node data.

[0022] Use the group setting module to set the blockchain network to have two organizations: 1 ,org 2 Each organization has three grouping storage (GS) nodes P 1 ~P 3 , set up a coding node for each group. Using the storage coding module, the coding node divides each block file into 4 data segments m 1 ~m 4 Next, the data segment is multiplied by the generator matrix G to obtain 6 coding segments s, including 2 check segments c 1 ,c 2 and the original 4 data segments m 1 ~m 4Each organization in the storage coding module saves the coding segment of each block file. Each GS node in the group retains 2 coding segments. The 6 nodes in the whole network have 2 complete block files. The storage space of each coding segment is 1 / 4 of the original block file. Each node stores 2 coding segments, so the storage space of each node is 1 / 2 of the original. Using the collection decoding module, org 1 ,org 2 Sufficient number of coded segments can be obtained from each node in the group or each node across the organization, and then the block files can be restored using decoding technology. Assuming that the data of any 4 nodes is completely lost, that is, 2 / 3 of the nodes in the entire network are lost, we can still restore the original block files of all nodes to ensure that the block files are not lost. Using the synchronous recovery module, org 1 and org 2 The organization ledger snapshots can be compared with each other. If the comparison results are different, the latest block file of the block is copied to the other organization, and then the local organization ledger is updated. Therefore, this method reduces the redundant storage of blockchain data while also ensuring the availability and security of the blockchain.

[0023] Figure 2 This is a specific block file storage encoding module operation flow chart of the present invention. The erasure code technology used in this example is QC-LDPC. 1 There are 3 nodes for encoding storage. The specific steps of encoding storage are as follows:

[0024] 201. Get the complete block file. In the L node within the organization Org, synchronize the block file b of the blockchain network i , and get it. This solution does not encode the latest u-th block file, but chooses to delay the u-1 block file. Delaying the encoding of the block file can ensure that the block file will no longer record new blocks and change, thereby ensuring that the organization's ledger is synchronized and consistent.

[0025] 202. Block file encoding. After the L node obtains the block file, it encodes the block file bi using the QC-LDPC code encoding algorithm to generate the encoding segment b′. i . Assume that the number of data segments m is k, and the number of check segments c is r. When the encoding block file b i When b i Divide into k data segments m. If you want to divide b into fixed size S B / k, you need to fill the block file with 0 to S B Size. Generate the coded segment b′ using the data segmentation and erasure code generator matrix i =mG=s 0 ,s 1 ,…sk+r , the set of partially encoded segments s that each node finally stores.

[0026] 203. The block file is successfully stored. The L node calculates the coded segment set S corresponding to each P in the group, and then stores it in the corresponding node account book Z. Each Z is different and only stores part of the coded segment of each block file. All Z in the group will be combined into the organization account book GZ, and the GZ of each organization will remain consistent. After successfully storing the relevant information, the L node will store the original block file b i Delete it.

[0027] If we calculate the partial coding segment set S corresponding to different P nodes, we need to consider how to make each P store the coding segment s as little as possible and evenly, and let the Org have complete coding segment information. This paper designs a cyclic shift coding segment storage algorithm with the goal of balancing overhead, and calculates the number of coding segments that each P needs to store as n / and coded segment set

[0028] The specific algorithm steps are as follows:

[0029] 301. Input the number of nodes in the organization n, the number of data segments k, the number of verification segments r, and the block file number i;

[0030] 302. Initialization t = 0, b=0,γ=0, Storage for the first node Node number, when block file number 1 The default value is 1. Obtained through calculation, t records the number of nodes that have stored the encoding segment of block file i, The set of partial coded segments of the current organization node j storing block file i, where j∈[1,n], b=0 means that the partial coded segments need to be obtained by coding calculation, and γ=0 means that the block coded segments are numbered starting from 0 by default;

[0031] 303. Let β = (k + r) % n. If β = 0, it is divisible. The number of coded segments stored in each GS node in the group is calculated as n. / =(k+r) / n; otherwise, n / =(k+r) / n+1,;

[0032] 304. Calculate the first node number of the block file stored in this round and assign it to j. Calculate the first node number for storing the next block file

[0033] 305. Node i has its own encoding segment s for block file j @ ,γ∈[0,k+r), if n×n / = = k + r, indicating that the coding segment can be equally divided by n nodes, and step 306 is executed; otherwise n × n / >(k+r), execute step 307;

[0034] 306. If γ ≥ t × n / And γ<(t+1)×n / ,but t=t+1, j=(j%n)+1;

[0035] 307. If β≠0,γ≥t×n / And γ<(t+1)×n / ,but t=t+1,j=(j%n)+1,β=β-1;if β==0,γ≥t×(n / -1) and γ<(t+1)×(n / -1), then t=t+1, j=(j%n)+1;

[0036] 308. If t+1 == n, execute 309, otherwise return to 305 until all nodes are stored.

[0037] 309, Return Finish.

[0038] Figure 3 The operation flow of the block file collection decoding module in the present invention is as follows: Figure 3 As shown, two organizations are specifically designed in the block network, org 1 Execute the block file recovery process. The specific steps are as follows:

[0039] 401 initiates a transaction request to query the block file. First, the client C initiates a query request to the L node in the organization Org through the chain code to query whether the encoding segment set S has been stored in different P nodes in the Org.

[0040] 402. Collect coded segments. Coded segments s can be collected from within the organization or across organizations. If s is insufficient within the group, the L nodes within the organization will collect across organizations until the required s is collected, and then decode and restore to the corresponding b i . Generally, priority is given to collecting s within the group.

[0041] 402.1, Collection within the organization. The L node broadcasts a request to collect b to each P node in the Org iWhen the P node receives the broadcast message, it returns the corresponding s to the L node. If the L node collects enough s, it will broadcast the message to stop collecting s. The L node prioritizes collecting k data segments m, so that decoding can be avoided. In addition, the L node can also collect enough data segments and check segments to restore the block file using the erasure code decoding algorithm.

[0042] 402.2, cross-organization collection. If less than k s are collected in the Org, the missing s can be collected across organizations and returned to the L node. The L node sends a message to the A node in the Org to communicate with other organizations and obtain the missing s. After receiving the message, the A node of the other organization returns the corresponding s. Figure 3 , the A1 node in Org1 communicates with the A1 node in Org2 and obtains the corresponding s from Org2. It should be noted that the decoding matrix is ​​the inverse matrix of the linear combination corresponding to the encoding segment, which will be saved in the account book in advance.

[0043] 403. Decode and recover the block file. If the L node has collected enough coding segments s from the previous step, it can start to use QC-LDPC decoding to recover the block file, otherwise it returns an error. The coding segment s is divided into two types: data segment m and check segment c. If the L node collects enough k, it can recover the original block file b i Otherwise, it needs to be recovered through c decoding. i By calculating its hash value and comparing it with the value before encoding, you can confirm whether the recovered block file is accurate and verify whether it has been tampered with.

[0044] 404. Return the query result. The query result is returned from the L node, and the entire query process ends.

[0045] Figure 4 This is the operation flow of the block file synchronization recovery module in the present invention, such as Figure 4 As shown, the local organization is specifically designed to request ledger synchronization in the block network. The block file accounts for the largest storage overhead in the ledger. The present invention also optimizes the storage of block files. Therefore, synchronously updating block files is to synchronize blockchain ledgers. There may be the following situations, which require synchronous updating of block files.

[0046] (1) A certain P node is offline for a long time, and some of the corresponding coding segments of the block files are missing, that is, the node ledger Z is incomplete. Other P nodes in the organization Org are operating normally, and the node ledger Z is stored completely. The Org can only provide snapshots of the organization ledger through the index library and status database, but cannot provide complete original files for offline nodes to synchronize blocks.

[0047] (2) When other organizations are offline, they need to synchronize all current block files with adjacent organizations.

[0048] (3) When a block file is encoded and stored, all organizations must promptly and synchronously calculate and store the encoding segments of the block file and update the organization ledger.

[0049] The specific steps for synchronously restoring block files are as follows:

[0050] 501. Initiate synchronization and recovery ledger request. If any P node in the organization Org wants to synchronize the latest block file encoding segment, first initiate a synchronization recovery organization ledger GZ request to the L node. For the above three synchronization situations, only need any P node in the Org to initiate a synchronization request, you can achieve any node or the entire organization ledger update, you can also synchronize the new block file in the network.

[0051] 502. Restore block files. L node collects and restores block files according to the collection decoding module. It only needs to restore the block files in the last synchronization process. For example: the last time the block file b was synchronized i , which means that the previous i block files have been synchronized and have not been tampered with and do not need to be synchronized again. So this time, only b needs to be restored. i,1 to b L The latest generated block file can be used, and then its ledger snapshot is calculated. This method does not need to start recovery from the first block file. However, if the intermediate block files are missing, it may not be applicable, and you have to reconsider whether to restore all block files.

[0052] 503. Restore the organization's ledger. After receiving the synchronization request, other organizations can directly obtain and query their latest ledger snapshots in their index libraries and status databases.

[0053] 504. Broadcasting ledger snapshot. L node broadcasts the ledger snapshot in the Org to L nodes in other organizations. After receiving the ledger snapshots of other organizations, it compares them with the local organization’s ledger snapshot, comparing the highest hash, block number, worldstate and other parameters. Other organizations return the comparison results.

[0054] 505. Request for synchronization of block files. If the comparison results are inconsistent, the L node of the local Org initiates a request for synchronization of block files to the corresponding organization. The corresponding organization sends the corresponding missing block files or encoding segments to the L node.

[0055] 506. Update local ledger. Finally, the L node obtains the new block file, and synchronously updates the index and status database, as well as the node ledger of each node in the new group.

Claims

1. A blockchain data group storage optimization structure based on erasure code, characterized in that: include: Node grouping setting module, encoding storage module, collection decoding module and synchronization recovery module; The node grouping setting module divides the network into organizations, and based on this, the encoding storage module is used to encode and store data in groups. After the storage is successful, the decoding data can be collected and decoded by the collection and decoding module. Finally, considering the synchronization problem of the local node account book, the synchronization recovery module is used to restore the node data; In the node grouping setting module, set the number of organizations in the blockchain network to n o , divide several nodes into n o The module also sets each Org's node responsible for encoding and storing block files as L nodes, initializes the organization ledger GZ, and sets the number of nodes in the current organization n p , node number, number of encoded block files, number of unencoded block files, generation matrix G, and check matrix H are stored in GZ; In the encoding storage module, the L node first divides each block file b into data segments m of the same size, encodes m using the erasure coding algorithm, multiplies m with the generator matrix G, and calculates the check segment c; m and c are combined to form the encoding segment s; The L node evenly stores the s corresponding to each b in each node according to the set cyclic shift storage coding segmentation algorithm; the blockchain b contains multiple blocks, and each block will be placed in b in order after it is generated, until the maximum size of the b space is exceeded, the next b will be generated, so the latest u-th block file cannot be encoded, and the u-1th block must be encoded; each node makes the block the same through the consensus mechanism, so the b recorded by each node is also the same; finally, when each node in the Org is stored in a certain b's coding segment, the corresponding GZ will also update the coding storage information of a certain b, the L node will record the hash value of the original b, and each node will delete the original b; the cyclic shift storage coding segmentation algorithm, according to the input coding segment s and the number of nodes n p The parameters,s,are evenly stored in the nodes of the organization after being calculated by the algorithm, and the numbers are not repeated, and no overhead of adding zeros is added. Ultimately, the storage overhead of each node in the organization is even and no additional storage space is required; In the collection and decoding module, when the encoded segment s needs to be restored to the original block file, the L node of the current organization will first collect the corresponding s from the organization. If s is insufficient, the L node needs to collect the missing s from other organizations across organizations; After the collection is successful, the L node restores the original block file according to the erasure code decoding algorithm in the decoding module, and compares the hash of the data block with the block file before encoding to see whether the recovery is successful; In the synchronization recovery module, when a node or organization needs to update its corresponding node ledger and organization ledger, the L node of the current organization initiates a request to synchronize and recover the organization ledger GZ; After receiving the synchronization recovery request, other organizations query their respective GZ ledger snapshots; L nodes recover block files based on the collected decoding module, calculate their ledger snapshots, and then broadcast them to other organizations; After receiving the ledger snapshot, other organizations compare it with the local organization's ledger snapshot and return the comparison result to the L node. If the comparison results are inconsistent, the local organization's L node initiates a request to synchronize block files with the corresponding organization. The corresponding organization sends the corresponding missing block file or encoding segment to the L node.

2. An optimization method based on the block chain data group storage optimization structure based on erasure code as claimed in claim 1, characterized in that: The steps include: (1) The blockchain grouping setting module groups the nodes of the network, organizes the number of organizations, initializes the organizational ledger of each organization, sets the encoding nodes, the number of nodes, the node number, the number of encoded block files, the number of unencoded block files, the generation matrix G, and the check matrix H; (2) The encoding node uses the block encoding module to encode each block file, generates encoding segments, and uses the cyclic shift storage encoding segment algorithm to allow each node in the organization to store the encoding segments; (3) When the block file needs to be restored, the collection decoding module is used to collect the required coding segments within or across organizations, and the block file is restored through the erasure code decoding algorithm; (4) When the local organization synchronizes the ledger, it uses the synchronization recovery module to send synchronization requests to other organizations, compare the local ledger snapshot with the ledger snapshots of other organizations, and obtain the latest block file to synchronize the local ledger if the comparison results are different.

Images