A blockchain storage optimization method and system based on erasure code

Abstract

The present invention discloses a blockchain storage optimization method and system based on erasure code, the method comprising the following steps: obtaining blockchain network nodes; grouping blockchain network nodes using a dynamic close-range node grouping algorithm based on a fuzzy c-means clustering algorithm to obtain consensus units; based on the consensus units, using a random coding block placement strategy based on node performance to store and allocate coding blocks, and globally processing block files. The present invention reduces the storage burden of a single node, improves the problems of blockchain losing centralization, data security, and data authenticity due to high node storage pressure.

Description

Technical Field

[0001] The present invention belongs to the field of blockchain technology and erasure code technology, and specifically relates to a blockchain storage optimization method and system based on erasure code. Background Art

[0002] Blockchain is a record-keeping technology that stores transactions and valuable information in the form of blocks. Blockchain has important features that distinguish it from centralized systems, such as decentralization, immutability, security, and anonymity, which has enabled it to be successfully applied in financial technology, public governance, supply chain, and other fields.

[0003] The first application scenario of erasure coding technology is distributed storage system, which aims to solve the problem of data recovery when the distributed storage system fails. The storage redundancy strategy of erasure coding can ensure that the distributed system minimizes the redundant storage overhead with the same reliability. The blockchain system is essentially a special distributed system. In theory, erasure coding can be applied to the blockchain system. Pal Rudrashhish et al. used digital fountain code to solve the problem of blockchain copy redundancy and improved the storage performance of the system. Zhao Guofeng et al. proposed a block file storage model for blockchain system based on erasure coding to improve the storage performance of Hyperledger, but its defect is that the object of use is strictly limited and can only be applied to various blockchain platforms under the alliance chain Hyperledger. Perard proposed a low-storage blockchain node based on erasure coding. The article takes nodes with small storage space as the main research object. Although this scheme has certain advantages in reducing blockchain storage consumption, it reduces the level of decentralization of the blockchain system. Qi et al. designed the BFT-Store storage engine by combining BFT consensus and erasure coding technology. Although this design scheme realizes the scalability of blockchain, its availability is poor. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a blockchain storage optimization method and system based on erasure codes, which adopts a fuzzy c-means clustering algorithm and a coding block placement node set selection strategy to reduce the storage burden of a single node and improve the problems of blockchain losing centralization, data security, and data authenticity caused by the high node storage pressure when using a full copy mechanism to achieve the consistency of distributed ledgers.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A blockchain storage optimization method based on erasure coding comprises the following steps:

[0007] Obtaining blockchain network nodes, each of which stores a global and complete distributed ledger;

[0008] The blockchain network nodes are grouped using a dynamic close-distance node grouping algorithm based on a fuzzy c-means clustering algorithm to obtain consensus units;

[0009] Based on the consensus unit, a random coding block placement strategy based on node performance is adopted to allocate storage for coding blocks and perform global processing on block files.

[0010] Preferably, the method of grouping the blockchain network nodes using a dynamic close-distance node grouping algorithm based on a fuzzy c-means clustering algorithm includes:

[0011] Select the central node;

[0012] Based on the fuzzy c-means clustering algorithm, the central node sends a request message to each participating node, and the participating node returns a response message to the central node;

[0013] The distance between two points is estimated according to the round trip delay of the request and response messages, and the participating nodes are added to the group that meets the preset requirements according to the estimated distance.

[0014] Preferably, the method for globally processing the block file includes: block encoding, block data storage, block reconstruction and block data synchronization;

[0015] Block encoding is the process of encoding blocks using RS erasure code; block data storage is the process of randomly storing coded blocks in a specified set of nodes; block reconstruction is the reverse process of block encoding, which realizes the recovery of block files; block data synchronization is the process of synchronizing the latest block data to new nodes when new nodes join.

[0016] Preferably, in the block encoding, the specific process of using RS erasure code encoding processing on the block includes:

[0017] Let the i-th block in the blockchain be B (i) ;

[0018] In block B (i) Before slicing, if block B (i) If k equal-size partitions are not supported, fill block B with 0. (i) ;

[0019] If block B (i) Support k equal-sized splits, then use the node to split block B (i) Divide into k equal-sized file blocks Block B (i) Then use RS(n,k) erasure code to generate n coded blocks The n coding blocks are randomly distributed and stored in designated n nodes for storage.

[0020] Preferably, in the block data storage, the specific process of randomly storing the coded blocks in the designated node set includes:

[0021] Add k nodes that reach the threshold T in the consensus unit to the candidate node set N;

[0022] The k nodes in the set N are sorted in descending order based on the performance response feedback using the superior and inferior solution distance method to make performance index decisions;

[0023] The first n nodes in the sorting result are selected as the coding block to place the node set S.

[0024] Preferably, in the block reconstruction, the reverse process of block encoding, the specific process of realizing the recovery of block files includes: fault detection stage, recovery node selection stage, coding block reading stage, original data decoding stage, new data block encoding stage and metadata update stage.

[0025] Preferably, during block data synchronization, when a new node joins, the specific process of synchronizing the latest block data to the new node includes:

[0026] When a new node joins the blockchain system for the first time, the new node is added to a consensus unit through a dynamic proximity grouping algorithm;

[0027] The new node sends a broadcast request to the other nodes in the corresponding consensus unit to obtain the maximum block height of the blockchain in the current state;

[0028] The new node sequentially reconstructs the block data from the genesis block to the maximum block high block interval, and temporarily stores T time complete blocks locally;

[0029] After time T is up, the block data storage process is repeated. The performance of the nodes is calculated based on TOPSIS. If the ranking of the new node performance decision index meets the preset requirements, some coding blocks are retained.

[0030] The present invention also provides a blockchain storage optimization system based on erasure codes, comprising: an acquisition module, a grouping module and a processing module;

[0031] The acquisition module is used to acquire blockchain network nodes, each of which stores a global and complete distributed ledger;

[0032] The grouping module is used to group the blockchain network nodes using a dynamic close-distance node grouping algorithm based on a fuzzy c-means clustering algorithm to obtain consensus units;

[0033] The processing module is used to allocate storage for coding blocks based on the consensus unit and adopt a random coding block placement strategy based on node performance to perform global processing on block files.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention improves the block encoding speed and block reconstruction speed, while reducing system space occupation.

[0036] (1) Block encoding speed

[0037] In different RS encoding schemes, when the block size is the same, the lower the fault tolerance rate, the faster the encoding speed; in different RS encoding schemes, when the block size and fault tolerance rate are the same, there are some differences in the encoding speed under different encoding combinations; in the same RS encoding scheme, as the block file size increases, the encoding speed also increases.

[0038] (2) Block reconstruction speed

[0039] Within the fault tolerance range of RS erasure codes, the fixed block size remains unchanged, the fewer the number of coded blocks lost by the system, the faster the block reconstruction speed; within the fault tolerance range of RS erasure codes, the number of fixed data block losses remains unchanged, the higher the fault tolerance rate, the faster the reconstruction speed.

[0040] (3) Comparison of system storage space usage

[0041] When the number of blocks remains unchanged, as the number of system nodes increases, the storage space occupied by Hyperledger Fabric is proportional to the number of system nodes. Compared with Hyperledger Fabric, the BSO-EC proposed in this article can bring lower storage consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0043] Figure 1 Detailed operation mechanism diagram of BSO-EC in an embodiment of the present invention;

[0044] Figure 2 This is a flow chart of a blockchain storage optimization method based on erasure coding in an embodiment of the present invention;

[0045] Figure 3It is a block coding principle diagram in an embodiment of the present invention;

[0046] Figure 4 A coding block placement node set selection strategy diagram in an embodiment of the present invention;

[0047] Figure 5 is a flow chart of BSO-EC data synchronization in an embodiment of the present invention;

[0048] Figure 6 This is an overall framework diagram of a blockchain storage optimization system based on erasure coding in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] Embodiment 1

[0052] The present invention provides a blockchain storage optimization method based on erasure codes (Blockchain Storage Optimization Based on Erasure Codes, BSO-EC), the detailed operation mechanism is as follows Figure 1 As shown, the steps are as follows:

[0053] Step 1: System initialization. In the initial operation phase of the BSO-EC system, each participating node in the network stores a global and complete distributed ledger, which facilitates the subsequent block encoding operations and block data synchronization and other block processing processes.

[0054] Step 2: Consensus unit generation. The nodes in the BSO-EC network establish logical connections with each other based on landmark clustering, and then group the nodes in the entire network based on the dynamic close-range node grouping algorithm. Each small group after grouping is called a consensus unit. The grouping operation facilitates structured management between small groups, shortens the interaction delay between nodes, and improves communication efficiency.

[0055] Step 3: Block processing. This step mainly includes sub-steps such as block encoding, block storage, block reconstruction and block synchronization. Block encoding is to use the RS encoding scheme to encode the complete block file. Block storage is to store the encoded blocks in a decentralized manner on each participating node of the consensus unit. Block reconstruction is the process of obtaining the available number of encoded blocks to restore the original block when some node data is damaged. Block synchronization is to update the data and status of the nodes that have joined BSO-EC to the latest.

[0056] Specifically, Figure 2 As shown in Table 1, first, BSO-EC uses a dynamic close-distance node grouping algorithm based on the fuzzy c-means clustering algorithm to group nodes. Blockchain network node grouping is convenient for sub-network structured management, and can also reduce the interaction delay between nodes and improve communication efficiency. The dynamic close-distance node grouping algorithm is shown in Table 1. The idea of ​​the algorithm is to first select the central node, and then perform topology optimization around the central node based on the fuzzy c-means clustering algorithm. The central node sends a request message to each participating node, and the participating node returns a response message to the central node. The distance between the two points is estimated based on the round-trip delay of the request and response messages. The node is added to a suitable group based on the estimated distance. The dynamic close-distance grouping algorithm can ensure that the delay of nodes in the same group is much smaller than the delay between nodes in different groups.

[0057] Table 1

[0058]

[0059] Then, a coding block placement node set selection strategy is proposed to allocate storage for coding blocks and perform global processing on block files. The global processing flow of block files is: block encoding, block data storage, block reconstruction, and block data synchronization. Block encoding is to encode blocks using RS erasure code. Block data storage is to randomly store coding blocks in a specified node set. Block reconstruction is the reverse process of block encoding, which can realize the recovery of block files. Block data synchronization is to synchronize the latest block data to the new node when a new node joins BSO-EC. The random coding block placement strategy based on node performance can effectively avoid the great instability of the random selection strategy and maximize the guarantee that the group of nodes has good computing power and storage capacity. In the process of ordinary participating nodes being elected as candidate nodes, the random number generation mechanism based on digital signatures and hash algorithms can effectively ensure the fairness, impartiality, openness and transparency of the election process.

[0060] In this embodiment, the global processing flow of the block file includes four steps: block encoding, block data storage, block reconstruction, and block data synchronization.

[0061] Furthermore, the i-th block in the blockchain is denoted as B(i) Then B (0) is the first block, that is, the genesis block. The encoding mechanism of RS erasure code in distributed storage system is still applicable to the encoding of blocks in the BSO-EC blockchain system in this paper. The specific block encoding implementation is as follows Figure 3 shown.

[0062] In block B (i) Before slicing, if block B (i) Does not support k-partitioning with equal size. Need to fill block B with 0 first (i) , and then continue encoding related operations. First, the node will block B (i) Divide into k equal-sized file blocks Then, block B (i) Then use RS(n,k) erasure code to generate n coded blocks The last n coded blocks are randomly distributed and stored in the specified n nodes.

[0063] Furthermore, block data storage refers to distributing and storing the coded blocks obtained by the block coding step in each node in the consensus unit, so that each node in the consensus unit can collaboratively store the coded blocks. The present invention proposes a random coded block placement strategy based on node performance. Now let's abstract the problem of selecting the node set for coded block placement: find a node set S to store block B (i) There are n coding blocks such that S∈M and |S|=n, where M={1,2,…,m} represents a consensus unit with m nodes and satisfies the condition m>n.

[0064] The coding block placement node set selection strategy process is as follows Figure 4 shown.

[0065] First, add the k nodes that reach the threshold T in the consensus unit to the candidate node set N. Then, the k nodes in the set N are determined by the Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS) for performance index, and are sorted in order from high to low according to the performance response feedback. Finally, the first n nodes in the sorting result are selected as the coding blocks to place the node set S. Since the performance of the node changes dynamically with the network, this selection method can avoid the storage of the coding block on the node with extremely poor performance due to the completely random storage of the shards. Therefore, each data distribution ensures that the coding block is stored on the node with better performance to the greatest extent, making full use of high-performance nodes.

[0066] Furthermore, block reconstruction can be understood as the reverse process of block encoding. The reconstruction principle is: the node reconstructs the block by reading multiple available encoding blocks in the consensus unit during the block recovery phase. The encoding block placement node set selection strategy is shown in Table 2.

[0067] Table 2

[0068]

[0069]

[0070] The specific steps are as follows:

[0071] (1) Fault detection stage

[0072] The system needs to regularly or in real time check whether the data blocks on each node of the consensus unit are complete. If data loss or damage is found, the block reconstruction process is triggered.

[0073] (2) Recovery node selection phase

[0074] The system needs to select one or more nodes as recovery nodes according to certain strategies, that is, nodes used to store newly generated data blocks. The recovery node should have sufficient space, bandwidth and computing power, and try to maintain load balance with other online nodes in the consensus unit.

[0075] (3) Coded block reading stage

[0076] First, a request for obtaining a coded block is sent to all online nodes in the consensus unit. Then the system reads k or more coded blocks from n nodes in the consensus unit M, and the online nodes transmit the relevant coded blocks stored locally to the recovery node.

[0077] (4) Raw data decoding stage

[0078] Because RS erasure codes satisfy the MDS property, the original block B can be reconstructed when the recovery node successfully downloads more than k coded blocks. (i) .

[0079] (5) New data block encoding stage

[0080] The recovery node needs to re-encode the newly reconstructed block according to the RS erasure code algorithm and store the encoded block in the set of encoding block placement nodes.

[0081] (6) Metadata update phase

[0082] The system needs to update metadata information, record the location, size and other information of the newly generated coding blocks, and delete lost or damaged coding blocks.

[0083] Furthermore, when a new node is first added to the BSO-EC blockchain system, the block height of the node will usually lag behind other nodes in the network. At this time, the node status must be updated to the latest through block data synchronization. The block data synchronization process is as follows: Figure 5 shown.

[0084] from Figure 5 It can be seen that when a new node joins the BSO-EC blockchain system for the first time, it is added to a consensus unit through a dynamic close-proximity grouping algorithm. Then the new node sends a broadcast request to the rest of the nodes in the group to obtain the maximum block height of the blockchain in the current state. After that, the new node sequentially reconstructs the block data from the genesis block to the maximum block height block, and temporarily stores these complete blocks locally for T time. After T time is up, the block data storage process is repeated, and the performance of the node is calculated based on TOPSIS. If the new node performance decision index is ranked high, some coding blocks can be retained.

[0085] Embodiment 2

[0086] like Figure 6 As shown, the present invention provides a blockchain storage optimization system based on erasure code, including: an acquisition module, a grouping module and a processing module;

[0087] The acquisition module is used to acquire blockchain network nodes, each of which stores a global and complete distributed ledger;

[0088] The grouping module is used to group blockchain network nodes using a dynamic close-distance node grouping algorithm based on the fuzzy c-means clustering algorithm to obtain consensus units;

[0089] The processing module is used to allocate storage for coding blocks based on the consensus unit and to perform global processing on block files using a random coding block placement strategy based on node performance.

[0090] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A blockchain storage optimization method based on erasure coding, characterized in that: The following steps are involved: Obtaining blockchain network nodes, each of which stores a global and complete distributed ledger; The blockchain network nodes are grouped using a dynamic close-distance node grouping algorithm based on a fuzzy c-means clustering algorithm to obtain consensus units; The method of grouping the blockchain network nodes using a dynamic close-distance node grouping algorithm based on a fuzzy c-means clustering algorithm includes: Select the central node; Based on the fuzzy c-means clustering algorithm, the central node sends a request message to each participating node, and the participating node returns a response message to the central node; The distance between two points is estimated according to the round-trip delay of the request and response messages, and the participating nodes are added to the group that meets the preset requirements according to the estimated distance; Based on the consensus unit, a random coding block placement strategy based on node performance is adopted to allocate storage for coding blocks and globally process block files; The method of global processing of block files includes: block encoding, block data storage, block reconstruction and block data synchronization; Block encoding is the process of encoding blocks using RS erasure code; block data storage is the process of randomly storing coded blocks in a specified set of nodes; block reconstruction is the reverse process of block encoding, which realizes the recovery of block files; block data synchronization is the process of synchronizing the latest block data to new nodes when new nodes join.

2. The blockchain storage optimization method based on erasure coding according to claim 1 is characterized in that: In block encoding, the specific process of using RS erasure code encoding to process blocks includes: Let the i-th block in the blockchain be B (i) ; In block B (i) Before slicing, if block B (i) If k equal-size partitions are not supported, fill block B with 0. (i) ; If block B (i) Support k equal-sized splits, then use the node to split block B (i) Divide into k equal-sized file blocks Block B (i) Then use RS(n,k) erasure code to generate n coded blocks The n coding blocks are randomly distributed and stored in designated n nodes for storage.

3. The blockchain storage optimization method based on erasure coding according to claim 1 is characterized in that: In the block data storage, the specific process of randomly storing the coded block to the specified node set includes: Add k nodes that reach the threshold T in the consensus unit to the candidate node set N; The k nodes in the set N are sorted in descending order based on the performance response feedback using the superior and inferior solution distance method to make performance index decisions; The first n nodes in the sorting result are selected as the coding block to place the node set S.

4. The blockchain storage optimization method based on erasure coding according to claim 1 is characterized in that: In the block reconstruction, the reverse process of block encoding, the specific process of realizing the recovery of block files includes: fault detection stage, recovery node selection stage, coding block reading stage, original data decoding stage, new data block encoding stage and metadata update stage.

5. The blockchain storage optimization method based on erasure coding according to claim 1 is characterized in that: In block data synchronization, when a new node joins, the specific process of synchronizing the latest block data to the new node includes: When a new node joins the blockchain system for the first time, the new node is added to a consensus unit through a dynamic proximity grouping algorithm; The new node sends a broadcast request to the other nodes in the corresponding consensus unit to obtain the maximum block height of the blockchain in the current state; The new node sequentially reconstructs the block data from the genesis block to the maximum block high block interval, and temporarily stores T time complete blocks locally; After time T is up, the block data storage process is repeated. The performance of the nodes is calculated based on TOPSIS. If the ranking of the new node performance decision index meets the preset requirements, some coding blocks are retained.

6. A blockchain storage optimization system based on erasure coding, characterized in that: include: Acquisition module, grouping module and processing module; The acquisition module is used to acquire blockchain network nodes, each of which stores a global and complete distributed ledger; The grouping module is used to group the blockchain network nodes using a dynamic close-distance node grouping algorithm based on a fuzzy c-means clustering algorithm to obtain consensus units; The method of grouping the blockchain network nodes using a dynamic close-distance node grouping algorithm based on a fuzzy c-means clustering algorithm includes: Select the central node; Based on the fuzzy c-means clustering algorithm, the central node sends a request message to each participating node, and the participating node returns a response message to the central node; The distance between two points is estimated according to the round-trip delay of the request and response messages, and the participating nodes are added to the group that meets the preset requirements according to the estimated distance; The processing module is used to allocate storage for coding blocks based on the consensus unit and adopt a random coding block placement strategy based on node performance to perform global processing on block files; The method of global processing of block files includes: block encoding, block data storage, block reconstruction and block data synchronization; Block encoding is the process of encoding blocks using RS erasure code; block data storage is the process of randomly storing coded blocks in a specified set of nodes; block reconstruction is the reverse process of block encoding, which realizes the recovery of block files; block data synchronization is the process of synchronizing the latest block data to new nodes when new nodes join.

Images