Method and system for blockchain storage optimization based on erasure coding

Abstract

Disclosed in the present invention are a method and system for blockchain storage optimization based on erasure coding. The method comprises the following steps: acquiring blockchain network nodes; grouping the blockchain network nodes by using a dynamic short-distance node grouping algorithm based on a fuzzy C-means clustering algorithm so as to obtain a consensus unit; and on the basis of the consensus unit, performing storage allocation on coding blocks by using a random coding block placement policy based on node performance, and performing global processing on a block file. According to the present invention, the storage load of a single node is reduced, and the problem of loss of features such as decentralization, data security, data authenticity, and data credibility of the blockchain due to heavy storage load of nodes is mitigated.

Description

A blockchain storage optimization method and system based on erasure code Technical Field

[0001] The present invention belongs to the fields of blockchain technology and erasure code technology, and specifically relates to a blockchain storage optimization method and system based on erasure code. Background Art

[0002] Blockchain is a record-keeping technology that stores transactions and valuable information in blocks. Blockchain possesses key characteristics that distinguish it from centralized systems, such as decentralization, immutability, security, and anonymity. This has led to its successful application in areas such as fintech, public governance, digital currency, and supply chains.

[0003] Erasure coding was first applied in distributed storage systems, aiming to address data recovery issues in the event of failures. Erasure coding storage redundancy strategies ensure that distributed systems maintain the same reliability while minimizing redundant storage overhead. Blockchain systems are essentially specialized distributed systems, and erasure coding can theoretically be applied to them. Pal Rudrashhish et al. used digital fountain codes to address blockchain replica redundancy, improving system storage performance. Zhao Guofeng et al. proposed a block file storage model for blockchain systems based on erasure coding, improving the storage performance of Hyperledger. However, this model is strictly limited in its applicability and can only be applied to blockchain platforms under the consortium Hyperledger blockchain. Perard proposed a low-storage blockchain node based on erasure coding, focusing on nodes with limited storage space. While this solution has certain advantages in reducing blockchain storage consumption, it also reduces the level of decentralization of the blockchain system. Qi et al. combined BFT consensus and erasure coding technologies to design a BFT-Store storage engine. While this design achieves blockchain scalability, it suffers from poor usability.

[0004] Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention proposes a blockchain storage optimization method and system based on erasure codes. It adopts a fuzzy c-means clustering algorithm and a coding block placement node set selection strategy to reduce the storage burden of a single node and improve the problems of blockchain losing centralization, data security, and data authenticity due to the high node storage pressure when using a full copy mechanism to achieve the consistency of distributed ledgers.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A blockchain storage optimization method based on erasure coding includes the following steps:

[0008] Obtaining blockchain network nodes, each of which stores a global and complete distributed ledger;

[0009] The blockchain network nodes are grouped using a dynamic close-distance node grouping algorithm based on a fuzzy c-means clustering algorithm to obtain consensus units;

[0010] Based on the consensus unit, a random coding block placement strategy based on node performance is adopted to allocate storage for coding blocks and perform global processing on block files.

[0011] Preferably, the method for grouping the blockchain network nodes using a dynamic close-range node grouping algorithm based on a fuzzy c-means clustering algorithm includes:

[0012] Select the central node;

[0013] Based on the fuzzy c-means clustering algorithm, the central node sends a request message to each participating node, and the participating node returns a response message to the central node;

[0014] The distance between two points is estimated based on the round trip delay of the request and response messages, and the participating nodes are added to the group that meets the preset requirements based on the estimated distance.

[0015] Preferably, the method for globally processing the block file includes: block encoding, block data storage, block reconstruction and block data synchronization;

[0016] Block encoding is the process of encoding blocks using RS erasure code; block data storage is the process of randomly storing encoded blocks in a specified set of nodes; block reconstruction is the reverse process of block encoding, which realizes the recovery of block files; block data synchronization is the process of synchronizing the latest block data to new nodes when new nodes join.

[0017] Preferably, in block encoding, the specific process of using RS erasure code encoding processing on the block includes:

[0018] Let the i-th block in the blockchain be B (i) ;

[0019] In block B (i) Before slicing, if block B (i) If k equal-size splits are not supported, fill block B with 0. (i) ;

[0020] If block B (i) Support k equal-sized splits, then use the node to split block B (i) Split into k equal-sized file blocks Block B (i)Then use RS(n,k) erasure code to generate n coding blocks The n coding blocks are randomly distributed and stored in n designated nodes.

[0021] Preferably, in the block data storage, the specific process of randomly storing the coded blocks in the designated node set includes:

[0022] Add k nodes that reach the threshold T in the consensus unit to the candidate node set N;

[0023] The k nodes in the set N are sorted in descending order based on the performance response feedback using the superior and inferior solution distance method to determine the performance index.

[0024] The first n nodes in the sorting result are selected as the coding block to place the node set S.

[0025] Preferably, in the block reconstruction, the reverse process of block encoding, and the specific process of realizing the recovery of the block file include: fault detection stage, recovery node selection stage, coding block reading stage, original data decoding stage, new data block encoding stage and metadata update stage.

[0026] Preferably, during block data synchronization, when a new node joins, the specific process of synchronizing the latest block data to the new node includes:

[0027] When a new node joins the blockchain system for the first time, it is added to a consensus unit through a dynamic proximity grouping algorithm;

[0028] The new node sends a broadcast request to the other nodes in the corresponding consensus unit to obtain the maximum block height of the blockchain in the current state;

[0029] The new node sequentially reconstructs the block data from the genesis block to the maximum block high block, and temporarily stores T complete blocks locally;

[0030] After time T is up, the block data storage process is repeated. The performance of the nodes is calculated based on TOPSIS. If the ranking of the new node performance decision index meets the preset requirements, some coding blocks are retained.

[0031] The present invention also provides a blockchain storage optimization system based on erasure codes, comprising: an acquisition module, a grouping module, and a processing module;

[0032] The acquisition module is used to acquire blockchain network nodes, each of which stores a global and complete distributed ledger;

[0033] The grouping module is used to group the blockchain network nodes using a dynamic close-distance node grouping algorithm based on a fuzzy c-means clustering algorithm to obtain consensus units;

[0034] The processing module is used to allocate storage for coding blocks based on the consensus unit and adopt a random coding block placement strategy based on node performance to perform global processing on block files.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention improves both the block encoding speed and the block reconstruction speed, while reducing system space occupation.

[0037] (1) Block encoding speed

[0038] In different RS encoding schemes, when the block size is the same, the lower the fault tolerance, the faster the encoding speed; in different RS encoding schemes, when the block size and fault tolerance are the same, the encoding speed under different encoding combinations also varies to some extent; in the same RS encoding scheme, as the block file size increases, the encoding speed also increases.

[0039] (2) Block reconstruction speed

[0040] Within the fault tolerance range of RS erasure codes, the fixed block size remains unchanged. The fewer the number of coding blocks lost by the system, the faster the block reconstruction speed. Within the fault tolerance range of RS erasure codes, the number of fixed data blocks lost remains unchanged. The higher the fault tolerance rate, the faster the reconstruction speed.

[0041] (3) Comparison of system storage space usage

[0042] As the number of system nodes increases, the storage space occupied by Hyperledger Fabric increases proportionally to the number of system nodes, while the number of blocks remains unchanged. Compared to Hyperledger Fabric, the BSO-EC proposed in this article can achieve lower storage consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] FIG1 is a diagram showing a detailed operation mechanism of BSO-EC in an embodiment of the present invention;

[0045] FIG2 is a flow chart of a blockchain storage optimization method based on erasure coding in an embodiment of the present invention;

[0046] FIG3 is a block coding principle diagram in an embodiment of the present invention;

[0047] FIG4 is a diagram of a coding block placement node set selection strategy in an embodiment of the present invention;

[0048] FIG5 is a flow chart of BSO-EC data synchronization in an embodiment of the present invention;

[0049] FIG6 is an overall framework diagram of a blockchain storage optimization system based on erasure codes in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Example 1

[0053] This invention provides a blockchain storage optimization method based on erasure codes (BSO-EC). The detailed operation mechanism is shown in Figure 1. The steps are as follows:

[0054] Step 1: System Initialization. During the initial operation phase of the BSO-EC system, each participating node in the network stores a global and complete distributed ledger, which facilitates subsequent block encoding operations and block data synchronization, among other block processing procedures.

[0055] Step 2: Consensus Unit Generation. Nodes in the BSO-EC network establish logical connections based on landmark clustering. Subsequently, nodes across the network are grouped using a dynamic proximity node grouping algorithm. Each subgroup is called a consensus unit. This grouping facilitates structured management between subgroups, reduces inter-node interaction latency, and improves communication efficiency.

[0056] Step 3: Block Processing. This step primarily includes block encoding, block storage, block reconstruction, and block synchronization. Block encoding uses the RS encoding scheme to encode the complete block file. Block storage involves distributing the encoded blocks across participating nodes in the consensus unit. Block reconstruction involves recovering the original block by obtaining the available encoded blocks in the event of data corruption on some nodes. Block synchronization involves updating the data and status of nodes participating in the BSO-EC to the latest version.

[0057] Specifically, as shown in Figure 2, BSO-EC first groups nodes using a dynamic proximity node grouping algorithm based on the fuzzy c-means clustering algorithm. Blockchain network node grouping not only facilitates structured management of subnetworks, but also reduces inter-node interaction latency, improving communication efficiency. The dynamic proximity node grouping algorithm, as shown in Table 1, first selects a central node and then performs topology optimization around it using the fuzzy c-means clustering algorithm. The central node sends a request message to each participating node, which then returns a response message to the central node. The distance between the two nodes is estimated based on the round-trip latency of the request and response messages. Nodes are then assigned to appropriate groups based on the estimated distance. The dynamic proximity node grouping algorithm ensures that the latency between nodes in the same group is significantly lower than the latency between nodes in different groups.

[0058] Table 1

[0059] Then, a strategy for selecting a node set for code block placement is proposed to allocate code block storage and perform global block file processing. The global block file processing process consists of block encoding, block data storage, block reconstruction, and block data synchronization. Block encoding involves encoding the block using RS erasure code. Block data storage involves randomly storing the code block on a designated set of nodes. Block reconstruction is the reverse process of block encoding and enables block file recovery. Block data synchronization involves synchronizing the latest block data to new nodes when they join the BSO-EC. A random code block placement strategy based on node performance effectively avoids the significant instability of random selection strategies and maximizes the computing and storage capabilities of the node group. During the election of regular participating nodes as candidate nodes, a random number generation mechanism based on digital signatures and hash algorithms effectively ensures the fairness, impartiality, openness, and transparency of the election process.

[0060] In this embodiment, the global processing flow of the block file includes four steps: block encoding, block data storage, block reconstruction, and block data synchronization.

[0061] Furthermore, the i-th block in the blockchain is denoted as B (i) Then B(0) This is the first block, also known as the genesis block. The RS erasure code encoding mechanism in distributed storage systems is still applicable to the block encoding of the BSO-EC blockchain system described in this paper. The specific block encoding implementation is shown in Figure 3.

[0062] In block B (i) Before slicing, if block B (i) Does not support k-partitioning with equal size. You need to fill block B with 0 first. (i) , and then continue with the encoding related operations. First, the node will block B (i) Split into k equal-sized file blocks Then, block B (i) Then use RS(n,k) erasure code to generate n coding blocks The last n coding blocks are randomly distributed and stored on the specified n nodes.

[0063] Furthermore, block data storage refers to distributing and storing the coded blocks obtained by the block encoding step in each node in the consensus unit, so that each node in the consensus unit can collaboratively store the coded blocks. The present invention proposes a random coded block placement strategy based on node performance. Now let's abstract the problem of selecting the node set for coded block placement: find a node set S to store block B. (i) n coding blocks such that S∈M and |S|=n, where M={1,2,…,m} represents a consensus unit with m nodes and satisfies the condition m>n.

[0064] The coding block placement node set selection strategy process is shown in Figure 4.

[0065] First, k nodes that reach the threshold T in the consensus unit are added to the candidate node set N. Then, the k nodes in the set N are subjected to the Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS) method to make performance indicator decisions, and are sorted in order from high to low based on performance response feedback. Finally, the top n nodes in the sorting result are selected as coding blocks and placed in the node set S. Since the performance of the node changes dynamically with the network, this selection method can avoid the situation where the coding block is stored on a node with extremely poor performance due to completely random storage of the shards. Therefore, each data distribution ensures that the coding block is stored on a node with better performance to the greatest extent possible, making full use of high-performance nodes.

[0066] Furthermore, block reconstruction can be understood as the reverse process of block encoding. The reconstruction principle is as follows: During the block recovery phase, nodes reconstruct blocks by reading multiple available encoding blocks within the consensus unit. The strategy for selecting the node set for encoding block placement is shown in Table 2.

[0067] Table 2

[0068] The specific steps are as follows:

[0069] (1) Fault detection stage

[0070] The system needs to regularly or in real time check whether the data blocks on each node of the consensus unit are complete. If data loss or damage is found, the block reconstruction process is triggered.

[0071] (2) Recovery node selection phase

[0072] The system needs to select one or more nodes as recovery nodes based on a certain strategy. This is the node used to store newly generated data blocks. The recovery node should have sufficient space, bandwidth, and computing power, and try to maintain load balancing with other online nodes in the consensus unit.

[0073] (3) Coded block reading stage

[0074] First, a code block acquisition request is sent to all online nodes in the consensus unit. Then the system reads k or more code blocks from n nodes in the consensus unit M. The online nodes transmit the relevant code blocks stored locally to the recovery node.

[0075] (4) Original data decoding stage

[0076] Because RS erasure codes satisfy the MDS property, the original block B can be reconstructed when the recovery node successfully downloads more than k coded blocks. (i) .

[0077] (5) New data block encoding stage

[0078] The recovery node needs to re-encode the newly reconstructed block according to the RS erasure code algorithm and store the encoded block in the encoding block placement node set.

[0079] (6) Metadata update phase

[0080] The system needs to update metadata information, record the location, size and other information of newly generated coding blocks, and delete lost or damaged coding blocks.

[0081] Furthermore, when a new node first joins the BSO-EC blockchain system, its block height will usually lag behind other nodes in the network. At this time, it is necessary to update the node status to the latest through block data synchronization. The block data synchronization process is shown in Figure 5.

[0082] As shown in Figure 5, when a new node first joins the BSO-EC blockchain system, it is added to a consensus unit using a dynamic proximity grouping algorithm. The new node then broadcasts a request to the remaining nodes in the group to obtain the maximum block height of the blockchain in its current state. The new node then sequentially reconstructs the block data from the genesis block to the maximum block height block, temporarily storing these complete blocks locally for a period of time T. After T time has expired, the block data storage process repeats. Based on the TOPSIS calculation of node performance, the new node that ranks high on the performance decision metric is allowed to retain some of the coded blocks.

[0083] Example 2

[0084] As shown in FIG6 , the present invention provides a blockchain storage optimization system based on erasure coding, comprising: an acquisition module, a grouping module, and a processing module;

[0085] The acquisition module is used to obtain blockchain network nodes, each of which stores a global and complete distributed ledger;

[0086] The grouping module is used to group blockchain network nodes using a dynamic close-range node grouping algorithm based on the fuzzy c-means clustering algorithm to obtain consensus units;

[0087] The processing module is used to allocate storage for coding blocks based on the consensus unit and perform global processing on block files using a random coding block placement strategy based on node performance.

[0088] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A blockchain storage optimization method based on erasure coding, characterized in that: The following steps are involved: Obtaining blockchain network nodes, each of which stores a global and complete distributed ledger; The blockchain network nodes are grouped using a dynamic close-distance node grouping algorithm based on a fuzzy c-means clustering algorithm to obtain consensus units; Based on the consensus unit, a random coding block placement strategy based on node performance is adopted to allocate storage for coding blocks and perform global processing on block files.

2. The blockchain storage optimization method based on erasure coding according to claim 1 is characterized in that: The method of grouping the blockchain network nodes using a dynamic close-distance node grouping algorithm based on a fuzzy c-means clustering algorithm includes: Select the central node; Based on the fuzzy c-means clustering algorithm, the central node sends a request message to each participating node, and the participating node returns a response message to the central node; The distance between two points is estimated according to the round trip delay of the request and response messages, and the participating nodes are added to the group that meets the preset requirements according to the estimated distance.

3. The blockchain storage optimization method based on erasure coding according to claim 1 is characterized in that: The method of global processing of block files includes: block encoding, block data storage, block reconstruction and block data synchronization; Block encoding is the process of encoding blocks using RS erasure code; block data storage is the process of randomly storing coded blocks in a specified set of nodes; block reconstruction is the reverse process of block encoding, which realizes the recovery of block files; block data synchronization is the process of synchronizing the latest block data to new nodes when new nodes join.

4. The blockchain storage optimization method based on erasure coding according to claim 3 is characterized in that: In block encoding, the specific process of using RS erasure code encoding to process blocks includes: Let the i-th block in the blockchain be B (i) ; In block B (i) Before slicing, if block B (i) If k equal-size partitions are not supported, fill block B with 0. (i) ; If block B (i) Support k equal-sized splits, then use the node to split block B (i) Divide into k equal-sized file blocks Block B (i) Then use RS(n,k) erasure code to generate n coded blocks The n coding blocks are randomly distributed and stored in designated n nodes for storage.

5. The blockchain storage optimization method based on erasure coding according to claim 3 is characterized in that: In the block data storage, the specific process of randomly storing the coded block to the specified node set includes: Add k nodes that reach the threshold T in the consensus unit to the candidate node set N; The k nodes in the set N are sorted in descending order according to the performance response feedback by using the superior and inferior solution distance method to make performance index decisions; The first n nodes in the sorting result are selected as the coding block to place the node set S.

6. The blockchain storage optimization method based on erasure coding according to claim 3 is characterized in that: In the block reconstruction, the reverse process of block encoding, the specific process of realizing the recovery of block files includes: fault detection stage, recovery node selection stage, coding block reading stage, original data decoding stage, new data block encoding stage and metadata update stage.

7. The blockchain storage optimization method based on erasure coding according to claim 3 is characterized in that: In block data synchronization, when a new node joins, the specific process of synchronizing the latest block data to the new node includes: When a new node joins the blockchain system for the first time, the new node is added to a consensus unit through a dynamic proximity grouping algorithm; The new node sends a broadcast request to the other nodes in the corresponding consensus unit to obtain the maximum block height of the blockchain in the current state; The new node sequentially reconstructs the block data from the genesis block to the maximum block high block interval, and temporarily stores T time complete blocks locally; After time T is up, the block data storage process is repeated, and the performance of the nodes is calculated based on TOPSIS. If the ranking of the new node performance decision index meets the preset requirements, some coding blocks are retained.

8. A blockchain storage optimization system based on erasure coding, characterized in that: include: Acquisition module, grouping module and processing module; The acquisition module is used to acquire blockchain network nodes, each of which stores a global and complete distributed ledger; The grouping module is used to group the blockchain network nodes using a dynamic close-distance node grouping algorithm based on a fuzzy c-means clustering algorithm to obtain consensus units; The processing module is used to allocate storage for coding blocks based on the consensus unit and adopt a random coding block placement strategy based on node performance to perform global processing on block files.

Images