A highly reliable and low-overhead data storage system for blockchain log storage

Abstract

The present invention discloses a highly reliable and low-overhead data storage system and method for blockchain log storage. The method includes the following steps: dividing the log file to be stored into two storage stages, and the two storage stages include: a preparation stage and a submission stage; storing the log file in stages into the blockchain log storage system. By dividing the submission of the log file into a preparation stage and a submission stage to complete the storage of the log file, the present invention can effectively reduce the storage overhead and expand the storage capacity of the system by reusing the physical nodes where the blockchain system is located as off-chain storage nodes without adding additional machines. At the same time, combined with the redundancy provided by erasure coding, it can avoid data being deleted or tampered with by Byzantine nodes.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed storage, and more specifically, to a highly reliable and low-overhead data storage system and method for blockchain log storage. Background Art

[0002] Blockchain is a technology that combines methods such as distributed consensus, encryption, and timestamping to achieve peer-to-peer transactions, coordination, and collaboration without relying on any third-party centralized institution. However, blockchain has the problem of excessive storage overhead, which has become a bottleneck for the implementation of blockchain applications. To address the problem of excessive storage overhead in blockchain, the existing solutions can be divided into on-chain storage and off-chain storage. On-chain storage stores data on the blockchain without the need for an additional off-chain storage system. Each node only needs to store the corresponding data according to a pre-set rule, thereby reducing the storage overhead. On-chain storage can be divided into a collaborative storage mode and a lightweight node mode. Off-chain storage transfers the data in the block body to a storage system outside the blockchain. At this time, the blockchain only stores non-data information such as pointers to these data. The storage space occupied by non-data information is small, so the scalability problem of blockchain storage can be solved. The original data is stored in a non-blockchain system, and at the same time, a unique identifier of the data is generated according to a certain rule and returned to the blockchain system. When accessing the complete data, the original data is searched for in the non-blockchain system through the unique identifier of the data.

[0003] The prior art discloses a method, system, and device for data access and storage. When storing data, first determine a data storage instruction, and then, according to the identity identifier carried in the data storage instruction, determine the corresponding blockchain and key pair. Finally, according to the key pair, store the data to be stored in the blockchain. When querying data, first determine a data query instruction, and then, according to the identity identifier corresponding to the data query instruction, determine the corresponding blockchain and private key. Finally, according to the private key, decrypt and query the data in the blockchain. It can be seen that through the method provided by the embodiments of the present application, when accessing and storing data corresponding to the identity identifier, there is no need to access multiple databases. Only by accessing the blockchain corresponding to the identity identifier and storing data through a key pair, the complexity of the operation can be simplified while ensuring data security, and the efficiency of data access and storage can be improved. This solution stores data in the blockchain in the form of full replicas. When a blockchain node fails, any node that has not failed can continue to work, and the failed blockchain node can be repaired through a normal node. However, on-chain storage in the form of full replicas has the problem of excessive storage overhead. In a system with n nodes, each piece of data will be stored n times. Storing cold data that is rarely accessed (such as log records, etc.) in this way will cause excessive storage space waste. Summary of the Invention

[0004] To overcome the defects of high cost and easy waste of storage resources in the existing log storage solutions, the present invention provides a highly reliable and low-cost data storage system and method for blockchain log storage.

[0005] The primary object of the present invention is to solve the above technical problems, and the technical solution of the present invention is as follows:

[0006] In the first aspect of the present invention, a highly reliable and low-cost data storage system for blockchain log storage is provided, including: a log client and a log consortium chain node. The log client is communicatively connected to the log consortium chain node through an RPC interface. Among them, the log client is used to encode a log file to obtain encoded blocks, and the log consortium chain node is used to store the encoded blocks and the metadata of the log file.

[0007] Furthermore, the log client includes: an erasure code engine, a blockchain light client, and a first RPC interface. The erasure code engine is used to perform erasure code encoding on the log file, the blockchain light client is used to verify the existence of a transaction, and the first RPC interface is used for data interaction with the log consortium chain node.

[0008] Furthermore, the process of the erasure code engine encoding the log file is as follows:

[0009] Each log file is split into n - 2f data blocks, and 2f parity blocks are generated through (n - 2f, 2f)-RS encoding; the n - 2f data blocks and the 2f parity blocks are collectively referred to as encoded blocks. Subsequently, the n - 2f encoded blocks are uploaded to the consortium chain node, where n represents the total number of nodes and f represents the number of malicious nodes that can be tolerated.

[0010] Furthermore, the block header in the blockchain light client is synchronized with the full node, and the Merkle root is stored in the block header. When the light client queries a certain transaction from the full node, the full node will return this transaction and the Merkle proof of the transaction. The existence of this transaction can be verified through the Merkle proof. Further, the log consortium chain node includes: a log storage layer, a blockchain layer, and a second RPC interface. The log storage layer is used to store the encoded blocks, the blockchain layer is used to store the metadata of the log, and the second RPC interface is used for data interaction with the log client; among them, the blockchain layer includes: a PBFT consensus engine, a transaction execution module, and a P2P network. The PBFT consensus engine is used to execute the existing PBFT consensus algorithm, the transaction execution module is used to collect and package transactions, and the P2P network is a peer-to-peer network composed of distributed and independent blockchain nodes in the blockchain.

[0011] The second aspect of the present invention provides a highly reliable and low-overhead data storage method for blockchain log storage. The method is applied to the highly reliable and low-overhead data storage system for blockchain log storage, and is characterized by including the following steps:

[0012] Divide the log file to be stored into two storage stages, and the two storage stages include: a preparation stage and a submission stage;

[0013] Deposit the log file into the blockchain log storage system in stages.

[0014] Furthermore, in the preparation stage, the blockchain protocol does not work, and the blockchain server acts as a storage server. The client sends encoded blocks to the server and saves the signature returned by the server;

[0015] In the submission stage, the client sends a submission request to the blockchain and attaches the signature returned by the server in the preparation stage to the request. The blockchain system verifies the signature to ensure that the system has saved enough encoded blocks.

[0016] Furthermore,

[0017] In the two storage stages of the log file, the consortium chain nodes play two different roles in the two stages. In the first stage, the consortium chain is logically a storage node off the chain and runs the protocol of the log storage layer; after the log client encodes the log file and uploads it to the consortium chain node, the consortium chain node will check the encoded block and store it in the KV database on the node. When the storage is successful, the node will sign the encoded block with its own private key, and the signature will be returned and saved in the blockchain to prove that the node has successfully saved this log encoded block.

[0018] Furthermore, when the log client collects more than n - f signatures, it indicates that at least n - 2f non-malicious nodes have successfully saved the log encoded block at this time. At this time, the log client can initiate a transaction and upload the signature and other log file metadata to the log consortium chain in the form of a transaction. The blockchain node will check the submitted transaction, package it and add it to the blockchain after the check, and there will also be a status database to index the log file;

[0019] When the user queries a specific log through the client, first obtain the metadata from any blockchain, and then obtain n - 2f correct encoded blocks through the log storage layer and decode and recover the log at the client.

[0020] Furthermore, the specific steps of the client log file storage process are as follows:

[0021] A1: Take a single log file as an input parameter, encode the data of the log file using an erasure coding function to obtain the same number of encoded blocks as the number of clusters, and calculate the checksum through hash operation;

[0022] A2: The consortium chain nodes Ni are sorted from 1 to n according to the public key. The client sends the corresponding encoded block Ci, the checksum of the log file, and the checksum of the encoded block Ci to the consortium chain node Ni, where n represents the number of clusters;

[0023] A3: The consortium chain node checks whether the encoded block is consistent with the checksum; and ensures that the log file corresponding to the encoded block has not been submitted;

[0024] A4: Define key-value pairs, use the checksum of the log file in the encoded block as the key, store the encoded block on the local object data of the node, and use the data associated with the log file as the value. The value includes the data itself of the encoded block file, the checksum information, and the metadata covering the client information, log timestamp, and application path information. When the value of the metadata is empty, it means that the log file is in the preparation stage and has not been submitted;

[0025] A5: The consortium chain node returns the signature of the checksum information of the log file and the checksum information of the encoded file to indicate that the encoded block has been received and saved. The client will submit the signature in the next stage as a voucher for the normal reception of the node;

[0026] A6: Detect whether the client has collected more than n - 2f ready messages returned by the consortium chain nodes. If so, wake up the main thread;

[0027] A7 The client submits the log file metadata and the signature returned by the consortium chain node as a transaction.

[0028] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0029] The present invention completes the storage of the log file by dividing the submission of the log file into a preparation stage and a submission stage. Without adding additional machines, by reusing the physical nodes where the blockchain system is located as off-chain storage nodes, it can effectively reduce the storage overhead, expand the storage capacity of the system, and at the same time combine the redundancy provided by erasure coding to avoid data being deleted or tampered with by Byzantine nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a block diagram of a highly reliable and low-overhead data storage system for blockchain log storage according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0031] To more clearly understand the above objects, features, and advantages of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0032] In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0033] Embodiment 1

[0034] As Figure 1 shown, the present invention provides a highly reliable and low-overhead data storage system for blockchain log storage, including: a log client and a log consortium chain node. The log client is communicatively connected to the log consortium chain node through an RPC interface. Among them, the log client is used to encode a log file to obtain encoded blocks, and the log consortium chain node is used to store the encoded blocks and the metadata of the log file.

[0035] In the embodiment of the present invention, the log file is encoded into encoded blocks after encoding by the client, and the encoded blocks are uploaded to the log consortium node for storage; when enough nodes return messages indicating successful storage, the log metadata is sent as a transaction to the blockchain; when reading data, the client sends a request to the consortium chain node to read the encoded blocks and restores the log file at the client.

[0036] Further, the log client includes: an erasure code engine, a blockchain light client, and a first RPC interface. Among them, the erasure code engine is used to perform erasure code encoding on the log file, the blockchain light client is used to verify the existence of a transaction, and the first RPC interface is used for data interaction with the log consortium chain node.

[0037] Further, the process of the erasure code engine encoding the log file is as follows:

[0038] Each log file is split into n - 2f data blocks, and 2f parity blocks are generated through (n - 2f, 2f)-RS encoding; the n - 2f data blocks and the 2f parity blocks are collectively referred to as encoded blocks. Subsequently, the n - 2f encoded blocks are uploaded to the consortium chain node, where n represents the total number of nodes, and f represents the number of malicious nodes that can be tolerated.

[0039] It should be noted that before submitting log metadata to the blockchain in the form of a transaction, the client will save the log file to off-chain storage. In the embodiments of the present invention, in order to increase the reliability of off-chain storage, the peer nodes in the consortium blockchain are used as off-chain storage nodes, and the log file is written in the form of erasure code encoded blocks. The log chain of the new domain name resolution system uses PBFT as the consensus protocol. A distributed system using PBFT can tolerate no more than 1 / 3 malicious nodes. When the total number of nodes is n, if f malicious nodes need to be tolerated, it is necessary to satisfy n≥3f + 1. When reading a piece of data successfully submitted through the PBFT protocol, it can only be guaranteed that the correct result can be read on n - 2f nodes. For example, assume that when submitting message m through the protocol, f malicious nodes disguise themselves as normal nodes. If exactly f non-malicious nodes in the system fail and go down at this time, the system can still complete the submission of message m and reach an agreement. At this time, message m is written into n - f nodes, where n - 2f are normal nodes and the other f nodes are malicious nodes. When reading message m, the malicious nodes can return error messages, so it can only be guaranteed that the correct information can be read from n - 2f nodes. In the present invention, the maximum number of Byzantine nodes f will be determined and configured before running, which determines the subsequent erasure code encoding mode used. The client encodes each log file using (n - 2f, 2f)-RS. Specifically, each log file is split into n - 2f data blocks, and 2f parity blocks are generated through (n - 2f, 2f)-RS encoding; the n - 2f data blocks and 2f parity blocks are collectively referred to as encoded blocks, and then the n - 2f encoded blocks are uploaded to the consortium blockchain nodes (saved off-chain); when reading, only n - 2f correct encoded blocks need to be obtained to recover the log file.

[0040] Furthermore, the block header in the blockchain light client is synchronized with the full node, and the Merkle root is saved in the block header. When the light client queries a certain transaction from the full node, the full node will return this transaction and the Merkle proof of the transaction. The existence of this transaction can be verified through the Merkle proof.

[0041] In the embodiments of the present invention, the uploading and downloading of logs are implemented in the client of the log blockchain. The client in the present invention interacts with the blockchain node through the light client protocol. Different from the full node, the light client does not participate in mining and does not store the block body, and the requirements for computing power and storage capacity are lower than those of the full node. The block header in the light client is synchronized with the full node, and the Merkle root is stored in the block header. When the light client queries a certain transaction from the full node, the full node will return this transaction and the Merkle proof of the transaction, and the existence of this transaction can be verified through the Merkle proof. The light client protocol enables users to participate in the blockchain at a very low cost. Since only the block header needs to be obtained during synchronization, the load on the blockchain network is also reduced.

[0042] In the present invention, the log consortium chain node includes: a log storage layer, a blockchain layer, and a second RPC interface. The log storage layer is used to store encoded blocks, the blockchain layer is used to store the metadata of the logs, and the second RPC interface is used for data interaction with the log client.

[0043] In the embodiments of the present invention, the blockchain layer includes: a PBFT consensus engine, a transaction execution module, and a P2P network. The PBFT consensus engine is used to execute the existing PBFT consensus algorithm, the transaction execution module is used to collect transactions and package transactions, and the P2P network is a peer-to-peer network composed of distributed and independent blockchain nodes in the blockchain.

[0044] Embodiment 2

[0045] The present invention also provides a highly reliable and low-overhead data storage method for blockchain log storage. The method is applied to the highly reliable and low-overhead data storage system for blockchain log storage, and includes the following steps:

[0046] The log file to be stored is divided into two storage stages, and the two storage stages include: a preparation stage and a submission stage;

[0047] The log file is stored in the blockchain log storage system in stages.

[0048] Further, in the preparation stage, the blockchain protocol does not work, the blockchain server acts as a storage server, the client sends the encoded block to the server, and saves the signature returned by the server;

[0049] In the submission stage, the client sends a submission request to the blockchain, and attaches the signature returned by the server in the preparation stage to the request. The blockchain system verifies the signature to ensure that the system has saved enough encoded blocks.

[0050] Further, during the two storage phases of the log file, the consortium blockchain nodes play two different roles in the two phases. In the first phase, the consortium blockchain logically serves as an off-chain storage node and runs the protocol of the log storage layer. After the log client encodes the log file and uploads it to the consortium blockchain node, the consortium blockchain node will check the encoded block and store it in the KV database on the node. After successful storage, the node will sign the encoded block using its private key, and the signature will be returned and saved in the blockchain to prove that the node has successfully saved this log encoded block.

[0051] Further, when the log client has collected more than n - f signatures, it indicates that at least n - 2f non-malicious nodes have successfully saved the log encoded block at this time. At this time, the log client can initiate a transaction to upload the signature and other log file metadata to the log consortium blockchain in the form of a transaction. The blockchain node will check the submitted transaction, and after completion of the check, it will be packaged and added to the blockchain. At the same time, there will be a state database to index the log file;

[0052] When a user queries a specific log through the client, the user first obtains the metadata from any blockchain, and then obtains n - 2f correctly encoded blocks through the log storage layer and decodes and restores the log at the client.

[0053] Embodiment 3

[0054] In the present invention, the specific steps for storing the client log file are as follows:

[0055] A1: Take a single log file as an input parameter, encode the data of the log file using an erasure coding function to obtain the same number of encoded blocks as the number of clusters, and calculate the checksum through a hash operation;

[0056] A2: Sort the consortium blockchain nodes Ni in ascending order of public keys from 1 to n. The client sends the corresponding encoded block Ci, the checksum of the log file, and the checksum of the encoded block Ci to the consortium blockchain node Ni, where n represents the number of clusters;

[0057] A3: The consortium blockchain node checks whether the encoded block is consistent with the checksum; and ensures that the log file corresponding to the encoded block has not been submitted before;

[0058] A4: Define a key-value pair, use the checksum of the log file in the encoded block as the key, and store the encoded block on the local object data of the node. Use the data associated with the log file as the value. The value includes the data itself of the encoded block (denoted as d), the checksum information (denoted as dc), and the metadata covering the client information, log timestamp, and application program path information (denoted as meta). When the value of the metadata is empty, it indicates that the log file is in the preparation stage and has not been submitted yet;

[0059] A5: The consortium chain node returns the signature of the file checksum of the encoded log file to indicate that it has received and saved the encoded block. The client will submit the signature in the next phase as a voucher for the normal reception by the node.

[0060] A6: Detect whether the client has collected more than n - 2f "ready" messages returned by the consortium chain nodes. If so, wake up the main thread.

[0061] A7: The client submits the log file metadata and the signature returned by the consortium chain node as a transaction.

[0062] It should be noted that in the submission phase, the submission message includes: log file metadata, log file checksum, and the signature returned by the consortium chain node in the preparation phase. The log metadata includes information such as client information, log time range, application program path, etc., which can be used to retrieve the log. The log file checksum and the signature returned by the consortium chain node can be used to verify the authenticity of the log file. More specifically, before the transaction submitted by the client enters the memory pool, the check_tx function will be called. The check_tx function checks the number of correctly signed nodes. After determining that more than 2f nodes have temporarily stored the encoded block, it will return SUCCESS, and the transaction will enter the memory pool and be submitted through the PBFT protocol. After the submission is completed and the block is packaged, each blockchain node will call the deliver_tx function. The deliver_tx function is used to update the state machine in the blockchain. The update of the state machine means writing the metadata of the log file. The writing of the metadata is to facilitate the client to query the corresponding log file through the metadata, and at the same time indicates that the log file has been successfully submitted.

[0063] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, rather than limiting the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation manners here. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A highly reliable and low - overhead data storage system for blockchain log storage, characterized in that, Including: A log client and a log consortium chain node. The log client and the log consortium chain node are communicatively connected through an RPC interface. Among them, the log client is used to encode a log file to obtain encoded blocks, and the log consortium chain node is used to store the encoded blocks and the metadata of the log file; The system adopts the following method, including: Dividing the log file to be stored into two storage stages, and the two storage stages include: a preparation stage and a submission stage; Storing the log file in the blockchain log storage system in stages; In the preparation stage, the blockchain protocol does not work. The blockchain server acts as a storage server. The client sends encoded blocks to the server and saves the signature returned by the server; In the submission stage, the client sends a submission request to the blockchain and attaches the signature returned by the server in the preparation stage to the request. The blockchain system verifies the signature to ensure that the system has saved enough encoded blocks.

2. The highly reliable and low - overhead data storage system for blockchain log storage according to claim 1, characterized in that, The log client includes: an erasure code engine, a blockchain light client, and a first RPC interface. Among them, the erasure code engine is used to perform erasure code encoding on the log file, the blockchain light client is used to verify the existence of a transaction, and the first RPC interface is used for data interaction with the log consortium chain node; 3. The highly reliable and low - overhead data storage system for blockchain log storage according to claim 2, characterized in that, The process of the erasure code engine encoding the log file is as follows: Each log file is cut into n - 2f data blocks, and 2f parity blocks are generated through (n - 2f, 2f)-RS encoding; the n - 2f data blocks and the 2f parity blocks are collectively referred to as encoded blocks. Subsequently, the n - 2f encoded blocks are uploaded to the consortium chain node. n represents the total number of nodes, and f represents the number of malicious nodes that can be tolerated.

4. The highly reliable and low - overhead data storage system for blockchain log storage according to claim 3, characterized in that, The block header in the blockchain light client is synchronized with the full node, and the Merkle root is saved in the block header. When the light client queries a certain transaction from the full node, the full node will return this transaction and the Merkle proof of the transaction. The existence of this transaction can be verified through the Merkle proof.

5. The highly reliable and low - overhead data storage system for blockchain log storage according to claim 1, characterized in that, The log consortium chain node includes: a log storage layer, a blockchain layer, and a second RPC interface. The log storage layer is used to store encoded blocks, the blockchain layer is used to store the metadata of the log, and the second RPC interface is used for data interaction with the log client; among them, the blockchain layer includes: a PBFT consensus engine, a transaction execution module, and a P2P network. The PBFT consensus engine is used to execute the existing PBFT consensus algorithm, the transaction execution module is used to collect transactions and package transactions, and the P2P network is a peer-to-peer network composed of distributed independent blockchain nodes in the blockchain.

6. The highly reliable and low - overhead data storage system for blockchain log storage according to claim 1, characterized in that, During the two storage stages of the log file, the consortium chain node plays two different roles in the two stages. In the first stage, the consortium chain logically acts as an off-chain storage node and runs the protocol of the log storage layer; after the log client encodes the log file and uploads it to the consortium chain node, the consortium chain node will check the encoded blocks and store them in the KV database on the node. After successful storage, the node will sign the encoded blocks with its own private key, and the signature will be returned and saved in the blockchain to prove that the node has successfully saved this log encoded block.

7. The highly reliable and low - overhead data storage system for blockchain log storage according to claim 6, characterized in that, When the log client has collected more than n - f signatures, it indicates that at least n - 2f non-malicious nodes have successfully saved the log encoding blocks. At this time, the log client can initiate a transaction to upload the signatures and other log file metadata to the log consortium blockchain in the form of a transaction. The blockchain nodes will check the submitted transaction, and after completion, package and add it to the blockchain. Meanwhile, there will be a status database to index the log files; When a user queries a specific log through the client, they first obtain the metadata from any blockchain, and then retrieve n - 2f correctly encoded blocks from the log storage layer to decode and restore the log on the client side.

8. The highly reliable and low - overhead data storage system for blockchain log storage according to claim 7, characterized in that, The specific steps for the client to store log files are as follows: A1: Take a single log file as the input parameter, use the erasure coding function to encode the data of the log file to obtain the same number of encoding blocks as the number of clusters, and calculate the checksum through hash operation; A2: The consortium blockchain nodes Ni are sorted from 1 to n according to the public key. The client sends the corresponding encoding block Ci, the checksum of the log file, and the checksum of the encoding block Ci to the consortium blockchain node Ni, where n represents the number of clusters; A3: The consortium blockchain nodes check whether the encoding block is consistent with the checksum; and ensure that the log file corresponding to the encoding block has not been submitted before; A4: Define key-value pairs, use the checksum of the log file in the encoding block as the key, and store the encoding block on the local object data of the node. Use the data associated with the log file as the value, where the value includes the data itself of the encoded block file, the checksum information, and the metadata covering the client information, log timestamp, and application program path information. When the value of the metadata is empty, it indicates that the log file is in the preparation stage and has not been submitted yet; A5: The consortium blockchain node returns the signature of the checksum information of the log file and the checksum information of the encoded file to indicate that it has received and saved the encoding block. The client will submit the signature in the next stage as a voucher for the node to receive it normally; A6: Detect whether the client has collected more than n - 2f messages indicating readiness returned by the consortium blockchain nodes. If so, wake up the main thread; A7: The client submits the log file metadata and the signature returned by the consortium blockchain nodes as a transaction.

Images