A blockchain optimization method based on snapshot and data dump
By using snapshot files to replace the Genesis block as the starting point of synchronization in the blockchain system and dumping historical block data to IPFS, the synchronization workload and storage pressure problems caused by the large amount of data in the blockchain system are solved. The system is scalable and security improvement is achieved.
Patent Information
- Application Number
- CN202310217424.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-02
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2043-03-02
AI Technical Summary
The huge amount of data in the blockchain system has limited system scalability, and the workload of synchronization of new nodes is large, and off-chain storage methods are difficult to reduce storage pressure and synchronous workload at the same time.
By regularly persisting unspent transactions local to full nodes, replacing Genesis blocks with snapshot files as a synchronization starting point, easing the workload of synchronization of new nodes, and dumping historical block data to IPFS to reduce the burden of full node storage , and at the same time, the system security is ensured in combination with security means.
The synchronization time of the entire node is shortened, the storage overhead of the entire node is reduced, and the system is improved without jeopardizing the system's security.
Smart Images

Figure CN116723201B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of blockchain technology, and in particular to a blockchain optimization method based on snapshot and data dump. Background Art
[0002] With the rise of blockchain technology, the current size of the blockchain has reached an astonishing 447GB, and this amount of data has shown a steady growth trend over time. Any full node must download and process this data in order to have the latest set of unspent transaction outputs to quickly verify the correctness of the transaction. Therefore, the huge amount of data has seriously affected the scalability of the system.
[0003] For the above problems, there is a proposal to use off-chain storage to reduce the storage burden of nodes in the network, but this method has two disadvantages: 1. Loss of control over data; 2. New nodes still need to be synchronized from the genesis block, and the synchronization workload has not been improved. It can be seen that the current off-chain storage method is difficult to meet the two requirements of reducing storage pressure and synchronization workload at the same time. Summary of the invention
[0004] The purpose of this application is to propose a blockchain optimization method based on snapshots and data dumps. In order to solve the problem of huge data volume and excessive synchronization workload of new nodes in the blockchain system, the latest unspent transaction output set snapshots of all nodes in the network are regularly persisted, and the snapshot files are used to replace the genesis block as the synchronization starting point, thereby reducing the synchronization workload of new nodes. In addition, the historical block data is dumped to the off-chain IPFS to reduce the storage burden of the full node, and security measures are combined to ensure the security of the blockchain system.
[0005] In order to achieve the above purpose, the technical solution of this application is as follows:
[0006] A blockchain optimization method based on snapshots and data dumps, wherein a block storing a hash value of a snapshot file is used as a snapshot block. The blockchain optimization method based on snapshots and data dumps comprises:
[0007] After the first snapshot block, all nodes determine whether the next block meets the preset snapshot block conditions. If so, all full nodes in the entire network create a snapshot file based on the local unspent transaction output set, and use the created snapshot file as the input of the hash function to obtain the hash value of the snapshot file;
[0008] All full nodes execute the consensus algorithm to compete for the right to publish blocks. The full node that obtains the right to publish blocks stores the hash value of the snapshot file in a block that meets the preset snapshot block conditions, and uses the block that stores the hash value of the snapshot file as the second snapshot block. The second snapshot block is then published, and other full nodes synchronize the second snapshot block and update the local unspent transaction output set based on the transaction data therein;
[0009] The full node that subsequently obtains the right to publish the block adds the hash value of the snapshot file to the block with the preset cumulative number of confirmed blocks after the second snapshot block;
[0010] All full nodes determine whether historical block data needs to be transferred. If transfer is required, the full node stores the first snapshot block and the historical blocks between the first snapshot block and the second snapshot block in the Interstellar File System, saves the hash index returned by the Interstellar File System, and saves the correspondence between "hash index, block height" in each corresponding block.
[0011] Furthermore, the blockchain optimization method based on snapshot and data dump also includes:
[0012] Step F1: The newly joined full node downloads the latest snapshot file and snapshot block from the neighboring full node;
[0013] Step F2: Use the hash value of the snapshot file in the latest snapshot block to verify the latest snapshot file downloaded. If the verification passes, proceed to the next step; otherwise, return to step F1;
[0014] Step F3: After verification, download the latest snapshot block from the neighboring node and then preset the block with the cumulative confirmed block number, and extract the hash value of the snapshot file from it;
[0015] Step F4, use the hash value of the extracted snapshot file to verify the latest downloaded snapshot file, if the verification passes, proceed to the next step, otherwise return to step F1;
[0016] Step F5: After verification, the unspent transaction output data in the latest snapshot file is imported into the newly added full node, and an unspent transaction output set is constructed locally on the newly added full node;
[0017] Step F6: Continue to request the remaining blocks from neighboring full nodes, synchronize the transactions therein, and update the local unspent transaction output set.
[0018] Furthermore, the determining whether the next block satisfies a preset snapshot block condition includes:
[0019] If the block height of the next block is the sum of the block height of the first snapshot block and the preset first value, it is determined that the next block meets the preset snapshot block condition.
[0020] Furthermore, the preset first value is greater than a preset cumulative confirmed block quantity.
[0021] This application proposes a blockchain optimization method based on snapshots and data dumps. When a new full-node client joins the blockchain system, the original synchronization process requires the full node to synchronize from the genesis block until the latest block, which is extremely time-consuming. After applying the technical solution of this application, the unspent transaction output set snapshot file replaces the genesis block as the synchronization starting point, greatly reducing the synchronization time of the full node. In order to maintain the security of the blockchain system, the full node is required to store the complete blockchain ledger data, which puts huge storage pressure on the full node. This application dumps the historical block body data between the old and new snapshot blocks to IPFS, and stores the IPFS hash in the corresponding block body, without endangering the security of the blockchain system, greatly reducing the storage overhead of the full node. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 A flowchart of the blockchain optimization method based on snapshots and data dumps for this application;
[0023] Figure 2 A structural relationship diagram between blocks in an embodiment of the present application;
[0024] Figure 3 This is a schematic diagram of the new full node synchronization process for this application. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0026] In the blockchain system, the data structure containing transaction information is called a normal block. If the block is the first block in the blockchain, it is called a genesis block. In this application, based on the concept of a normal block, if the coinbase field of a normal block stores the hash value of the snapshot file of the unspent transaction output set, the normal block is called a snapshot block. A block is divided into two parts: a block header and a block body. The block header is used to store the hash value (PreHash) of the previous block, the hash value (Merkle Hash) of the root of the transaction tree in the current block body, and the timestamp (TimeStamp), etc. The block body is used to store detailed transaction data (Data).
[0027] There are many client nodes in the blockchain system, among which the full node has all the functions of the system and is the most powerful node, so it places high requirements on the hardware running the full node client. The full node of the blockchain system must download and store all blocks, which is very unfriendly to the full node client with scarce storage resources.
[0028] The technologies used in this application include unspent transaction output set snapshot files and interplanetary file systems. Among them, the unspent transaction output set snapshot file is obtained by persisting the local unspent transaction output set by the full node of the blockchain network. It will replace the genesis block as the synchronization starting point of the full node newly added to the system, which can greatly reduce the synchronization workload of the full node. The full name of the interplanetary file system is Interplanetary File System, referred to as IPFS. IPFS is a network transmission protocol designed to create persistent and distributed storage and file sharing, and is also a peer-to-peer hypermedia distribution protocol addressed according to file content. The nodes in the IPFS network together constitute a decentralized distributed file system. After the file is stored in IPFS, an IPFS hash will be obtained. The IPFS hash can be used as an index to access the file and can also be used to check whether the file content has been tampered with. It is based on this feature of IPFS that it is selected as a third-party storage platform for block data dump in the scheme. Files stored in IPFS consist of two parts: one is data, which consists of a data block of up to 256KB; the other is connection, which connects the array of these data blocks. If the stored file is larger than 256KB, it will be split into 256KB blocks, which are linked through array links to make the file logically complete. Since the size of a block is usually larger than 256KB, a block will be split and stored.
[0029] In one embodiment, Figure 1 As shown in the figure, a blockchain optimization method based on snapshot and data dump is proposed, including:
[0030] Step S1: After the first snapshot block, all nodes determine whether the next block meets the preset snapshot block conditions. If so, all full nodes in the entire network create a snapshot file based on the local unspent transaction output set, and use the created snapshot file as the input of the hash function to obtain the hash value of the snapshot file.
[0031] A node is a network node in a blockchain distributed system. It is a server, computer, phone, etc. connected through a network. For blockchains of different natures, the way to become a node will also be different. Taking blockchain as an example, participating in a transaction constitutes a node. A full node is a node with a complete blockchain ledger. A full node needs to occupy memory to synchronize all blockchain data, can independently verify all transactions on the blockchain and update data in real time, and is mainly responsible for the broadcast and verification of blockchain transactions.
[0032] This application uses the block that stores the hash value of the snapshot file as the snapshot block. This step determines whether the next block meets the preset snapshot block condition after a snapshot block. For example, after the first snapshot block A, it determines whether the next block meets the preset snapshot block condition.
[0033] In a specific embodiment, if the block height of the next block is the sum of the block height of the first snapshot block and a preset first value, it is determined that the next block meets the preset snapshot block condition.
[0034] That is, it is judged based on the block height, which is the number of blocks linked to the main chain, that is, the number of blocks connected to the blockchain. For example, the block height of the first snapshot block A is s. After X blocks, the block with a block height of s+X is considered to meet the preset snapshot block condition. X is a preset parameter (preset first value), and the value of X must be greater than the number of cumulative confirmed blocks in subsequent steps. For example, in one embodiment, the number of cumulative confirmed blocks is set to 6, then the value of X must be greater than 6.
[0035] If the next block does not meet the preset snapshot block conditions, it is a normal block and is executed according to the original rules of the blockchain system, skipping the remaining steps of this application, that is, there is no need to add the hash value of the snapshot file to the block. If it is determined that the next block meets the preset snapshot block conditions, all full nodes in the entire network will persist the local unspent transaction output set as a snapshot file.
[0036] For example, the block height of snapshot block A is s. After judgment, it is found that block K meets the conditions, then all full nodes create a snapshot file and calculate the hash value of the snapshot file.
[0037] Since the data in the unspent transaction output set will change after publishing or synchronizing the transaction records in the snapshot block, in order to create consistent snapshot files for all full nodes, it is required that before publishing or synchronizing a new snapshot block, all full nodes in the entire network must create an unspent transaction output set snapshot file based on the local unspent transaction output set.
[0038] The full node then uses the snapshot file as input to the hash function SHA-256 to obtain a 256-bit hash value for the snapshot file.
[0039] It should be noted that the first snapshot block in this embodiment does not refer to the first snapshot block in the blockchain system. The first and second are only used to distinguish the two snapshot blocks before and after, which will not be repeated below.
[0040] Step S2, all full nodes execute the consensus algorithm to compete for the right to publish blocks. The full node that obtains the right to publish blocks stores the hash value of the snapshot file in a block that meets the preset snapshot block conditions, and uses the block that stores the hash value of the snapshot file as the second snapshot block. Then, the second snapshot block is published, and other full nodes synchronize the second snapshot block and update the local unspent transaction output set according to the transaction data therein.
[0041] In the blockchain system of this embodiment, the full node uses the POW consensus algorithm to determine whether it has obtained the right to publish a block. The full node that has obtained the right to publish a block publishes the block to the entire blockchain system, and a full node is selected from the entire network to publish the block.
[0042] The full node that obtains the right to publish a block stores the hash value of the snapshot file in the coinbase field of the snapshot block to be published (for example, block K) as the second snapshot block. The second snapshot block is then published, and other full nodes synchronize the second snapshot block and update the local unspent transaction output set based on the transaction data in the second snapshot block.
[0043] When a new full node joins the blockchain network, the hash value of the snapshot file can be used to verify the integrity of the unspent transaction output set snapshot file downloaded by the new full node from the neighboring full node, thereby ensuring the security of the system.
[0044] It should be noted that other full nodes synchronize the second snapshot block in the same way as synchronizing ordinary blocks, and update the local unspent transaction output set through the transactions in the snapshot block. After the update, the data status of the local unspent transaction output set changes, which will not be repeated here.
[0045] Step S3: The full node that subsequently obtains the block publishing right adds the hash value of the snapshot file to a block with a preset cumulative number of confirmed blocks after the second snapshot block.
[0046] Because there are many full nodes in the whole network, in order to prevent a full node from doing evil and putting the hash value of the maliciously modified snapshot file into the snapshot block, this application sets a preset number of cumulative confirmation blocks, for example, 6, then the value of X is greater than 6. The hash value of the snapshot file is attached to the 6 blocks after the snapshot block (defined as cumulative confirmation blocks), which is similar to testifying, improving the credibility of the snapshot file.
[0047] That is, after the second snapshot block is determined, the full node that obtains the block publishing right continues to publish blocks, and the full node that publishes blocks adds the hash value of the snapshot file to the 6 blocks after the second snapshot block.
[0048] Step S4, all full nodes determine whether it is necessary to transfer historical block data. If transfer is required, the full node stores the first snapshot block and the historical blocks between the first snapshot block and the second snapshot block in the Interstellar File System, and saves the hash index returned by the Interstellar File System, and saves the correspondence between "hash index, block height" in each corresponding block.
[0049] After the second snapshot block, if the hash value of the snapshot file has been added to 6 blocks cumulatively, that is, the hash value of the snapshot file has been added to the 6 cumulative confirmation blocks. After that, any full node in the blockchain system will determine whether its storage space is sufficient. If it is sufficient, it can continue to store blocks locally. If the storage space is insufficient, the first snapshot block and the historical blocks between the first snapshot block and the second snapshot block will be stored in the Interstellar File System IPFS.
[0050] For example, if there are 9 other ordinary blocks (ordinary blocks BJ) between the first snapshot block A and the second snapshot block K, the block body data in the first snapshot block A and ordinary block BJ will be stored in the Interstellar File System IPFS. The full node dumps the block body data between the new and old snapshot blocks from the local blockchain to IPFS, and deletes the transaction data in the locally saved block body, thereby reducing the storage overhead of the full node. Afterwards, the IPFS hash value corresponding to the block body is stored in the corresponding block body.
[0051] In this embodiment, IPFS returns a hash index after storage, that is, the address index of the block data storage, indicating the address of a block data storage, and then saves it to the corresponding block according to the <hash index, block height> correspondence. For example, the <hash index, block height> correspondence of block B is saved in the block body of block B.
[0052] It is difficult to ensure the security of the system by relying solely on the hash value in the snapshot block. According to the principle that the possibility of data being tampered with after 6 blocks of cumulative confirmation in the blockchain system is almost zero, this application appends the hash value of the snapshot file in the 6 blocks after the snapshot block, which can greatly improve the security of the system.
[0053] The above embodiment describes the method for publishing snapshot blocks and synchronizing snapshot blocks by the existing full nodes in the blockchain system. However, for the full nodes newly added to the blockchain system, which have no local data, it is necessary to download the unspent transaction output set snapshot file from the existing full nodes in the system to build a local unspent transaction output set.
[0054] In another embodiment, a blockchain optimization method based on snapshots and data dumps, such as Figure 3 As shown, it also includes:
[0055] Step F1: The newly joined full node downloads the latest snapshot file and snapshot block from the neighboring full node.
[0056] The newly joined full node needs to download all blocks, but only needs to use the transactions in all blocks after the latest snapshot block to update the local unspent transaction output data status. In order to create local unspent transaction output data in the newly joined full node, in this step, the newly joined full node downloads the latest snapshot file and snapshot block from the neighboring full node. The snapshot file and snapshot block created most recently to the newly joined full node are the latest.
[0057] Step F2: Use the hash value of the snapshot file in the latest snapshot block to verify the latest snapshot file downloaded. If the verification passes, proceed to the next step; otherwise, return to step F1.
[0058] This step performs the first hash value verification, that is, the latest snapshot file downloaded is verified using the snapshot file hash value in the latest snapshot block. Assuming that the latest snapshot block is block K, block K stores the hash value corresponding to the latest snapshot file. The hash value obtained from block K is compared with the hash value of the snapshot file calculated by the hash algorithm to perform the verification, which will not be repeated here.
[0059] Step F3: After verification, download the latest snapshot block from the neighboring node, preset the block with the cumulative confirmed block number, and extract the hash value of the snapshot file from it.
[0060] Step F4: Use the hash value of the extracted snapshot file to verify the latest downloaded snapshot file. If the verification passes, proceed to the next step; otherwise, return to step F1.
[0061] This step performs a second hash value check, using the snapshot file hash value extracted in step F3 to check the latest snapshot file downloaded in step F1. If they are consistent, the check passes.
[0062] Step F5: After verification, the unspent transaction output data in the latest snapshot file is imported into the newly added full node, and an unspent transaction output set is constructed locally on the newly added full node.
[0063] This step imports the unspent transaction outputs in the latest snapshot file into the newly added full node, and constructs an unspent transaction output set locally on the newly added full node.
[0064] Step F6: Continue to request the remaining blocks from neighboring nodes, synchronize the transactions therein, and update the local unspent transaction output set.
[0065] In this embodiment, the new full node downloads the unspent transaction output snapshot file from the neighboring full node through the network, and uses the hash value in the snapshot block and its subsequent 6 cumulative confirmation blocks to verify the integrity of the snapshot file. After the integrity verification is passed, the remaining block data is downloaded from the neighboring full node with the snapshot file as the starting point, and the transactions in these blocks are replayed until the latest block, indicating that the synchronization is complete.
[0066] The remaining blocks include transaction data, which are used to update the local unspent transaction output set, that is, the transaction data is removed from the local unspent transaction output data, and the newly generated unspent transaction output data is added.
[0067] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A blockchain optimization method based on snapshot and data dump, characterized in that: The block storing the hash value of the snapshot file is used as the snapshot block. The blockchain optimization method based on snapshot and data dump includes: After the first snapshot block, all nodes determine whether the next block meets the preset snapshot block conditions. If so, all full nodes in the entire network create a snapshot file based on the local unspent transaction output set, and use the created snapshot file as the input of the hash function to obtain the hash value of the snapshot file; All full nodes execute the consensus algorithm to compete for the right to publish blocks. The full node that obtains the right to publish blocks stores the hash value of the snapshot file in a block that meets the preset snapshot block conditions, and uses the block that stores the hash value of the snapshot file as the second snapshot block. The second snapshot block is then published, and other full nodes synchronize the second snapshot block and update the local unspent transaction output set based on the transaction data therein; The full node that subsequently obtains the right to publish the block adds the hash value of the snapshot file to the block with the preset cumulative number of confirmed blocks after the second snapshot block; All full nodes determine whether historical block data needs to be transferred. If transfer is required, the full node stores the first snapshot block and the historical blocks between the first snapshot block and the second snapshot block in the Interstellar File System, saves the hash index returned by the Interstellar File System, and saves the correspondence between "hash index, block height" in each corresponding block.
2. The blockchain optimization method based on snapshot and data dump according to claim 1, characterized in that: The blockchain optimization method based on snapshot and data dump also includes: Step F1: The newly joined full node downloads the latest snapshot file and snapshot block from the neighboring full node; Step F2: Use the hash value of the snapshot file in the latest snapshot block to verify the latest snapshot file downloaded. If the verification passes, proceed to the next step; otherwise, return to step F1; Step F3: After verification, download the latest snapshot block from the neighboring node and then preset the block with the cumulative confirmed block number, and extract the hash value of the snapshot file from it; Step F4, use the hash value of the extracted snapshot file to verify the latest downloaded snapshot file, if the verification passes, proceed to the next step, otherwise return to step F1; Step F5: After verification, the unspent transaction output data in the latest snapshot file is imported into the newly added full node, and an unspent transaction output set is constructed locally on the newly added full node; Step F6: Continue to request the remaining blocks from neighboring full nodes, synchronize the transactions therein, and update the local unspent transaction output set.
3. The blockchain optimization method based on snapshot and data dump according to claim 1, characterized in that: The determining whether the next block meets the preset snapshot block condition includes: If the block height of the next block is the sum of the block height of the first snapshot block and the preset first value, it is determined that the next block meets the preset snapshot block condition.
4. The blockchain optimization method based on snapshot and data dump according to claim 3 is characterized in that: The preset first value is greater than the preset cumulative confirmed block quantity.
