A blockchain data oriented multi-version verifiable query method

Abstract

This invention discloses a method for multi-version verifiable querying of blockchain data, including an index structure supporting efficient multi-version queries and a verifiable structure supporting rapid generation and verification of multi-version proofs. First, the index structure supporting efficient multi-version queries can simultaneously and efficiently locate all historical versions in multi-version tracing queries across multiple blocks. This index structure includes an LSM-Tree-based index structure and a novel filter based on Huffman coding. The filter allows for a one-time query of the ordered string table containing different versions of data, significantly improving the efficiency of multi-version queries. Second, the verifiable structure supporting rapid generation and verification of multi-version proofs can generate proofs for multiple versions of data in parallel and efficiently aggregate them, reducing the proof size for batch verification. Through this scheme, this invention effectively optimizes the query and verification process for multi-version data, achieving rapid data location and efficient generation and verification of proofs.

Description

Technical Field

[0001] This invention belongs to the field of blockchain technology, and in particular relates to a multi-version verifiable query method for blockchain data. Background Technology

[0002] Blockchain, as a distributed ledger technology, possesses the crucial characteristics of data immutability and traceability. With the widespread application of blockchain technology across various fields, the demand for data integrity, traceability, and security is increasing daily. However, as blockchain systems expand in scale and application scenarios diversify, traditional methods of data storage, retrieval, and verification may face challenges. Especially when dealing with multi-version data, efficient storage, retrieval, and verification become urgent issues. Existing blockchain data storage primarily falls into two categories: one separates the underlying persistent storage from the verifiable index structure, and the other directly builds a verifiable structure on top of the underlying persistent storage.

[0003] The approach of separating the underlying persistent storage from the verifiable index structure results in a separate architecture where the key-value database (level database) handles the persistent storage of the underlying data, while the verifiable index structure is used for indexing the upper layers and providing verifiable queries. This separation leads to the underlying database storing a large amount of data to provide indexes for the verifiable structure, with relatively little actually used for data storage. Redundant storage in intermediate index nodes results in a smaller portion of the LSM-Tree actually used for storing account data. The block-based index structure requires traversal to handle multi-version data queries, reducing the efficiency of multi-version queries. Because the start and end positions of state changes cannot be determined, multi-version data queries are often handled through traversal, starting from the latest block and going to the oldest block to determine the block where the state change occurred. The block-based verifiable structure often provides multi-version proofs by merging multiple single-version proofs, causing the size of the multi-version proof to increase linearly with the number of versions.

[0004] While building verifiable structures directly on the underlying persistent storage reduces redundant storage in the index nodes, it doesn't improve the performance of multi-version queries (MLMs). Existing work combines verifiable structures with LSM-Trees to reduce redundant storage in the index nodes by removing upper-level indexes. However, since updating the verification structure often incurs significant overhead, this approach leads to excessive overhead in rebuilding the verification structure during compression. MLMs still require traversal, reducing their efficiency. With the introduction of more efficient consensus algorithms and transaction concurrency control algorithms, these problems in blockchain storage have become a major bottleneck restricting the performance of blockchain systems. Summary of the Invention

[0005] The purpose of this invention is to propose a multi-version verifiable query method for blockchain data, aiming to solve the problems of low query efficiency, excessively large proof size, and high verification cost in existing blockchain systems. To achieve rapid location of multi-version data, this invention designs an efficient structure supporting multi-version indexing. This structure enables rapid retrieval of all versions of data through effective organization and management of different versions in the blockchain. Simultaneously, to improve the efficiency of multi-version queries, this invention designs a novel filter based on Huffman coding, accelerating the data location process. This invention also designs a new verifiable structure specifically for the generation and verification of multi-version proofs. By introducing vector commitments, the proof size is reduced, supporting parallel proof generation and proof aggregation. Furthermore, a batch verification mechanism is introduced, enabling multiple version proofs to be efficiently verified in a single verification process, significantly reducing verification overhead.

[0006] The specific technical solution for achieving the objective of this invention is as follows:

[0007] A method for efficient and verifiable multi-version querying of blockchain data includes: an index structure supporting efficient multi-version querying and a verifiable structure supporting fast creation and verification of multi-version proofs; wherein,

[0008] The index structure supporting efficient multi-version queries is implemented based on LSM-Tree. The multi-version index relies on various LSM-Tree components, avoiding additional maintenance of the index structure and reducing redundant hash storage overhead through merging. New filters are designed to reduce the access overhead of the original multi-layered filters, accelerating multi-version queries.

[0009] The verifiable structure supporting rapid generation and verification of multi-version proofs adopts a scheme combining vector commitments and Merkle hash trees. This retains the advantage of constant-level proof size of vector commitments while minimizing their use. Proofs are generated in parallel for multiple versions of data and efficiently aggregated to reduce proof size and facilitate batch verification.

[0010] The index structure that supports efficient multi-version queries specifically includes:

[0011] A1: During the write phase, index data is written to the components of the LSM-Tree, including the in-memory table MemTable, the immutable in-memory table ImmTable, and the disk-based SSTable;

[0012] A2: During the merging phase, different versions of the same key are merged to reduce storage overhead; in the compaction process of LSM-Tree, this method increases the size of key-value pairs while reducing the storage of duplicate keys.

[0013] A3: During the query phase, all version information of a given key is retrieved at once; the in-memory table MemTable and the immutable in-memory table ImmTable are searched, and the SSTable on disk is also retrieved to ensure that all historical version information of the key is obtained;

[0014] The verifiable structure that supports rapid generation and verification of multi-version proofs specifically includes:

[0015] B1: Constructing a Merkle hash tree and vector commitment (VC);

[0016] B2: Proof and verification of Merkle hash trees and vector commitments (VC);

[0017] B3: Only persist the upper-level vector commitment (VC).

[0018] Furthermore, A1 specifically includes:

[0019] A11: Write to memory, write index data to the MemTable in memory;

[0020] A12: When the size of the memory table MemTable reaches a predetermined threshold, it is converted into an immutable memory table ImmTable, and then persisted as an SSTable and written to disk;

[0021] A13: When the data in the memory table is flushed to the disk, the fingerprint of the corresponding key and the SSTable number need to be inserted into the Cuckoo Filter. It should be noted that the number is not inserted directly, but the number after Huffman coding is inserted into the Cuckoo Filter to reduce the storage overhead of the SSTable number.

[0022] Furthermore, A3 specifically includes:

[0023] A31: Memory query; First, use a skip list to query MemTable to return whether there is any key change information in MemTable; then, query ImmTable to check if there is any corresponding key change information.

[0024] A32: Based on the design filter, determine all SSTables containing the key; in the SSTables that meet the criteria, use binary search to retrieve the corresponding value and return it.

[0025] Furthermore, B1 specifically includes:

[0026] B11: First, sort the input status list by account address in units of blocks, then construct the corresponding Merkle hash tree for each block and calculate its corresponding Merkle hash tree root;

[0027] B12: Use the Merkle hash tree root of each block as the data for the vector commitment, and save some Merkle hash tree nodes to the cache during the calculation process to facilitate the subsequent storage of the verification structure on the disk;

[0028] B13: Construct a vector commitment VC from the root hash of the Merkle hash tree for all blocks based on the input public parameter pp, and compute the corresponding commitment and the proof for each position.

[0029] Furthermore, B2 specifically includes:

[0030] B21: Find the account address to be verified in the state list and generate the location information of the account to be verified, including its Merkle hash tree index m_id and its position pos in the Merkle hash tree. Since the state list is already sorted by account address, the query can be performed quickly.

[0031] B22: Provide proof of existence for all accounts in the proof list. Each proof includes a Merkle hash tree proof and a vector commitment proof. The Merkle hash tree proof requires reconstructing the necessary parts of the Merkle hash tree based on the state list and the Merkle hash tree generated during construction. For the vector commitment proof, the proofs pre-computed during construction are read directly.

[0032] The beneficial effects of this invention include:

[0033] This invention aims to address the problems of low efficiency in multi-version data querying, excessively large proof sizes, and high verification costs in existing blockchain systems. It focuses on developing an efficient and verifiable multi-version query method for blockchain data. The goal is to quickly locate multi-version data, generate multi-version proofs, and effectively verify these proofs. Specifically, this invention designs an efficient indexing mechanism that can simultaneously and efficiently locate all historical versions in multi-version tracing queries across multiple blocks. By merging LSM-based multi-version indexes, the storage overhead of duplicate hash values ​​is reduced. A new filter is designed to query all SSTables containing key-value pairs at once, accelerating multi-version queries. Furthermore, an efficient verifiable structure is designed that can simultaneously generate proofs for multiple versions of data in parallel and perform efficient aggregation to reduce the proof size and facilitate batch verification. Combining Merkle trees and vector commitments improves verification efficiency while reducing the size of multi-version proofs. Attached Figure Description

[0034] Figure 1 This is a flowchart of the invention;

[0035] Figure 2 This is an example diagram of the encoding of filters in the multi-version index proposed in this invention;

[0036] Figure 3 This is a schematic diagram of the Level ID encoding algorithm;

[0037] Figure 4 This is an example diagram of the multi-version query proposed in this invention;

[0038] Figure 5 This is a schematic diagram of the verifiable structure of multiple versions of the present invention. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0040] This invention discloses an efficient and verifiable multi-version query method for blockchain data, supporting rapid location of multi-version data, generation of multi-version proofs, and effective verification of these proofs. The method comprises two core components: first, an index structure supporting efficient multi-version queries, enabling simultaneous and efficient location of all historical versions in multi-version tracing queries across multiple blocks; second, a verifiable structure supporting rapid generation and verification of multi-version proofs, simultaneously generating proofs for multiple versions of data in parallel and efficiently aggregating them to reduce proof size and facilitate batch verification. Through a series of optimization schemes, this invention efficiently optimizes the verifiable query process for multi-version data, rapidly locating multi-version data, generating multi-version proofs, and verifying the proofs.

[0041] In the efficient and verifiable multi-version query method for blockchain data, the transaction lifecycle will go through the following three stages: (1) execution stage; (2) writing, merging and querying stage; (3) building a verifiable structure.

[0042] Execution Phase: In the transaction execution phase of the blockchain, firstly, verified transactions are packaged into a new block. This block contains multiple transaction records, the hash value of the previous block, a timestamp, and other relevant metadata. Nodes verify the new block using a consensus mechanism (such as Proof-of-Work or Proof-of-Stake) to ensure its validity. Each transaction triggers a state update for the corresponding account, and these state changes are reflected in the blockchain's state database. Once all transactions have been executed, nodes update their local state database to reflect the latest account balances and contract states.

[0043] The write, merge, and query phases proceed to the write sub-phase once transaction execution is complete. This phase utilizes an index structure that supports efficient multi-version queries. First, index data is written to the in-memory table (MemTable) in the LSM-Tree-related components. When the in-memory table reaches a predetermined threshold size, it is converted to an immutable in-memory table (ImmTable) and persisted to disk as an SSTable. Next, the system enters the merge sub-phase, where different versions of data with the same key are merged to ensure blockchain data integrity. This process reduces storage redundancy while increasing the size of key-value pairs. Simultaneously with data flushing to disk, filters must be updated to include the latest information on key-value pairs in the SSTable. Finally, the system enters the query sub-phase, which aims to retrieve all version information for a given key at once, collect all relevant version data, and output it to the log, preparing for the next step of building a verifiable structure.

[0044] In the verifiable structure construction phase, a scheme combining vector commitments and Merkle hash trees was adopted. This approach retains the advantage of constant-level proof size of vector commitments while minimizing their use, thus mitigating performance issues caused by excessive computational overhead. Based on log information from the previous phase, Merkle trees were constructed on a block-by-block basis, with vector commitments used to connect blocks. This reduced the single-version proof query overhead by decreasing the tree height. Through a maintainable and aggregatable vector commitment scheme, proof generation and verification were aggregated across blocks, reducing proof size and accelerating verification speed.

[0045] Figure 1This paper illustrates the overall flow of the multi-version verifiable query method proposed in this invention. Blocks contain account transaction data, which can trigger modifications to the account state. Account information is stored in an LSM-Tree for indexing. The LSM-Tree efficiently manages write and query operations, supporting fast indexing and retrieval. Corresponding state data is appended to a state log file, recording the account state and change history after each transaction for subsequent auditing and querying. To address the issue of redundant data storage, storage consumption is reduced and index access speed is improved by deleting intermediate index nodes and merging indexes in the LSM-Tree. A novel filter is designed to support one-time queries on all storage tables (SSTables), effectively locating relevant data and solving the problem of low efficiency in multi-version queries. The verifiable structure is built based on the state log, combining Merkle trees and vector commitments to accelerate proof computation as much as possible while reducing the proof size.

[0046] Figure 2 This example illustrates the encoding process of filters in a multi-version index. The LSM-Tree has a three-level structure. The first and second levels contain four SSTables, and the last level contains one SSTable. The proportion of each SSTable in each level within the entire LSM-Tree structure can be calculated. For example, SSTable1 has a proportion of 1 / 125, SSTable5 has a proportion of 5 / 125, and SSTable9 has a proportion of 100 / 125. Using Huffman coding to encode the Level ID of all SSTables, we obtain... Figure 2 The Huffman tree shown in the upper right corner. SSTable1 contains k1, SSTable5 contains k2 and k3, and SSTable9 contains k4, k5, k6, and k7. Assuming the original filter doesn't use encoding and directly uses int32 bytes to store the corresponding Level IDs, then three int32 bytes are needed to store the Level IDs of k1, k2, and k4, totaling 12 bytes. However, using Huffman encoding, k1 requires 6 bits, k2 requires 3 bits, and k4 only requires 1 bit. This reduces the total storage overhead to 1.25 bytes, significantly decreasing the storage cost of the Level IDs.

[0047] like Figure 3As shown, Algorithm 1 illustrates the Level ID encoding process. Level IDs are encoded based on L (the number of levels in the LSM-Tree), K (the number of SSTables in each level except the last), Z (the number of SSTables in the last level), and T (the size ratio of the next level to the previous level). The proportion of each SSTable in each level within the entire LSM-Tree is calculated (CalculateSSTableRatio). Based on the proportion of each SSTable in the LSM-Tree, Huffman coding (EncodeLevelID) is performed on the SSTable. Tables at the bottom with more key-value pairs correspond to shorter Level ID encoding lengths, thus reducing the storage overhead of Level IDs and consequently reducing the storage overhead of the Cuckoo Filter.

[0048] like Figure 4 As shown, for index queries, when a user provides an account (Acc), the system needs to return all version information for that account. This process differs from the LSM-Tree query process, which only needs to find the latest data that meets the criteria, while multi-version queries need to retrieve all version information for a given account at once. Figure 3 The document demonstrates the actual storage structure, including MemTable, ImmTable, and the SSTables in the first two levels, Level 0 and Level 1. It describes the specific query process, such as querying all state values ​​for Acc6, which is divided into two stages: the first stage queries the data in memory, and the second stage queries the data on disk. When querying the MemTable using a skip list, it returns that there is no change information related to Acc6 in the MemTable. Similarly, when querying the ImmTable, there is no change information corresponding to Acc6. First, based on the FHE filter, it queries the SSTables corresponding to Acc6, which are {1, 3}. A binary search is then performed in the SSTables to retrieve the corresponding value and return it.

[0049] Figure 5This paper proposes a novel verifiable structure, BVM (Block Vector Merkle), which combines vector commitments and Merkle trees to reduce the size of multi-version proofs (MPPs) while maximizing the performance of MPP generation and verification. Verifiable structures are crucial in tamper-proof storage because they support verifiable queries, ensuring the correctness of query results based on provided proofs. Existing blockchain storage typically designs verifiable structures based on Merkle hash trees, verifying data integrity and immutability through the difficulty of hash inversion. Another approach is based on vector commitments, improving verification efficiency by generating small, fixed-size proofs. However, using only one approach has limitations. Merkle tree proofs are relatively large, while vector commitments have high initialization and verification overhead. Therefore, BVM combines the advantages of both approaches, providing a more efficient method for generating and verifying MPPs.

[0050] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of the inventive concept are included in this invention and are protected by the appended claims.

Claims

1. A method for multi-version verifiable query on blockchain data, characterized in that, include: It supports an index structure that enables efficient multi-version queries and a verifiable structure that supports rapid generation and verification of multi-version proofs; among which, The index structure that supports efficient multi-version queries specifically includes: A1: During the write phase, index data is written to the components of the LSM-Tree, including the in-memory table MemTable, the immutable in-memory table ImmTable, and the disk-based SSTable; A2: During the merging phase, different versions of the same key are merged to reduce storage overhead; in the compaction process of LSM-Tree, this method increases the size of key-value pairs while reducing the storage of duplicate keys. A3: During the query phase, all version information of a given key is retrieved at once; the in-memory table MemTable and the immutable in-memory table ImmTable are searched, and the SSTable on disk is also retrieved to ensure that all historical version information of the key is obtained; The verifiable structure that supports rapid generation and verification of multi-version proofs specifically includes: B1: Constructing a Merkle hash tree and vector commitment (VC); B2: Proof and verification of Merkle hash trees and vector commitments (VC); B3: Only persist the upper-level vector commitment (VC); The A1 specifically includes: A11: Write to memory, write index data to the MemTable in memory; A12: When the size of the memory table MemTable reaches a predetermined threshold, it is converted into an immutable memory table ImmTable, and then persisted as an SSTable and written to disk; A13: When the data in the memory table is refreshed to the disk, the fingerprint of the corresponding key and the SSTable number need to be inserted into the Cuckoo Filter; instead of inserting the number directly, the number after Huffman coding is inserted into the Cuckoo Filter to reduce the storage overhead of the SSTable number. The A3 specifically includes: A31: Memory lookup; First, use a skip list to query MemTable to return whether there is any key change information in MemTable; then, query ImmTable to check if there is any corresponding key change information. A32: Based on the design filter, determine all SSTables containing the key; in the SSTables that meet the conditions, use binary search to retrieve the corresponding value and return it; Specifically, B1 includes: B11: First, sort the input status list by account address in units of blocks, then construct the corresponding Merkle hash tree for each block and calculate its corresponding Merkle hash tree root; B12: Use the Merkle hash tree root of each block as the data for the vector commitment, and save some Merkle hash tree nodes to the cache during the calculation process to facilitate the subsequent storage of the verification structure on the disk; B13: Construct a vector commitment VC from the root hash of the Merkle hash tree for all blocks based on the input public parameter pp, and compute the corresponding commitment and the proof for each position.

2. The multi-version verifiable query method as described in claim 1, characterized in that, Specifically, B2 includes: B21: Find the account address to be proven in the status list, generate the location information of the account to be proven, including its Merkle hash tree number m_id and its position pos in the Merkle hash tree; B22: Provide proof of existence for all accounts in the proof list. Each proof includes a Merkle hash tree proof and a vector commitment proof. The Merkle hash tree proof requires reconstructing the necessary parts of the Merkle hash tree based on the state list and the Merkle hash tree generated during construction. For the vector commitment proof, the proofs pre-computed during construction are read directly.

Images