Method for writing base page / incremental page in memory into persistent storage medium, method for compressing incremental page and computer equipment

Abstract

The invention discloses a method for writing a base page / incremental page in a memory into a persistent storage medium, a method for compressing the incremental page and computer equipment. The method for writing the basic page / incremental page in the memory into the persistent storage medium comprises the steps that in the memory, for a sub-tree formed by index data divided into logic pages, each logic page generates a basic page according to a memory page which reaches preset times to be modified; when the basic page / increment page of a certain version is generated in the upper layer of logic page, the increment page of the corresponding version is generated in the adjacent lower layer of logic page; and when the basic pages / incremental pages in the memory reach a preset upper limit, writing the basic pages / incremental pages in the memory into a persistent storage medium according to a version sequence.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of blockchain technology, and in particular, relate to a method for writing a base page / incremental page in a memory to a persistent storage medium, a method for compressing an incremental page, and a computer device. Background Art

[0002] Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. In the blockchain system, data blocks are combined into a chain data structure in a sequential manner according to time sequence, and a distributed ledger that cannot be tampered with or forged is guaranteed by cryptography. Due to the characteristics of blockchain such as decentralization, information cannot be tampered with, and autonomy, blockchain has also received more and more attention and application. Summary of the invention

[0003] The object of the present invention is to provide a method for writing a base page / delta page in a memory to a persistent storage medium, a method for compressing delta pages, and a computer device, comprising:

[0004] A method for writing a base page / delta page in a memory to a persistent storage medium, comprising:

[0005] In memory, for a subtree formed by index data divided into logical pages, each logical page generates a base page according to the memory page that has been modified a predetermined number of times; when a base page / incremental page of a certain version is generated in the upper-level logical page, an incremental page of the corresponding version is generated in the adjacent lower-level logical page;

[0006] When the base page / incremental page in the memory reaches a predetermined upper limit, the base page / incremental page in the memory is written to the persistence in the order of version sorting.

[0007] A method for compressing incremental pages based on the above method of writing base pages / incremental pages in memory to a persistent storage medium comprises:

[0008] For an incremental page corresponding to a logical page in a persistent storage medium, searching for a plurality of adjacent continuous incremental pages whose sum reaches a predetermined size;

[0009] A merge operation is performed on the plurality of consecutive delta pages whose sum reaches a predetermined size.

[0010] A computer device comprising:

[0011] processor;

[0012] and a memory storing a program, wherein when the processor executes the program, the operations in the above method are performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0014] Figure 1 is a schematic diagram of deploying a smart contract in one embodiment;

[0015] Figure 2 is a schematic diagram of calling a smart contract in one embodiment;

[0016] Figure 3 is a schematic diagram of a block storage structure in one embodiment;

[0017] Figure 4 is a schematic diagram of a block storage structure in one embodiment;

[0018] Figure 5 is a schematic diagram of an MPT tree in one embodiment;

[0019] Figure 6 It is a schematic diagram of the modules involved in the transaction processing process and the relationship between the CPU, memory and disk in one embodiment;

[0020] Figure 7 It is a schematic diagram of the modules involved in the transaction processing process and the relationship between the CPU, memory and disk in one embodiment;

[0021] Figure 8 is a schematic diagram of the logical relationship of the state database in one embodiment;

[0022] Fig. 9 is a flow chart of reading status data in one embodiment;

[0023] Fig.10 is a schematic diagram illustrating reading status data in combination with a tree structure in an embodiment;

[0024] Fig.11 is a schematic diagram illustrating reading status data in combination with a tree structure in an embodiment;

[0025] Fig.12 is a schematic diagram illustrating reading status data in combination with a tree structure in an embodiment;

[0026] Fig.13 is a schematic diagram illustrating reading status data in combination with a tree structure in an embodiment;

[0027] Fig.14is a schematic diagram illustrating reading status data in combination with a tree structure in an embodiment;

[0028] Fig.15 is a schematic diagram of a block storage structure in one embodiment;

[0029] Fig.16 is a schematic diagram of a block storage structure in one embodiment;

[0030] Fig.17 is a schematic diagram of a block storage structure in one embodiment;

[0031] Fig.18 is a schematic diagram of a block storage structure in one embodiment;

[0032] Fig.19 is a schematic diagram of a block storage structure in one embodiment;

[0033] Fig. 20 is a schematic diagram of a block storage structure in one embodiment;

[0034] Fig.21 is a schematic diagram of a block storage structure in one embodiment;

[0035] Fig. 22 is a schematic diagram of a block storage structure in one embodiment;

[0036] Fig.23 is a schematic diagram of a block storage structure in one embodiment;

[0037] Fig.24 is a schematic diagram of a block storage structure in one embodiment;

[0038] Fig.25 is a schematic diagram of a block storage structure in one embodiment;

[0039] Fig.26 is a schematic diagram of a block storage structure in one embodiment;

[0040] Fig. 27 is a schematic diagram of a block storage structure in one embodiment;

[0041] Fig.28 is a schematic diagram of a block storage structure in one embodiment;

[0042] Fig.29 is a schematic diagram of a block storage structure in one embodiment;

[0043] Fig.30 is a schematic diagram of a block storage structure in one embodiment;

[0044] Fig.31 is a schematic diagram of a block storage structure in one embodiment;

[0045] Fig.32 is a schematic diagram of a block storage structure in one embodiment;

[0046] Fig.33 is a schematic diagram of a block storage structure in one embodiment;

[0047] Fig.34 is a schematic diagram of a block storage structure in one embodiment;

[0048] Fig.35 is a schematic diagram of a block storage structure in one embodiment;

[0049] Fig.36 is a schematic diagram of a block storage structure in one embodiment;

[0050] Fig.37 is a schematic diagram of a block storage structure in one embodiment;

[0051] Fig.38 is a schematic diagram of a block storage structure in one embodiment;

[0052] Fig.39 is a schematic diagram of a block storage structure in one embodiment;

[0053] Fig.40 is a schematic diagram of a block storage structure in one embodiment;

[0054] Fig.41 is a schematic diagram of a block storage structure in one embodiment;

[0055] Fig.42 is a schematic diagram of a block storage structure in one embodiment. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0057] Blockchain is generally divided into three types: public blockchain, private blockchain and consortium blockchain. In addition, there are many types of combinations, such as private blockchain + consortium blockchain, consortium blockchain + public blockchain and other different combinations. Among them, the public chain has the highest degree of decentralization. Public chains are represented by Bitcoin and Ethereum. Participants who join the public chain can read data records on the chain, participate in transactions, and compete for the right to record new blocks. Moreover, each participant (reflected as the participant's node on the blockchain) can freely join and exit the network and perform related operations. On the contrary, the private chain has the write permission of the network controlled by a certain organization or institution, and the data reading permission is stipulated by the organization. In short, the private chain can be a weakly centralized system with strictly limited and few participating nodes. This type of blockchain is more suitable for use within a specific organization. The consortium chain is a blockchain between the public chain and the private chain, which can achieve "partial decentralization". Each node in the alliance chain usually has a corresponding entity or organization; participants join the network through authorization and form a stakeholder alliance to jointly maintain the operation of the blockchain.

[0058] Whether it is a public chain, a private chain or a consortium chain, in addition to supporting the transfer of native assets on the blockchain between accounts, such as the transfer of Bitcoin and Ethereum, it can also provide smart contract functions. Smart contracts on the blockchain are contracts that can be triggered and executed by transactions on the blockchain system. Smart contracts can be defined in the form of code.

[0059] Taking Ethereum as an example, supporting users to create and call some complex logic in the Ethereum network is the biggest challenge that distinguishes Ethereum from Bitcoin blockchain technology. The core of Ethereum as a programmable blockchain is the Ethereum Virtual Machine (EVM), and each Ethereum node can run EVM. EVM is a Turing-complete virtual machine, which means that various complex logics can be implemented through it. Users publish and call smart contracts in Ethereum on EVM. In fact, the virtual machine directly runs the virtual machine code (virtual machine bytecode, hereinafter referred to as "bytecode"). Smart contracts deployed on the blockchain can be in the form of bytecode.

[0060] For example Figure 1 As shown in the figure, after Bob sends a transaction containing the information of creating a smart contract to the Ethereum network, the EVM of node 1 can execute this transaction and generate the corresponding contract instance. "0x6f8ae93..." in Figure 1 represents the address of this contract. The data field of the transaction can store bytecode, and the to field of the transaction is an empty account. After the nodes reach an agreement through the consensus mechanism, the contract is successfully created and subsequent users can call this contract.

[0061] After the contract is created, a contract account corresponding to the smart contract is generated on the blockchain, and has a specific address. The contract code and account storage will be saved in the contract account. The behavior of the smart contract is controlled by the contract code, and the account storage of the smart contract saves the state of the contract. In other words, the smart contract generates a virtual account on the blockchain that contains the contract code and account storage.

[0062] As mentioned above, the data field of the transaction that creates a smart contract can store the bytecode of the smart contract. The bytecode consists of a series of bytes, each of which can identify an operation. Based on development efficiency, readability and other considerations, developers can choose a high-level language to write smart contract code instead of writing bytecode directly. The smart contract code written in a high-level language is compiled by a compiler to generate bytecode, which can then be deployed on the blockchain. Ethereum supports many high-level languages, such as Solidity, Serpent, LLL, etc.

[0063] Taking Solidity as an example, contracts written in it are very similar to classes in object-oriented programming languages. Multiple members can be declared in a contract, including state variables, functions, function modifiers, events, etc. State variables are values ​​stored in the account storage of smart contracts and are used to save the state of the contract.

[0064] The following is a code example of a simple smart contract written in Solidity language 1:

[0065] Contract SimpleStorage {

[0066] string storedData;

[0067] function set(string s){

[0068] storedData = s;

[0069] }

[0070] function get()constant returns(string){

[0071] return storedData;

[0072] }

[0073] }

[0074] In addition, if Figure 2As shown, still taking Ethereum as an example, after Bob sends a transaction containing information about calling a smart contract to the Ethereum network, the EVM of node 1 can execute this transaction and generate a corresponding contract instance. The from field of the transaction in Figure 2 is the address of the account that initiates the call to the smart contract, the "0x6f8ae93..." in the to field represents the address of the smart contract being called, and the data field of the transaction stores the method and parameters for calling the smart contract. In addition, the value field can also be included to represent the value of the Ether in the transaction. After calling the smart contract, the value of storedData may change. Later, a client can call a blockchain node (for example Figure 2 Node 6) in the view the current value of storedData.

[0075] Smart contracts can be executed independently on each node in the blockchain network in a prescribed manner, and all execution records and data are stored on the blockchain. Therefore, when such a transaction is completed, the blockchain stores transaction credentials that cannot be tampered with or lost.

[0076] As mentioned above, the storedData in the above example is a state variable, which is stored in the account storage of the smart contract. In various blockchain networks that introduce smart contracts, such as Ethereum, accounts can usually include two types:

[0077] Contract account: stores the executed smart contract code and the value of the state in the smart contract code, and can usually only be activated by calling an external account;

[0078] Externally owned account: A user's account, such as an Ethereum owner account.

[0079] The design of external accounts and contract accounts is actually a mapping from account addresses to account status. Account status usually includes fields such as Nonce, Balance, Storage root, CodeHash, etc. Nonce and Balance exist in both external accounts and contract accounts. CodeHash and Storage root properties are generally only valid on contract accounts.

[0080] Nonce: Counter. For external accounts, this number can represent the number of transactions sent from the account address; for contract accounts, it can be the number of contracts created by the account.

[0081] Balance: The amount of ether owned by this address.

[0082] Storage root: The hash of the root node of an MPT tree, which organizes the storage of state variables of the contract account.

[0083] CodeHash: The hash value of the smart contract code. For contract accounts, this is the hash value of the smart contract; for external accounts, since they do not include smart contracts, the CodeHash field can generally be an empty string / all-0 string.

[0084] MPT stands for Merkle Patricia Tree, which is a tree structure that combines Merkle Tree and PatriciaTree (compressed prefix tree, a more space-saving Trie tree, dictionary tree). Merkle Tree, the Merkle tree algorithm calculates a hash value for each transaction, and then connects two by two to calculate the hash again until the top Merkle root. Ethereum uses an improved MPT tree, such as a 16-way tree structure, which is usually referred to as MPT tree.

[0085] The data structure of the Ethereum MPT tree includes a state trie. The state trie contains key-value pairs (also written as key-value, abbreviated as kv or kv) of the storage content corresponding to each account in the Ethereum network. The "key" in the state trie can be a 160-bit identifier (such as the address of an Ethereum account or part of the hash value of an address, collectively referred to as an account address below), which is distributed in the storage from the root node of the state trie to the leaf node. The "value" in the state trie is generated by encoding the information of the Ethereum account (using the Recursive-Length Prefix encoding (RLP) method). As mentioned above, for external accounts, the values ​​include nonce and balance; for contract accounts, the values ​​include nonce, balance, codehash, and storageroot.

[0086] The contract account is used to store the states related to the smart contract. After the smart contract is deployed on the blockchain, a corresponding contract account will be generated. This contract account generally has some states, which are defined by the state variables in the smart contract and generate new values ​​when the smart contract is created and executed. The smart contract generally refers to a contract that can automatically execute clauses defined in the form of code in the blockchain environment. Once an event triggers the clauses in the contract (meets the execution conditions), the code can be automatically executed. In the blockchain, the relevant states of the contract are saved in the storage trie, and the hash value of the storage trie root node is stored in the above storageroot, so that all states of the contract are locked to the contract account through hash. The storage trie is also an MPT tree structure that stores the key-value mapping from the state address to the state value. Part of the information from the root node to the leaf node of the storage trie tree is arranged in sequence to store the address of a state, and the leaf node stores the value of the state.

[0087] like Figure 3 In some blockchain data storages shown, the block header of each block includes several fields, such as the previous block hash previous_Hash (Prev Hash in the figure), the random number Nonce (in some blockchain systems, this Nonce is not a random number, or in some blockchain systems, the Nonce in the block header is not enabled), the timestamp Timestamp, the block number Block Num, the state root hash State_Root, the transaction root hash Transaction_Root, the receipt root hash Receipt_Root, etc. Among them, the Prev Hash in the block header of the next block (such as block N+1) points to the previous block (such as block N), which is the hash value of the previous block. In this way, the next block on the blockchain is locked to the previous block through the block header. Among them, State_Root, Transaction_Root and Receipt_Root lock the state set, transaction set and receipt set respectively. The state set, transaction set and receipt set organize the state, transaction and receipt in the form of a tree respectively. Generally, it can be the same tree structure or different tree structures. For example, in Ethereum, the same MPT structure is used. In some tree structures including the state set of smart contracts such as Ethereum, there are two levels of MPT structures: the leaf nodes of the upper-level MPT structure include two types: external accounts and contract accounts; each contract account includes the MPT structure of the lower level, and the leaf nodes of the lower level include the value of the state in the contract account.

[0088] Figure 4This is a schematic diagram of the structure of blockchain data storage. Taking Ethereum as an example, we can combine Figure 3 As shown in the figure, state_root is the hash value of the root of the MPT tree composed of the status of all accounts in the current block, that is, the state trie pointing to state_root is a state tree in the form of MPT. The root node of this MPT tree is generally an extension node (Extension Node) or a branch node (Branch Node), and the hash value of this root node is generally stored in state_root. The root node can be connected to one or more layers of Extension Node / Branch Node below, and these multi-layer tree nodes can be collectively referred to as intermediate nodes (Internal Node). From the root node of this MPT to the leaf node, a part of the value in each node is connected in sequence to form an account address and serve as a key. The account information stored in the leaf node is the value corresponding to this account address, thus forming a key-value pair. This key can also be a part of sha3(Address), that is, a part of the hash value of the account address (the hash algorithm uses the sha3 algorithm, for example), and the value stored can be rlp(Account), that is, the rlp encoding of the account information. The account information is

[0089] A tuple consisting of [nonce, balance, storageRoot, codeHash]. As mentioned above, for external accounts, there are generally only two items, nonce and balance, and the storageRoot and codeHash fields store empty strings / all zero strings by default. In other words, external accounts do not store contracts, nor do they store state variables generated after contract execution. Contract accounts generally include Nonce, Balance, Storage root, and CodeHash. Among them, Nonce is the transaction counter of the contract account; Balance is the account balance; Storage root corresponds to another MPT, through which the storage root can be linked to the information of the contract-related status; CodeHash is the hash value of the contract code. Whether it is an external account or a contract account, its account information is generally located in a separate leaf node (Leaf Node). From the Extension Node / Branch Node of the root node to the Leaf Node of each account, there may be several branch nodes and extension nodes in the middle.

[0090] The state trie can be a tree in the form of MPT, generally a 16-way tree, that is, each layer can have up to 16 child nodes. For Extension Node, it is used to store common prefixes, and it generally has 1 child node, which can be Branch Node. For Branch Node, it can have up to 16 child nodes, which may include ExtensionNode and / or Leaf Node.

[0091] Among them, for a contract account in the state trie, its storage_Root points to another tree in the form of MPT, which stores the data of the state variables involved in the contract execution. The MPT-style tree pointed to by this storage_Root is Storage Trie, that is, the hash value of the root node of Storage Trie. Generally, this Storage Trie tree also stores key-value pairs. The key indicates the address of the state variable, and its value can be the result obtained after the position of the state variable declaration in the contract (the value counting from 0) is processed by certain rules, such as sha3 (the position of the state variable declaration) or sha3 (the contract name + the position of the state variable declaration). The value is used to store the value of the state variable (for example, the value encoded by RLP). A part of the data stored on the path from the root node to the intermediate node to the leaf node is connected to form the key, and the value is stored in the leaf node. As mentioned above, the Storage trie can also be a tree in the form of MPT, which is generally a 16-way tree, that is, for a Branch Node, it can have up to 16 child nodes, which may include Extension Nodes and / or Leaf Nodes. For an ExtensionNode, it can generally have 1 child node, which can be a Branch Node or a Leaf Node.

[0092] For example Figure 4The Leaf Node Account P of the state trie in the contract is a contract account, and its Storage Root locks all states in the contract storage. These states are organized into an MPT tree, and the tree structure is like the Storage trie linked to the Storage Root. In this linked Storage trie, taking Leaf Node StateVariable N as an example, for example, the value of storedData in the aforementioned contract code example, its key is sha3 (the declaration position of storedData, that is, the second line of the code), and its value is s (for simplicity, the encoding format of value is omitted here, for example, RLP, which will be similar in the future and will not be repeated). Among them, the key values ​​are distributed sequentially from the root node to the leaf node (that is, Leaf Node Variable N) of the storage trie.

[0093] For another example, Figure 4 Leaf Node Account C in the state Trie is an external account, and its key is sha3(Address C), which is the hash value of the address of account C (the hash algorithm uses the sha3 algorithm, for example). The stored value value can be (Account), where the account information Account is a tuple consisting of [nonce, balance]. As mentioned above, since Account C is an external account, its account information is two items, nonce and balance (codehash and storage root are omitted here, and the following are similar). For example, an external account, whose nonce is 20 and Balance is 4550, then the Leaf Node State Variable C stores nonce=20 and balance=4550. The address of Account C is the key, and its values ​​are distributed sequentially from the root node to the leaf node (i.e., Leaf NodeVariable C) of the state Trie.

[0094] These states, including the kv of external accounts and the kv of contract accounts, are ultimately stored in the database. The storage in the database does not directly store the states of these accounts, that is, it does not directly store the kv of these accounts, but stores the kv value of each tree node itself.

[0095] like Figure 5As shown in the example, in the upper-level MPT structure, for leaf node A1, the key of the leaf node is composed of a7 of the shared nibble in the root node A8 (Extension Node)—slot 1 of the intermediate node A7 (Branch Node)—1335 of the key-end in the leaf node A1, which is a711335, and Balance=45.0ETH and Nonce=n1 are stored in the leaf node. For leaf node A2, the key of the leaf node is composed of a7 of the shared nibble in the root node A8 (Extension Node)—slot 7 of the intermediate node A7 (Branch Node)—d3 of the shared nibbles in the node A6 (Extension Node)—slot 3 of the intermediate node A5 (Branch Node)—7 of the key-end in the leaf node A2, which is a77d337, and Balance=1.00WEI and Nonce=n2 are stored in the leaf node. For leaf node A3, the key of the leaf node is composed of a7 of the shared nibble in root node A8 (Extension Node), slot f of intermediate node A7 (Branch Node), and 9365 of key-end in leaf node A3, which is a7f9365. Balance = 1.1ETH and Nonce = n3 are stored in the leaf node. For leaf node A4, the key of the leaf node is composed of a7 of the shared nibble in root node A8 (Extension Node), slot 7 of intermediate node A7 (Branch Node), d3 of the shared nibbles in node A6 (Extension Node), slot 9 of intermediate node A5 (Branch Node), and 7 of key-end in leaf node A4, which is a77d397. Balance = 0.12ETH, Nonce = n4, CodeHash = c1, and Storage root = s1 are stored in the leaf node. s1 can be H(A10), which is the hash of the root node A10 of the next tree layer. The leaf nodes of A1, A2 and A3 store the information of external accounts, and the leaf node of A4 stores the information of contract accounts. For contract accounts, it contains the next level MPT, forming a Storage Trie, which is used to store the state variables in the contract account.

[0096] like Figure 5As shown in the example, in the next-level MPT structure, for leaf node A11, slot 3 in root node A10 (Branch Node) - 35b2e4 of key-end in leaf node A11 are sequentially combined to form the key of the leaf node, that is, 335b2e4, and "Zhang San_A = 20" is stored in the leaf node, for example, indicating that the share of type A digital assets defined in the contract belonging to Zhang San is 20, that is, the balance of Zhang San's type A assets is 20. For leaf node A12, slot 7 in root node A10 (Branch Node) - c25988 of key-end in leaf node A12 are sequentially combined to form the key of the leaf node, that is, 7c25988, and "Li Si_B = 20" is stored in the leaf node, for example, indicating that the share of type B digital assets defined in the contract belonging to Li Si is 50, that is, the balance of Li Si's type B assets is 50. For leaf node A15, the key of the leaf node is composed of slot f in root node A10 (Branch Node) - a in shared nibble in intermediate node A13 (Extension Node) - slot 6 in intermediate node A14 (Branch Node) - be33 of key-end in leaf node A15, which is fa6be33, and "storedData=s" is stored in the leaf node. For leaf node A16, the key of the leaf node is composed of slot f in root node A10 (Branch Node), a in shared nibble in intermediate node A13 (Extension Node), slot 9 in intermediate node A14 (Branch Node), and 9365 in key-end in leaf node A16, which is fa99365. "Wang Wu_A=35" is stored in the leaf node, which means that the share of type A digital assets defined in the contract belonging to Wang Wu is 35, that is, the balance of Wang Wu's type A assets is 35.

[0097] In the node structure of the above MPT tree, the prefix prefix is ​​used to indicate the tree node type, for example, 0 indicates an Extension Node containing an even number of shared nibbles (shared half bytes), 1 indicates an Extension Node containing an odd number of shared nibble(s), 2 indicates a Leaf Node containing an even number of nibbles, and 3 indicates a Leaf Node containing an odd number of nibble(s).

[0098] In the above node structure, the hash value of the entire content of the next tree node is filled in the corresponding position of the previous tree node. In the database, the key-value mapping of each tree node is actually stored, where the value includes the content stored in this tree node, and the corresponding key is the hash value of the entire content of this tree node. In this way, the tree node kv actually stored in the database is as follows:

[0099]

[0100]

[0101] Table 1. Tree node kv actually stored in the database

[0102] In Table 1 above, H() is used to represent hash calculation. In this way, the hash value of the next tree node is anchored in the previous tree node. Through such layer-by-layer hashing, the root hash of the entire state trie tree is obtained, and the root hash is locked in the state root field of the block header.

[0103] In some blockchain systems, the code of the blockchain platform may include a P2P (Peer to Peer) module, a consensus module, an execution module, and a storage module. P2P is a way of forming a computer network. Unlike common web networks, P2P is decentralized. The P2P module can complete the distributed dissemination of data. For blockchain nodes, the P2P module can be used to disseminate and receive receipts in a peer-to-peer manner. Different participants can establish a distributed blockchain network through deployed nodes. The ledger constructed using a chain block structure is stored on each node (or most nodes, such as consensus nodes) in the distributed blockchain network. This is also called a decentralized (or multi-centralized) distributed ledger. Such a blockchain system needs to solve the problem of consistency and correctness of the ledger data on multiple decentralized (or multi-centralized) nodes. The same blockchain platform program is running on each node. Under certain fault tolerance requirements, the consensus module can ensure that all loyal nodes have the same transactions, thereby ensuring that all loyal nodes have consistent execution results for the same transaction, and packaging the transactions and execution results to generate blocks. The current mainstream consensus mechanisms include: Proof of Work (POW), Proof of Stake (POS), Delegated Proof of Stake (DPOS), Practical Byzantine Fault Tolerance (PBFT) algorithm, Honey Badger BFT algorithm, etc. During the consensus process, the consensus module can generally generate the timestamp of the block corresponding to the current transaction set. The execution module can execute transactions, including ordinary transfer transactions and transactions involving contracts, before or after the consensus module completes the consensus. For transactions involving contracts, the execution module can introduce a virtual machine to execute the code of the smart contract, such as the Ethereum Virtual Machine (EVM), so that the differences in the hardware configuration and software environment of each node are shielded by the EVM to ensure that the process and results of executing the smart contract on each node are the same, and the sandbox environment is used to prevent the execution of the smart contract from affecting the blockchain platform code, other programs or operating systems on the host. For a situation of a consortium chain, the nodes can determine the transaction content and transaction order in a transaction set through the consensus module, and then output a deterministic transaction set of the consensus result to the execution module. The execution module generates the execution result by executing ordinary transfer transactions / transactions involving contracts, and sends it to the storage module. The storage module can be responsible for storing the execution result in the local persistent storage medium of the node.

[0104] like Figure 6 In the blockchain node shown, the physical components include CPU, memory, disk, etc. The blockchain platform code executed by this blockchain node may include P2P module, consensus module, execution module and storage module. The functional implementation of P2P module, consensus module and execution module generally requires the participation of CPU and memory. The storage module may include a tree building module, a block header generation module, a WAL (Write Ahead Log) module and a state database module. Among them, the tree building module is used to build a tree (for example, an MPT tree) based on the state kv passed in by the execution module, such as the aforementioned state trie and storage trie, so as to obtain the kv of the tree node, which generally requires the participation of CPU and memory. The block header generation module is used to generate a block header based on the root node of the tree built by the tree building module and some other data (such as the hash of the previous block, timestamp, block number, etc.), which generally requires the participation of CPU and memory. The WAL module is used to persistently store the leaf node kv of the tree generated by the tree building module before writing the leaf node kv of the tree generated by the tree building module to the state database module, so as to prevent data loss caused by power failure and other situations during the process of writing the leaf node kv of the tree generated by the tree building module to the state database module, and to restore the data when this happens, which generally requires the participation of the CPU, memory and disk. The state database module is used to store the tree node kv in Table 1 constructed by the tree building module on a persistent storage device; since the tree node data will eventually be written to a persistent storage medium (such as the disk in the figure), the state database module generally requires the participation of the disk in addition to the CPU and memory.

[0105] From the perspective of storage structure, the above-mentioned Merkle tree structure, such as Ethereum's MPT and Libra's SMT (SparseMerkle Tree, similar to MPT), can be located in the tree construction module in the form of the corresponding relationship in Table 1 above and stored in the memory. Among them, the upper Merkle tree is a prefix tree (dictionary tree), which can realize the organization of data and obtain a unique Merkle root for the organized data. The leaf node can save the state Value, and the root node to the intermediate node to the leaf node implements the lexicographic index of the state key. These tree nodes are encoded as Key according to certain rules, and their contents are encoded as Value, and finally stored in the lower database. Most databases use NoSQL Key-Value DB (DataBase, database; Key-Value DB is also referred to as KVDB) with LSM (Log-Structured Merge-Tree, log structure merge tree) class structure, which is located in the state database module and finally saved on disk. Specifically, the database is, for example, Ethereum's levelDB and Libra's RocksDB. Both KVDBs are based on the LSM storage engine.

[0106] The LSM storage engine is a hierarchical, ordered, disk-oriented storage engine. It draws on the characteristics of continuous appending (rather than modification) of Logs. The core idea is to make full use of the fact that sequential batch writes of disks are much more efficient than random writes, and give up some read efficiency in exchange for maximum write efficiency. Generally speaking, the way to maximize the use of disk characteristics is to read or write a fixed-size block of data at one time and reduce random addressing operations as much as possible. The design idea of ​​LSM is based on this characteristic of disks and assumes that there is enough memory. It does not need to write data to disk every time it is updated. Instead, the latest data resides in memory first. After the amount of data accumulates enough, the data in memory is merged with the data on disk using merge sort and appended to disk in batches.

[0107] The following Figure 7As an example, the storage process of the state data module is further explained. After the tree building module generates the tree node kv data and sends it to the storage module, these kv data are first written to WAL in an Append-Only manner and stored on the disk. Then, the tree node kv data is passed to the state database module. As mentioned above, writing to WAL first can prevent data loss caused by power outages and other situations during the process of writing the tree node kv generated by the tree building module to the state database module, and restore the data when this happens. After the state database module receives the tree node kv data, it is first stored in the memory, specifically in the MemTable in the memory. MemTable can provide concurrent read and write operations. There can be multiple MemTables in the memory. When the amount of data in the MemTable reaches a certain threshold, for example, when it reaches 256MB, the data in the MemTable can be written (flush) to the disk. In order to avoid the blockage of flush caused by the operation of writing MemTable, this MemTable is converted into an immutable Immutable Memtable, that is, Immutable Memtable is set to read-only, and a new MemTable is generated to receive the newly passed in tree node kv. This new MemTable can provide concurrent read and write operations (such as New MemTable in the figure). The storage engine writes the data in Immutable MemTable to disk.

[0108] In the disk, KVDB is stored in multiple levels of SST (Sorted String Table, SSTable) files. SST comes from the paper of Google Bigtable. BigTable is a distributed data storage system designed by Google, a non-relational database used to process massive amounts of data. SST can be used to store data files inside Bigtable, and is an efficient key-value file storage format. Content stored in persistent storage media can also be called files.

[0109] The SST may include multiple layers, for example, 3 layers, 4 layers, 5 layers, 6 layers, 7 layers or more. Figure 7The figure shows the situation of three layers, namely level 0, level 1, and level 2. Generally, the total capacity of the SST of the previous layer is significantly smaller than that of the next layer. As an example, the total capacity of the SST of level 0 is 1GB, the total capacity of the SST of level 1 is 10GB, and the total capacity of the SST of level 2 is 100GB. Assume that the capacity of MemTable is 256MB. The tree module can be written with the tree node kv corresponding to the block, and then written to MemTable. After the tree kv corresponding to one or more blocks is written to MemTable, the space occupied by MemTable may reach 256MB. Then, this MemTable is converted into Immutable MemTable, and the data in ImmutableMemTable can be written (flush, here refers to the operation of writing data in memory to disk) to disk. Specifically, the data in MemTable can be flushed to the SST of Level 0 on the disk. On the other hand, as mentioned above, a new MemTable is generated to receive the newly incoming tree node kv and provide concurrent read and write operations.

[0110] As mentioned above, each layer of disk storage, that is, each level of disk storage, can include multiple SST files. The kv stored in the SST of Level 1 and lower levels is generally stored in the order of the key, and the key values ​​in different SSTs of the same layer are also related in size and are not repeated. For example, Level 1 includes 3 SSTs, the SST on the left 11 It includes 1000 kv, for example <k1-v1> <k2-v2> <k3-v3>... <k 1000 -v 1000 >, and arranged in order from small to large k, for example, k1 = 10b43c032a46, k2 = 1118d46f41e1, k3 = 1b44942a3d6e, ..., k 1000 =2018df9326a7. It can be seen that the SST on the left 11 k1<k2<k3<...<k 1000 , but k1, k2, k3, ... k 1000 are not necessarily continuous. Similarly, the SST in the middle 12 For example, it includes 800 kv, <k1-v1> <k2-v2> <k3-v3>... <k 800 -v 800 >, and arranged in order from small to large k, for example, k1 = 21a514c0a7e1, k2 = 21b46b994214, k3 = 21b489571b6d, ..., k 800 =284a5df3fca7. It can be seen that the SST in the middle 12 k1<k2<k3<...<k 800 , but similarly, k1, k2, k3, ... k 800 are not necessarily continuous. Similarly, the SST on the right 13 For example, it includes 1100 kv, <k1-v1> <k2-v2> <k3-v3>... <k 1100 -v 1100 >, where k is arranged in ascending order, for example, k1 = 3a5e55ac3fe1, k2 = 3a5e55ac3fe2, k3 = 3afb8ed31526, ..., k 1100 =400a51ef1405. It can be seen that the SST on the right 13 k1<k2<k3<...<k 1100 , but similarly, k1, k2, k3, ... k 1100 Moreover, overall, the left SST 11 The minimum value of k is less than the left SST 11 The maximum value of k is less than the intermediate SST 12 The minimum value of k < the intermediate SST 12 The maximum value of k is less than the right SST 13 The minimum value of k is less than the right SST 13 The same is true for each layer of Level 2, Level 3, etc., that is, within each SST in each layer, there is a size order relationship in the value of k, and between multiple SSTs in each layer, there is a size relationship in the range of k.

[0111] The kv in an SST in Level 0 can be sorted according to the size of k, and there may be no size relationship between k in multiple SSTs. This is because MemTable generally stores multiple kvs corresponding to each block in the order of blocks from old to new (or block numbers from small to large, because the block numbers of new blocks are larger and the block numbers of old blocks are smaller), and these k are the hash values ​​of tree nodes. The changes in the content of tree nodes are likely to cause the hash values ​​of tree nodes (i.e., k values) to be very different. The kv in an SST in Level 0 can be sorted according to the size of k. Such sorting is efficient and feasible in memory; of course, it is also possible not to sort according to the size of k values, which is not excluded here. Taking sorting as an example, when the capacity of MemTable reaches 256MB, it may store multiple kvs corresponding to multiple blocks, and this MemTable is converted to Immutable MemTable. These kvs in Immutable MemTable can be sorted in the order of k and written into the SST on the left of Level 0. There may be multiple different Immutable MemTables written to the same SST before the SST on the left side of Level 0 is written. 01 Once it is full, the SST on the right side of Level 0 can be written in a similar way. 02 . Although the kv written by a single Immutable MemTable to a certain SST in Level 0 is sorted, since the kv sets written by different Immutable MemTables in the same SST in Level 0 are generally not sorted by size, and the kvs between different SSTs (such as the left and right SSTs) are also not sorted by size, that is, the ranges of k in the left and right SSTs in Level 0 may overlap. Similarly, since the kv written to an SST in Level 0 may be written after multiple MemTables are full, and the kv ranges in the writing process after multiple MemTables are full generally overlap, there is generally no specific order between the multiple kvs stored in each SST in Level 0, and there is generally no specific range order between the SSTs in Level 0. Of course, if an SST is written by an Immutable MemTable in Level 0, since the Immutable MemTable sorts the kvs when writing to the SST, the kvs stored in the SST are also sorted by k.

[0112] Furthermore, when the storage capacity of Level 0 reaches or approaches the upper limit, the data of Level 0 is written to Level 1 through a process called "compaction". During this compaction process, the kv in each SST in Level 0 and the kv of some or all SSTs in Level 1 can be transferred to the memory first, sorted in the memory, and then written to the SST of Level 1. Since sorting is performed during the compaction process, there is an order relationship in the value of k in the SST in Level 1, and there is a size relationship in the range of k between multiple SSTs in Level 1. In other words, in each SST in Level 1, kv is sorted according to <k1-v1> <k2-v2> <k3-v3>... <k n -v n >, and k1<k2<k3<...<k n , and k1, k2, k3, ... k n They are not necessarily continuous. Moreover, overall, for two adjacent SSTs, the minimum value of k of the left SST < the maximum value of k of the left SST < the minimum value of k of the right SST < the maximum value of k of the right SST. Figure 8 as shown in .

[0113] During the compaction process, the kvs in each SST in Level 0 and the kvs in some SSTs in Level 1 can be transferred to the memory for sorting. The said some SSTs in Level 1 can be the SSTs in Level 1 that have overlapping ranges with the kvs in Level 0, that is, the SSTs in Level 1 that have overlapping ranges with the kvs in each SST in Level 0. Specifically, for example, in the above example, Level 1 includes 3 SSTs, SST 11 The key range is 10b43c032a46~2018df9326a7, SST 12 The key range is 21a514c0a7e1~284a5df3fca7, SST 13 The key range is 3a5e55ac3fe1~400a51ef1405. For example, the key range of each SST in Level 0 is 18c49a405e32~24d35f39ab56. It can be seen that there are SSTs in Level 1 that overlap with the key range in Level 0. 11 and SST 12 In this way, during the compaction process, the kv in each SST in Level 0 and the SST in Level 1 11 、SST 12 The kv can be transferred to the memory and sorted, and then the sorted results are written into the SST of Level 1.

[0114] Similarly, when the storage capacity of Level 1 reaches the upper limit, the data of Level 1 is written to Level 2 through the compaction process. Similarly, during this compaction process, the kvs in each SST in Level 1 and the kvs of some or all SSTs in Level 2 can be first transferred to the memory, sorted in the memory, and then written to the SST of Level 2. Similarly, if there is Level 3 under Level 2, when the storage capacity of Level 2 reaches the upper limit, the data of Level 2 is written to Level 3 through the compaction process. Similarly, during this compaction process, the kvs in each SST in Level 2 and some or all SSTs in Level 3 can be first transferred to the memory, sorted in the memory, and then written to the SST of Level 3. And so on.

[0115] In this way, overall, the data stored in the upper SST is newer than that in the lower SST. The latest data is stored in memory, the next latest data is stored in Level 0, and the oldest data is stored in the SST in the lowest level. In addition, compaction can also be used to perform garbage collection, such as deleting data marked for deletion.

[0116] The sorted kv can be retrieved efficiently, for example, by binary search. Binary search is also called binary search, and the algorithm is as follows: define left as the leftmost element position of the sequence list, right as the rightmost element position of the sequence list, define mid = (left + right) / 2, that is, the middle position of the sequence list; compare the value being searched with the value at the mid position; since the list is ordered, if the value being searched is smaller than mid, then only the first half of the table needs to be searched, otherwise only the second half of the table needs to be searched (if the two values ​​are found to be equal in the first comparison, the position of the current value is directly returned). And so on, until the value being searched is found or it is determined that the value being searched is not in the list (that is, the search fails).

[0117] Such a storage engine has read amplification and write amplification problems when reading and writing data, which are described in detail below.

[0118] In the blockchain system using the above storage engine, when the execution module executes a transaction, it may be necessary to read an existing state. For example, for a general transfer transaction, when external account A transfers a certain amount of assets (for example, f) to external account B, it is necessary to increase or decrease the account balance. Assuming that before the execution of the transaction, the balance of account A is a and the balance of account B is b, then the execution of the transaction is to set the balance of account A to the value of (af) and the balance of account B to the value of (b+f). Then this transaction needs to read the values ​​of the two state variables a and b when it is executed. For another example, in a transaction that calls a contract, external account C initiates a call to contract D, then C initiates a transaction pointing to the address of contract account D, and specifies the called function and input parameters in the data field of the transaction. Assuming that the logic of the function called in contract D includes judging the value of a global state d in the contract, and assuming that the value of this d is a Boolean type, the judgment logic includes, if it is true, execute branch 1, and if it is false, execute branch 2. Then this transaction that calls the contract needs to read the value of the state d when it is executed.

[0119] During the transaction execution process, whether it is reading the values ​​of the state variables a and b as mentioned above, or reading the value of the state variable d, the execution module can first search whether the value of the state variable exists in the memory of the execution module. If not, a read instruction is sent to the storage module. The read instruction may include the address of the state to be read, that is, the key value of the state. After receiving the read instruction, the storage module finds the root node of the state tree node in the construction tree module, and matches the state key value to be searched with the shared nibble(s) field (for Extension Node) or slot (for Branch Node) of the root node from the beginning, and then reads the hash of the next layer of tree nodes (which can be Internal Node or Leaf Node, taking Internal Node as an example) from the matching position, and then first searches for the hash in the memory of the construction tree module. If it is not found, it searches in the memory of the state database module. If it is still not found, it needs to be searched in the file of the state database module. Specifically, you can search in the SST in Level 0 first. If you still can't find it, continue to search in a certain SST in Level 1. Continue to search in a certain SST in the lower level until you find the next Internal Node pointed to by the hash value. After unlocking this Internal Node, continue to match the remaining fields of the state address key value to be read from the front to the back. If it matches, read the hash value from the matching point and jump to the next level tree node pointed to by the hash value. Repeat this process, unlock the Internal Node level by level and match the remaining fields of the state address key value to be read from the front to the back. The hash value at the matching point is used as the basis for the next search for the intermediate node or leaf node until the Leaf node is matched, so as to read the value of the state from the Leaf node. In this process, some of the kvs of the tree nodes at each level may not exist in the memory of the tree construction module, but in the state database module. Therefore, it may be necessary to read from the state database module to the memory for operation. You can first search for this key in the MemTable or Immutable MemTable of the state database module. If it does not exist, you need to further search the SST files at all levels on the disk. For the Internal Node and Leaf node that exist on the disk, you need to load them into the memory and decode (deserialize) them to get the content. Specifically, for example, first search the sst files of level 0 one by one. If each SST of Level 0 is sorted, you can do a binary search in each SST based on the hash value of the intermediate node / leaf node. After finding it, load v into the memory and decode and match it.If the key is not found in the SST of Level 0, a binary search is performed in Level 1 based on the hash value of the intermediate node / leaf node. If the key is not found, a binary search is performed in Level 2. If the key is found, the v corresponding to the k is loaded into memory and decoded. This process is repeated until the leaf node of the tree structure is matched and the value is read from it. When searching in Level 1, Level 2, and lower levels, since the overall SST in these levels is sorted, a binary search can generally be performed in this level.

[0120] From the above process of reading a state kv, it can be seen that this reading process needs to perform binary search in SSTs of multiple different levels according to the hash value of the tree node. This may require reading a large number of files in SSTs of multiple levels and loading them into memory, which brings about the problem of read amplification. Assuming that the database includes Level 0, Level 1, Level 2, ..., Level M-1, a total of M layers, then in the worst case, reading the data of a state requires at least M accesses to the disk (greater than M because in the worst case, the N SSTs of Level 0 may require more than 1 and less than or equal to N searches).

[0121] In addition, if the tree building module lacks the kv of a certain tree node in the process of building the kv of all tree nodes on the path to the root node according to the state kv, it may also need to do a binary search from Level 0, Level 1, Level 2, ..., etc., and the principle is similar to the above process. In the above transaction example, for example, after the transaction is executed, the value of the execution result (af) / the value of (b+f) / the value of the contract state d needs to be written into the state database. For the above values ​​to be written to the database, the execution module calls the storage interface of the storage module and outputs the key-value value of the state to the storage module. The tree building module in the storage module generates the tree node kv from the root of the tree structure to the intermediate node and the leaf node according to the state kv, and also writes these tree node kvs into the WAL module, and then writes the kv of this series of tree nodes into the MemTable of the state database module and marks it as dirty data. Dirty data usually refers to data that has been written to the MemTable but has not yet been written to the persistent storage (such as SSTable). Dirty data can be written to the disk in the aforementioned flush process. The state database module then writes some kvs in MemTable, including the tree node kv, into the SST file on disk. Once MemTable is persisted to SSTable, the data in it is no longer dirty data, that is, it becomes "clean" data. It can be seen that the above process of persistently storing a state kv is relatively complicated, consumes more computing and memory resources, and brings about the problem of write amplification.

[0122] The complexity of the above-mentioned reading and writing processes and the problems of read amplification and write amplification pose challenges to the high performance requirements and large-scale applications of blockchain.

[0123] The present application provides a method embodiment for reading blockchain state data. The method embodiment can be applied to a storage module of a blockchain node. The storage module constructs a world state with a tree structure in a memory. The leaf nodes of the tree structure may include the content of the state data or the position index of the state data in a data file in a persistent storage medium.

[0124] In an example, the tree structure as a whole can be Fig.10 As shown, this is also consistent with Figure 5 In particular, if the leaf node in memory contains state data, then Figure 5 As shown, for example Figure 5 The leaf node A11 in the memory has state data, that is, Zhang San_A=20. If there is no state data in the leaf node in the memory, the position index of the state data in the data file can be used.

[0125] In this application, the actual data and index are separated, which helps to improve the flexibility and performance of the system. The basic concepts involved here include data files and index files. Among them, the data file is a file that stores the actual content; the index file is an index that stores the location of the data file. Its working principle is: write the data content to the data file, record the location of the data in the file (such as file name, offset and length); then, create an entry in the index file, including: a key (a unique identifier for search) pointing to a reference to the data file (file name, offset, etc.). In this way, when retrieving data, the data location can be quickly located through the index file, and then the actual content can be read from the data file according to the location information. The advantages of this method are: ① flexibility, data and indexes can be managed and optimized independently; ② performance, indexes are usually small and can be fully loaded into memory to speed up queries; ③ space efficiency: avoid storing a large amount of duplicate data in the index; ④ update efficiency: when modifying data, only the location information in the index needs to be updated.

[0126] For example Fig.10 In the example, location is used to represent the location index. Still taking the A11 node as an example, if there is no state data in the memory, the location field included in it can be used to store the location index of the data file where the state data is located. The location index is, for example, <2,750,210>. This <2,750,210> can indicate that the state data is located in "file 2" in the data file, the starting position is the 750KB of "file 2", and the length is 210KB. Fig.10 As shown in the data file in the lower left corner, assuming that the size of each file is 1024KB, the logical address range of the index is 0KB to 1023KB. In this way, the status data of A11 actually occupies the range from 750KB to 959KB in file 2.

[0127] like Fig. 9 As shown, this embodiment includes:

[0128] S910: The storage module of the blockchain node receives an instruction for the state key to be searched, and searches for a leaf node corresponding to the state key in the tree structure in the memory.

[0129] As mentioned above Figure 6 As shown in , the execution module can generate an execution result by executing a normal transfer transaction / a transaction involving a contract, and send it to the storage module. Specifically, for example, the value of a state variable needs to be read during the execution of a contract. The execution module can first search the memory of the execution module to see whether the value of the state variable exists. If not, a read instruction is sent to the storage module. The read instruction may include the address of the state to be read, that is, the key value of the state.

[0130] The storage module includes a tree construction module. The tree construction module constructs a tree structure with the world state in the memory. If the blockchain node is cold started, that is, there is no tree structure of the world state in the memory, then the tree nodes of the tree structure can be loaded from the disk and organized into a tree in the memory. This process also involves loading the leaf nodes in the tree structure. Here, an example is taken to illustrate the case where there is an incomplete tree structure in the tree construction module. The case of cold start loading is similar. Since the capacity of the memory is generally limited, the LRU (Least Recently Used, a commonly used page replacement algorithm, selects the page that has not been used for the longest time to be eliminated) algorithm is generally used to eliminate tree nodes that have not been accessed for a long time from the memory. In this way, each leaf node on the tree structure constructed in the memory by the tree construction module has a certain possibility of being eliminated. For the leaf nodes of the world state tree, some may be in the memory and some may not be in the memory. In addition, for the content in the tree node, such as the state data in the leaf node, a similar elimination mechanism can also be used, or the state data may not be kept in the memory.

[0131] For the state key to be found, if the memory of the tree building module has the leaf node corresponding to the state key, it can be found directly in the memory. In this case, you can refer to Figure 5 , directly find the key of the corresponding leaf node in the memory, and then read the value from it. Then, the state data can be read from the value. If the memory of the tree building module does not have the leaf node corresponding to the state key, or even if there is the leaf node, the state data to be found does not exist in it, then it is necessary to further search from the state database module. For the case where the state database uses an LSM tree, as mentioned above, you can first search in the MemTable, and if it is not found, search in the file on the disk.

[0132] S920: Obtain a position index contained in a leaf node corresponding to the tree node key, where the position index points to a position in a data file in the persistent storage medium.

[0133] If the leaf node to be searched is present in the memory of the tree building module, but the state data does not exist in it, it is necessary to further search from the state database module. As mentioned above, if there is no state data in the memory, the location field included in it can be used, and its content is, for example, Fig.10 The location of the leaf node A11 is <2, 750, 210>. This <2, 750, 210> is the location index, which indicates that the status data is located in file 2 of the data file, starting at the 750KB and with a length of 210KB. Fig.10 As shown in the data file in the lower left corner, assuming that the size of each file is 1024MB, the logical address range of the index is 0KB to 1023KB. In this way, the status data of A11 actually occupies the range from 750KB to 959KB in file 2.

[0134] If the leaf node to be found is not in the memory of the tree building module, it is necessary to find and load the leaf node from the state database module. Finding and loading the leaf node from the state database module specifically includes loading the index file and data file of the leaf node. Assume Fig.10 The corresponding underlying database stores the kv of all intermediate nodes in the tree structure, as shown in the following table:

[0135]

[0136] Table 2. Tree node kv actually stored in the database

[0137] In Table 2 above, similar to Table 1, H() is used to represent hash calculation. In this way, the hash value of the next tree node is anchored in the previous tree node. Through such layer-by-layer hashing, the root hash of the entire state trie tree is obtained, and the root hash is locked in the state root field of the block header. Assume that the kv pairs in this table are stored in the underlying state database module, saved on disk, and use a similar LSM structure. In this way, similar to the above, after the storage module receives the read instruction, it can find the root node of the state tree node in the construction tree module, and match the state key value to be searched with the shared nibble(s) field (for Extension Node) or slot (for Branch Node) of the root node from the beginning, read the hash of the next layer of tree nodes from the matching position, and then, preferentially search for the hash in the memory of the construction tree module. If it has not been found, search in the memory of the state database module. If it still has not been found, it needs to be searched in the file of the state database module. Specifically, you can search in the SST in Level 0 first. If you still can't find it, continue to search in a certain SST in Level 1. Continue to search in a certain SST in the lower level until you find the next Internal Node pointed to by the hash value. After unlocking this Internal Node, continue to match the remaining fields of the state address key value to be read from the front to the back. If it matches, read the hash value from the matching point and jump to the next level tree node pointed to by the hash value. Repeat this process, unlock the Internal Node level by level and match the remaining fields of the state address key value to be read from the front to the back. The hash value at the matching point is used as the basis for the next search for the intermediate node or leaf node until the Leaf node is matched, so as to read the value of the state from the Leaf node. In this process, some of the kvs of the tree nodes at each level may not exist in the memory of the tree construction module, but in the state database module. Therefore, it may be necessary to read from the state database module. You can first search for this key in the MemTable or Immutable MemTable of the state database module. If it does not exist, you need to further search the SST files at all levels on the disk. For the Internal Node and Leaf node that exist on the disk, you need to load them into the memory and decode (deserialize) them to get the content. Specifically, for example, first search the sst files of level 0 one by one. If each SST of Level 0 is sorted, you can do a binary search in each SST based on the hash value of the intermediate node / leaf node. After finding it, load v into the memory and decode and match it.If the key is not found in the SST of Level 0, a binary search is performed in Level 1 according to the hash value of the intermediate node / leaf node. If the key is not found, a binary search is performed in Level 2. If the key is found, the v corresponding to the key is loaded into the memory and decoded. This process is repeated until the key is matched to the leaf node of the tree structure and the value is read from it. When searching in Level 1, Level 2, and lower levels, since the overall SST in these levels is sorted, a binary search can generally be performed in this level.

[0138] In this way, the tree node kv of the found leaf node can be loaded into the memory. For example, A11 loads the content prefix:2,Key-end:35b2e4,location:(2,750,210) in the leaf node value into the memory. The <2,750,210> is the position index of the value content in the file.

[0139] S930: Load the value of the data file location in the persistent storage medium pointed to by the location index into the memory and read the status data therein.

[0140] Still taking A11 as an example, the position index <2,750,210> points to the position in the data file in the persistent storage medium. In this way, the value of the position of the data file in the persistent storage medium pointed to by the position index <2,750,210> can be loaded into the memory and the status data therein can be read, that is, the data of file 2 in the data file starting from the starting position 750KB and the length of 210KB is loaded into the memory, thereby reading the status data therein, that is, "Zhang San_A=20".

[0141] Through the position index, you can directly locate the position in the data file, so that you can directly load the data file and read the status data from it.

[0142] In the case where the leaf node to be searched is present in the memory of the tree building module but the state data does not exist, it can be directly read from the corresponding data file according to the location of the leaf node in the memory. In this way, the kv of the leaf node is not stored in the persistent storage SST file, and the binary search process in the SST is required multiple times.

[0143] In the case where the leaf node to be searched does not exist in the memory of the tree building module, it is necessary to search and load the leaf node from the state database module. In fact, it is also necessary to read a certain amount of content from the disk into the memory before searching. In the implementation method corresponding to Table 1, the kv of the tree node is stored in the disk, and the value of the leaf node includes the state data, for example, H(11) in Table 1 corresponds to Zhang San_A=20 in the content, H(12) corresponds to Li Si_B=20 in the content, H(15) corresponds to storedData=s in the content, and H(16) corresponds to Wang Wu_A=20 in the content. Generally speaking, the space occupied by state data is relatively large, accounting for more than 90% of the total space of a kv. In the implementation method corresponding to Table 2, the index and data are separated. The index stores the location of the data file where the status data is located, but not the status data itself. For example, in Table 2, H(11) corresponds to location (2,750,210) in the content, H(12) corresponds to location (3,350,210) in the content, H(15) corresponds to location (5,760,140) in the content, and H(16) corresponds to location (5,170,210) in the content. The actual status is stored in the corresponding position in the data file, such as Fig.10 As shown in the data files and status data in . In the process of searching for leaf nodes, only index files can be searched. In this way, more index files can be loaded into the memory within the unit space for searching. The search efficiency of memory is significantly higher than that of disk. Generally, the content to be searched will be loaded into the memory before searching. The implementation method of Table 2 can find leaf nodes faster than the implementation method of Table 1 because more indexes can be loaded into the memory space of unit capacity. However, in the implementation method of Table 1, for example, after reading a batch of kv into the memory, if the leaf node is not found, another batch of kv needs to be re-read into the memory for searching again. Moreover, due to the state content that occupies a large space, the number of kv in the memory space of unit capacity is relatively small, so the number of times to read from the disk will generally be more frequent, and the reading speed of the disk is much slower than that of the memory. Therefore, the search efficiency of the implementation method of Table 1 is low.

[0144] The above combination Fig.10 Explained Fig. 9 The specific process of the method embodiment. Fig.10 The structure of the MPT tree is still two levels, and the structure of the intermediate nodes of each level of the tree includes two types of tree nodes: Extension Node and Branch Node. In a variation, a tree node such as Branch Node can be used as an intermediate node and a root node, for example Fig.11 As shown in . In this way, from the root node through the intermediate node to the leaf node, the sequential combination of the slots in the branch nodes passed through can constitute part of the content in the state key. The hash value of the next level node can be stored in the slot of the previous level node. Fig.11 In the example, a two-level tree structure can still be used. The leaf nodes in the upper tree can store account status (including the status of external accounts and contract accounts), and the leaf nodes in the lower tree can store the status of contracts. In the leaf nodes that store external account status, contract account status, or contract status, position indexes can be used. For example, the leaf nodes of the upper tree include account status in the following four rows: H(M1), H(M2), H(M3), and H(M4):

[0145]

[0146] Table 3. Tree node kv actually stored in the database

[0147] Through the position index, you can directly locate the position in the data file, so that you can directly load the data file and read the status data from it. Furthermore, the contract status included in the leaf nodes of the next level tree is as shown in the four rows of H(Z1), H(Z2), H(Z3), and H(Z4) in the table above.

[0148] Data files such as Fig.11 As shown in , the content in the data file is as follows:

[0149] M1: Nonce: n1, balance: 45.0 ETH

[0150] M2: Nonce: n2, balance: 1.00WEI

[0151] M3: Nonce: n3, balance: 1.1ETH

[0152] M4:Nonce:n1,balance:0.12ETH,codehash:c1,storage:s1

[0153] Z1: Zhang San_A=20

[0154] Z2: Li Si_B = 50

[0155] Z3: Wang Wu_A=35

[0156] Z4: storedData = s

[0157] Whether it is the previous level tree or the next level tree, the location in the data file can be quickly located through the location in the leaf node. In this way, the kv of the leaf node is not stored in the persistent storage SST file, and the binary search process in the SST is required multiple times according to the key (hash value of the content) of the leaf node. Fig.11 In the example, according to Fig. 9 The blockchain state data can be read in the same way as in the previous step. According to the position index, the position in the data file can be directly located. Therefore, the content at the corresponding position in the data file can be directly loaded into the memory. This also avoids the read amplification problem caused by reading a large number of files in multiple levels of SST and loading them into the memory. Fig.10 The storage method shown in Fig.11 Stored in the same way.

[0158] For example, external accounts and contract accounts use fixed-length strings as keys, and the key of the contract status can be a string of the same length. Specifically, for example, the key of an external account or contract account can be an account address, while the key of the contract status can be a string of the same length. For another example, the key of an external account or contract account can be a hash value of an account address according to a certain hash algorithm, such as the sha3 algorithm; and the key of the contract status can be sha3 (contract name + location of the state variable declaration). The above-mentioned external accounts, contract accounts, and contract status are strings of the same length and format, collectively referred to as state keys. For example, the length of the state key is 4 bytes, or 32 bits. In this way, assuming that the root node (B0), intermediate nodes (..., C5, ..., C13, ..., C5D1, ..., C5D9, ..., C5D14, ..., C13D2, ..., C13D10, ...) are each a hexadecimal tree, then on the one hand, the slot of each intermediate node (such as 0, 1, 2, ..., d, e, f in B0, a total of 16 slots) can be filled with the hash value of the next tree node, and on the other hand, the position of the slot of this root node / intermediate node can represent a 4-bit key. 4 bits constitute a half-byte, so a total of 8 layers of root nodes and intermediate nodes can be loaded with 32 bits, a total of 8 bytes, that is, all state keys. The 32 bits of the state key can be stored in one half-byte of the key from the intermediate node at the top level to the intermediate node at the bottom level to the leaf node. In this way, for the case where the intermediate nodes are fixedly expanded, the last 4 bits of the key can be stored in the leaf node. In the case where the intermediate nodes are fixedly expanded, Fig.11 In the example, there are 8 layers from C* to C13D*, all the way to the bottom layer M*. In another case, the prefix and / or infix and / or suffix compression method can also be adopted, so that one or some leaf nodes plus the intermediate nodes above can be less than 8 layers.

[0159] In addition, a tree structure can also be used to store account status and contract status, for example Fig.12 As shown in . Since the key of the contract status is a hash value, based on the discrete characteristics of the hash algorithm, the status key of different contract status may be the path from the root node to different leaf nodes in this layer of tree structure. In this way, the value of different states of the same contract may exist in different leaf nodes at the bottom of the tree structure, and will no longer be gathered under the contract account to which it belongs. In fact, with a one-layer tree structure, the account status and contract status are scattered in the leaf nodes at the bottom.

[0160] In addition, Fig.12 On the basis of Fig.13 In the case of infix compression, for example, the tree node key of M1 is 0x52d1479e (0x indicates that each bit is represented by a hexadecimal number), and the tree node key of Z1 is 0x52d9ab36, then M1 and Z1 have a common infix 2d. In addition to the 1 and 9 separated by the hexadecimal tree, the tree node C5 can also have a compressed infix 2d, such as Fig.13 In the case of suffix compression, for example, key-end is used to represent the key-end in the tree node M1, and key-end in the tree node Z1 is 36. By using prefix and / or infix compression, the number of tree layers can be reduced.

[0161] Regardless of whether it is compressed or not, for the state key being searched, you can start from the root node of the tree structure and match the path of the state key segment by segment until you find the leaf node of the tree structure corresponding to the state key. In one way, the root node and the intermediate node are stored in a key-value format, where the key is the hash value of the value, similar to the structure in Table 3. In this way, you can match the hash of the path of the state key in order starting from the previous tree node, and search for the next tree node based on the hash until you find the leaf node of the tree structure corresponding to the state key.

[0162] In addition, the key of the tree node kv actually stored in the database may not be the hash value of the value, but a partial value on the state key path, such as the value of a partial key from the root node to the current tree node (combined with Fig.12 , without compressed prefixes / infixes, as shown in Table 4), or the value of a partial key after the previous tree node (combined with Fig.13 , with compressed prefixes / infixes, as shown in Table 5).

[0163]

[0164]

[0165] Table 4. Tree node kv actually stored in the database

[0166] According to Table 4, combined Fig.12 For example, if the state key to be searched is 0x59...36 (leaf node Z1), starting from the root node of the tree structure, the tree node with key 0x5 in the database can be matched according to the first digit 5 ​​from the left of the state key (h(C5) in the value of the tree node with key 0x5 can be used to verify with the hash value on slot 5 in the root node. If they are the same, it means that the tree node 0x5 is the child node in slot 5 of the 0x tree node); further, the tree node with key 0x9 in the database can be matched according to the second digit 9 from the left of the state key (h(C5D9) in the value of the tree node with key 0x9 can be used to verify with the hash value on slot 9 in the parent node. If they are the same, it means that the tree node 0x9 is the child node of the 0x5 tree node); and so on. Finally, the last digit 6 of the state key can be matched to the tree node with key 0x6 in the database (h(Z1) in the value of the tree node with key 0x6 can be used to verify with the hash value on slot 6 in the parent node. If they are the same, it means that the tree node 0x6 is the child node of the parent node).

[0167]

[0168]

[0169] Table 5. Tree node kv actually stored in the database

[0170] According to Table 5, for example, if the state key to be searched is 0x52d9...36 (leaf node Z1), starting from the root node of the tree structure, the first bit 5 from the left of the state key can be matched in the database. If the same key is not matched, or the hash value in the value of the tree node that matches the key is inconsistent with h(C5) in the root node, then the state key is matched again in the database according to the first-second bits 52 from the left, and the cycle continues until a match is found. When the tree node with the key 0x52d is matched in the database according to the first-third bits 52d from the left of the state key, and the H(C5) in its value is the same as the hash value on slot 5 in the root node, it means that a match is found. Then, we can match the database according to the 1st to 4th bits 52d9 from the left of the state key. If the same key is not matched, or the hash value in the value of the tree node of the matched key is inconsistent with h(C5) in the root node, we can match the database according to the 1st to 2nd bits 52 from the left of the state key again. If a match is found, and H(C5D9) in the matched tree node value is the same as the hash value on slot 9 in the parent node, it means that the match is found. Furthermore, we match the database according to the 1st to 5th bits from the left of the state key, match the database according to the 1st to 6th bits from the left of the state key, and so on, until a leaf node is matched.

[0171] In another implementation, the complete state key or the corresponding complete state key can be stored in the leaf node. For example, the complete state key is stored in the leaf node. Fig.14 As shown in , the key-full field of the leaf node can store the complete key of the corresponding state. The key-value of the leaf node stored in the database is the complete key of the corresponding state of the leaf node. In this way, when searching for a state key, you can directly match the key of the leaf node in the database according to the key without matching the key of the intermediate node. Obviously, this search method is extremely efficient for query operations. In fact, the above Figure 10-13 In the example above, the complete state key can also be used as the key of the leaf node.

[0172] Above Figure 9-14 In the embodiment, the location index location is stored in the leaf node, and the position of the value of the leaf node in the data file can be quickly located through the location index after the leaf node is found. In this way, the content of the corresponding position in the located data file can be directly loaded into the memory, which also avoids the read amplification problem caused by reading a large number of files in multiple Levels of SST and loading them into the memory. This also avoids the kv of the leaf node being stored in the persistently stored SST file, and the process of performing binary searches in the SST multiple times according to the key (hash value of the value content) of the leaf node. Moreover, similarly, for the leaf node to be found that is not in the memory of the tree construction module, the above embodiment can only search the index file, so that more index files can be loaded into the memory within the unit space for search, and the leaf node can be found faster.

[0173] As mentioned above, the state root in the block header stores the hash value of the root of the underlying MPT tree to which it is anchored. This hash value is actually obtained by calculating the hash from the leaf node layer by layer, as described above. Generally, the storage module of the blockchain node will build the world state of the tree structure in memory through the tree construction module. Specifically, in addition to the above Figures 12-14 In addition to the leaf nodes, it also includes intermediate nodes and root nodes, as shown in the above Tables 2 to 5.

[0174] Moreover, the chain data structure of the blockchain is realized by anchoring the hash of the next block to the PrevHash of the previous block, such as Figure 3 , 4 As shown in , there are blocks identified by different block numbers on the blockchain. These block numbers are also called versions. Under different versions of blocks, the corresponding tree structure may change.

[0175] People in this field know that the essence of blockchain is a chain structure composed of a series of data blocks (blocks) linked by hashes. Each block has a world state corresponding to the block, but each block does not necessarily contain a complete world state, but in most cases contains a reference to the previous world state. The following explains this:

[0176] In the block structure of Ethereum, when the value in a key-value changes, the content of the leaf node of the corresponding MPT tree changes. Accordingly, the relevant hash values ​​stored in a series of intermediate nodes and the root node above the leaf node will also change. In the structure shown in Table 1, the changed leaf nodes, intermediate nodes, and root nodes will also be added to the end of Table 1 in an appended manner. At the same time, the values ​​of more state variables in a contract storage may not change.

[0177] like Fig.15 The Ethereum block structure diagram shown in Fig.15 and Figure 3 The difference is that the references of the root nodes and intermediate nodes of the two-level Merkle tree under the State Root in block N+1 to the state variables that have not changed in block N are added. For example, in the contract storage of a contract, the value of the same state variable is changed from 29 in block N to 45 in block N+1. Assuming that other state variables have not changed, in block N+1, except for the leaf node that stores the value changed to 45 and the intermediate nodes and intermediate nodes related to it, the other leaf nodes and intermediate nodes that have not changed in block N+1 directly reference the same leaf nodes and intermediate nodes in block N. Two adjacent blocks N and N+1 are shown here. It can be understood that for the block structure of multiple consecutive blocks on the blockchain, the leaf nodes and intermediate nodes in block N+1 that have not changed relative to the previous blocks all reference the same leaf nodes and intermediate nodes that were last updated to the value in block N+1. The same leaf node and intermediate node that were last updated to the value, for example, the same leaf node and related intermediate node that were updated to the value in block N-1, the same leaf node and related intermediate node that were updated to the value in block 5, and so on. The different blocks here can also be called different versions, and the block number can be the version number. The reference here is achieved by recording the hash value of the lower node in the upper node. The benefit of this is obvious, that is, the leaf nodes and intermediate nodes that have not changed will not be repeatedly stored in the underlying database.

[0178] Here, we still use the change of the state in the same contract storage in version N+1 relative to version N as a simple explanation. For example, if the content of a tree node changes, its hash will change accordingly. MPT uses the hash as the index in the database, which means that for each value, there is a certain record in the database. MPT associates parent and child nodes based on the node hash. Therefore, whenever the content of a node changes, ultimately for the parent node, only a hash index value changes. The content of the parent node also changes, creating a new higher-level parent node, and recursively passing this impact to the root node. Ultimately, a change creates a new path from the modified node to the root node, and the old node can still be accessed through the old path based on the old root node.

[0179] like Fig.15 As shown in the figure, when the content of a node in MPT changes from 29 in version N to 45 in version N+1, assuming that the other states in the contract state have not changed in the previous and next two versions, the changed 45 in the world state of version N+1 will create a new path to the root node (Storage root) of this contract. In addition, the other leaf nodes and intermediate nodes that have not changed reuse the leaf nodes and / or intermediate nodes in the previous version (i.e. the version with block number N) through hash pointers, thereby constructing a new MPT tree for the storage of the contract in version N+1, while still retaining its old path. Fig.18 In the same contract storage, the leaf nodes / intermediate nodes whose values ​​in version N+1 have not changed relative to version N include V, U, T, S, etc. Therefore, in the storage of the contract in version N+1, in addition to creating a new path from 45 to the Storage root, the Storage root of version N+1 is indexed to V, U, T, S in version N through hash, but does not include the leaf node with a value of 29 and related intermediate nodes. Assuming that the contract is deployed in version N, CodeHash is similar. The contract code corresponding to CodeHash exists in the state database (StateDB) corresponding to version N, and CodeHash of version N+1 also points to the contract code in version N.

[0180] In addition to contract storage, other account storage (including storage of external accounts and contract accounts) can also adopt similar methods, such as Fig.15 For the account storage that has not changed in version N, the State Root in the block header of N+1 points to the storage of external accounts or contract accounts such as L, M, N, P, Q, and R in version N. Only the leaf nodes and related intermediate nodes that have changed are stored in the account storage corresponding to version N+1, which will not be described one by one.

[0181] All states under the current block, in fact, many of them are likely to have been updated in a previous block and have not changed until the current block. This is similar to Ethereum, which is also called current state and is stored in currentDB. CurrentDB is used to store the latest "current state" data, such as the latest account status, contract code, and contract storage. Moreover, as the block grows, the data in currentDB can be continuously modified.

[0182] As mentioned above Figure 3 and above Fig.15 The content of the blockchain is very redundant. It is an efficient way to store data by referencing the same leaf node and intermediate node that were last updated to the same value in the previous block. Moreover, the state changed in each block is likely to be part of the global state. This is also similar to Ethereum, also known as history state, which is stored in historyDB. historyDB is used to store historical state data, thereby providing a complete historical view of the entire chain.

[0183] The present application provides a method for constructing a world state based on a tree structure, comprising:

[0184] S170: Receive status key-value.

[0185] S172: Generate leaf nodes of the tree structure corresponding to the state key-value according to the construction rules of the tree structure; calculate the hash value of the leaf node according to the content in the leaf node, and generate or update the upper tree node according to the hash value of the leaf node and the state key, until the root node of the tree.

[0186] The leaf node may be in the form of the aforementioned data file or the aforementioned KVDB. For the aforementioned data file, the leaf node may generate a hash value including the position index of the state value in the data file in the persistent storage medium and the state value content.

[0187] S174: Generate corresponding memory pages for the generated / updated tree nodes according to the divided logical page structure, and the memory pages are used to represent the contents of all corresponding tree nodes in the logical page.

[0188] In the present application, the entire Merkle tree structure can be divided into multiple LogicalPages (logical pages). According to the node association relationship of the tree structure, several upper and lower adjacent tree nodes can be aggregated into one LogicalPage, such as a 2-level 16-branch dictionary tree with a total of 256 child nodes aggregated into one LogicalPage. Among them, there are multiple layers of LogicalPages from the root node to the leaf node of the Merkle tree, and there are parent-child and brother relationships between LogicalPages, thus forming a complete Merkle tree. In this way, the tree structure can be divided into multiple logical pages, and a logical page contains at least one tree node. Different logical pages contain different tree nodes, and the logical page maintains the content of the corresponding tree node of the current latest version.

[0189] Inside each LogicalPage, a memory page (MemoryPage) can be maintained to represent the contents of all tree nodes corresponding to the logical page of the latest version of the logical page. Furthermore, the contents of the tree nodes maintained by the logical page can be generated into base pages and delta pages according to the generation / modification behavior based on the continuous version update. Specifically, BasePage (base page) and DeltaPage (delta page) are generated according to the generation / modification behavior. For example, for continuous state changes, such as the state variables a=5, b=8, c=3 included in version N, a=5, b=8, c=3 can be used as the contents of the base page BasePage. For a=6 in version N+1 and a=8 in version N+2, assuming that b and c have not changed, a=6 in version N+1 and a=8 in version N+2 can be used as DeltaPages based on this BasePage, such as DeltaPage1, and the case where b and c have not changed is not included in DeltaPage1. For a=11 in version N+3 and a=15 in version N+4, a=11 in version N+3 and a=15 in version N+4 can be used as DeltaPages based on this BasePage, such as DeltaPage2. Similarly, assuming that b and c have not changed, b and c are not included in DeltaPage2, and so on. Among them, BasePage and DeltaPage can be used as the smallest unit of the tree's memory structure. In addition, BasePage can be used as the smallest unit of the tree's disk persistence. On this basis, DeltaPage can also be used as the smallest unit of the tree's disk persistence.

[0190] It should be noted that the BasePage here is a checkpoint of the entire LogicalPage content, which can contain the full content of the LogicalPage at that time, such as all the tree node information contained. Checkpoint is generally a snapshot of a specific state saved in a computer system, so that it can be restored from this saved state and continue execution when the system fails or fails. In many cases, checkpoints are used for long-running computing tasks, safety-critical systems, distributed systems, and database systems.

[0191] For BasePage, a BasePage can be generated every predetermined number of state modifications (such as m times), such as Fig.16 As shown in the basepage2 on the right side of the lower right red box. As in the above example, in version N, the state variable a=5 can be used as the content of the base page BasePage1. For version N+1, a=6, and version N+2, a=8, a=6 in version N+1 and a=8 in version N+2 are used as DeltaPage1 based on this BasePage. For version N+3, a=11, and version N+4, a=15, a=11 in version N+3 and a=15 in version N+4 are used as DeltaPage2 based on this BasePage. For version N+5, a=20, assuming that the modification reaches the predetermined number of times 5, the value a=20 in version N+5, b=8, and c=3 can be used as the content of this BasePage2.

[0192] Similarly, DeltaPage describes several version modifications of LogicalPage, and aggregates multiple modifications into a set, such as generating a DeltaPage every M modifications, and creating a new DeltaPage to collect the subsequent version modification operations. As in the above example, for a=6 in version N+1 and a=8 in version N+2, a=6 in version N+1 and a=8 in version N+2 are used as DeltaPage1 based on this BasePage. For a=11 in version N+3 and a=15 in version N+4, a=11 in version N+3 and a=15 in version N+4 can be used as DeltaPage2 based on this BasePage. It can be seen that each DeltaPage is set to create an incremental page for 2 version modification operations.

[0193] For each modification of BasePage and DeltaPage, corresponding dirty data is generated in the memory. When the memory occupied by BasePage and DeltaPage reaches a predetermined amount, BasePage and DeltaPage can be batch-persisted to disk, thereby avoiding persisting all dirty data for each modification and continuously occupying memory and CPU resources. The memory generation and disk persistence of BasePage and DeltaPage can be performed in the background.

[0194] When the storage module persists the world state of the tree structure built in the memory through the structure of the logical page, it can cache the state data according to the predetermined strategy based on the hot and cold characteristics of the state data, such as the (Least Recently Used, LRU) strategy, with the logical page as the granularity. In this way, if the MemoryPage is eliminated, the corresponding BasePage and DeltaPage are read to replay the MemoryPage.

[0195] The above scheme provided by this application describes a scheme for dividing the world state of a tree structure according to Page segmentation, dividing the tree structure into LogicalPage, and managing MemoryPage / BasePage / DeltaPage internally in LogicalPage, optimizing each state tree modification into multiple version modifications at the granularity of LogicalPage, and reducing the overall write amplification. For the current version, it can be directly obtained through the memory page, and for the historical version, the BasePage and several associated DeltaPages can be read to replay. The entire MemoryPage is cached in memory without affecting the contract execution performance.

[0196] In the above scheme, a LogicalPage internally manages MemoryPage / BasePage / DeltaPage, and as mentioned above, several upper and lower adjacent tree nodes can be aggregated into a LogicalPage according to the node association relationship of the tree structure, such as aggregating 256 child nodes in a 2-level 16-branch dictionary tree into a LogicalPage. Among them, there are multiple layers of LogicalPages from the root node to the leaf node of the Merkle tree, and there are parent-child and brother relationships between LogicalPages, thus forming a complete Merkle tree. In this way, the tree structure can be divided into multiple logical pages, and a logical page contains at least one tree node. Different logical pages contain different tree nodes, and the logical page maintains the content of the corresponding tree node of the current latest version.

[0197] Then, when constructing a world state based on a tree structure, in addition to generating a leaf node of the tree according to the state key-value, changes will also occur from the generated leaf node to the intermediate node to the root node, such as the hash value described in S172 above. Moreover, the key of the state also needs to be designed, especially for the LogicalPage and the corresponding MemoryPage / BasePage / DeltaPage.

[0198] Therefore, this application also proposes the following method for constructing a world state based on a tree structure, including:

[0199] S180: Receive status key-value.

[0200] S182: Generate leaf nodes of the tree structure corresponding to the state key-value according to the construction rules of the tree structure; calculate the hash value of the leaf node according to the content in the leaf node, and generate or update the upper tree node according to the hash value of the leaf node and the state key, until the root node of the tree.

[0201] The leaf node may be in the form of the aforementioned data file or the aforementioned KVDB. For the aforementioned data file, the leaf node may generate a hash value including the position index of the state value in the data file in the persistent storage medium and the state value content.

[0202] The process of S182 can be specifically referred to in the above content, which is briefly described again as follows:

[0203] Like Ethereum, when the storage module receives a key-value pair (kv) request to write the state, the key (k) will first be hashed (such as Keccak-256). In addition, the state key in S182 here can also be the state key received in S180, that is, it can also be not hashed. Next, use this key to search in the MPT. This search process starts from the root node of the MPT and traverses and matches down along the path of the tree in the lexicographic order of the key. In this process, the key is generally compared bit by bit (or half a byte by half a byte, also known as a nibble, generally a hexadecimal number of one bit) with the path stored in the node. During the traversal and matching process, three types of nodes may be encountered: branch nodes (if the path forks at a certain point), extension nodes (if there is a shared prefix), and leaf nodes (storing the final value). Branch nodes are used to fork in the tree, extension nodes are used to optimize storage (through path compression), and leaf nodes store actual values. If a complete lexicographic path for a matching key is found to exist during the search process, a leaf node will be reached. At this point, the value (v) stored in this leaf node can be updated. If the search and matching process finds that only part of the path is matched, it may be necessary to create a new node (which may be a branch node, an extension node, or a leaf node) to complete the remaining path and eventually reach the leaf node, and store or update the value in the leaf node.

[0204] Once a leaf node is updated or created, a backtracking process begins. This process starts with the modified leaf node and recalculates the hash value of the node. Then, it moves up to the parent node, updates the relevant reference in the parent node (this reference may be the direct content of the child node, or the hash value of the child node content, depending on the size of the child node), and recalculates the hash value of the parent node. This process continues until the root node of the MPT is reached.

[0205] Eventually, when the backtracking process reaches the root node, the root hash value of the entire MPT will be updated. This new root hash value represents the new state of the entire state tree, which will be included in the block header of the next block as the new state root. In this way, each block not only contains transaction information, but also saves the hash value of the global state after executing these transactions through the state root.

[0206] S184: Update the version of the generated / updated tree node in the corresponding memory page according to the divided logical page structure, the memory page is used to represent the content of all corresponding tree nodes in the logical page, the logical page contains a page identifier, and the page identifier includes the lexicographic order content and version of the upper tree node in the logical page.

[0207] It should be noted that the process of S184 can occur after S182 or during S182.

[0208] Fig.18 It is a complete diagram of another embodiment of the present application. Fig.18 and Fig.13 The overall structure is similar, with the main difference being that, for example, the intermediate node labeled C5 and the intermediate nodes labeled C5D1, C5D9, and C5D14 below it together form a LogicalPage, and in the leaf nodes M1, M2, M3, and M4, key-end is stored instead of key-full, and in addition to the position index contained in the leaf node, it also contains the hash value of the corresponding state value content. Of course, in this embodiment, as mentioned above, the leaf nodes M1, M2, M3, and M4 can also be key-full.

[0209] The leaf node may also include a hash value (valuehash) of the state value content and a key-end field, wherein the hash value of the leaf node is calculated based on the content in the leaf node, including the calculation based on the hash value of the state value content in the leaf node and the key-end field, but does not include the location index of the state value in the data file in the persistent storage medium.

[0210] The leaf node may also include a hash value and a key-full field of the state value content, wherein the hash value of the leaf node is calculated based on the content in the leaf node, including the calculation based on the hash value of the state value content in the leaf node and the key-full field, but does not include the position index of the state value in the data file in the persistent storage medium. Fig.18 The valuehash in the leaf node is omitted. And, Fig.18 The data structure of the tree node is used instead of Fig.14 A graphical description of .

[0211] In the aforementioned S182, a tree node is generated / updated. The updated tree node is integrated into the corresponding memory page according to the aforementioned logical page structure, indicating the world state tree structure under the current block number. In this application, a version number can be set for the logical page. The version number can typically be a block number. In addition to the block number, the version number can also be data such as a block hash, timestamp, state root, transaction root, receipt root, consensus round number, etc. that can uniquely identify a block. For simplicity, the following is explained in terms of block number.

[0212] The version number of the logical page can be used as the content in the page identification of the logical page. In addition to the version number, the page identification of the logical page can also include the lexicographical content of the tree nodes included. Specifically, the page identification can include the lexicographical content of the upper tree nodes in the logical page. Fig.18 As shown in , in a LogicalPage composed of the intermediate node labeled C5 and the intermediate nodes labeled C5D1, C5D9, and C5D14 below it, the upper tree node has a node ID (NodeID), for example, 0xd2345 (0x represents hexadecimal), and the intermediate nodes C5D1, C5D9, and C5D14 below it also have a NodeID. The PageID of the LogicalPage can be set to the NodeID of the upper tree node. These page IDs are the lexicographic content from the root node to the upper tree node in the LogicalPage (that is, the memory page), that is, the key of the lexicographic content distribution from the root node through the intermediate node to the upper tree node in the LogicalPage. Similarly, NodeID is the lexicographic content from the root node to the tree node, that is, the key of the lexicographic content distribution from the root node through the intermediate node to the tree node.

[0213] As described above, in the process of generating / updating the current version of the tree nodes, the changed tree nodes constitute dirty data. And as mentioned above, the base page and incremental page can be generated based on the changes in the logical pages of the continuous version. On this basis, the base page can be used as the minimum unit of disk persistence of the tree, and the incremental page can also be used as the minimum unit of disk persistence of the tree. A base page can be generated once for every predetermined number of state modifications, and an incremental page can be generated once for every predetermined number of state modifications. When the base page and incremental page occupy a certain amount of memory, the base page and incremental page are batched for disk persistence.

[0214] The base page and incremental page are persistent objects of the memory page corresponding to the logical page, and the memory page is the memorized object of the logical page. The base page generally corresponds to the memory page, contains global state variables, and can have the same page ID as the corresponding memory page, and the memory page has the same page ID as the corresponding logical page. The incremental page can also correspond to the memory page, containing tree nodes in one or more consecutive versions of the memory page that have changed relative to the previous adjacent base page / incremental page / version, which also contains the changed state variables, but generally not the global state variables, and the structure of the tree node in the incremental page itself generally does not need to change, so the page ID of the corresponding memory page can be the same as the NodeID of the upper tree node.

[0215] In addition, the base page and the incremental page may have versions. After the above-mentioned S184 operation, the base page and the incremental page are generated based on the changes of the continuous versions of the memory page, and the corresponding versions of the base page and / or the incremental page may be generated based on the changes of the continuous versions of the memory page. Specifically, the base page generally corresponds to a memory page, including all the tree nodes in the corresponding memory page, and also including the global state variables, and therefore may have the same version as the corresponding memory page. The incremental page may correspond to one or more memory pages, including tree nodes in one or more continuous versions of memory pages that have changed relative to the previous adjacent base page / incremental page, and also including the changed state variables, which are generally not global state variables, so the version of an incremental page may be the lowest or highest version in the one or more memory pages corresponding thereto.

[0216] For example, a base page with version 4 contains state variables a=1, b=2, c=3 and corresponding intermediate nodes and root nodes, and the subsequent incremental page may include a=2 with version 5 and corresponding intermediate nodes and root nodes, a=1 with version 6 and corresponding intermediate nodes and root nodes, and b=4 with version 7 and corresponding intermediate nodes and root nodes. Among them, although a=1 with version 6 is the same as the state variable a=1 contained in the base page, it is different from the previous adjacent a=2 with version 5, so a=1 with version 6 and corresponding intermediate nodes and root nodes are also included in the incremental page. In addition, for the subsequent incremental page, such as the incremental page containing tree nodes with versions 7, 8 and 9, it is actually in order relative to the previous adjacent incremental page and version of the tree node. The tree nodes with versions 5, 6, and 7 correspond to the memory pages with versions 5, 6, and 7 respectively. The incremental page containing tree nodes with versions 5, 6, and 7 can have a version of 5 or 7, that is, it can be the lowest or highest version among the corresponding multiple memory pages.

[0217] Furthermore, a page type may be set in the base page and the incremental page to distinguish between the base page and the incremental page.

[0218] The above scheme is a multi-version Page segmentation scheme for the Merkle state data dictionary tree. The tree is switched to LogicalPage. Both the tree and LogicalPage maintain the version (such as the block number). MemoryPage / BasePage / DeltaPage are managed internally. BasePage / DeltaPage maintains the memory structure and file persistence by version. By reducing the overall write amplification through page aggregation, the version ordering feature is utilized to avoid compaction after the page is dropped to the disk, which can reduce the write amplification of the page being dropped to the disk. At the same time, through version-based replay, it supports the reading of state data of any historical version and SPV functions. In addition, combined with the aforementioned Figures 9 to 14 The state value can be stored in a data file of a persistent storage medium, and the leaf node includes a position index of the state value in the data file of the persistent storage medium, thereby further reducing the problem of write amplification.

[0219] Corresponding to the world state constructed in the above manner, the present application also provides a reading method, including:

[0220] S190: Receive the read status key and version.

[0221] S192: According to the state key and version, match the version and page ID in the page identifier of the memory page in lexicographic order. After matching, if the memory page includes at least two layers of tree nodes, continue to match the version and page ID of the tree nodes below the top tree node in lexicographic order until a leaf node is found.

[0222] For the received state key and version, the version and page ID in the page identifier of the memory page can be matched in lexicographic order. For example, for Fig.18 The intermediate node labeled C5 and the intermediate nodes labeled C5D1, C5D9, and C5D14 below it together constitute a LogicalPage. The corresponding memory page can be found first according to the version and page ID, and then the received state key can be used to further match the lower tree nodes in the memory page.

[0223] For example, for the received status key 0xd2345167890ab and version 20, the version + page ID of a memory page is 20 + 0xd2345, then Fig.18 The middle node labeled C5 and the middle nodes labeled C5D1, C5D9, and C5D14 below it together form a memory page composed of a LogicalPage, which can be matched to (the first 5 bits of 0xd2345167890ab, i.e., 0xd2345). Furthermore, in the lower tree nodes in the memory page, assuming that the tree node C5D1 is matched, that is, the NodeID in the tree node C5D1 is matched to 0xd23451; further, the memory page below the memory page can be matched. Fig.19 As shown, the lower memory page is matched. According to 0xd2345167890ab, the memory page with PageID 0xd234516 and the same version is matched. The memory page includes upper and lower tree nodes E5 and E5F2, E5F7, and E5F8. Furthermore, in the lower tree node of the memory page, it is assumed that the tree node E5F7 is matched, that is, the NodeID in the tree node E5F7 is matched to 0xd2345167. This goes back and forth, matching the version and page ID in the page identifier of the memory page in lexicographic order until the last memory page and the leaf node therein are found, and the search is completed.

[0224] As mentioned above, a memory page is a memorized object of a logical page. However, based on the cache elimination strategy in the memory, there may be no currently matching memory page. At the same time, as mentioned above, base pages and incremental pages can be generated based on the changes in the continuous versions of the logical page, and the base pages and incremental pages are persistent objects of the memory pages corresponding to the logical pages. Therefore, in the case where the memory page does not exist in the memory, the corresponding base page and incremental page can also be loaded from the persistent storage medium to the memory according to the page identifier, and the loaded base page and incremental page can be played back to obtain the memory page.

[0225] Specifically, for example, if no memory page with the same version and matching the remaining lexicographic content is found in the memory, the corresponding base page and incremental page matching the remaining lexicographic content can be found in the persistent storage medium according to the page identifier. As mentioned above, since base pages and incremental pages also have versions, and base pages generally have the same version as the corresponding memory pages, and the version of an incremental page can be the lowest version among one or more corresponding memory pages, there are the following two situations:

[0226] 1. In the case where a base page with the same version is found in the persistent storage medium, the corresponding base page can be loaded from the persistent storage medium to the memory according to the page identifier, and then the loaded base page can be restored to the memory page.

[0227] 2. In the case where the base page with the same version is not found in the persistent storage medium, the corresponding base page and incremental page are loaded from the persistent storage medium to the memory according to the page identifier; the loaded base page and incremental page are replayed to obtain the memory page.

[0228] The page identifiers of the base page and the incremental page may also include page types.

[0229] In this way, the loading of the corresponding base page from the persistent storage medium to the memory according to the page identifier can be loading a base page of the maximum version not greater than the read version, with the same page ID and the page type of the base page from the persistent storage medium to the memory. For the above situation 1, that is, the situation where the base page with the same version is found in the persistent storage medium, the base page with the same version, the same page ID and the page type of the base page can be loaded into the memory. For example, the version to be searched is 20, and there is no memory page with version 20 in the memory, and the base page with the same version 20 is found in the persistent storage medium, and the base page with the same version 20, the same page ID and the page type of the base page can be loaded into the memory. For the above situation 2, that is, the situation where the base page with the same version is not found in the persistent storage medium, the base page of the maximum version not greater than the read version, the same page ID and the page type of the base page can be loaded from the persistent storage medium to the memory. For example, the version to be found is 20, but there is no memory page with version 20 in the memory. Of course, the base page with version 20 is not found in the persistent storage medium. In fact, there are BasePage1 with version 17, DeltaPage1 with version 18, DeltaPage2 with version 20, and BasePage2 with version 22 in the persistent storage medium. DeltaPage1 with version 18 can include tree nodes with version 18 and tree nodes with version 19, and DeltaPage2 with version 20 can include tree nodes with version 20 and tree nodes with version 21. Then, BasePage1 with a version not greater than 20, a consistent page ID, and a page type of the base page can be loaded into the memory. That is, the version of the base page that meets the conditions here is 17.

[0230] For the above situation 1, that is, the situation where the base page with the same version is found in the persistent storage medium, the base page with the same version, the same page ID and the page type of the base page can be loaded into the memory, and then the memory page with the same version can be restored. For example, if the version to be found is 20, and there is no memory page with version 20 in the memory, and the base page with the same version 20 is found in the persistent storage medium, the base page with the same version 20, the same page ID and the page type of the base page can be loaded into the memory, and then the loaded base page with version 20 can be restored to the memory page with version 20.

[0231] For the above situation 2, that is, the situation where the base page with the same version as the version is not found in the persistent storage medium, a base page with a maximum version not greater than the read version, a consistent page ID and a page type of the base page can be loaded from the persistent storage medium to the memory, and then the loaded base page and incremental page can be replayed to obtain the memory page. The loaded base page and incremental page can be replayed to obtain the memory page, specifically, a base page with a maximum version not greater than the read version, a consistent page ID and a page type of the base page can be loaded from the persistent storage medium to the memory, and then an incremental page with a minimum version not less than the read version, a consistent page ID and a page type of the incremental page can be loaded from the persistent storage medium to the memory. For example, if the version to be searched is 20, but there is no memory page with version 20 in the memory, then of course there is no base page with version 20 in the persistent storage medium. In fact, there are BasePage1 with version 17, DeltaPage1 with version 18, DeltaPage2 with version 20, and BasePage2 with version 22 in the persistent storage medium. DeltaPage1 with version 18 can include tree nodes with version 18 and tree nodes with version 19, and DeltaPage2 with version 20 can include For tree nodes with version 20 and tree nodes with version 21, BasePage1 with a version not greater than 20, consistent page ID and page type of the base page with a maximum version of 17 can be loaded into memory, that is, the version of the eligible base page here is 17, and then DeltaPage1 with version 17 and later and a minimum version 20 not less than the read version 20, consistent page ID and page type of the incremental page with version 18 and DeltaPage2 with version 20 are loaded into memory, and then the loaded base page and incremental page are played back to obtain the memory page.

[0232] For the above situation 2, specifically, the loaded base page and incremental page can be replayed in the order of page type and version to obtain the memory page. Still according to the above example, the loaded base page is BasePage1 with version 17, and the loaded incremental pages include DeltaPage1 with version 18 and DeltaPage2 with version 20. Then, the incremental pages can be replayed in the order of versions, that is, based on the BasePage1 with version 17, in the order of DeltaPage1 with version 18 → DeltaPage2 with version 20 to obtain the memory page. Since DeltaPage2 with version 20 includes tree nodes with version 20 and tree nodes with version 21, after playing back the page in sequence to the incremental page with the maximum version not greater than the read version, that is, playing back the loaded DeltaPage1 with version 18, when playing back in sequence to DeltaPage2 with the maximum version 20 loaded, not all tree nodes in DeltaPage2 are played back, but the tree nodes in the incremental page with the maximum version loaded are continued to be played back in sequence until they are the same as the read version, that is, the tree node with version 20 in DeltaPage2 is played back, thereby obtaining the memory page.

[0233] In another example, assuming that an incremental page contains a version of a tree node, that is, the version of the incremental page is the same as the version of the tree node contained therein, then based on the base page, the incremental page can be replayed in version order to an incremental page equal to the read version to obtain a memory page.

[0234] The reason why it is necessary to replay from the base page to the incremental page to obtain the memory page is due to the trustless characteristics of the blockchain, which requires not only providing data but also verifiable data. Specifically, based on the leaf nodes and all related intermediate nodes of the global state in the base page, the incremental page is replayed in sequence, that is, the leaf nodes, intermediate nodes and root nodes in the base page are updated according to the leaf nodes, intermediate nodes and root nodes contained in the incremental page in version order. Only then can all tree nodes in the restored memory page maintain the hash anchor relationship from the leaf node to the intermediate node to the root node, such as the hash value of the correct lower-level tree node stored in the branch node slot, so that data can be verified.

[0235] As mentioned above, the version of a delta page can be the lowest or highest version among the one or more memory pages corresponding to it. In the above example, the case where the version of a delta page is the lowest version among the multiple memory pages corresponding to it is given. The following describes the case where the version of a delta page is the highest version among the multiple memory pages corresponding to it.

[0236] For the aforementioned situation 2, i.e., the situation where the base page with the same version as the version is not found in the persistent storage medium, a base page with a maximum version not greater than the version read, a consistent page ID and a page type of a base page can be loaded from the persistent storage medium to the memory. On this basis, an incremental page with a version after the base page and a minimum version not less than the version read, a consistent page ID and a page type of an incremental page can also be loaded from the persistent storage medium to the memory and loaded. For example, the version to be searched is 20, but there is no memory page with version 20 in the memory. Of course, the base page with the same version as 20 is not found in the persistent storage medium. In fact, there are, for example, BasePage1 with version 17, DeltaPage1 with version 19, DeltaPage2 with version 21 and BasePage2 with version 22 in the persistent storage medium. DeltaPage1 with version 19 can include tree nodes with version 18 and tree nodes with version 19, and DeltaPage2 with version 21 can include For tree nodes with version 20 and tree nodes with version 21, BasePage1 with a version not greater than 20, consistent page ID and page type of the base page with a maximum version of 17 can be loaded into memory, that is, the version of the eligible base page here is 17, and then version 17 of BasePage1 is loaded to the minimum version 21 not less than the read version 20, the page ID is consistent and the page type is the incremental page with version 19 and DeltaPage2 with version 21 into memory, and then the loaded base page and incremental page are played back to obtain the memory page.

[0237] For the above situation 2, specifically, the loaded base page and incremental page can be replayed in the order of page type and version to obtain the memory page. Still according to the above example, the loaded base page is BasePage1 with version 17, and the loaded incremental pages include DeltaPage1 with version 19 and DeltaPage2 with version 21. Then, the incremental pages can be replayed in the order of versions, that is, based on the BasePage1 with version 17, in the order of DeltaPage1 with version 19 → DeltaPage2 with version 21 to obtain the memory page. Since DeltaPage2 with version 21 includes tree nodes with version 20 and tree nodes with version 21, after playing back the page in sequence to the incremental page with the maximum version smaller than the read version, that is, playing back DeltaPage1 with the loaded version 19, when playing back in sequence to DeltaPage2 with the loaded maximum version 21, not all tree nodes in DeltaPage2 are played back, but the tree nodes in the incremental page with the loaded maximum version are continued to be played back in sequence until they are the same as the read version, that is, the tree node with version 20 in DeltaPage2 is played back, thereby obtaining the memory page.

[0238] S194: Read the status in the leaf node.

[0239] As mentioned above, the leaf node may be in the form of the aforementioned data file or the aforementioned KVDB. In the form of the aforementioned data file, the leaf node may include the location index of the state value in the data file in the persistent storage medium and the hash value of the state value content. Thus, the location index contained in the leaf node can be obtained, and the location index points to the location in the data file in the persistent storage medium. Then, the value of the location of the data file in the persistent storage medium pointed to by the location index can be loaded into the memory and the state data therein can be read. In the form of KVDB, the cache can be searched according to the hash value of the leaf node. If the cache is not found, the aforementioned Figure 7 The search in the state database module shown may involve a binary search in the sst of each Level, as described above and will not be repeated here.

[0240] As mentioned above, the essence of blockchain is a chain structure composed of a series of data blocks (blocks) linked by hashes. A memory page can include all states in the current version. In fact, many of these states are likely to have been updated in a previous block and have not changed until the current block. This is similar to Ethereum, which is also called current state. CurrentDB is used to store the latest "current state" data, such as the latest account status, contract code, and contract storage. Moreover, as the block grows, the data in currentDB can be continuously modified. There is a large redundancy in the chain structure of multiple blocks in the blockchain. It is an efficient way to store data by referencing the same leaf node and intermediate node that was last updated to the value in the previous block. Moreover, the changed state in each block is likely to be part of the global state. This is also similar to Ethereum, also called history state.

[0241] In some current implementations of blockchain technology, there is a practice of storing the current state and history state in currentDB and historyDB respectively. These two databases are usually independent entities. In some implementations, both databases are persisted independently. However, this dual database architecture has obvious defects: First, it leads to redundant storage of state data, which increases storage overhead. For example, if currentDB occupies 10GB of disk space and historyDB occupies 100GB, the total storage requirement reaches 110GB. Second, this architecture increases the complexity of the system. The usual operation process is to update currentDB first, then generate a binary log (Binlog), then synchronize Binlog to historyDB, and finally execute Binlog by historyDB to generate the corresponding version of the state record. This mechanism of keeping the dual databases synchronized not only increases the complexity of the system, but also may introduce potential risks of data consistency.

[0242] On the other hand, some blockchain implementations adopt a solution that only retains currentDB without maintaining historyDB. Although this method greatly reduces storage requirements (for example, it only takes up 10GB of disk space), it also brings new problems. When it is necessary to restore to the state of a historical version, if the read-write set is saved, the system needs to start from the genesis block and replay the read-write set in version order until the required historical version state is reached; if the read-write set is not saved, it is necessary to replay all transactions starting from the genesis block until the target state is reached. This method is extremely inefficient, time-consuming and computationally intensive when performing historical state recovery, especially when dealing with long-running blockchain systems, this problem becomes more prominent.

[0243] This application is in the above Figures 9 to 19 Based on the scheme, a method for storing the world state in a tree structure is provided, including:

[0244] S200: Generate corresponding memory pages for the generated / updated tree nodes according to the divided logical page structure, and the memory pages are used to represent the contents of all the corresponding tree nodes in the logical page; the logical page is the result of aggregating several upper and lower adjacent tree nodes according to the node association relationship of the tree structure, and a logical page contains at least one tree node, and different logical pages contain different tree nodes.

[0245] S202: Generate a base page and an incremental page based on changes in the contents of all tree nodes in the logical pages of the successive versions;

[0246] S200 and S202 are similar to the above and will not be described in detail.

[0247] S204: persisting the basic page and the incremental page in one database in a manner that does not distinguish between currentDB and historyDB.

[0248] In this application, a database is set. When data is written to disk, historyDB and currentDB are no longer distinguished, but the base page and incremental page are written into this database. In this way, unlike the prior art, two databases, currentDB and historyDB, are used, so data redundancy and storage overhead can be reduced. Moreover, unlike the prior art, only currentDB is retained without maintaining historyDB, so restoring any version status becomes faster and more efficient.

[0249] Furthermore, as described above, a base page may be generated once for each predetermined number of state modifications, and a delta page may be generated once for each predetermined number of state modifications.

[0250] The base page and the incremental page may contain a page identifier, which includes a version, a page ID, and a page type; the logical page and the memory page contain a page identifier, which includes a version and a page ID.

[0251] Moreover, the state value may be stored in a data file of a persistent storage medium, and the leaf node includes a position index of the state value in the data file of the persistent storage medium.

[0252] Based on the solutions in Figures S200 to S202 above, this application also provides a method for restoring the state of any version of the world, including:

[0253] S210: Read the version to be restored;

[0254] S212: Reading a basic page of the same version as the restored version from the database;

[0255] S214: If a base page of the same version as the restored version is read from the database, the base page of the restored version is loaded from the database into the memory to obtain a memory page;

[0256] S216: If the base page of the same version as the restored version is not read in the database, a base page of a maximum version not greater than the restored version and a page type of a base page is loaded from the database to the memory, and an incremental page of a version after the base page to a minimum version not less than the restored version and a page type of an incremental page is loaded from the database to the memory; the loaded base page and incremental page are replayed in the order of page type and version to obtain the memory page of the restored version;

[0257] S218: For a memory page including a leaf node, read the status of the leaf node into the memory.

[0258] For restoring the world state of any version, such as restoring the current version or any historical version, the above recovery process can be performed according to the version that needs to be restored. Of course, for the memory pages existing in the memory, as mentioned above, based on the elimination mechanism in the cache, it is also possible that some MemoryPages are eliminated. If a MemoryPage that needs to be restored is eliminated, the corresponding BasePage and DeltaPage can be read to replay the MemoryPage. As mentioned above, the base page and incremental page can be generated based on the changes in the logical page of the continuous version, and the base page and incremental page are persistent objects of the memory page corresponding to the logical page. Therefore, in the case where the memory page does not exist in the memory, the corresponding base page and incremental page can also be loaded from the persistent storage medium to the memory, and the loaded base page and incremental page can be replayed to obtain the memory page.

[0259] In addition, when the blockchain node is cold started, there are generally no memory pages in the memory. At this time, the solution of the present application can be used to load the base page and incremental page corresponding to the current latest version of the block from the persistent storage medium to the memory, and the loaded base page and incremental page can be replayed to obtain the memory page, thereby obtaining the current state.

[0260] In the prior art, for restoring any historical version, in addition to the above-mentioned solution of replaying all transactions from the genesis block until the target state is reached, it is also possible to start from the root node of the MPT tree locked by the state root of the specified historical version and search for tree nodes in the database. This process involves decoding and then continuing to search for the next level of tree nodes according to the slot relationship. It also involves searching layer by layer in LevelDB, including traversal and a large number of binary search operations. As mentioned above, it is also computationally intensive and has the problem of read amplification. In fact, the same problem exists for restoring the current version.

[0261] The reading solution provided in the present application adopts the method of marking the base page and incremental page of the version. When searching for the base page and incremental page in the database, the base page and incremental page are generally distributed in the order of the version, so the search can be done quickly, thereby improving the recovery efficiency.

[0262] The method may also include: verifying the hash locking relationship between the upper and lower adjacent tree nodes from the leaf node to the root node, and verifying the consistency of the root node hash value with the state root in the latest version of the block. This is because, in the world state of the tree structure in the blockchain, the upper layer of tree nodes anchors the hash value of the lower layer of tree nodes, thereby achieving verifiable storage as a whole. Therefore, after restoring a version of the world state in the above manner, further verification of the anchored hash can ensure the correctness of the world state.

[0263] As mentioned above, the state value can be stored in a data file of a persistent storage medium, and the leaf node includes the position index of the state value in the data file of the persistent storage medium. Correspondingly, reading the state in the leaf node here can be to obtain the position index contained in the leaf node, and the position index points to the position in the data file of the persistent storage medium, and then load the value of the data file position in the persistent storage medium pointed to by the position index into the memory and read the state data therein.

[0264] As in the example mentioned above, the leaf node may include a hash value (valuehash) of the state value content, wherein the hash value of the leaf node is calculated based on the content in the leaf node, including the calculation based on the hash value of the state value content in the leaf node, but excluding the position index of the state value in the data file in the persistent storage medium. Here, based on the valuehash field included in the leaf node, this solution can also verify that the hash value of the data file in the persistent storage medium pointed to by the position index is consistent with the valuehash in the leaf node.

[0265] The loaded base page and incremental page are replayed in the order of page type and version to obtain the memory page of the restored version. Similar to the above, it can be based on the base page, and the incremental page is replayed in the order of version to the incremental page of the maximum version smaller than the restored version, and the tree nodes in the loaded incremental page of the maximum version are continued to be replayed in sequence until they are the same as the restored version to obtain the memory page.

[0266] For example, if the version to be restored is 20, and the base page with the same version as 20 is not found in the database, as in the above example, BasePage1 with a version not greater than 20 and a page type of the base page with a maximum version of 17 can be loaded into the memory, that is, the version of the eligible base page here is 17, and then the version 17 of the BasePage1 to the minimum version 21 not less than the read version 20, the page ID is consistent and the page type is the incremental page with a version of 19 DeltaPage1 and a version of 21 DeltaPage2 are loaded into the memory, and then the loaded base page and incremental page are replayed to obtain the memory page. Specifically, the loaded base page and incremental page can be replayed in the order of page type and version to obtain the memory page. Still according to the above example, the loaded base page is BasePage1 with version 17, and the loaded incremental pages include DeltaPage1 with version 19 and DeltaPage2 with version 21. Then, the incremental pages can be replayed in the order of versions, that is, based on the BasePage1 with version 17, in the order of DeltaPage1 with version 19 → DeltaPage2 with version 21, to obtain the memory page. Since DeltaPage2 with version 21 includes tree nodes with version 20 and tree nodes with version 21, after replaying the page in sequence to the incremental page with the maximum version smaller than the read version, that is, replaying the loaded DeltaPage1 with version 19, when replaying in sequence to DeltaPage2 with the maximum version 21 loaded, all tree nodes in DeltaPage2 are not replayed, but the tree nodes in the incremental page with the maximum version loaded are continuously replayed in sequence until the version is the same as the read version, that is, the tree nodes with version 20 in DeltaPage2 are replayed, thereby obtaining the memory page.

[0267] It should be noted that for multiple logical pages, although a base page is generated for each predetermined number of state modifications, and an incremental page is generated for each predetermined number of state modifications, and this predetermined number is generally set to be the same, different logical pages contain different tree nodes. Therefore, for a tree structure containing multiple logical pages, the number of tree nodes in different logical pages may not be synchronized. In this way, there may be Fig. 20 In the case of , the tree node containing version 17 is located in the DeltaPage with version number 17 in the left LogicalPage, and in the BasePage with version number 17 in the right LogicalPage, that is, the version numbers of the BasePage and DeltaPage in different logical pages are not synchronized. When restoring the world state of a specified version, the aforementioned S214 and S216 may be parallel operations, and there is no order of precedence.

[0268] In addition, in the embodiment provided by the present application, the specific design of LogicalPage can be adopted as follows: LogicalPage is divided into an intermediate page (InternalPage) and a leaf page (LeafPage). Among them, one intermediate page contains one intermediate node as the root of this logical page and multiple intermediate nodes as child nodes of this root. For example, one InternalPage contains one InternalNode as the root of this LogicalPage and at most 16 InternalNodes as child nodes of this root. One leaf page contains one leaf node, or contains at least two leaf nodes and at least one intermediate node to form a subtree. On the premise of forming a complete tree structure, at least one slot of any of the above InternalNodes is not empty. A typical example is shown in the attached Fig.18 In the example, the InternalNode C5 and the three InternalNodes C5D1, C5D9 and C5D14 in the lower layer form an InternalPage, and C5, C5D1, C5D9 and C5D14 form a subtree. At this time, the three slots of the InternalNode C5 are not empty, that is, slot1 stores the hash value of the InternalNode C5D1, slot9 stores the hash value of the InternalNode C5D9, and slote stores the hash value of the InternalNode C5D14. Each of the InternalNodes C5D1, C5D9 and C5D14 has at least one slot that is not empty, because as an InternalNode, it must be connected to the InternalNode in the lower layer, and the specific number of slots that are not empty and which slot is not empty depends on the number and position of the connected lower layer InternalNodes.

[0269] The aforementioned application also proposes a method for constructing a world state based on a tree structure, such as S180 to S184 and S200 to S204. In S184, the generated / updated tree node is updated according to the divided logical page structure, and the memory page is used to represent the content of all corresponding tree nodes in the logical page. The logical page contains a page identifier, and the page identifier includes the lexicographic order content and version of the upper tree node in the logical page.

[0270] Based on this, the aforementioned present application also provides a reading method. The reading method involves searching for corresponding base pages and incremental pages from a disk. The base page and incremental page may include a page identifier, and the page identifier includes a version, a page type, and a page ID. In the database, the scheme can match the base page and incremental page according to the page identifier on the disk according to the state key and version to be found, and after matching, the base page and incremental page are loaded into the memory and played back to obtain a MemoryPage.

[0271] First, if Fig.21 As shown, a State Trie locked to the State Root in the block header of block number N is shown. This State Trie can be the aforementioned Figures 11 to 14 Any of the above. The State Trie tree structure adopts the aforementioned LogicalPage structure. Among them, MemoryPage is used to represent the contents of all tree nodes corresponding to the logical page of the current latest version, and is a memorized object of the logical page. The contents of the tree nodes maintained by the logical page are updated according to the contents of the tree nodes maintained by the logical page in the continuous version, and BasePage and DeltaPage can be generated. If MemoryPage is eliminated, the corresponding BasePage and DeltaPage can be read to replay the MemoryPage. To read the corresponding BasePage and DeltaPage, the corresponding BasePage and DeltaPage need to be found on the disk. BasePage and DeltaPage can be used as the minimum unit of disk persistence of the tree. Each BasePage and DeltaPage on the disk has a specific position on the disk, such as the disk block number, the starting position and the continuous length, so that the BasePage / DeltaPage can be located on the disk. As mentioned above, the page identifier can typically include the version, page type and page ID. Then, an index file can be set to store the corresponding relationship between the page identifier and the position of the page file (i.e., the data file of the page) on the disk. When searching for any BasePage and DeltaPage on disk, the index file can be loaded first, that is, the index file is loaded into memory, and matching is performed based on the loaded index file. After matching the corresponding page identifier, the corresponding disk location can be found, and the BasePage and DeltaPage are loaded from the corresponding disk location. It should be noted that, generally speaking, the entries in the index can be sorted according to the page identifier, so that binary search can be implemented.

[0272] The index files stored on disk are similar to BasePage and DeltaPage. Fig.21 As shown in . In particular, a BasePage generally contains the contents of the full amount of tree nodes in a memory page, so the version of the top-level tree node in the BasePage is the version of the BasePage, and the lexicographic order of the contents from the root node to the top-level tree node in the BasePage can be used as the PageID of the BasePage. A DeltaPage can be a collection of multiple consecutive modifications to the MemoryPage, so that a DeltaPage can correspond to multiple consecutive versions, and for any version, it is generally not the full amount of tree node content, and generally speaking, a DeltaPage does not contain the full amount of tree node content. The version of an incremental page can be the lowest version in the corresponding one or more memory pages, or it can be the highest version in the corresponding one or more memory pages. The following is explained using the lowest version as an example.

[0273] After receiving the read state key and version, for example, after the storage module receives the read state key and version, a series of page IDs corresponding to LogicalPage can be constructed according to the state key, and the memory page can be matched according to the version and page ID. If the memory page does not exist in the memory, the corresponding base page and incremental page can be further searched and loaded from the persistent storage medium according to the page identifier.

[0274] Specifically, for the page ID of the version build, if there is a BasePage corresponding to the version and page ID on the disk, it can be found in the index. For example, to find the version 24 and the same page ID as MemoryPage B5 in the figure, you can traverse or perform a binary search in the index according to the version number, find several entries matching 24, and search for entries matching PageID in these matching entries. For PageID, you can match the suffix. Assuming that PageID also matches, that is, the PageType in the matching entry is BasePage, it means that the full tree node content for restoring the MemoryPage is already available in the BasePage, so the Page file on the disk can be read and loaded according to the disk location corresponding to the version + page type + ID, thereby restoring the MemoryPage.

[0275] If the search version is 25 and the same page ID as MemoryPage B5 in the figure, you can traverse or perform a binary search in the index according to version + page type + PageID. If you find several adjacent entries that match version 25, and further search for entries with matching PageID in these adjacent entries, as mentioned above, perform a suffix match with PageID in the adjacent entries of version 25 + page type + PageID in the index. Fig.21 As shown in , since the versions of the DeltaPage corresponding to B5 are 25 and 27 respectively, that is, 25 exists, assuming that PageID also matches, this search is a hit, and the hit PageType is DeltaType. Then, further search for the previous DeltaPage or BaseType. The version 25 can be -1 operated to get 24, and then search in the index with version 24 and PageID. The result is as described above, and the file with PageType of BasePage in the matching entry is found. After finding the file of the BaseType type, since the BasePage file already has the full tree node content of a MemoryPage, the BaseType with version 24 and PageID matching can be combined with the DeltaType with version 25 and PageID matching to restore the target MemoryPage with version 25, thereby ending the search. Then, the BaseType with version 24 and PageID matching and the DeltaType with version 25 and PageID matching can be loaded, and the BaseType with version 24 and PageID matching and the DeltaType with version 25 and PageID matching can be replayed in the order of versions, thereby restoring the MemoryPage with version 25. The specific recovery method is as described in S192 or S216 above, and will not be repeated here.

[0276] Similarly, if the search version is 26 and the same page ID as MemoryPage B5 in the figure, you can traverse or perform a binary search in the index according to the version number. If you find several entries matching 26, you can further search for entries with the same PageID among these matching entries. Fig.21 As shown in , since the IDs of the DeltaPage corresponding to B5 are 25 and 27 respectively, and 26 does not exist, this search did not hit. Since the example here sets the version of the incremental page to the lowest version of the corresponding one or more memory pages, if there is no entry matching the ID of version 26, the version is subtracted by 1 and searched again. As shown in the previous paragraph, the BaseType with version 24 and PageID matching and the DeltaType with version 25 and PageID matching are finally found. Then, the BaseType with version 24 and PageID matching and the DeltaType with version 25 and PageID matching can be loaded, and the BaseType with version 24 and PageID matching and the DeltaType with version 25 and PageID matching can be replayed in version order, thereby restoring the MemoryPage with version 26. The specific recovery process is as described above and will not be repeated here.

[0277] Similarly, if the search version is 27 and the same page ID as MemoryPage B5 in the figure, the index can be traversed or binary searched according to version + page type + PageID. If several adjacent entries matching version 27 are found, and further search for entries with matching PageID in these adjacent entries, as mentioned above, suffix matching is performed with PageID in the adjacent entries of version 27 + page type + PageID in the index. Fig.21 As shown in , since the versions of the DeltaPage corresponding to B5 are 25 and 27 respectively, that is, 27 exists, assuming that PageID also matches, this search is a hit, and the hit PageType is DeltaType. Then, further search for the previous DeltaPage or BaseType. The version 27 can be -1 operated to get 26, and then search in the index with version 26 and PageID. After recursive search, as described in the above paragraphs, the file with PageType as BasePage in the matching entry is finally found. After finding the file of the BaseType type, since the BasePage file already has the full tree node content of a MemoryPage, the BaseType with version 24 and PageID matching can be combined with the DeltaType with versions 25 and 27 and PageID matching to restore the target version 27 of MemoryPage, thereby ending the search. Furthermore, the BaseType with version 24 and PageID matching and the DeltaType with version 25 and 27 PageID matching can be loaded, and the BaseType with version 24 and PageID matching and the DeltaType with version 25 and 27 PageID matching can be replayed in version order, thereby restoring the MemoryPage with version 27. The specific recovery process is as described above and will not be repeated here. The above search process shows that if a matching incremental page is found in the index file of the base page / incremental page according to the version and page ID, the following operations are recursively performed: the version is decremented, and the corresponding base page / incremental page is searched in the index file according to the decremented version and page ID until the base page is found, and the search process ends.

[0278] Similarly, if the search version is 28 and the same page ID as MemoryPage B5 in the figure, you can traverse or perform a binary search in the index according to the version number. If you find several entries matching 28, you can further search for entries with the same PageID among these matching entries. Fig.21 As shown in , since the IDs of the DeltaPage corresponding to B5 are 25 and 27 respectively, and 28 does not exist, this search did not hit. Since the example here sets the version of the incremental page to the lowest version of the one or more memory pages corresponding to it, if there is no entry matching the ID of version 28, the version is minus 1 and searched again. After recursive search, as shown in the above paragraphs, the BaseType with version 24 and PageID matching and the DeltaType with version 25 and 27 PageID matching are finally found. Then, the BaseType with version 24 and PageID matching and the DeltaType with version 25 and 27 PageID matching can be loaded, and the BaseType with version 24 and PageID matching and the DeltaType with version 25 and 27 PageID matching can be replayed in version order, thereby restoring the MemoryPage with version 28. The specific recovery process is as described above and will not be repeated here. The above search process shows that if no matching incremental page is found in the index file of the base page / incremental page based on the version and page ID, the following operations are recursively performed: the version is decremented, and the corresponding base page / incremental page is searched in the index file based on the decremented version and page ID until the base page is found, and the search process ends.

[0279] In the above search process, if the PageType in the matching entry is DeltaPage, it means that the tree node content for recovering part of the increment of the MemoryPage is already available in the DeltaPage, but the possible previous version of DeltaPage and BasePage are also needed. Therefore, it is necessary to decrement the version again and continue to traverse in the index or perform a binary search with the suffix ID until a BasePage with the same page ID and lower than the largest version in the version is found. After finding all the BasePages and DeltaPages, they are loaded into memory, and can be played back in order according to these BasePages and DeltaPages, and finally the specified version of MemoryPage is recovered.

[0280] The above solution is a general feasible solution. For each version and PageID to be found, it is necessary to match them in the index according to the version and PageID, which is a relatively complex task involving a large amount of searching and comparison.

[0281] Furthermore, the present application proposes a method for matching base pages and delta pages to speed up the search process in the disk. In terms of chronological order, BasePage / DeltaPage is generated by time, or in other words, by version order. In this way, the generation time of the subsequent version is later, and the generation time of the predecessor version is earlier. In this scheme, the data file storage location of the previous predecessor version can be set in the Page file of the disk, so that as the subsequent version is formed, a series of forward linked lists pointing to the previous predecessor file are formed in each version, such as Fig. 22 shown.

[0282] The method for finding a base page and / or an incremental page proposed in the present application includes:

[0283] S220: receiving the read status key and version;

[0284] S222: Match the version and page ID in the page identifier of the logical page in lexicographic order according to the state key and version;

[0285] S224: searching for an incremental page in the index file according to the version and the page ID;

[0286] S226: After the incremental page is found, the file position of the base page / incremental page of the predecessor stored in the found incremental page is recursively read until the base page is found;

[0287] S228: Read the corresponding data file from the persistent storage medium according to the persistent storage medium position of the found base page / incremental page.

[0288] The data files here are the base pages / incremental pages linked by the forward linked list.

[0289] S220 and S222 are similar to the above and will not be described in detail.

[0290] The above process mentioned:

[0291] If a matching incremental page is found in the index file of the base page / incremental page according to the version and page ID, the following operations are recursively performed: the version is decremented, and the corresponding base page / incremental page is searched in the index file according to the decremented version and page ID until the base page is found, and the search process ends; and,

[0292] If no matching incremental page is found in the index file of the base page / incremental page based on the version and page ID, the following operations are recursively performed: the version is decremented, and the corresponding base page / incremental page is searched in the index file based on the decremented version and page ID until the base page is found, and the search process ends.

[0293] From the above content, we can know that for the version and page ID that are being searched, there is a corresponding incremental page on disk. According to the above method, the corresponding incremental page entry can always be found first in the index. Specifically, it includes:

[0294] If a matching delta page entry is found in the index file based on the version and page identifier, or,

[0295] If no matching incremental page entry is found in the index file according to the version and page ID, the following operations are recursively performed: the version is decremented, and the corresponding incremental page entry is searched in the index file according to the decremented version and page ID, until an incremental page entry matching the version is found.

[0296] Next, the recursive search for the corresponding predecessor base page / incremental page entry in the index file according to the decremented version and page ID can be avoided, and the file location of the predecessor base page / incremental page stored in the first found incremental page entry can be read.

[0297] like Fig. 22 As shown in , for example, if the search is for Page 28 pointing to B5 (the details of PageID are ignored here), the incremental page DeltaPage27 pointing to Page B5 can be first found through S220, S222 and S224. Since DeltaPage27 stores the file location of its predecessor DeltaPage, that is, DeltaPage25, it is possible to avoid searching in the index again, and instead find DeltaPage27 directly on the disk based on the file location of its predecessor DeltaPage25 stored in DeltaPage27. Similarly, since DeltaPage25 stores the file location of its predecessor DeltaPage, that is, DeltaPage24, it is possible to avoid searching in the index again, and instead find BasePage24 directly on the disk based on the file location of its predecessor BasePage24 stored in DeltaPage25. At this point, the base page is found, and as mentioned above, after finding the file of the BaseType type, since the BasePage file already has the full tree node content of a MemoryPage, the BaseType with version 24 and PageID matching can be combined with the DeltaType with versions 25 and 27 of PageID matching to restore the target MemoryPage with version 28, thus ending the search. Furthermore, the BaseType with version 24 PageID matching and the DeltaType with versions 25 and 27 of PageID matching can be loaded, and the BaseType with version 24 and PageID matching and the DeltaType with versions 25 and 27 of PageID matching can be replayed in version order, thereby restoring the MemoryPage with version 28.

[0298] It should be noted that, in the embodiment of the present method, since the BasePage file already has the full tree node content of a MemoryPage, the file storage location of the previous predecessor version can be set in the file of the incremental page instead of setting the file storage location of the previous predecessor version in the file of the base page.

[0299] In order to avoid matching each version and PageID to be searched in the index according to the version and PageID, that is, to reduce the work of searching and comparing, the present application also proposes a method embodiment for matching the base page and the incremental page. In this method embodiment, the previous predecessor version can be recorded in the incremental page entry of the index, including the version of the previous predecessor base page / incremental page. As the subsequent version is formed, its predecessor version is recorded in each incremental page entry in the index, such as Fig.23 shown.

[0300] The method for finding a base page and / or an incremental page proposed in the present application includes:

[0301] S230: receiving the read status key and version;

[0302] S232: Match the version and page ID in the page identifier of the logical page in lexicographic order according to the state key and version;

[0303] S234: searching for an incremental page entry in the index file according to the version and the page ID;

[0304] S236: After finding the incremental page entry, recursively read the version of the predecessor base page / incremental page stored in the found incremental page entry and search in the index file according to the found version and page ID until the base page is found;

[0305] S238: Determine the location of the corresponding data file in the persistent storage medium according to the found version of the base page / incremental page and the page ID, and read the corresponding data file from the persistent storage medium.

[0306] S230 and S232 are similar to the above and will not be described in detail.

[0307] In S234, the incremental page entry may be first searched in the index file according to the version and the page ID. If a matching incremental page entry is found in the index file according to the version and the page ID, the version of the predecessor base page / incremental page may be read from the found incremental page entry.

[0308] In this way, after finding the incremental page entry in S236, the version of the predecessor base page / incremental page stored in the found incremental page entry can be recursively read and searched in the index file according to the found version and page ID until the base page entry is found.

[0309] like Fig.23 As shown in, for example, if the page with version 27 pointing to B5 (the details of PageID are ignored here), the delta page entry DeltaPage27 pointing to Page B5 can be first found through S230, S232 and S234. Since the version of the previous predecessor base page / delta page is set in the delta page entry of the index, that is, the entry DeltaPage27 stores the version 25 of its predecessor DeltaPage, DeltaPage25, so the predecessor version 25 in the index can be located. In the next search, only the entry with version 25 needs to be searched, thereby avoiding the search in all index files.

[0310] In addition, in S234, the incremental page entry is searched in the index file according to the version and the page ID. If no matching incremental page entry is found in the index file according to the version and the page ID, the following operations are recursively performed: the version is decremented, and the corresponding incremental page entry is searched in the index file according to the decremented version and the page ID until an incremental page entry with a matching version is found. Thus, the version of the predecessor base page / incremental page stored in the found incremental page entry can be read.

[0311] In this way, after finding the incremental page entry in S236, the version of the predecessor base page / incremental page stored in the found incremental page entry can be recursively read and searched in the index file according to the found version and page ID until the base page entry is found.

[0312] like Fig.23 As shown in , for example, if the page with version 28 pointing to B5 (the details of PageID are ignored here) is searched, then the incremental page entry DeltaPage28 pointing to Page B5 can be first found through S230, S232 and S234. Since there is no DeltaPage with version 28 and matching PageID in the index, the version is decremented, and here the version is 28 after decrement. The corresponding incremental page entry is searched in the index file according to the decremented version 28 and the page ID, and the incremental page entry with matching version can be found (PageID also matches). Furthermore, since the version of the previous predecessor base page / incremental page is set in the incremental page entry of the index, that is, the entry DeltaPage27 stores its predecessor DeltaPage, that is, version 25 of DeltaPage25, so the predecessor version 25 in the index can be located. In the next search, it is only necessary to search in the entry with version 25, thereby avoiding searching in all index files.

[0313] Furthermore, as described in S238, the position of the corresponding data file in the persistent storage medium can be determined based on the found base page / incremental page version and the page ID in the index, and the corresponding data file can be read from the persistent storage medium. That is, the position of the corresponding data file in the persistent storage medium can be determined based on the found base page version 24 and the entry of the page ID in the index, and the position of the corresponding data file in the persistent storage medium can be determined based on the entry of the incremental page 25 and the page ID in the index, and the position of the corresponding data file in the persistent storage medium can be determined based on the entry of the incremental page 27 and the page ID in the index, and the corresponding data file can be read from the persistent storage medium.

[0314] The above search in the index file according to the version and the page ID may be a search in the index file according to the version, and further a search according to the page ID in the index file entry matching the found version. The search may be a traversal search or a binary search.

[0315] As mentioned above, the search process starts from the root node of the MPT, and traverses and matches downward along the path of the tree in the lexicographic order of the key. The base page and incremental page can be used as the smallest unit of disk persistence of the tree. In order to speed up the search for the base page, the application also provides a method for searching the base page / incremental page. In this method, the version of the base page / incremental page where the child node of the tree node contained therein is located can be set in the entry of the base page / incremental page of the index file.

[0316] like Fig.24 As shown, the index file contains many entries. In each entry, in addition to setting the corresponding relationship between the page identifier and the location of the page file on the disk, the version of the base page where the child node corresponding to the tree node in the page is located can also be set. For example Fig.24 For MemoryPage B4, the index contains the corresponding relationship between the page ID of the page and the location of the page file on the disk. For example, for the page DeltaPage28, an entry in the index is 28+DeltaPage(type)+ID→(). In addition, the page DeltaPage28 contains two versions of tree nodes, 28 and 29. In this way, the version of the base page where the child nodes corresponding to the tree nodes 28 and 29 in the page are located can be set in the index. For example, if the version of the base page where the child nodes are located is 24, Fig.24 In the index, this meaning is represented by sub-version. Similarly, the DeltaPage26 page contains two versions of tree nodes, 26 and 27. The version of the base page where the sub-nodes corresponding to the tree nodes 26 and 27 in the page are located can be set in the index, such as the version of the base page where the sub-nodes are located is 24.

[0317] Similar, such as Fig.25 As shown, the index file contains many entries. In each entry, in addition to setting the correspondence between the page identifier and the location of the page file on the disk, the version of the incremental page where the child node corresponding to the tree node in the page is located can also be set. For MemoryPage B4, the page DeltaPage26 contains two versions of tree nodes, 26 and 27. In this way, the version of the incremental page in the next layer MemoryPage B5 where the child nodes corresponding to the tree nodes with versions 26 and 27 in the page are located can be set in the index. For example, the child node of the tree node with version 26 is also 26, and the child node of the tree node with version 27 is also 27, but the child nodes 26 and 27 are respectively located in DeltaPage25 and DeltaPage27 in the lower MemoryPage B5 layer. Fig.25 This meaning is indicated by the sub-version in the index.

[0318] Combining the above two situations, if Fig.25 As shown in , similarly, for the tree node containing two versions 24 and 25 in DeltaPage24 of MemoryPage B4, the version of the base page / delta page where the child nodes corresponding to the tree nodes 24 and 25 in the page are located can be set in the index, and the version of the base page / delta page in the next MemoryPage B5 layer where the child nodes corresponding to the tree nodes with versions 24 and 25 in the page are located can be set in the index. For example, the child node of the tree node with version 24 is also 24, and the child node version of the tree node with version 25 is also 25, but the child nodes 24 and 25 are respectively located in BasePage24 and DeltaPage25 in the lower MemoryPageB5 layer.

[0319] Combining the above two situations, if Fig.24 As shown in , similarly, for the tree node with version 30 in the BasePage30 page in MemoryPage B4, the version of the base page where the child node corresponding to the tree node 30 in the page is located can be set in the index. Specifically, the version of the base page in the next MemoryPage B5 layer where the child node corresponding to the tree node with version 30 in the page is located can be set in the index. For example, the child node of the tree node with version 30 is also 30, and the child node 30 is located in BasePage30 in the lower MemoryPage B5 layer.

[0320] Based on this, the present application provides a method for searching a base page / incremental page, including:

[0321] S240: receiving the read status key and version;

[0322] S242: Match the version and page ID in the page identifier of the logical page in lexicographic order according to the state key and version;

[0323] S244: searching for a base page / incremental page entry in the index file according to the version and the page ID;

[0324] S246: Search the index file for the lower-level base page / incremental page entry according to the version of the base page / incremental page where the child node recorded in the tree node in the found base page / incremental page entry is located.

[0325] Furthermore, according to the persistent storage medium position in the found base page / incremental page entry of the child node, the corresponding data file can be read from the persistent storage medium.

[0326] In particular, the search for the incremental page in the index file according to the version and page ID in S244 may or may not be found. If a matching incremental page is found in the index file according to the version and page ID, S246 may be continued. Fig.24 In the process, incremental page entry 26 is found in the index file according to version 26 and page ID, and execution of S246 continues.

[0327] If no matching incremental page is found in the index file according to the version and page ID, the following operations may be performed recursively: the version is decremented, and the corresponding incremental page entry is searched in the index file according to the decremented version and page ID until an incremental page with a matching version is found. Fig.24 In the example, if no incremental page entry matching the MemoryPage B4 layer is found in the index file according to version 27 and the page ID, then version 27 is reduced by 1 to obtain 26, and the incremental page corresponding to the MemoryPage B4 layer is searched in the index file according to the decremented version 26 and the page ID, and DeltaPage26 can be found. Then continue to execute S246. The details are as described above and will not be repeated here. It should be noted that, in the case where an incremental page contains multiple version tree nodes, the above process of decrementing and searching the version may be performed recursively until an incremental page matching the version is found.

[0328] In addition, you can Fig.23 , 24 , 25 solutions, such as Fig.26 In this combined solution, the base page / incremental page entry of the index file can be set with the base page / incremental page version of the child node of the tree node, and the incremental page entry of the index file can be set with the previous predecessor incremental page version.

[0329] Based on this, the present application also provides a method embodiment for finding a base page / incremental page, including:

[0330] S260: receiving the read status key and version;

[0331] S262: Match the version and page ID in the page identifier of the logical page in lexicographic order according to the state key and version;

[0332] S264: searching for an incremental page entry in the index file according to the version and the page ID;

[0333] S266: On the one hand, the lower-level base page / incremental page entry is searched in the index file according to the version of the base page / incremental page where the child node recorded in the tree node in the found incremental page entry is located; on the other hand, the version of the predecessor base page / incremental page is recursively read in the found lower-level incremental page entry and searched in the index file according to the found version and page ID until the lower-level base page is found.

[0334] As mentioned above, the search process starts from the root node of the MPT, and traverses and matches along the tree path in the lexicographic order of the key. For example, to find the tree node with version number 27, according to the relationship between the tree node and BasePage and DeltaPage, it may involve first searching in the BasePage and DeltaPage of MemoryPage B4, and then searching in the BasePage and DeltaPage of the lower MemoryPage B5. Here, we want to find the BasePage and DeltaPage corresponding to MemoryPage B5 with version number 27.

[0335] Specifically, after receiving the read state key and version, the version and page ID in the page identifier of the logical page can be matched in lexicographic order according to the state key and version. Then, the base page / incremental page entry is searched in the index file according to the version and page ID. Here, for the tree node with version 27 in the MemoryPage B4 layer, it exists in DeltaPage26, that is, no matching incremental page is found in the index file according to the version and page identifier. At this time, the following operations can be performed recursively: decrement the version, and search the corresponding incremental page in the index file according to the decremented version and page ID until the incremental page with matching version is found. Specifically, after 27 is subtracted by 1, 26 is obtained, and the corresponding incremental page entry can be searched in the index file according to the version 26 and page ID. As mentioned above, the version of DeltaPage26 matches. Assuming that the page ID also matches, the DeltaPage 26 can be found here. Since the version of the base page / incremental page where the child node of the tree node is located is set in the incremental page entry of the index file, the version of the base page / incremental page where the child node of the tree node record is located can be found, which is BasePage 24 and DeltaPage 27 in the MemoryPage B5 layer. On the other hand, since the version of the previous predecessor base page / incremental page is set in the incremental page entry of the index file, the lower base page / incremental page entry can be found in the index file. Specifically, the version of its predecessor base page / incremental page can be recursively read in the found lower incremental page entry and searched in the index file according to the found version and page ID, so as to find DeltaPage 27 and BasePage 24. Therefore, in the lower MemoryPage B5 layer, BasePage 24, DeltaPage 25 and DeltaPage 27 can be found.

[0336] Furthermore, the MemoryPage with version 27 can be restored based on the found BasePage 24, DeltaPage 25 and DeltaPage 27. For the specific restoration process, please refer to the above.

[0337] In another example, to find a tree node with version number 28, here for the tree node with version 28 in the MemoryPage B4 layer, it exists in DeltaPage 28, that is, the matching incremental page can be found in the index file according to the version and page ID. Since the version of the base page / incremental page where the child node of the tree node is located is set in the incremental page entry of the index file, then, according to the version of the base page / incremental page where the child node recorded in the found incremental page entry is located, BasePage 24 and DeltaPage 27 in the MemoryPage B5 layer here can be found. On the other hand, since the version of the previous predecessor base page / incremental page is set in the incremental page entry of the index file, the lower-level base page / incremental page entry can be found in the index file. Specifically, the version of its predecessor base page / incremental page can be recursively read in the found lower-level incremental page entry and searched in the index file according to the found version and page ID, so as to find DeltaPage 27 and BasePage 24. Therefore, in the lower layer MemoryPage B5, BasePage 24, DeltaPage 25, and DeltaPage 27 can be found.

[0338] The present application also provides a method embodiment for finding a base page and / or an incremental page, still as Fig.24 and 25 As shown, the difference is that the data file of the base page / incremental page is provided with the version of the base page / incremental page where the child node of the tree node is located, and the method includes:

[0339] S270: receiving the read status key and version;

[0340] S272: Match the version and page ID in the page identifier of the logical page in lexicographic order according to the state key and version;

[0341] S274: Searching for a base page / incremental page entry in the index file according to the version and page ID;

[0342] S276: After the base page / incremental page entry is found, the version of the base page / incremental page where the child node of the tree node is stored is read in the found base page / incremental page data file, and the lower-level incremental page entry is searched in the index file based on the found version and page ID;

[0343] Furthermore, according to the persistent storage medium position found in the lower-level base page / incremental page stripe, the corresponding data file can be read from the persistent storage medium.

[0344] In particular, in the above S274, the base page / incremental page may be found or not found in the index file according to the version and page ID. If a matching base page / incremental page is found in the index file according to the version and page ID, S276 may be continued. Fig.24 In the process, the base page / incremental page entry 26 is found in the index file according to the version 26 and the page ID, and then execution of S276 is continued.

[0345] If no matching incremental page is found in the index file according to the version and page ID, the following operations may be performed recursively: the version is decremented, and the corresponding incremental page entry is searched in the index file according to the decremented version and page ID until an incremental page with a matching version is found. Fig.24 In the example, if no incremental page entry matching the MemoryPage B4 layer is found in the index file according to version 27 and the page ID, then version 27 is reduced by 1 to obtain 26, and the incremental page corresponding to the MemoryPage B4 layer is searched in the index file according to the decremented version 26 and the page ID, and DeltaPage26 can be found. Then continue to execute S276. The details are as described above and will not be repeated here. It should be noted that, in the case where an incremental page contains multiple version tree nodes, the above process of decrementing and searching the version may be performed recursively until the base page / incremental page matching the version is found.

[0346] Similarly, based on the above S270 to S278, Fig.23 , 24 , 25 solutions, still Fig.26 In this combined solution, the data file of the base page / incremental page is provided with the version of the base page / incremental page where the child node of the tree node is located, and the version of the previous predecessor incremental page can be provided in the incremental page entry of the index file.

[0347] Based on this, the present application also provides a method embodiment for finding a base page / incremental page, including:

[0348] S280: receiving the read status key and version;

[0349] S282: Match the version and page ID in the page identifier of the logical page in lexicographic order according to the state key and version;

[0350] S284: Searching for a base page / incremental page entry in the index file according to the version and the page ID;

[0351] S286: On the one hand, the lower-level base page / incremental page entry is searched in the index file according to the version of the base page / incremental page where the child node of the tree node is located recorded in the base page / incremental page data file corresponding to the found base page / incremental page entry; on the other hand, the version of the predecessor base page / incremental page is recursively read in the found lower-level incremental page entry and searched in the index file according to the found version and page ID until the lower-level base page is found.

[0352] As mentioned above, the search process starts from the root node of the MPT, and traverses and matches along the tree path in the lexicographic order of the key. For example, to find the tree node with version number 27, according to the relationship between the tree node and BasePage and DeltaPage, it may involve first searching in the BasePage and DeltaPage of MemoryPage B4, and then searching in the BasePage and DeltaPage of the lower MemoryPage B5. Here, we want to find the BasePage and DeltaPage corresponding to MemoryPage B5 with version number 27.

[0353] Specifically, after receiving the read state key and version, the version and page ID in the page identifier of the logical page can be matched in lexicographic order according to the state key and version. Then, the base page / incremental page entry is searched in the index file according to the version and page ID. Here, for the tree node with version 27 in the MemoryPage B4 layer, it exists in DeltaPage26, that is, no matching incremental page is found in the index file according to the version and page identifier. At this time, the following operations can be performed recursively: decrement the version, and search the corresponding incremental page in the index file according to the decremented version and page ID until the incremental page with matching version is found. Specifically, after 27 is subtracted by 1, 26 is obtained, and the corresponding incremental page entry can be searched in the index file according to the version 26 and page ID. As mentioned above, the version of DeltaPage26 matches. Assuming that the page ID also matches, the DeltaPage 26 can be found here. Since the version of the base page / incremental page where the child node of the tree node is located is set in the data file of the base page / incremental page, the lower base page / incremental page entry can be found in the index file according to the version of the base page / incremental page where the child node of the tree node is located recorded in the base page / incremental page data file corresponding to the found base page / incremental page entry. Here, the base page / incremental page entry corresponds to the base page / incremental page data file, including a tree node with version 26 and a tree node with version 27. The version of the child node of the tree node with version 26 recorded in the data file corresponds to the incremental page version 25. The version of the child node of the tree node with version 27 recorded in the data file corresponds to the incremental page version 27. Moreover, the version of the lower base page corresponding to the tree node with version 26 recorded in the data file is 24. In this way, BasePage24 and DeltaPage 27 in the lower MemoryPage B5 layer are found. On the other hand, since the version of the previous predecessor base page / incremental page is set in the incremental page entry of the index file, the lower base page / incremental page entry can be found in the index file. Specifically, the version of the predecessor base page / delta page can be recursively read in the found lower-level delta page entry and searched in the index file according to the found version and page ID to find DeltaPage 27 and BasePage 24. Therefore, BasePage24, DeltaPage 25, and DeltaPage 27 can be found in the lower layer MemoryPageB5.

[0354] Furthermore, the MemoryPage with version 27 can be restored based on the found BasePage 24, DeltaPage 25 and DeltaPage 27. For the specific restoration process, please refer to the above.

[0355] In another example, to find a tree node with version number 28, here the tree node with version 28 in the MemoryPage B4 layer exists in DeltaPage 28, that is, the matching incremental page can be found in the index file according to the version and page identifier. Since the base page / incremental page data file is provided with the version of the base page / incremental page where the child node of the tree node is located, the lower-level base page / incremental page entry can be found in the index file according to the base page / incremental page entry corresponding to the base page / incremental page data file where the child node of the tree node is located, which is BasePage 24 and DeltaPage 27 in the MemoryPage B5 layer. On the other hand, since the previous predecessor base page / incremental page version is set in the incremental page entry of the index file, the lower-level base page / incremental page entry can be found in the index file. Specifically, the version of the predecessor base page / delta page can be recursively read in the found lower-level delta page entry and searched in the index file according to the found version and page ID to find DeltaPage 27 and BasePage 24. Therefore, BasePage 24, DeltaPage 25, and DeltaPage 27 can be found in the lower-level MemoryPage B5 layer.

[0356] Similarly, the data file location of the base page / incremental page where the child node of the tree node is located can be set in the data file of the base page / incremental page. Based on this, the present application also provides an embodiment of a method for searching a base page / incremental page, including:

[0357] S290: receiving the read status key and version;

[0358] S292: Match the version and page ID in the page identifier of the logical page in lexicographic order according to the state key and version;

[0359] S294: Searching for a base page / incremental page entry in the index file according to the version and the page ID;

[0360] S296: After finding the base page / incremental page entry, read the data file position of the base page / incremental page where the child node of the tree node is stored in the found base page / incremental page data file, and read the data file of the base page / incremental page where the child node is located based on the read data file position.

[0361] Among them, in the data file of the basic page / incremental page, the data file location of the basic page / incremental page where the child node is located can be recorded in the slot connected to the child node in the tree node.

[0362] Similarly, the data file location of the base page / incremental page where the child node of the tree node is located can be set in the data file of the base page / incremental page, and the version of the previous predecessor base page / incremental page can be set in the incremental page entry of the index file. Based on this, the present application also provides an embodiment of a method for searching a base page / incremental page, including:

[0363] S300: receiving the read status key and version;

[0364] S302: Match the version and page ID in the page identifier of the logical page in lexicographic order according to the state key and version;

[0365] S304: Searching for a base page / incremental page entry in the index file according to the version and the page ID;

[0366] S306: Read the base page / incremental page data file according to the found base page / incremental page entry, and then obtain the data file position of the base page / incremental page where the child node of the tree node is located from the read data file, so as to find the lower-level base page / incremental page data file; recursively read the version of its predecessor base page / incremental page in the found lower-level incremental page entry and search in the index file according to the found version and page ID until the lower-level base page is found.

[0367] The searching for the incremental page entry in the index file according to the version and the page ID may include:

[0368] If a matching delta page is found in the index file based on the version and page identifier, or,

[0369] If no matching incremental page is found in the index file according to the version and page ID, the following operations are recursively performed: the version is decremented, and the corresponding incremental page is searched in the index file according to the decremented version and page ID, until an incremental page with a matching version is found;

[0370] The data file location of the predecessor base page / delta page stored in the delta page entry found is read.

[0371] Among them, in the data file of the basic page / incremental page, the data file location of the basic page / incremental page where the child node is located can be recorded in the slot connected to the child node in the tree node.

[0372] In blockchains such as Ethereum, the world state is a huge data structure that contains the balances, contract codes, and storage data of all accounts. As the network continues to run, this state continues to grow, resulting in a sharp increase in storage requirements and increasing hardware requirements for full blockchain nodes. Reducing the hard disk space required to run a full node can reduce the hardware requirements for a full blockchain node, and the increase in data volume will result in longer node synchronization time and lower query speed. Therefore, reducing storage requirements and maintaining performance is an important requirement for long-term operation of the blockchain. To meet this demand, Ethereum can implement world state pruning (State Pruning), which is a technology for optimizing the storage and management of blockchain data. The means is generally to only retain the complete state of the latest N blocks and delete the state in the old blocks. After pruning, the storage space required for full nodes can be reduced by more than 80%, which can significantly reduce storage requirements. Furthermore, more participants can run full nodes, which can enhance the decentralization of the network. This can also improve the synchronization speed. For example, a new node can complete synchronization in a few hours instead of a few days. In terms of query efficiency, state access and verification become faster.

[0373] It should be noted that while state pruning can significantly reduce storage requirements, it also brings some trade-offs. For example, very old historical states cannot be directly accessed, which may affect some applications or analysis tools. Therefore, the Ethereum community has been looking for the best solution to balance storage efficiency and functional integrity.

[0374] In general, world state pruning is an important component of the scalability and sustainability of blockchain projects such as Ethereum, and is critical to the long-term health of the network.

[0375] In Ethereum's state data pruning solution, snapshots are generally used for pruning. The snapshots generated by the latest 128 versions of written data are retained in memory, and all KV data in RocksDB (a variant of the aforementioned LevelDB) are traversed to prune data that does not exist in the snapshot. The disadvantage of this is that all historical data needs to be traversed, which consumes a lot of disk I / O and is time-consuming.

[0376] As mentioned above, the present application provides a solution for separating actual data (state value) and index by means of data files and index files, and the tree node as a whole can be used as the index of the state value. On this basis, it also provides the correspondence of tree nodes to MemoryPage according to the divided LogicalPage structure. Specifically, a MemoryPage can be maintained inside a LogicalPage, and MemoryPage is a memorized object of the LogicalPage. Moreover, for the LogicalPage, different versions of BasePage and DeltaPage can be generated according to the generation / modification behavior. BasePage and DeltaPage can be used as the smallest unit of disk persistence of the tree, that is, BasePage and DeltaPage are persistent objects of MemoryPage corresponding to LogicalPage. The persistent objects of the index file here include BasePage and DeltaPage.

[0377] The pruning of the world state includes the pruning of persistent index files. Generally speaking, a pruning version can be specified, and the goal is to prune at least some or all index files before the version.

[0378] The present application provides a method for tailoring a world state, including:

[0379] S310: receiving a specified cropped version;

[0380] S312: searching for the largest version base page that is not greater than the specified version in each logical page according to the version;

[0381] S314: For each logical page, the maximum version base page that is not greater than the specified version is used as a safe trimming point, and the base page / incremental page of any version in the logical page before the safe trimming point is deleted.

[0382] like Fig.21 As shown, for example, a series of BasePages and DeltaPages are persistently stored in the disk. Assume that the specified trimmed version received is 26, that is, the goal is to trim some or all index files before the specified version, and consider that the specific version after the trimmed version can be restored after trimming, so it is necessary to determine the actual trimmed version of different LogicalPages, which may be lower than the specified version in different LogicalPages.

[0383] In S312, the search can be performed step by step from the InternalPage where the root node is located to the LeafPage direction. The specific implementation is similar to the above and will not be repeated here. In particular, in order to speed up the search, the solution provided by the above-mentioned present application includes establishing another layer of index for BasePage and DeltaPage, and accordingly, the content of BasePage and DeltaPage becomes the data file corresponding to the index file. In this way, for a specified cropped version, at least part or all of the content before the specified version can be cropped. At least how much of the content before the specified version is cropped needs to be considered in order to be able to restore the specific version after the cropped version after cropping, so the actual cropped version may be lower than the specified version. In other words, the actual cropped version and all versions before it will generally be cropped through the cropping work, and the version retained after the actual cropped maximum version is generally called the safe cropping point, and the version of the safe cropping point is generally less than or equal to the specified version.

[0384] The specific implementation of S312 can be to search the index file for the maximum version base page that is not greater than the specified version in each logical page according to the version. Specifically, for a certain LogicalPage, the first search may directly find the base page equal to the specified version, and the base page is the safe clipping point. In addition, the first search may not find the base page equal to the specified version, but find the incremental page equal to the specified version, then there is no hit this time, and it is necessary to continue searching in the index file. For the latter, the following operations can be recursively performed: decrement the version, and search the corresponding base page in the index file according to the decremented version until the base page with a matching version is found. It is also possible that neither the base page equal to the specified version nor the incremental page equal to the specified version is hit. In this case, the following operations can be recursively performed: decrement the version, and search the corresponding base page entry in the index file according to the decremented version until the base page entry with a matching version is found.

[0385] In addition, for any version of the incremental page, the data file storage location of the previous predecessor version can be set in its data file. In this way, in the implementation of S312, if the incremental page is first found in the index file according to the version, the data file location of the base page / incremental page of the stored predecessor can be recursively read in the found incremental page until the data file location of the base page is found.

[0386] In addition, the version of the previous predecessor base page / incremental page can also be set in the incremental page entry of the index. In this way, if there is no hit in the index for the base page equal to the specified version, but an incremental page equal to the specified version is hit, the version of the predecessor base page / incremental page stored in the found incremental page entry can be recursively read and searched in the index file according to the found version until the base page is found; if there is neither a hit for the base page equal to the specified version nor a hit for the incremental page equal to the specified version, the following operations can be recursively performed: decrement the version, and search the corresponding base page / incremental page entry in the index file according to the decremented version until the base page / incremental page entry with a matching version is found. If in the process of recursively executing the above operations, the incremental page is first found instead of the base page, the version of the predecessor base page / incremental page stored in the found incremental page entry can be recursively read and searched in the index file according to the found version until the base page is found.

[0387] The above-mentioned searching in the index file according to the version may include a traversal search or a binary search.

[0388] In summary, after the above S312, it can be determined that:

[0389] The version of the largest base page in InternalPage B4 that is not greater than the specified version 26 is 23;

[0390] The version of the largest base page in InternalPage B5 that is not greater than the specified version 26 is 24;

[0391] The version of the largest base page in LeafPage B6 that is not greater than the specified version 26 is 25;

[0392] The version of the largest base page in InternalPage B7 that is not greater than the specified version 26 is 24;

[0393] The version of the largest base page in InternalPage B8 that is not greater than the specified version 26 is 24;

[0394] The version of the base page in LeafPage B9, which is the largest version not greater than the specified version 26, is 23.

[0395] In this way, S314 can use the maximum version base page determined in each LogicalPage as a safe clipping point to clip the BasePage and DeltaPage before the safe clipping point (the safe clipping point is retained and not clipped). Specifically:

[0396] For InternalPage B4, the BasePage and DeltaPage before version 23 are trimmed, while the BasePage and DeltaPage after version 23 are retained;

[0397] For InternalPage B5, the BasePage and DeltaPage before version 24 are trimmed, while the BasePage and DeltaPage after version 24 are retained;

[0398] For LeafPage B6, the BasePage and DeltaPage before version 25 are trimmed, while the BasePage and DeltaPage after version 25 are retained;

[0399] For InternalPage B7, the BasePage and DeltaPage before version 24 are trimmed, while the BasePage and DeltaPage after version 24 are retained;

[0400] For InternalPage B8, the BasePage and DeltaPage before version 24 are trimmed, while the BasePage and DeltaPage after version 24 are retained;

[0401] For LeafPage B9, BasePage and DeltaPage before version 23 are trimmed, while BasePage and DeltaPage of version 23 and later are retained.

[0402] In this application, BasePage and DeltaPage are used as the smallest units of disk persistence of the tree. Pruning them as a whole can greatly reduce the disk I / O of the pruning work.

[0403] After S314, any version of entries before the safe clipping point in the index file may be further deleted.

[0404] In addition, as mentioned above, the entry of the base page / incremental page of the index file can set the version of the underlying base page where the child node of the tree node is located. Based on this, the present application also provides a method for pruning the world state, including:

[0405] S320: receiving a specified cropped version;

[0406] S322: Search in the index file base page / incremental page entry according to the version;

[0407] S324: searching the lower-level base page entry in the index file according to the version of the base page where the child node of the tree node record in the found base page / incremental page entry is located;

[0408] S326: For the lower-level logical page, the base page corresponding to the found base page entry is used as a safe clipping point, and any version of the base page / incremental page data file in the logical page before the safe clipping point is deleted.

[0409] Each sub-LogicalPage is searched from top to bottom from the root node in the tree structure (that is, the LogicalPage where the root node is located), so that the BasePage corresponding to each sub-LogicalPage on the persistent storage medium can be found. Specifically, in the embodiment here, the index file can be searched from top to bottom from the root node in the tree structure (that is, the LogicalPage where the root node is located) index file in the index entry, so that the index file of each sub-LogicalPage can be further found in the index file, and then the data file of the BasePage corresponding to the persistent storage medium can be found.

[0410] From the upper LogicalPage, the lower LogicalPage can be searched from top to bottom. As mentioned above, it can be searched in the index file according to the version and page ID. As for the page ID, as mentioned above, there is a lexicographic order from top to bottom. LogicalPages of different layers have different lexicographic order contents. The above embodiments provide a variety of implementation schemes, one of which can be implemented as follows. Fig.19 as shown in .

[0411] For the specific implementation of S322 and S324, please refer to the above and will not be repeated here.

[0412] Similarly, after the method, any version of the entry before the safe clipping point in the index file may be deleted.

[0413] As mentioned above, the data file of the base page / incremental page is provided with the version of the base page / incremental page where the child node of the tree node is located. Based on this, the present application also provides a method for pruning the world state, including:

[0414] S330: receiving a specified cropped version;

[0415] S332: Search in the index file base page / incremental page entry according to the version;

[0416] S334: After the base page / incremental page entry is found, the version of the lower-level base page where the child node of the tree node is stored is read in the found base page / incremental page data file, and the lower-level base page entry is searched in the index file based on the found version;

[0417] S336: For the lower-level logical page, the base page corresponding to the found base page entry is used as a safe clipping point, and any version of the base page / incremental page data file in the logical page before the safe clipping point is deleted.

[0418] For the specific implementation of S332 and S334, please refer to the above and will not be repeated here.

[0419] Similarly, after the method, any version of the entry before the safe clipping point in the index file may be deleted.

[0420] The above processes of S310 to S314, S320 to S326 and S330 to S336 are search and deletion performed in units of LogicalPage. Generally speaking, the BasePage and DeltaPage stored in the disk can be arranged in order according to the version, specifically in the order of version + page type + page ID in this application, and the BasePage and DeltaPage of each version corresponding to all LogicalPages are arranged in order. In other words, the data files of BasePage and DeltaPage stored in the disk are not always gathered together according to LogicalPage. In the above methods, for the upper and lower layers of LogicalPage, the safe clipping points of LogicalPages of different layers may be different, and even the safe clipping points of different LogicalPages in the same layer may be different. In this way, the BasePage and DeltaPage data files arranged in order of version (including version + page type + page ID) on the disk may be selectively deleted from the continuous data files. And this selective deletion generally requires reading some data files from the disk into the memory, marking the data files to be deleted with deletion marks, and then writing them to the disk. Subsequently, the marked data files are actually deleted through the compaction process of the disk. The data files written to the disk after being marked for deletion make the valid files on the disk discontinuous for a period of time before they are actually deleted, so that the valid data files on the disk are also continuous, such as Fig.28 as shown in . Fig.28 In the figure, for simplicity, the base page is represented by a square in the tree structure on the left, and the incremental page is represented by a triangle in the tree structure on the left. The figure also draws a line between the upper and lower LogicalPage layers to represent the lower base page corresponding to the upper base page. In addition, the data files in the disk on the right that have a regular dot distribution represent examples of data files to be deleted, while those that have no regular dot distribution represent data files to be retained.

[0421] Based on this, in order to delete continuous space in the disk as much as possible, the present application also provides an improved method for trimming the world state, corresponding to the above S310 to S314, including:

[0422] S340: receiving a specified cropped version;

[0423] S342: searching for the largest version base page that is not greater than the specified version in each logical page according to the version;

[0424] S344: For the largest version base page that is not greater than the specified version among all the found logical pages, take the lowest version base page as the safe trimming point, and delete the base page / incremental page of any version in the logical page before the safe trimming point.

[0425] Similarly, corresponding to the above S320 to S326, the present application also provides a method for pruning the world state, wherein the entry of the base page / incremental page of the index file is provided with a version of the lower base page where the child node of the tree node is located, and the method comprises:

[0426] S350: receiving a specified cropped version;

[0427] S352: Search in the index file base page / incremental page entry according to the version;

[0428] S354: After the base page / incremental page entry is found, the version of the lower-level base page where the child node of the tree node is stored is read in the found base page / incremental page data file, and the lower-level base page entry is searched in the index file based on the found version;

[0429] S356: For the largest version base page that is not greater than the specified version among all the found logical pages, take the lowest version base page as the safe trimming point, and delete the base page / incremental page of any version in the logical page before the safe trimming point.

[0430] Similarly, corresponding to the above S330 to S336, the present application also provides a method for pruning the world state, wherein the data file of the base page / incremental page is provided with a version of the lower base page where the child node of the tree node is located, and the method comprises:

[0431] S360: receiving a specified cropped version;

[0432] S362: Search in the index file base page / incremental page entry according to the version;

[0433] S364: After the base page / incremental page entry is found, the version of the lower-level base page where the child node of the tree node is stored is read in the found base page / incremental page data file, and the lower-level base page entry is searched in the index file based on the found version;

[0434] S366: For the largest version base page that is not greater than the specified version among all the found logical pages, take the lowest version base page as the safe trimming point, and delete the base page / incremental page of any version in the logical page before the safe trimming point.

[0435] In the above-mentioned embodiments, the lowest version of the base page among the maximum version base pages that are not greater than the specified version in all the found logical pages is used as a safe trimming point, and the base page / incremental page of any version in the logical page before the safe trimming point is deleted, so that continuous base page / incremental page data files can be trimmed in the disk, so that the retained base page / incremental page data files are also continuous in the disk.

[0436] In the above embodiments of S31*, S32*, S33*, S34*, S35*, and S36*, redundancy exists for the specified trimming version. In the embodiments of S34*, S35*, and S36*, since the lowest version of the base page of the largest version of all logical pages that are not greater than the specified version is used as the safe trimming point, the base page / incremental page of any version of the logical page before the safe trimming point is deleted, that is, there are still several versions between the specified version and the safe trimming point in the retained data file, so generally speaking, the redundancy is relatively large. In the embodiments of S31*, S32*, and S33*, although for a LogicalPage, the base page with the largest version not greater than the specified version is used as the safe trimming point, or the base page corresponding to the found base page entry is used as the safe trimming point, and the base page / incremental page data file of any version in the logical page before the safe trimming point is deleted, similarly, several versions between the specified version and the safe trimming point are still retained in the data file. However, in this case, since it is for one LogicalPage, rather than taking the base page with the lowest version among the base pages with the largest version not greater than the specified version in all logical pages, the redundancy is relatively smaller, but there is still a certain amount of redundancy.

[0437] To address this problem, the present application also provides a method for tailoring the world state, including:

[0438] S370: receiving a specified cropped version;

[0439] S372: for each logical page, according to the specified version, search for a base page with a maximum version that is not greater than the specified version in the logical page as the first base page and search for an incremental page with a matching version;

[0440] S374: For the logical page, based on the base page and the incremental pages after the base page and before the matching version, play back in version order to obtain the memory page of the specified version;

[0441] S376: Convert the memory page into a second base page, and use the second base page as a safe clipping point, and delete any version of the base page / incremental page data file in the logical page before the safe clipping point.

[0442] In addition, the incremental pages after the safe trim point may be regenerated.

[0443] For example Fig.29 The left side of the figure shows the world state before clipping, and the right side of the figure shows the InternalPage after being processed by S372 and S374.

[0444] In InternalPage C1, the BasePage version is 100, and the versions of the two consecutive DeltaPages are 101 and 150 respectively. The versions contained in DeltaPage with version 101 include 101 to 149, and the versions contained in DeltaPage with version 150 include 150 to 199. Here, according to the specified version, the largest base page of the logical page that is not greater than the specified version is searched as the first base page, that is, the BasePage with version 100; the incremental page with the same version is searched, that is, the incremental page containing the specified version, here is the DeltaPage with version 101, which contains version 105. When the specified trimming version is 105, Fig.29 The InternalPage C1 on the right, which has been processed by S372 and S374, contains a BasePage with version 105. This BasePage with version 105 is obtained by replaying the InternalNodes with versions 101, 102, 103, 104 and 105 in the BasePage with version 100 and the DeltaPage with version 101 before clipping. In addition, the incremental page after the safe clipping point can also be regenerated, such as the DeltaPage with version 106 here, which contains InternalNodes with versions 106 to 149.

[0445] Similarly, in InternalPage C2, the BasePage version is 80, and the versions of the two consecutive DeltaPages are 81 and 150 respectively. The versions contained in DeltaPage with version 81 include 81 to 149, and the versions contained in DeltaPage with version 150 include 150 to 199. Here, according to the specified version, the largest base page of the logical page that is not greater than the specified version is searched as the first base page, that is, the BasePage with version 80; the incremental page with the matching version is searched, that is, the incremental page containing the specified version, here is the DeltaPage with version 81, which contains version 105. When the specified trimming version is 105, Fig.29 The InternalPage C2 on the right, after being processed by S372 and S374, contains a BasePage with version 105. This BasePage with version 105 is obtained by replaying the InternalNodes with versions 81 to 105 in the BasePage with version 80 and DeltaPage with version 81 before clipping. In addition, the incremental page after the safe clipping point can also be regenerated, such as the DeltaPage with version 106 here, which contains InternalNodes with versions 106 to 149.

[0446] Similarly, in InternalPage C3, the BasePage version is 90, and the versions of the two consecutive DeltaPages are 91 and 100 respectively. The versions contained in DeltaPage with version 91 include 91 to 99, and the versions contained in DeltaPage with version 100 include 100 to 159. Here, according to the specified version, the largest base page of the logical page that is not greater than the specified version is searched as the first base page, that is, the BasePage with version 90; the incremental page with the matching version is searched, that is, the incremental page containing the specified version, here is the DeltaPage with version 100, which contains version 105. When the specified trimming version is 105, Fig.29 The InternalPage C3 on the right, which has been processed by S372 and S374, contains a BasePage with version 105. The memory page of the specified version is obtained by replaying the base page and the incremental pages after the base page and before and after the matching version in version order. This BasePage with version 105 is obtained by replaying the InternalNodes with versions 91 to 99 in the BasePage with version 90 and the DeltaPage with version 91 before trimming, and the InternalNodes with versions 100 to 105 in the DeltaPage with version 100. In addition, the incremental pages after the safe trimming point can also be regenerated, such as the DeltaPage with version 106 here, which contains InternalNodes with versions 106 to 159.

[0447] Similarly, in LeafPage C4, the BasePage version is 50, and the versions of the two consecutive DeltaPages are 51 and 90 respectively. The versions contained in DeltaPage with version 51 include 51 to 89, and the versions contained in DeltaPage with version 90 include 90 to 119. Here, according to the specified version, the largest base page of the logical page that is not greater than the specified version is searched as the first base page, that is, the BasePage with version 50; the incremental page with the matching version is searched, that is, the incremental page containing the specified version, here is the DeltaPage with version 90, which contains version 105. When the specified trimming version is 105, Fig.29 The LeafPage C4 on the right, which has been processed by S372 and S374, contains a BasePage with version 105. The memory page of the specified version is obtained by playing back the base page and the incremental pages after the base page and before the matching version in version order. This BasePage with version 105 is obtained by playing back the LeafNodes with versions 51 to 89 (or also including InternalNodes) in the BasePage with version 50 and DeltaPage with version 51 before trimming, and the LeafNodes with versions 90 to 105 (or also including InternalNodes) in the DeltaPage with version 90. In addition, the incremental pages after the safe trimming point can also be regenerated, such as the DeltaPage with version 106 here, which contains LeafNodes with versions 106 to 119 (or also including InternalNodes).

[0448] Similarly, in LeafPage C5, the BasePage version is 50, and the versions of the two consecutive DeltaPages are 61 and 100 respectively. The versions contained in DeltaPage with version 61 include 61 to 99, and the versions contained in DeltaPage with version 100 include 100 to 119. Here, according to the specified version, the largest base page of the logical page that is not greater than the specified version is searched as the first base page, that is, the BasePage with version 60; the incremental page with the matching version is searched, that is, the incremental page containing the specified version, here is the DeltaPage with version 100, which contains version 105. When the specified trimming version is 105, Fig.29 The LeafPage C5 on the right, which has been processed by S372 and S374, contains a BasePage with version 105. The memory page of the specified version is obtained by playing back the base page and the incremental pages after the base page and before the matching version in version order. This BasePage with version 105 is obtained by playing back the LeafNodes with versions 61 to 99 (or also including InternalNodes) in the BasePage with version 60 and DeltaPage with version 61 before trimming, and the LeafNodes with versions 100 to 105 (or also including InternalNodes) in the DeltaPage with version 100. In addition, the incremental pages after the safe trimming point can also be regenerated, such as the DeltaPage with version 106 here, which contains LeafNodes with versions 106 to 119 (or also including InternalNodes).

[0449] Similarly, in LeafPage C6, the BasePage version is 60, and the versions of the two consecutive DeltaPages are 61 and 90 respectively. The versions contained in DeltaPage with version 61 include 61 to 89, and the versions contained in DeltaPage with version 90 include 90 to 109. Here, according to the specified version, the largest base page of the logical page that is not greater than the specified version is searched as the first base page, that is, the BasePage with version 60; the incremental page with the matching version is searched, that is, the incremental page containing the specified version, here is the DeltaPage with version 90, which contains version 105. When the specified trimming version is 105, Fig.29 The LeafPage C6 on the right, which has been processed by S372 and S374, contains a BasePage with version 105. The memory page of the specified version is obtained by playing back the base page and the incremental pages after the base page and before the matching version in version order. This BasePage with version 105 is obtained by playing back the LeafNodes with versions 61 to 89 (or also including InternalNodes) in the BasePage with version 60 and DeltaPage with version 61 before trimming, and the LeafNodes with versions 90 to 105 (or also including InternalNodes) in the DeltaPage with version 90. In addition, the incremental pages after the safe trimming point can also be regenerated, such as the DeltaPage with version 106 here, which contains LeafNodes with versions 106 to 109 (or also including InternalNodes).

[0450] Similarly, after the above method, the base page entries before the safe clipping point in the index file can also be deleted / updated. Fig.28 As shown in , for example, for the BasePage with version 100 in InternalPage C1 before pruning, since the corresponding data file no longer exists after pruning, but the BasePage data file with version 105 is regenerated as above, the BasePage with version 100 in InternalPage C1 originally corresponding to the index file can be deleted.

[0451] Similarly, after the above method, the incremental page entry of the safe clipping point in the index file can also be updated. Fig.28 As shown in , for example, for the DeltaPage with version 101 in InternalPage C1 before pruning, since the corresponding data file no longer exists after pruning, and the specified version 105 corresponds to the DeltaPage 101, which is also the DeltaPage data file with version 106 regenerated as above, the DeltaPage with version 101 in InternalPage C1 originally corresponding to the index file can be updated to DeltaPage 106.

[0452] Similarly, after the above method, the incremental page entries before the safe trim point in the index file can also be deleted. Fig.28 As shown in , for example, for the DeltaPage with version 61 in InternalPage C6 before pruning, since the corresponding data file no longer exists after pruning, the DeltaPage with version 61 in InternalPage C6 originally corresponding to the index file can be deleted.

[0453] In addition, similar to the above:

[0454] The version of the lower-level base page where the child node of the tree node is located can be set in the entry of the base page / incremental page of the index file, and the step of searching the first base page with the largest version not greater than the specified version in the logical page according to the specified version may include:

[0455] According to the version, search in the base page / incremental page entry of the index file; according to the version of the lower base page where the child node of the tree node record in the found base page / incremental page entry is located, search in the index file for the lower base page entry, and use the lower base page as the first base page.

[0456] The version of the lower-level base page where the child node of the tree node is located can be set in the data file of the base page / incremental page, and the method of searching the first base page with the largest version not greater than the specified version in the logical page according to the specified version may include:

[0457] According to the version, search in the base page / incremental page entry of the index file; after finding the base page / incremental page entry, read the version of the lower base page where the child node of the tree node stored is located in the found base page / incremental page data file, and search the lower base page entry in the index file based on the found version, and the lower base page is used as the first base page.

[0458] If the base page of the specified version is not found, the following operations can be recursively performed: the version is decremented, and the corresponding base page is searched in the index file according to the decremented version until a base page with a matching version is found; and the found base page is used as the first base page.

[0459] For any version of the incremental page, in which the data file of the previous predecessor version is set with the data file storage location, searching the logical page for the largest version base page that is not greater than the specified version according to the version may include:

[0460] If the incremental page is first found in the index file according to the version, the data file position of the base page / incremental page of its predecessor stored in the found incremental page is recursively read until the data file position of the base page is found; the found base page is used as the first base page.

[0461] For a version of a previous predecessor base page / delta page set in a delta page entry of an index, searching for a maximum version base page in a logical page that is not greater than the specified version according to the version may include:

[0462] Recursively read the version of the predecessor base page / incremental page stored in the found incremental page entry and search in the index file according to the found version until the base page is found; the found base page is used as the first base page.

[0463] The solution of this embodiment is to search for the first base page and the corresponding incremental page according to the specified version for each logical page, and replay the memory page of the specified version based on the base page and the incremental pages after the base page, the matching version and the previous version in version order. The version of the second base page obtained after the replay is the specified version, and use the second base page as a safe trimming point and delete any version of the base page / incremental page data file in the logical page before the safe trimming point, thereby achieving zero redundancy and the most accurate trimming.

[0464] The above solution is to prune the index file in the world state. In fact, it also involves pruning or updating the data file. The update here includes the process of selecting the qualified data file, removing the junk data in it, and then rewriting the data file. The goal of pruning is also to delete data files that are no longer valuable, thereby improving the storage space utilization of the disk. Here, pruning or updating data files is collectively referred to as garbage collection.

[0465] In the prior art, such as the method of garbage collection of data files in the database field, the index file is generally searched according to the key in the data file. If the key is not found in the index file, it means that the index has expired for the key, so the key and the corresponding data in the data file can be deleted. However, in the above-mentioned embodiment using BasePage and DeltaPage, this method is not suitable. This is because, unlike the database field, which stores the current latest status, the world status stored in the blockchain contains historical versions. For example, Fig. 27 As shown in the figure, for LeafPage MemoryPage B6, if there is an operation on LeafNode with version 26 in DeltaPage26, such as adding a key and a corresponding value to LeafNode with version 26, or modifying the value of an existing key, the corresponding value will be stored in the state data file. Specifically, the NodeID of LeafNode with version 26 and the location of the corresponding data stored in the data file can be recorded in DeltaPage26 of LeafPage MemoryPage B6 in the index, such as through the aforementioned location<FileID,StartPosition,Length> Indicates that the index can quickly locate the position of the state value corresponding to the leaf node in the data file. Among them, FileID can represent the ID of the data file on the disk, StartPosition can represent the starting position in the data file, and Length can represent the length. Fig.10 The location of the leaf node A11 is <2, 750, 210>. This <2, 750, 210> is the location index, which indicates that the status data is located in file 2 of the data file, starting at the 750KB and with a length of 210KB. Fig.10 As shown in the data file in the lower left corner, assuming that the size of each file is 1024MB, the logical address range of the index is 0KB to 1023KB. In this way, the status data of A11 actually occupies the range from 750KB to 959KB in the data file with FileID 2.

[0466] A data file may contain data corresponding to indexes in multiple BasePage / DeltaPage. Fig.30 As shown, in the state data file (DataFile) of the disk, it is assumed that there are multiple data files, such as data file 1, data file 2, data file 3, ..., data file 233, data file 234, data file 235, .... Take data file 1 as an example, which contains data of multiple corresponding indexes, for example, data of Location 1 corresponding to 31+NodeID in DeltaPage with version 31 of LeafPage C4 in the index, data of Location 7 corresponding to 49+NodeID in DeltaPage with version 31 of LeafPage C4 in the index, data of Location 7 corresponding to 50+NodeID in BasePage with version 50 of LeafPage C4 in the index, and data of Location 12 corresponding to 50+NodeID in BasePage with version 50 of LeafPage C4 in the index. Of course, data file 1 may also contain other data. It should be noted that only part of the data files containing LeafPage C4 are shown here, and the omitted data files may not contain data files of LeafPage C4, but contain state data files of other LeafPages.

[0467] In the state index file (IndexFile), for LeafPage C4, the data file locations (Location) corresponding to different versions + tree nodes can be included in different versions of BasePage / DeltaPage. Fig.30 As shown in the figure, the DeltaPage 31+DeltaPage+PageID of LeafPage C4 in the upper right corner can contain the corresponding relationship between version+tree node and Location. The BasePage and DeltaPage of other LeafPage C4 are similar. The difference is that for BasePage, it contains the corresponding relationship between version+tree node and Location of one version, while for DeltaPage, it can contain the corresponding relationship between several versions+tree node and Location.

[0468] The above application provides a solution for separating actual data and index, wherein the data file is a file storing actual content; the index file is an index storing the location of the data file. Figures 10 to 14 The index in is different from Figures 21 to 26 Here, Figures 10 to 14 The index in corresponds to the tree structure, which is specifically reflected in the above-mentioned logical page, memory page, base page and incremental page. The leaf node stores the data file where the actual state data is located and its position in the data file, as described in the above location. The Value in the state data is stored in the data file. In this way, different from Figure 8 The traditional way of storing the kv of leaf nodes together, Figures 10 to 14 The index and data are separated in the database, and the index files and data files are stored separately in different databases (or data tables).

[0469] It needs to be clarified again that Figures 10 to 14 The data file itself in the KV of the world state, Figures 21 to 26 The index in is a re-index of the index file. The following embodiments are directed to Figures 10 to 14 The index and world state in .

[0470] like Fig.30 As shown in the tree structure on the left, LogicalPage can be divided into InternalPage and LeafPage. An InternalPage contains at least two layers of InternalNode; a LeafPage contains at least one LeafNode, or also contains at least one layer of InternalNode. InternalPage and LeafPage can contain BasePage and DeltaPage.

[0471] Corresponds to Fig.30 For the simplified 31-version DeltaPage of LeafPage C4 on the left, the version + the correspondence between the tree node and Location contained in the IndexFile on the right ranges from 31 to 49, which is smaller than the version 50 contained in the subsequent BasePage. For the 51-version DeltaPage of LeafPage C4, the version + the correspondence between the tree node and Location contained in it ranges from 51 to 89. For the 90-version DeltaPage of LeafPage C4, the version + the correspondence between the tree node and Location contained in it ranges from 90 to 119. It can be seen that the versions in each consecutive BasePage and DeltaPage are also consecutive and do not overlap.

[0472] As mentioned above, DeltaPage contains the increments generated by the generation / modification behavior of the tree nodes, and BasePage represents the contents of all the corresponding tree nodes in the memory page of the current version of the logical page, and also contains the increments generated by the generation / modification behavior of the tree nodes. In other words, in the index file of BasePage, some corresponding relationships are new corresponding relationships generated by the generation / modification behavior of the current version, that is, they do not exist in the previous memory pages, and some corresponding relationships are corresponding relationships generated by the generation / modification behavior of the previous version, that is, they exist in the previous memory pages. Fig.30 The BasePage of version 50 in the IndexFile on the right contains two corresponding relationships:

[0473] 50+NodeID→Location12 (1)

[0474] 50+NodeID→Location7 (2)

[0475] Among them, correspondence 1 is a new correspondence generated by the generation / modification behavior of the current version, that is, it does not exist in the previous memory page, so the Location12 of the index points to the DF position corresponding to L12 in the data file 1 in the DataFile alone (L12 represents Location12, and DF represents DataFile). Correspondence 2 is a correspondence generated by the generation / modification behavior in the previous DetaPage, that is, it no longer exists in the previous DeltaPage. Therefore, the DeltaPage of LeafPage C4 in version 31 also contains correspondence 2, and the BasePage of LeafPage C4 in version 50 also contains correspondence 2. Then, the correspondence 2 contained in the BasePage of LeafPage C4 in version 50 is not the correspondence generated by the generation / modification behavior in the memory page of the current version.

[0476] Here, the file usage can be recorded for the corresponding relationship generated by the generation / modification behavior in the current version of the memory page in BasePage. For example, the BasePage of the LeafPage C4 of version 50 mentioned above also includes the corresponding relationship 2, and a FileUsage can be recorded in IndexFile, in which the corresponding relationship 1 is recorded, but the corresponding relationship 2 is not recorded.

[0477] It should be noted that the NodeID in the above correspondence can be the state key for a leaf node. In other words, the complete state key can be recorded in the leaf node. For example Fig.14 and 19 The complete state key can be stored in the leaf node of the tree structure used as an index. In the data file corresponding to the Location, in addition to the state Value content, the complete state key can also be recorded. In addition, for other corresponding relationships included in the DeltaPage of the 31 version of LeafPage C4, and other corresponding relationships included in the BasePage of the 50 version of LeafPage C4, etc., the Location in any corresponding relationship not mentioned can also point to a location in other data files in the DataFile.

[0478] After the aforementioned trimming of the index file, it means that some data in the data file no longer has a corresponding index entry. These data that have lost their corresponding indexes become garbage data, which is generally difficult to be effectively used through indexing. When the proportion of garbage data or the amount of data in a data file reaches a certain threshold, the data file can be garbage collected. Therefore, the premise of garbage collection includes calculating the garbage ratio of the data file.

[0479] The present application proposes an embodiment of a method for calculating the garbage ratio of a data file, comprising: traversing the file usage of the incremental pages and base pages of the leaf pages in the index file, and according to the amount of valid data in the status key statistical data file, setting the difference between the proportion of the valid data amount in the data file and 1 as the garbage ratio of the data file.

[0480] Traversing this leaf page in the index file specifically includes traversing the base page and the incremental page in the leaf page. For the base page, the effective data volume in a data file can be counted by its file usage, which is set as the first effective data volume here. For the incremental page, the effective data volume of a data file can be counted by each entry in the index file, which is set as the second effective data volume here. For the same data file, the sum of the first effective data volume and the second effective data volume is the effective data volume of the data file.

[0481] For example Fig.30 As shown in , the leaf page LogicalPage C4 in the index file is traversed, specifically, the index file 31+DeltaPage+PageID, the index file 50+BasePage+PageID, the index file 51+DeltaPage+PageID, and the index file 90+DeltaPage+PageID of LogicalPage C4 can be traversed. It is assumed that Location 1 and Location 7 in the index file 31+DeltaPage+PageID of LogicalPage C4 correspond to data file 1; and it is assumed that Location 12 in the FileUsage of the index file 50+BasePage+PageID of LogicalPage C4 corresponds to data file 1, and Location 7 which is not in FileUsage is not considered, because this Location 7 is not the file usage generated by the generation / modification behavior in the memory page 50+BasePage+PageID, but directly references Location 7 in the index file of the incremental page 31+DeltaPage+PageID, so as to avoid repeated statistics.

[0482] Here, statistics are collected based on data files. For example, to collect statistics on DataFile1, Location1, Location7, and Location12 are collected. Fig.30 As shown, assuming that Location3, Location11, and Location25 all correspond to DataFile3, Location3, Location11, and Location25 are not counted in the effective data volume of DataFile1.

[0483] In this way, after the traversal is completed, the proportion of valid data in each DataFile will also be calculated, which is a fraction less than 1. Therefore, subtracting this proportion from 1 can get the proportion of garbage data in this DataFile, that is, the garbage ratio.

[0484] Based on the above-mentioned data file garbage ratio calculation method, the present application also provides a method embodiment for garbage collection of data files, including selecting valid data from data files whose garbage ratio reaches a threshold value and generating new data files.

[0485] Specifically, for a data file whose garbage ratio reaches a threshold, valid data in the data file may be selected, and a new data file may be generated based on the valid data. In addition, for several data files whose garbage ratios reach a threshold, valid data in the data files may be selected, and a new data file may be generated based on the valid data.

[0486] As mentioned above, the leaf node in the leaf page of the index file includes a state key. In addition, the data file may also include a state key. Based on this, another embodiment of the method for garbage collection of data files can also be specifically as follows:

[0487] S380: traverse the file usage of the base pages and all the keys in the incremental pages in all index files, obtain the corresponding data file location index for each traversed key, and obtain the reference information of the DataFile from the Location.

[0488] Get the reference information of DataFile from Location, including the FileID information of DataFile.

[0489] S382: Determine whether the FileID of the DataFile is consistent with the FileID of the data file whose garbage ratio reaches the threshold value. If they are consistent, add the corresponding key to the rewrite set.

[0490] For example, the data file whose garbage ratio reaches the threshold is DataFile_target, and its FileID is 3. In an index file traversed, that is, the file usage or incremental page of the base page of a leaf page traversed, contains the aforementioned version + NodeID → Location, and as mentioned above, the NodeID in this correspondence can be a state key for a leaf node. In other words, the complete state key can be recorded in the leaf node. In this way, the file usage or incremental page of the base page of the leaf page contains the mapping relationship from the state key to the Location. Here is an example:

[0491] 50+status key_W→Location12(FileID3,,) (3)

[0492] 51+status key_W→Location23(FileID3,,) (4)

[0493] 95+status key_W→Location37(FileID5,,) (5) ...

[0495] but:

[0496] In the above correspondence 3, the FileID is consistent with the data file whose garbage ratio reaches the threshold value, DataFile_target, and the FileID is 3;

[0497] In the above correspondence 4, the FileID is consistent with the data file whose garbage ratio reaches the threshold value, DataFile_target, and the FileID is 3;

[0498] In the above correspondence 3, FileID is 5, which is inconsistent with the data file whose garbage ratio reaches the threshold value, DataFile_target, which is 3.

[0499] In this way, if the data is consistent, it means that the current valid index contains a reference to the data file, so the data corresponding to the Location is valid data, so the data can be written into the rewrite set, and a new data file is generated based on the rewrite set later. If the data is inconsistent, it may be a different version, which does not need to be considered when processing DataFile_target.

[0500] S384: Rewrite into a new data file based on all keys and corresponding status data in the rewrite set.

[0501] After the above processes of S380 and S382 are completed, the rewrite set contains all valid key sets in the data file whose garbage ratio reaches the threshold, and all keys and corresponding state data in the rewrite set are rewritten into a new data file. In this way, garbage collection of DataFile is completed.

[0502] In addition, the data files whose garbage ratio reaches a threshold value may be deleted, or marked as deleted and then deleted uniformly later.

[0503] After S384, it can also include:

[0504] S386: Update the reference to the data file whose garbage ratio reaches the threshold in the index file to a reference to the new data file.

[0505] Update the Location reference to DataFile_target in the index file to a Location reference to the new data file, so that subsequent searches based on the index can find the correct new data file.

[0506] Different from Fig.30 In the embodiment, the present application also provides another method embodiment for calculating the garbage ratio of a data file. In the method embodiment, the incremental page contains the increments generated by the generation / modification behavior of the tree node, and the base page represents the contents of all the tree nodes corresponding to the memory page of the current version of the logical page, but if Fig.31 As mentioned above, the increments generated by the generation / modification behavior of tree nodes are not included.

[0507] Corresponds to Fig.31 The simplified 31-version DeltaPage of LeafPage C4 on the left, the version + the correspondence between the tree node and Location contained in the IndexFile on the right, the version range is 31 to 50, that is, it includes the version 50 in the subsequent BasePage. In this way, all the increments generated by the generation / modification behavior of the tree node can be included in the incremental page, so that the content of all the corresponding tree nodes in the memory page representing the current version of the logical page in the base page does not include the increments generated by the generation / modification behavior of the tree node. Similarly, for the DeltaPage of LeafPage C6, the version + the correspondence between the tree node and Location contained in the IndexFile on the right, the version range is 90 to 109, that is, it includes the version 109 in the subsequent BasePage.

[0508] Specifically, Fig.31 In , a data file may contain data corresponding to indexes in multiple BasePage / DeltaPage. Fig.31 As shown, in the state data file (DataFile) of the disk, it is assumed that there are multiple data files, such as data file 1, data file 2, data file 3, ..., data file 233, data file 234, data file 235, .... Take data file 1 as an example, which contains data of multiple corresponding indexes, for example, data of Location 1 corresponding to 31+NodeID in DeltaPage with version 31 of LeafPage C4 in the index, data of Location 7 corresponding to 49+NodeID in DeltaPage with version 31 of LeafPage C4 in the index, data of Location 12 corresponding to 50+NodeID in DeltaPage with version 50 of LeafPage C4 in the index, data of Location 7 corresponding to 50+NodeID in BasePage with version 50 of LeafPage C4 in the index, and data of Location 12 corresponding to 50+NodeID in BasePage with version 50 of LeafPage C4 in the index.

[0509] As mentioned above, DeltaPage contains the increments generated by the generation / modification behavior of the tree nodes, while BasePage represents the contents of all the corresponding tree nodes in the memory page of the current version of the logical page, but does not contain the increments generated by the generation / modification behavior of the tree nodes. In other words, in the index file of BasePage, all the corresponding relationships are new corresponding relationships generated by the generation / modification behavior in the previous version of DeltaPage. Fig.31 The BasePage of version 50 in the IndexFile on the right contains two corresponding relationships:

[0510] 50+NodeID→Location12 (6)

[0511] 50+NodeID→Location7 (7)

[0512] Among them, correspondence 6 is a new correspondence generated by the generation / modification behavior in the previous version of DeltaPage, that is, it already exists in the previous memory page, so the Location12 of this index also points to the DF position corresponding to L12 in data file 1 in DataFile (L12 represents Location12, DF represents DataFile). Correspondence 7 is also a correspondence generated by the generation / modification behavior in the previous DetaPage, that is, it already exists in the previous DeltaPage.

[0513] The method includes: traversing the incremental pages of the leaf pages in the index file, calculating the amount of valid data in the data file, and setting the difference between the proportion of the valid data in the data file and 1 as the garbage ratio of the data file; the leaf node contained in the leaf page includes the position index of the status data in the data file in the persistent storage medium.

[0514] Since DeltaPage contains the increments generated by generation / modification of tree nodes, and BasePage represents the contents of all tree nodes corresponding to the memory page of the current version of the logical page, but does not contain the increments generated by generation / modification of tree nodes, it is possible to traverse only the incremental pages of leaf pages in the index file, thereby counting the amount of valid data in the data file without traversing the base pages of leaf pages in the index file.

[0515] Similarly, the NodeID in the above correspondence can be a state key for a leaf node. After the aforementioned trimming of the index file, it means that some data in the data file no longer has a corresponding index entry. These data that have lost their corresponding indexes become garbage data, which are generally difficult to be effectively used through indexing. When the proportion of garbage data or the amount of data in a data file reaches a certain threshold, the data file can be garbage collected. Therefore, the premise of garbage collection includes calculating the garbage ratio of the data file.

[0516] The present application proposes another method embodiment for calculating the garbage ratio of a data file, including: traversing the incremental pages of the leaf pages in the index file, and according to the amount of valid data in the status key statistical data file, setting the difference between the proportion of the valid data amount in the data file and 1 as the garbage ratio of the data file.

[0517] Traversing the leaf page in the index file specifically includes traversing the incremental page in the leaf page, and the base page may not be traversed. For each incremental page traversed, the effective data volume of a data file can be counted through each entry of the index file, thereby obtaining the effective data volume.

[0518] For example Fig.31 As shown in , the leaf page LogicalPage C4 in the index file is traversed, specifically, the index file 31+DeltaPage+PageID, the index file 51+DeltaPage+PageID, and the index file 90+DeltaPage+PageID of LogicalPage C4 as shown in the figure can be traversed. It is assumed that Location 1, Location 7, and Location 11 in the index file 31+DeltaPage+PageID of LogicalPage C4 correspond to data file 1. Here, statistics are based on data files. For example, in order to count DataFile1, Location 1, Location 7, and Location 12 are counted here.

[0519] In this way, after the traversal is completed, the proportion of valid data in each DataFile will also be calculated, which is a fraction less than 1. Therefore, subtracting this proportion from 1 can get the proportion of garbage data in this DataFile, that is, the garbage ratio.

[0520] Based on the above-mentioned data file garbage ratio calculation method, the present application also provides a method embodiment for garbage collection of data files, including selecting valid data from data files whose garbage ratio reaches a threshold value and generating new data files.

[0521] Specifically, for a data file whose garbage ratio reaches a threshold, valid data in the data file may be selected, and a new data file may be generated based on the valid data. In addition, for several data files whose garbage ratios reach a threshold, valid data in the data files may be selected, and a new data file may be generated based on the valid data.

[0522] As mentioned above, the leaf node in the leaf page of the index file includes a state key. In addition, the data file may also include a state key. Based on this, another embodiment of the method for garbage collection of data files can also be specifically as follows:

[0523] S390: traverse all keys in the incremental pages of all index files, obtain the corresponding data file location index for each traversed key, and obtain reference information to the DataFile from the Location.

[0524] Get the reference information of DataFile from Location, including the FileID information of DataFile.

[0525] S392: Determine whether the FileID of the DataFile is consistent with the FileID of the data file whose garbage ratio reaches the threshold value. If they are consistent, add the corresponding key to the rewrite set.

[0526] For example, the data file whose garbage ratio reaches the threshold is DataFile_target, and its FileID is 3. In an index file traversed, that is, the incremental page of a leaf page traversed, it contains the aforementioned version + NodeID → Location, and as mentioned above, the NodeID in this correspondence can be the state key for the leaf node. In other words, the complete state key can be recorded in the leaf node. In this way, the incremental page of the leaf page contains the mapping relationship from the state key to the Location. Here is an example:

[0527] 50+status key_W→Location12(FileID3,,) (3)

[0528] 51+status key_W→Location23(FileID3,,) (4)

[0529] 95+status key_W→Location37(FileID5,,) (5) ...

[0531] but:

[0532] In the above correspondence 3, the FileID is consistent with the data file whose garbage ratio reaches the threshold value, DataFile_target, and the FileID is 3;

[0533] In the above correspondence 4, the FileID is consistent with the data file whose garbage ratio reaches the threshold value, DataFile_target, and the FileID is 3;

[0534] In the above correspondence 3, FileID is 5, which is inconsistent with the data file whose garbage ratio reaches the threshold value, DataFile_target, which is 3.

[0535] In this way, if the data is consistent, it means that the current valid index contains a reference to the data file, so the data corresponding to the Location is valid data, so the data can be written into the rewrite set, and a new data file is generated based on the rewrite set later. If the data is inconsistent, it may be a different version, which does not need to be considered when processing DataFile_target.

[0536] S394: Rewrite into a new data file based on all keys and corresponding status data in the rewrite set.

[0537] After the above processes of S390 and S392 are completed, the rewrite set contains all valid key sets in the data file whose garbage ratio reaches the threshold, and all keys and corresponding state data in the rewrite set are rewritten into a new data file. In this way, garbage collection is completed.

[0538] In addition, the data files whose garbage ratio reaches a threshold value may be deleted, or marked as deleted and then deleted uniformly later.

[0539] After S394, it can also include:

[0540] S396: Update the reference to the data file whose garbage ratio reaches the threshold in the index file to a reference to the new data file.

[0541] Update the Location reference to DataFile_target in the index file to a Location reference to the new data file, so that subsequent searches based on the index can find the correct new data file.

[0542] As mentioned in the aforementioned S314, S326, S336, S344, S356, S366 and S376, after determining the safe trimming point, the base page / incremental page data file of any version in the logical page before the safe trimming point can be deleted, that is, the trimming of the index file in the world state is completed. In addition to the trimming of the index file in the world state, the aforementioned garbage collection of the data file is also involved.

[0543] In fact, the data governance in the underlying database generally does not perform the above-mentioned deletion work in real time. Instead, the governance task will be hung in the background. When the traffic such as data query / modification in the foreground drops below a certain threshold, the background governance task will be started to complete the deletion and other trimming work.

[0544] More broadly and accurately, in modern database management systems, data governance is usually handled asynchronously. Instead of executing resource-intensive operations such as deletions immediately, the system puts these tasks into a background queue. The database management system continuously monitors system load and resource utilization, and triggers background governance tasks only when it detects that the foreground transaction processing load has dropped below a preset threshold. This approach is called "background cleanup" or "deferred maintenance". It effectively balances system performance and data consistency requirements to ensure that critical business operations are not affected. When implemented, it may involve multiple aspects such as garbage collection, index rebuilding, and statistics updates. Advanced database systems may also use intelligent scheduling algorithms to dynamically adjust the execution time and priority of governance tasks based on historical load patterns and current system status to maximize the overall efficiency of the system.

[0545] As mentioned above, the present application provides an implementation scheme for separating actual data from indexes. The tree node as a whole can be used as an index of the state value. The index file and the data file can be stored separately in different databases (or data tables). This involves pruning the index file and garbage collection of the data file. The aforementioned method embodiment for garbage collection aggregates valid data in data files whose garbage ratio reaches a threshold value into a newly generated data file, and deletes the data files whose garbage ratio reaches the threshold value. Both S386 and S396 mention that the reference to the data file whose garbage ratio reaches the threshold value in the index file is updated to a reference to the new data file, which involves the problem of index reconstruction.

[0546] Theoretically, the solution to rebuild the index is to replace all the old invalid Locations pointed to in the index file with new Locations pointing to the new data files generated after garbage collection. However, as mentioned above, the world state stored in the blockchain contains versions, such as the organization method of BasePage and DeltaPage proposed in the above application. Then, if the theoretical method is followed, it may involve replacing the old Locations in several BasePages and DeltaPages in the index file with new Locations, and this modification will cause a more serious write amplification problem when the amount is large.

[0547] Based on this, the present application provides an embodiment of a method for rebuilding an index, including:

[0548] S400: In the garbage collection process, invalid position mapping data in the index file is counted to generate remapping data including status key to new data file location.

[0549] The invalid position mapping data in the statistical index file can be counted in a separate process after the garbage collection process. In addition, the invalid position mapping data in the index file can also be counted during the garbage collection process to generate remapping data including the state key to the new data file position.

[0550] The aforementioned garbage collection process includes selecting valid data from the data files whose garbage ratios reach a threshold value and generating a new data file. That is, the valid data selected from the data files whose garbage ratios reach a threshold value will be centrally set in the new data file, and the data files whose garbage ratios reach a threshold value will be deleted subsequently, resulting in a change in the Location of the valid data after garbage collection, which is different from the Location before garbage collection. Here, selecting valid data from the data files whose garbage ratios reach a threshold value and generating a new data file may include the aforementioned processes of S380 to S382 or S390 to S392.

[0551] In S380-S382 or SS390-S392, the index file is traversed and the reference information of the DataFile in the Location is judged to be consistent with the FileID of the data file whose garbage ratio reaches the threshold. If they are consistent, the corresponding key is added to the rewrite set, which also means that the Location of the valid data after the garbage collection mentioned above has changed and is different from the Location before the garbage collection.

[0552] Therefore, the location mapping data in the index file can be recorded in the above garbage collection process, that is, the mapping relationship in the consistent index file can be recorded during the traversal of the index file and the judgment of whether the reference information of the DataFile in the Location is consistent with the FileID of the data file whose garbage ratio reaches the threshold. In this way, there is no need to perform the statistical process again separately, thereby reducing the consumption of computing resources. Statistics in a separate process is, for example, after garbage collection, which is specifically similar to counting the invalid location mapping data in the index file during the garbage collection process, and will not be repeated.

[0553] Still taking the example of counting invalid location mapping data in index files during garbage collection, for example, Fig.32 In the index file 31+DeltaPage+PageID of LeafPage C4 shown in the figure, the records include the following mapping relationships:

[0554] 49+NodeID→Location7

[0555] For example, Fig.32 In the index file 50+BasePage+PageID of LeafPage C4 shown in the figure, the records include the following mapping relationships:

[0556] 50+NodeID→Location7

[0557] The above two mapping relationships are only used as examples to illustrate the implementation process of the embodiment of the method of the present application. In fact, the index file may also include other mapping relationships, and the reference information of DataFile in Location in these mapping relationships is also consistent with the FileID of the data file whose garbage ratio reaches the threshold value. For example, the mapping relationship recorded in the index file 31+DeltaPage+PageID of LeafPage C4 also includes: 31+NodeID→Location1 and 50+NodeID→Location12, and the mapping relationship recorded in the index file 50+BasePage+PageID of LeafPage C4 also includes: 50+NodeID→Location12.

[0558] In the above mapping relationships, since DataFile1 pointed to by Location is determined as a data file whose garbage ratio reaches the threshold during the garbage collection process, these Locations need to be remapped to the positions in the new data file. Fig.32 As shown in , DataFile1, which is determined as a data file whose garbage ratio reaches the threshold, is displayed as a background with diagonal lines in the figure because it will be deleted later.

[0559] Still Fig.32 Taking the mapping relationship 49+NodeID→Location7 recorded in the index file 31+DeltaPage+PageID of LeafPage C4 and the mapping relationship 50+NodeID→Location7 recorded in the index file 50+BasePage+PageID of LeafPage C4 as examples, remapping data can be generated. These remapping data include the mapping relationship from the state key to the location of the new data file, such as:

[0560] NodeID→Location55

[0561] The NodeID here is the NodeID in 49+NodeID→Location7 and 50+BasePage+PageID mentioned above, and as mentioned above, as a LeafNode in LeafPage, this NodeID is the state key. And Location55 refers to the newly generated data file and its location, such as Fig.32 The position in the new data file 350 is shown in , for example (350, StartPosition, Length), where StartPosition and Length have specific starting position values ​​and length values ​​in the new data file with FileID 350.

[0562] like Fig.32 As shown in , for the source mapping data underlined by the blue arrow before garbage collection and the DataFile1 pointed to, since the garbage ratio of DataFile1 exceeds the preset and will be garbage collected later, the mapping data at the source underlined by the blue arrow is invalid data. Fig.32 As shown in , the source of the green dotted arrow is the corresponding relationship between the state key after processing in S400 and the position of the new data file, that is, the remapped data, which points to a specific position in the new data file 350.

[0563] In this way, the content in the new data file can be traversed, so as to obtain the corresponding relationship between the state key therein and the position of the new data file where the key is located.

[0564] In addition, you can Fig.33 As shown in Fig.33 and Fig.32 The main difference is that the version of the DeltaPage is the maximal version of the tree nodes contained in it.

[0565] S420: Generate a rewrite page containing the remapped data, and install the rewrite page in the index file;

[0566] As mentioned above, the generated remapped data may be aggregated into a rewrite page of a certain size, and then the rewrite page may be installed in the index file, such as Fig.32 RewritePage as shown in .

[0567] In particular, it is possible to aggregate according to NodeID, i.e., the state key, for example, aggregate the remapped data of the same state key into the same RewritePage. In addition, there may not be a lot of remapped data of the same state key. In order to efficiently utilize the capacity of RewritePage, the NodeID, i.e., the state key, can be set to the RewritePage in the form of hash bucketing. Specifically, for example, the NodeID can be hashed by sha256, and the hash values ​​obtained can be divided into the same bucket according to the first several bits that are the same, so that the remapped data in the bucket are aggregated into the same RewritePage. Moreover, based on the discrete characteristics of hash calculation, the number of buckets is generally average.

[0568] In addition, the generation of remapping data including the state key to the new data file location can generate remapping data including the version and the state key to the new data file location. In this way, the remapping data written into the RewritePage is also in the form of version+NodeID→new data file location. Since there may be a situation where multiple different versions of NodeID are remapped to the same new data file location, the version here can select the maximum version of the corresponding original NodeID, thereby indicating in the RewritePage that the version in the remapping data is the maximum version of the remapped original NodeID. For example, Fig.32 For example, the mapping relationship 49+NodeID→Location7 recorded in the index file 31+DeltaPage+PageID of LeafPage C4 and the mapping relationship 50+NodeID→Location7 recorded in the index file 50+BasePage+PageID of LeafPage C4, and 50+NodeID→Location5 in RewritePage, can represent the maximum valid version of the remapped data in RewritePage for the original BasePage / DeltaPage. Of course, you can also choose the minimum version of the corresponding original NodeID, or the version with the minimum NodeID version in the BasePage / DeltaPage.

[0569] In addition, considering that the page identifiers of BasePage and DeltaPage included in the index file include versions, here, the version of RewritePage can also be set, so as to maintain the order of the index file version. For example, the version of RewritePage can be set to the maximum version or minimum version corresponding to the NodeID contained. In this way, RewritePage is on the same level as BasePage / DeltaPage, and the same persistence and query logic can be reused without introducing a new persistence format and query logic.

[0570] S440: Setting the failed data file ID and the corresponding rewritten page ID in the metadata of the counted index file containing the failed position mapping data.

[0571] After the above S400 and S420, a mapping relationship between RewritePage and the new data file is established in the index file. In addition, a mapping relationship between the original index file and RewritePage needs to be established.

[0572] Here, the metadata of the original index file (including the original BasePage and DeltaPage) can be used. Metadata is called metadata, which is generally used to describe some attributes and characteristics of the data content, so metadata is also called "data about data". For example, it describes various attributes and characteristics of the data content. Metadata is, for example, manifest files, and issues in the form of Schema, YAML, XML, JSON, etc. Generally speaking, the metadata of a file will be much smaller than the volume of its data content. Moreover, although metadata and data content together constitute a complete file, it is usually possible to load / modify only the metadata without accessing the actual data content, thereby reducing memory usage, which can also greatly improve processing speed and efficiency, especially when the data file as a whole is very large.

[0573] The most common way to set metadata is to set it at the beginning of the file, followed by the actual data content. The advantage of this setting is that the metadata can be quickly accessed and the subsequent data can be quickly read. In addition, it is also possible to set it at the end of the file, or set it separately from the file, or set it mixed, or even embedded in the entire file. Here, we take the example of setting metadata in a file to explain.

[0574] like Fig.32 As shown in, in the MetaData of the index file 31+DeltaPage+PageID of LeafPage C4, the invalid FileID in the Location of the data content can be set, and the rewrite page ID corresponding to the invalid FileID can be set. For example, for the mapping relationship 49+NodeID→Location7(1,StartPosition,Length), the invalid FileID-1 and the corresponding rewrite page ID can be recorded in MetaData. The rewrite page can have an ID. The ID of this rewrite page, for aggregating remapped data of the same state key into the same RewritePage, can include the state key it contains; for aggregation by hash bucketing as described above, it can be that the rewrite page ID includes a preset number of hash value and position bit values, such as the first several bits of the hash value, the last several bits, and the middle specific position and length of several bits; etc. The ID of the rewrite page can also include a version, as mentioned above, which will not be repeated.

[0575] Similarly, for the mapping relationship 50+NodeID→Location7 recorded in the index file 50+BasePage+PageID of LeafPage C4, for the mapping relationship 50+NodeID→Location7(1,StartPosition,Length), the invalid FileID—1 and the corresponding rewritten page ID can be recorded in MetaData.

[0576] After generating a new data file according to the above garbage collection method embodiment and rebuilding the index according to the above index rebuilding method, the data file whose garbage ratio reaches the threshold value can be deleted, such as deleting Fig.32 DataFile1 with the slash as background.

[0577] In addition, you can Fig.34 shown. Fig.34 and Fig.33 Another major difference is that the metadata is specifically the metadata manifest of the database, rather than the metadata file of BasePage or DeltaPage. Specifically, the metadata manifest of the database can correspond to the database, that is, it includes all the files contained in the database. This manifest is generally a copy for a database. In this design, a higher-level metadata concept is introduced-the Database Metadata Manifest. This manifest is different from the metadata file of a single BasePage or DeltaPage, but a more macro data structure. Its scope covers the entire database and contains key information of all files in the database. Specifically, this manifest may include but is not limited to the following:

[0578] Basic information of the database (such as version number, creation time, etc.)

[0579] ●Index of all BasePages and DeltaPages

[0580] ●Logical relationship between files

[0581] ● Metadata summary of each file

[0582] Global statistics

[0583] It is worth noting that for each database, only one such manifest is usually maintained. This centralized metadata management approach has several significant advantages:

[0584] ● Improved metadata consistency and integrity

[0585] ●Simplified database-level operations (such as backup and recovery)

[0586] Optimized cross-file query performance

[0587] Based on the above index reconstruction method, the present application also provides a method for searching status data, including:

[0588] S410: receiving a status key of the status data to be searched;

[0589] S412: searching for mapping data in the base page / incremental page of the index file according to the status key to obtain the corresponding data file location;

[0590] S414: searching in the metadata of the base page / delta page based on the data file ID in the data file location, and if the rewrite page ID corresponding to the data file ID is found, searching for the corresponding remapped data based on the state key in the rewrite page pointed to by the rewrite page ID;

[0591] S416: Searching for status data in the new data file based on the corresponding re-mapped data found.

[0592] Through the above scheme, for the situation that may involve the same status key in several BasePages and DeltaPages in the index file, the redirection of the searched status key is achieved by modifying the MetaData in the data files of the BasePage and DeltaPage and a rewrite page, avoiding the modification of the data content in all the original BasePage and DeltaPage data files and maintaining a high search efficiency.

[0593] In addition, the state data to be searched in S410 may include a version in addition to the state key. In the case where the ID of the RewritePage includes a version, when the version of the state data to be searched + the state key are received, the BasePage / DeltaPage / RewritePage to be searched can be quickly located. Fig.32 As shown, for the case where the ID of RewritePage contains the version, since the version indicates the minimum version used by the remapping relationship in the RewritePage, it is convenient to perform a binary search when the BasePage, DeltaPage, and RewritePage are arranged in order by version in the index file, thereby reducing the time complexity of the search compared to the traversal search. For the case where the remapping relationship in RewritePage contains the version+NodeID→new data file location, similarly, it is also convenient to perform a binary search.

[0594] In addition, you can Fig.33 As shown in , in the case of using version + state key, when the version + state key of the state data to be searched is received, for the case where the ID of RewritePage contains the version, since the version indicates the maximum version used by the remapping relationship in the RewritePage, it is convenient to perform a binary search in the case where BasePage, DeltaPage, and RewritePage are arranged in order by version in the index file, thereby reducing the time complexity of the search compared to the traversal search. For the case where the remapping relationship in RewritePage contains version + NodeID → new data file location, similarly, it is also convenient to perform a binary search.

[0595] The above S400 to S440 provide an embodiment of a method for re-indexing. It has been mentioned that the NodeID, i.e., the state key, is set to the RewritePage by hash bucketing. In fact, reasonable use of bucketing can also save metadata and realize direct jump from the invalid data file ID to the corresponding rewrite page. The principle is as follows Fig.35 as shown, and specifically include:

[0596] S450: counting invalid position mapping data in the index file, and generating remapping data including the state key to the new data file position;

[0597] S460: Generate a rewrite page containing the remapped data, and install the rewrite page in the index file; wherein the remapped data is aggregated into the same rewrite page in a hash bucketing manner, and the ID of the rewrite page includes a preset number of bits of the hash value and a position.

[0598] For example, the state key is bucketed according to the first several bits after the hash is taken, so that the obtained hash values ​​with the same first several bits are bucketed in the same bucket, so that the remapped data in the bucket are aggregated in the same RewritePage.

[0599] The other contents are similar to the above-mentioned embodiments and will not be described in detail.

[0600] Correspondingly, based on S450 and S460, this application also provides a method for searching status data, which can also be referred to Fig.35 ,include:

[0601] S420: receiving the status key of the status data to be searched;

[0602] S422: searching for mapping data in the base page / incremental page of the index file according to the state key to obtain the corresponding data file location;

[0603] S424: Calculate a hash value for the state key and take a bit value of a preset number and position, search for the ID of the rewrite page according to the bit value, and if found, search for the corresponding remapped data based on the state key in the corresponding rewrite page;

[0604] S426: Searching for status data in the new data file based on the corresponding re-mapped data found.

[0605] For example, the remapping relationship aggregated in RewritePage is the result of bucketing the state key according to the first several bits after taking the hash. In the remapping relationship aggregated in the same RewritePage, each key is the same according to the first several bits after taking the hash, and the ID of RewritePage is set to the first several identical bits. Therefore, for the state key to be found, the same hash calculation can be performed, and several bits of the same position and length can be taken, and then the ID of RewritePage can be found based on the values ​​of these bits. If found, it can be further searched in the corresponding RewritePage, such as searching the remapping relationship in RewritePage based on the state key to be found. If the corresponding remapping relationship is found, the corresponding state data can be further found in the new data file location pointed to by the found remapping relationship.

[0606] As a distributed system, the core feature of blockchain is that each node continuously processes transactions and generates blocks to maintain the decentralization and transparency of the entire network. Under normal circumstances, nodes will verify and confirm transactions with each other and package them into blocks. However, even if no transactions occur for a period of time, the blockchain network will still generate empty blocks. These empty blocks can not only prevent the stagnation of the chain, but also effectively maintain the continuity of the blockchain ledger, ensuring the activity and availability of the network. The process of generating empty blocks is often regarded as the heartbeat mechanism of the blockchain, which means that the network is still running effectively. This mechanism is designed to ensure that the network can remain synchronized even when transaction activities are not frequent, reducing the risk of disconnection of each node. If some blockchain nodes fail to participate in the block generation process due to downtime, network failure or other reasons, the overall operation of the blockchain system can still continue, which reflects its high fault tolerance. However, how many nodes can be tolerated to be down and not participate in the operation of the blockchain depends on the consensus mechanism adopted and its settings. For example, in some PoW (proof of work)-based blockchains, the security of the entire network is closely related to the distribution of computing power, while in systems based on DPoS (delegated proof of stake) or PBFT (practical Byzantine fault tolerance), the system may be more tolerant of node failures. These consensus mechanisms ensure that the network continues to operate effectively even in the event of multiple node failures through well-designed algorithms. Therefore, choosing the right consensus mechanism is an important factor in ensuring that the blockchain system can still operate when facing problems such as downtime or delays.

[0607] Of course, in a blockchain system with considerable business traffic, empty blocks are a minority of cases, and more cases are normal business generating blocks. Most of the normal business will lead to changes in state, which will affect the continuous growth of state data and state index under the index and data separation architecture provided by the above application. For persistent storage, BasePage, DeltaPage and data files will continue to grow. In this way, the execution of the index reconstruction scheme provided by the above application itself requires traversal and statistics, generation of rewrite pages, installation of rewrite pages, etc., and this process will consume a relatively long time. In this process, the blockchain account book continues to grow and will not pause or stop. In the case where the RewritePage created by the above index reconstruction scheme contains a version, in the index file, RewritePage is in the same position as BasePage and DeltaPage, and is sorted by version together, so as to maintain the orderliness of the index file version. At the same time, due to the continuous growth of the aforementioned BasePage, DeltaPage and data files, it is possible that in the process of generating or installing RewritePage in the index file, BasePage, DeltaPage and corresponding data files with a version greater than RewritePage are generated. Of course, after installing RewritePage in the index file, BasePage, DeltaPage and corresponding data files with versions greater than RewritePage will be generated as the blockchain continues to run. After the new data file is generated by garbage collection, the data of the new data file is actually the result of migrating the valid data in the data file whose garbage ratio reaches the threshold before. In addition to generating new data files, index reconstruction and other tasks can be performed in the background through another independent thread. Generally, the data file whose garbage ratio reaches the threshold is deleted after the index reconstruction is completed. During this period, the BasePage or DeltaPage with a version number greater than (or equal to, depending on the sorting rules in the index file) the version of RewritePage is generated. In particular, the newly generated BasePage may contain mapping relationships that point to existing data content. At this time, there are two different options. One is to point to data files that have not been deleted but will be deleted after garbage collection, and the other is to point to new data files newly generated during the garbage collection process.

[0608] In addition, as mentioned above, in modern database management systems, data governance usually adopts an asynchronous processing mechanism. When the foreground transaction processing load drops below the preset threshold, the background governance task will be triggered. Similarly, the above-mentioned index reconstruction work can generally be triggered in the background governance task. For situations where the included mapping relationship may point to the content in the old invalid data file and need to be redirected by RewritePage, it is ideal to uniformly point to the same new data file from a certain point in time.

[0609] Then, for the newly generated index file mentioned above, which contains the mapping relationship pointing to the existing data content, choosing which data file to point to, and considering the background asynchronous governance mechanism, when is it appropriate to start adjusting the situation of pointing to the existing data content to uniformly point to the same new data file, careful design is required to converge the location of the invalid file within a specific range, thereby minimizing the delay caused by the jump.

[0610] The present application provides a method embodiment for converging the location of invalid files, and the background of the method embodiment includes: in the process of generating base pages and incremental pages based on the changes of all tree node contents in the leaf page logic page of the continuous version, and in the process of asynchronously performing garbage collection and rebuilding indexes. The method embodiment may include the following:

[0611] S430: When a new data file is generated during the garbage collection process, the maximum version of the base page / incremental page corresponding to the new data file is obtained, and the corresponding relationship between the state key in the new data file and the position of the new data file is obtained.

[0612] The generation of a new data file during the garbage collection process means that the new data file already exists in the persistent storage medium and has a data location in the persistent storage medium, that is, the information in the aforementioned Location. As mentioned earlier, the data content in these new data files comes from old data files whose garbage ratio reaches a threshold, and these old data files are located by the index file, specifically by the mapping relationship in the index file. The index file, such as the aforementioned BasePage or DeltaPage, has a version, and the mapping relationship therein may also have a version. Then, the corresponding BasePage or DeltaPage version can be obtained through the source of the new data file, thereby obtaining the maximum version of the corresponding BasePage or DeltaPage.

[0613] Specifically, the BasePage or DeltaPage version information based on which the new data file is generated can be recorded during the garbage collection process, such as recording the corresponding version during the process of traversing the index file and adding it to the rewrite set during the garbage collection process, as described in S380, S382 or S390, S392. In addition, it can also be an independent traversal process, and the specific implementation is similar to the above, which will not be repeated.

[0614] The data file may include the state key and the corresponding state value, so the state key can be obtained directly.

[0615] S432: During the index rebuilding process, the base page / delta page of the version before the obtained maximum version is re-indexed to generate a rewrite page.

[0616] Re-indexing the base page / incremental page of the version before the maximum version obtained includes: re-indexing the base page / incremental page of the version before the maximum version obtained based on the correspondence between the status key in the obtained new data file and the position of the new data file where it is located to generate a rewrite page, as described in the aforementioned re-indexing process. In this way, the generated RewritePage includes the remapping relationship from the status key to the new file position. Here, the base page / incremental page of the version before the maximum version obtained in S430 is re-indexed, and the base page / incremental page after the maximum version is not included.

[0617] S434: For the base page / incremental page of the version after the obtained maximum version, a mapping relationship in the base page / incremental page of the version after the maximum version is generated based on the correspondence between the state key in the obtained new data file and its position in the new data file.

[0618] Thus, after S434, the rewritten page forms a convergence point. The effect of this convergence point is as follows: Fig.36 As shown, the BasePage / DeltaPage of the later version of RewritePage (such as the BasePage with version number 120 in LeafPage C4, which is located after the version of RewritePage), and the corresponding relationship between the relevant NodeID and Location, the Location points to the new data file (such as Fig.36 In the new data file 350) or the data file generated later (indicated by the version of the new data file), there is no situation of pointing to an invalid data file. Before the convergence point, that is, in the BasePage / DeltaPage of the previous version of RewritePage, if remapping occurs, it will be remapped to the RewritePage. In this way, in the case where there is no remapping through the RewritePage before the convergence point, or the case where the RewritePage is remapped before the convergence point, the data file pointed to by Location or by Location after remapping is also controlled in the new data file or the previous data file, and will not point to the data file generated after the new data file. It can be seen that the convergence point forms a dividing line that distinguishes the remapped NodeID in the previous leaf page logical page and the related NodeID in the subsequent leaf page logical page.

[0619] Rewrite pages can remap the mapping relationship in the index file pointing to the invalid data file after garbage collection, so as to remap it to the new data file after garbage collection without making major changes to the existing index file (including the case of the pruned index file mentioned above). However, for the case of remapping through rewrite pages, one more jump is required, which will generate twice the reading delay compared to the case of locating the valid data file directly through the mapping relationship in BasePage / DeltaPage without remapping in the rewrite page.

[0620] As mentioned above, in data management using an asynchronous processing mechanism, when the foreground transaction processing load drops below a preset threshold, a background management task can be triggered. In addition to the aforementioned garbage collection, index rebuilding, and installation of rewrite pages, these background management tasks can also recycle rewrite pages, thereby eliminating the above-mentioned jumps and reducing the delay caused by jumps for subsequent reading.

[0621] Based on the above method for converging failed file locations, the present application also provides a method for reclaiming rewritten pages, including:

[0622] S440: Search for the rewrite page in the index file, and for the state key in the remapping relationship in the rewrite page, search for the mapping relationship of the same state key in the base page / delta page in the index file.

[0623] For convenience, the new data file location is referred to as newLocation here, and the original location pointing to the invalid data file is referred to as oldLocation.

[0624] Since the rewrite page contains the mapping relationship from the state key that originally pointed to oldLocation to newLocation, in order to eliminate such a jump, the rewrite page can be first searched in the index file.

[0625] After searching for the rewritten page in the index file, the mapping relationship of the same state key in the base page / incremental page can be searched in the index file based on the state key in the remapping relationship. Specifically, the mapping relationship of the same state ...

Claims

1. A method for writing a base page / incremental page in a memory to a persistent storage medium, comprising: In memory, for a subtree formed by index data divided into logical pages, each logical page generates a base page according to the memory page that has been modified a predetermined number of times; When a base page / incremental page of a certain version is generated in the upper-level logical page, an incremental page of the corresponding version is generated in the adjacent lower-level logical page; When the base page / incremental page in the memory reaches a predetermined upper limit, the base page / incremental page in the memory is written into a persistent storage medium in the order of version sorting.

2. The method as claimed in claim 1, wherein a delta page containing a version in an upper logical page is generated in a lower logical page in the tree structure.

3. The method according to claim 1, wherein writing the base page / delta page in the memory into the persistent storage medium in the order of version sorting comprises: Write the kv of the tree nodes contained therein in batches into the MemTable file in memory in units of base page / incremental page; When the amount of data in the MemTable reaches a certain threshold, the data in the MemTable is written to the persistent storage medium.

4. The method according to claim 3, wherein writing the data in the MemTable to the persistent storage medium comprises: Convert the MemTable into an immutable Memtable in memory, and generate a new MemTable to receive the newly passed in kv; Write the data in the Immutable MemTable into the persistent storage medium.

5. A method for compressing a delta page according to any one of claims 1 to 4, comprising: For an incremental page corresponding to a logical page in a persistent storage medium, searching for a plurality of adjacent continuous incremental pages whose sum reaches a predetermined size; A merge operation is performed on the plurality of consecutive delta pages whose sum reaches a predetermined size.

6. The method according to claim 5, wherein for a base page / delta page corresponding to a LogicalPage in a persistent storage medium, searching for a plurality of adjacent consecutive delta pages whose sum reaches a predetermined size comprises: For a base page / incremental page corresponding to a LogicalPage in a persistent storage medium, the last one of several adjacent continuous incremental pages is searched, and a preset number of adjacent incremental pages are searched forward in reverse order.

7. The method according to claim 6, for a base page / delta page corresponding to a LogicalPage in a persistent storage medium, searching for the last one of several adjacent continuous delta pages, comprising: For a base page / delta page corresponding to a LogicalPage in a persistent storage medium, a base page is found, and the delta page of the largest version corresponding to the logical page before the base page is found.

8. The method according to claim 7, for a base page / delta page corresponding to a LogicalPage in a persistent storage medium, searching for a base page and searching for a delta page of the largest version corresponding to the logical page before the base page, comprises: For a base page / delta page corresponding to a LogicalPage in the persistent storage medium, find a base page and find the delta page of the same version corresponding to the logical page before the base page.

9. The method according to claim 5, performing a merge operation on the plurality of consecutive delta pages whose sum reaches a predetermined size, comprising: Merging a preset number of incremental pages adjacent to the last one and the forward search, marking it as a merge operation; During compaction, merge operations are performed on delta pages that are marked for merge operations.

10. The method according to claim 5, wherein the predetermined size is a read / access unit size of the persistent storage medium.

11. A computer device comprising: processor; and a memory storing a program, wherein when the processor executes the program, the operation of any one of claims 1 to 10 is performed.

Images