Blockchain state data access and crash recovery method and system based on learning index

Abstract

The application discloses a blockchain state data access and crash recovery method and system based on a learning index, wherein a composite key is generated according to a state address and a block height, and the composite key and a state value form a composite key-value pair; the composite key is encoded by placing the state address in a high bit and the block height in a low bit, thereby reducing historical index redundancy caused by path node replication; secondly, the composite key-value pair to be written is first written into a memory level Merkle tree, and when the memory level satisfies a preset writing condition, the composite key-value pairs stored in the memory level Merkle tree are organized into ordered runs and written into a first persistent level, or when any persistent level satisfies a preset merging condition, all ordered runs in the persistent level are sorted, and the sorted ordered runs are written into a subsequent persistent level, thereby reducing direct disturbance of frequent writing to an underlying index structure.

Description

Technical Field

[0001] This invention belongs to the field of blockchain technology, and more specifically, relates to a method and system for accessing blockchain state data and recovering from crashes based on a learning index. Background Technology

[0002] A blockchain system maintains ledger data collaboratively through multiple nodes and ensures that the data is immutable, verifiable, and traceable through cryptographic hashes, consensus protocols, and authentication data structures. In a blockchain system, state data typically includes a state address and a state value. As blocks are continuously generated, the same state address may have different state values ​​at different block heights. Therefore, blockchain nodes not only need to query the latest state but also support historical state tracing queries and verification of query results.

[0003] Existing blockchain state indexes typically employ Merkle compressed prefix trees. This structure supports integrity verification through root hashes and Merkle paths, and allows for historical state queries by retaining historical nodes. However, when a state value is updated, the Merkle compressed prefix tree needs to be copied and persisted to the relevant nodes along the update path. As block height increases, a large number of historical path nodes accumulate, leading to high storage overhead in the index structure itself.

[0004] Learning indexes utilize models to learn the distribution patterns of ordered key sets and predict the position of target keys within the ordered data. While learning indexes can reduce the number of traditional index nodes, blockchain state storage scenarios have requirements such as frequent writes, continuous addition of historical versions, integrity verification, traceability queries, and consistency recovery after crashes. Directly using ordinary learning indexes for blockchain state data storage may result in high index maintenance overhead and low data access efficiency. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a blockchain state data access and crash recovery method and system based on learning index, thereby solving the technical problems of high index maintenance overhead and low data access efficiency in the existing technology.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a method for accessing and recovering blockchain state data based on a learning index is provided, comprising: Receive a blockchain state data write request and obtain the state address, state value, and block height from the write request; A composite key is generated based on the status address and the block height, and the composite key is combined with the status value to form a composite key-value pair; wherein, the composite key is encoded by placing the status address in the high bit and the block height in the low bit; Write the composite key-value pairs into a Merkle tree at the memory level; When the memory level meets the preset flushing condition, the composite key-value pairs stored in the Merkle tree of the memory level are organized into ordered runs and written to the first persistent level; when any persistent level meets the preset merging condition, all ordered runs in any persistent level are merged and sorted, and the merged ordered runs are written to the next persistent level. If a new ordered run is generated, a value file, a learning index file, and a Merkle file are built for the new ordered run, and the root hash list is updated after the value file, learning index file, and Merkle file are built.

[0007] According to the above-mentioned blockchain state data access and crash recovery method based on learning indexes, when the merging condition is met at any persistence level, all ordered runs in that persistence level are merged and sorted, and the merged ordered runs are written to the next persistence level. Specifically, this includes: All ordered composite key-value pairs in any persistent level are written into the merge queue in ascending order of composite key; wherein, composite keys are compared in order of priority of state address, followed by block height; if the same composite key is found, the most recently written composite key-value pair containing that composite key is written into the merge queue in the order it was written into any persistent level. When all ordered composite key-value pairs in any persistent level have been traversed, the composite key-value pairs in the merge queue are organized into ordered pairs and written to the next persistent level.

[0008] According to the above-described method for accessing and recovering blockchain state data based on a learning index, the new ordered learning index file includes a multi-level piecewise linear approximation model; any one level of the piecewise linear approximation model is constructed based on the following: If the current piecewise linear approximation model is the lowest level, then obtain the new ordered composite key-value pairs to construct the current key-value pair set; otherwise, obtain the starting key-value pairs of each linear segment in the previous piecewise linear approximation model to construct the current key-value pair set. Based on the optimal segmentation algorithm, each key-value pair in the current key-value pair set is mapped to a corresponding linear segment to obtain a piecewise linear approximation model containing multiple linear segments; wherein, for any linear segment, the error between the predicted position and the state value of the composite key in the key-value pair mapped to that linear segment is within a preset error range.

[0009] According to the above-described method for accessing and recovering blockchain state data based on a learning index, the method further includes: In response to a status query request, a target composite key is generated based on the status address to be queried and the target block height in the status query request; Based on the target composite key, query the Merkle tree in the memory level; If the target composite key is matched in the memory level, the status value of the matched composite key-value pair is retrieved and returned to the requester; otherwise, based on the target composite key, the ordered execution in each persistent level is queried in hierarchical order. Specifically, based on the target composite key, a query is performed in any ordered run at any persistent level to obtain the learning index file corresponding to that ordered run. Starting from the top-level piecewise linear approximation model of the learning index file, the linear segment corresponding to the target composite key in the current layer's piecewise linear approximation model is determined. Based on the linear segment, the predicted position of the target composite key in the current layer is predicted. Based on the predicted position and a preset error parameter, the linear segment corresponding to the target composite key in the next layer's piecewise linear approximation model is determined until the predicted position of the target composite key at the bottom layer is obtained. Based on the predicted position of the target composite key at the bottom layer, the state value corresponding to the target composite key is obtained from the value file of any ordered run and returned to the requester.

[0010] Based on the aforementioned blockchain state data access and crash recovery method based on learning indexes, the ordered execution in each persistent layer is queried sequentially according to hierarchical order, specifically including: Get the currently ordered execution in the current persistent hierarchy; Based on the Bloom filter corresponding to the current ordered operation, determine whether the target composite bond exists in the current ordered operation; If the Bloom filter determines that the target composite key is not in the current ordered run, it skips the current ordered run; otherwise, it queries the current ordered run based on the target composite key.

[0011] Based on the above-described method for accessing and recovering blockchain state data using a learning index, the Bloom filter corresponding to any ordered operation is constructed as follows: Determine the target false positive rate corresponding to any of the ordered operations; The length of the bit array and the number of hash functions are determined based on the target false positive rate and the number of composite key-value pairs in any ordered operation. Initialize the array elements in the bit array to 0, and obtain the corresponding number of hash functions; Traverse any ordered composite key-value pair, and for the current composite key-value pair, use each hash function to calculate the hash value corresponding to the composite key in the current composite key-value pair, and use the hash value corresponding to the composite key as the index of the bit array, and set the array element at the corresponding index to 1; The bit array, the parameters of the corresponding number of hash functions, the number of composite key-value pairs in any ordered run, and the target false positive rate are saved as the metadata of the Bloom filter corresponding to any ordered run.

[0012] According to the above-mentioned blockchain state data access and crash recovery method based on learning indexes, determining whether the target composite key exists in the current ordered operation based on the Bloom filter corresponding to the current ordered operation specifically includes: The hash value of the target composite key is calculated based on multiple hash functions of the Bloom filter corresponding to the current ordered operation, resulting in multiple bit array indices; Retrieve the array elements at the multiple bit array indices; If the array element at any array index is 0, then the target composite key is determined not to be in the current ordered process; If all the array elements at the multiple bit array indices are 1, then the target composite key is determined to exist in the current ordered process.

[0013] According to the above-described method for accessing and recovering blockchain state data based on a learning index, the method further includes: In response to a source tracing query request, obtain the source status address and block height range to be traced; A first boundary composite key is generated based on the state address to be traced and the lower bound of the block height interval, and a second boundary composite key is generated based on the state address to be traced and the upper bound of the block height interval. In the memory level, query composite key value pairs whose composite keys are located within the query range defined by the first boundary composite key and the second boundary composite key; If a matching composite key-value pair exists in the memory level, then obtain the Merkel path corresponding to the matching composite key-value pair. The system sequentially queries ordered runs in each persistent level to determine whether the composite key range of the corresponding ordered run intersects with the query range. For ordered runs where the composite key range intersects with the query range, it predicts the predicted positions of the first boundary composite key and the second boundary composite key at the lowest level based on the learning index file corresponding to the corresponding ordered run. Based on a preset error parameter and the predicted positions of the first boundary composite key and the second boundary composite key at the lowest level, it obtains the candidate ranges corresponding to the first boundary composite key and the second boundary composite key, and obtains the upper and lower bounds of the index from the candidate ranges corresponding to the first boundary composite key and the second boundary composite key. It reads the composite key value pairs in the corresponding ordered run whose composite keys are located between the upper and lower bounds of the index as the hit composite key value pairs, and generates the Merkle path corresponding to the hit composite key value pairs based on the Merkle file corresponding to the corresponding ordered run. The upper bound of the index is the first composite key greater than or equal to the first boundary composite key in the candidate ranges corresponding to the first boundary composite key and the second boundary composite key, and the lower bound of the index is the last composite key less than or equal to the second boundary composite key in the candidate ranges corresponding to the first boundary composite key and the second boundary composite key. Obtain the ordered runs of the currently valid root hash list references as a total set, remove the ordered runs that have been queried and generated Merkel paths from the total set, and obtain the root hashes of the remaining ordered runs in the total set; The verification proof is composed of the state values ​​of all the hit composite key-value pairs, the Merkel paths corresponding to the hit composite key-value pairs, and the remaining ordered root hashes in the total set. The state values ​​of all the hit composite key-value pairs and the verification proof are then returned to the requester.

[0014] According to the above-described method for accessing and recovering blockchain state data based on a learning index, the method further includes: When the blockchain system crashes and restarts, the hierarchical files of the memory level and each persistent level, as well as the root hash list, are reopened; the hierarchical files of the memory level and each persistent level, as well as the root hash list, are stored in persistent memory; the hierarchical files include the value file, the learned index file, the Merkle file, and the log file of the corresponding level; Based on the most recently valid root hash list, the most recently checked point, and the log files of each level, incomplete transactions since the most recently checked point are identified. These incomplete transactions include persistent level merge transactions involving new ordered runs of Merkle file root hashes that are not referenced by the most recently valid root hash list. The most recently checked point is recorded after the ordered runs of composite key-value pairs in the memory level have been written to the first persistent level, and the corresponding value file, learned index file, and Merkle file have been constructed, and the root hash list has been updated. Replay all unfinished transactions since the last checkpoint.

[0015] According to a second aspect of the present invention, a blockchain state data access and crash recovery system based on a learning index is provided for performing the method described in the first aspect.

[0016] According to a third aspect of the present invention, an electronic device is provided, comprising: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in the first aspect.

[0017] According to a fourth aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to perform the method as described in the first aspect.

[0018] According to a fifth aspect of the invention, a computer program product is provided, comprising a computer program or instructions that, when executed by a processor, implement the method as described in the first aspect.

[0019] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: By obtaining the status address, status value, and block height from the write request, a composite key is generated based on the status address and block height. The composite key is then combined with the status value to form a composite key-value pair. The composite key is encoded by placing the status address in the high-order bits and the block height in the low-order bits, which allows multiple historical versions of the same status address to be indexed in a unified key space, thereby reducing historical index redundancy caused by path node copying. Secondly, by first writing the composite key-value pairs to be written into the Merkle tree at the memory level, and when the memory level meets the preset flushing conditions, organizing the composite key-value pairs stored in the Merkle tree at the memory level into an ordered sequence and writing them into the first persistent level, or when the preset merging conditions are met at any persistent level, merging and sorting all the ordered sequences in that persistent level, and writing the merged ordered sequence into the next persistent level, the state data writing can be performed in a batch and hierarchical manner, thereby reducing the direct disturbance of the underlying index structure caused by frequent writes. Furthermore, after a new ordered run is generated during the hierarchical flushing and merging operations, a value file, a learned index file, and a Merkle file are constructed for it. After the value file, learned index file, and Merkle file are constructed, the root hash list is updated. On the one hand, this enables the multi-level learned index structure to generate block state root hashes and verification proofs. More importantly, updating the root hash list after the value file, learned index file, and Merkle file of the new ordered run are all constructed ensures that the root hash list does not reference incomplete new hierarchical data when a crash occurs, thereby guaranteeing the accuracy of crash recovery.

[0020] Furthermore, by generating a learning index file for ordered runs and using a piecewise linear approximation model to predict the position of composite keys in the value file, the query and tracing processes can quickly and accurately locate state values ​​by utilizing the distribution patterns of composite keys. Specifically, by first using a Bloom filter in the state query to determine whether the target composite key exists in the ordered run, ordered runs without the target composite key can be skipped, thereby reducing invalid hierarchical accesses. In addition, by generating boundary composite keys and composite key-value pairs between query boundary composite keys in the tracing query, and collecting Merkel paths and root hashes of unsearched ordered runs, the historical state value query results can be completely and accurately verified. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a blockchain state data access and crash recovery method based on a learning index, provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0023] This invention provides a method for accessing and recovering blockchain state data based on a learned index, such as... Figure 1 As shown, it includes: Step S1: Receive a blockchain state data write request and obtain the state address, state value, and block height from the write request; Step S2: Generate a composite key based on the state address and the block height, and combine the composite key with the state value to form a composite key-value pair; wherein, the composite key is encoded by placing the state address in the high bit and the block height in the low bit; Step S3: Write the composite key-value pair into a Merkle tree at the memory level; Step S4: When the memory level meets the preset flushing condition, organize the composite key-value pairs stored in the Merkle tree of the memory level into an ordered run and write it into the first persistent level; when any persistent level meets the preset merging condition, merge sort all the ordered runs in the any persistent level and write the merged ordered run into the next persistent level. Step S5: If a new ordered run is generated, construct a value file, a learning index file, and a Merkle file for the new ordered run, and update the root hash list after the value file, learning index file, and Merkle file are constructed.

[0024] Specifically, blockchain nodes are responsible for receiving write requests carrying blockchain state data. Blockchain state data refers to data in the blockchain system identified by state addresses and that changes with transaction execution, including at least a state address and a state value. A state address is a key used to identify blockchain state objects; in Ethereum-like systems, a state address can be used to locate accounts, contracts, or other state objects. A state value is the data content corresponding to a state address at a specific block height, where block height refers to the height or version identifier of a block in the blockchain, used to distinguish the state value of the same state address at different historical points in time. The state address (addr), state value (value), and block height (blk) of the blockchain state data to be written are obtained from the write request.

[0025] A composite key is generated based on the state address and block height. This composite key is used to simultaneously represent the state object and the historical version. In some embodiments, the composite key is encoded by placing the state address in the high-order bits and the block height in the low-order bits, so that historical versions corresponding to the same state address are arranged consecutively in the subsequent ordered execution. The composite key can be represented as K=<addr, blk> Where addr is the status address and blk is the block height. Then, this composite key K is combined with the status value value from the write request to form a composite key-value pair.<K, value> The composite key-value pair is written into a Merkle tree at the memory level. This memory level, denoted as L0, is the level that receives updates to the new blockchain state data. In some embodiments, the memory level uses a Merkle B+ tree to store the composite key-value pair. After inserting the composite key-value pair into the Merkle tree, the Merkle tree updates its node content and the root hash of the Merkle tree based on the insertion result.

[0026] When a memory level meets a preset flush condition, the composite key-value pairs stored in the Merkle tree of that memory level are organized into an ordered sequence and written to the first persistent level, thus achieving a level-by-level flush of the memory level. In some embodiments, the preset flush condition can be that the capacity of the memory level reaches a preset threshold, or the number of composite key-value pairs in the memory level reaches a preset number threshold. Therefore, if the memory level does not meet the flush condition, the composite key-value pairs remain in the memory level. In some embodiments, during level-by-level flushing, the leaf nodes of the Merkle tree in the memory level are scanned to obtain a set of composite key-value pairs arranged in composite key order. Then, the set of composite key-value pairs is organized into an ordered sequence. When comparing the size of composite keys, the comparison is performed in order of state address priority, followed by block height. For example, composite keys with smaller state addresses are placed earlier, while for composite keys with the same state address, composite keys with smaller block heights are placed relatively earlier.

[0027] Since a new ordered run is generated during hierarchical flushing, a corresponding value file, a learned index file, and a Merkle file are constructed for this new ordered run, and the root hash of the Merkle tree in the Merkle file is calculated. The value file is the data file that stores the state values ​​of each composite key-value pair in the ordered run; the learned index file is the data file that stores the learned index prediction model, used to predict the position of the composite key in the value file; and the Merkle file is the file that stores the Merkle tree used to authenticate the composite key-value pairs.

[0028] For persistent levels, if any persistent level meets a preset merging condition, all ordered runs in that persistent level are merged and sorted, and the merged ordered runs are written to the next persistent level. In some embodiments, the preset merging condition may be that the number of ordered runs in the persistent level reaches a preset upper limit, or that the persistent level's occupied capacity reaches a preset threshold. In other embodiments, for a persistent level that meets the preset merging condition, all ordered runs in that persistent level are obtained, and then the composite key-value pairs in all ordered runs in that persistent level are written to a merge queue in ascending order of composite key. The composite keys are compared in order of priority: state address first, then block height. If identical composite keys are found, the most recently written composite key-value pair containing that composite key is written to the persistent level in the order it was written. When all composite key-value pairs in ordered runs of that persistent level have been traversed, the composite key-value pairs in the merge queue are organized into ordered runs and written to the next persistent level, thus completing one level merge. Similar to the hierarchical flushing described above, since a new ordered run is generated during hierarchical merging, a corresponding value file, a learning index file, and a Merkle file are constructed for this new ordered run, and the root hash of the Merkle tree in the Merkle file is calculated.

[0029] In some embodiments, the new ordered learning index file includes a multi-level piecewise linear approximation model. Any of the multi-level piecewise linear approximation models is constructed based on the following: If the current piecewise linear approximation model is the lowest level, then obtain the new ordered composite key-value pairs to construct the current key-value pair set; otherwise, obtain the starting key-value pairs of each linear segment in the previous piecewise linear approximation model to construct the current key-value pair set. Based on the optimal segmentation algorithm, each key-value pair in the current key-value pair set is mapped to a corresponding linear segment, resulting in a piecewise linear approximation model containing multiple linear segments. For any linear segment, the error between the predicted position and the state value corresponding to the composite key in the key-value pair mapped to that segment is within a preset error range. Each linear segment includes a start key, slope, and intercept. When predicting the predicted position corresponding to a composite key, the state address and block height contained in the composite key can be converted into a sortable value, and then the corresponding linear segment is applied to this value for prediction.

[0030] In other embodiments, the learning index file stores the starting key, slope, intercept, and error parameters of each linear segment in the piecewise linear approximation model at each level, in hierarchical order. The error parameter is a preset parameter. Multi-level piecewise linear approximation models are constructed by recursively calling the above construction method. The recursion termination condition can be that the number of linear segments in the previous level's piecewise linear approximation model is less than a preset threshold, or that the number of constructed piecewise linear approximation models reaches a preset upper limit.

[0031] It should be noted that whenever a new ordered run is generated, the root hash list is updated after its corresponding value file, learning index file, and Merkle file are constructed. The root hash list is a data structure composed of the Merkle tree root hash at the memory level and the root hash of the Merkle tree in each ordered run's Merkle file at each persistent level. It is used to generate or update the block state root hash. Therefore, when a new ordered run is generated, this list can be updated based on the root hash of the Merkle tree in the constructed Merkle file.

[0032] For hierarchical flushing and merging operations, after the root hash list is updated, the old level's value files, learning index files, and Merkle files can be deleted or cleaned up. In some embodiments, the root hash list uses a dual-version record to maintain the current valid version identifier. When updating the root hash list, the new version of the root hash list is first written and persisted, and then the valid version identifier is switched; if a crash occurs before switching the valid version identifier, the old version of the root hash list continues to be used; if a crash occurs after switching the valid version identifier, the new version of the root hash list is used. If the deletion of the old level's value files, learning index files, and Merkle files fails, a cleanup identifier can be recorded for the corresponding files at the corresponding level, without affecting the completed root hash list update.

[0033] In some embodiments, in response to a status query request, a target composite key can be generated based on the address of the state to be queried and the target block height in the status query request. The construction method of the target composite key is the same as that of the composite key described above. Subsequently, based on the target composite key, the Merkle tree in the memory hierarchy is first queried. If the target composite key is found in the memory hierarchy (i.e., a composite key-value pair with the target composite key is stored in the memory hierarchy), the status value of the found composite key-value pair is retrieved and returned to the requester; otherwise, based on the target composite key, the ordered execution in each persistent hierarchy is queried sequentially according to the hierarchical order.

[0034] In some embodiments, when a query finds an ordered run in a persistent hierarchy, the Bloom filter corresponding to that ordered run can be used to determine whether the target composite key exists in that ordered run. If the Bloom filter determines that the target composite key is not in that ordered run, the ordered run is skipped, that is, the specific query process for that ordered run is not performed, and the next ordered run is processed instead; otherwise, the ordered run is queried based on the target composite key.

[0035] In other embodiments, the Bloom filter corresponding to any ordered operation is constructed based on the following: Determine the target false positive rate corresponding to the ordered run. By assigning a target false positive rate, the false positive probability of Bloom filters at different persistence levels can be controlled, thereby reducing invalid reads and query overhead. A lower target false positive rate typically requires more bit array space for the Bloom filter, but the probability of falsely judging "possible existence" is lower; a higher target false positive rate requires less space, but may lead to more invalid reads of learned index files and value files. In some embodiments, since different persistence levels have different access frequencies, number of ordered runs, and false positive costs, different target false positive rates can be assigned to different persistence levels, so that persistence levels with high access frequencies or high false positive costs have lower target false positive rates.

[0036] The length m of the bit array and the number of hash functions k are determined based on the target false positive rate and the number of composite key-value pairs in the ordered operation.

[0037] Initialize the array elements in the bit array above to 0, and obtain k different hash functions.

[0038] The algorithm iterates through the ordered composite key-value pairs. For each current composite key-value pair, it calculates the hash value corresponding to the composite key using various hash functions. The k hash values ​​corresponding to this composite key are then used as indices for a bit array, and the array element at the corresponding index is set to 1. Specifically, when calculating the hash value corresponding to a particular composite key, the state address and block height contained in that composite key can be converted into a numerical value, and then the corresponding hash function can be applied to this value for hash calculation.

[0039] Finally, the bit array, the parameters of the k hash functions, the number of composite key-value pairs in the ordered run, and the target false positive rate are saved as metadata for the Bloom filter corresponding to the ordered run, so that the Bloom filter can call them when determining whether the target composite key exists in the ordered run.

[0040] In other embodiments, when it is necessary to determine whether a target composite key exists in an ordered run using a Bloom filter corresponding to that ordered run, the hash value of the target composite key can be calculated based on multiple hash functions of the Bloom filter corresponding to that ordered run, resulting in multiple bit array indices. The array elements at these bit array indices are then retrieved and judged: if any array element at any bit array index is 0, it is determined that the target composite key is not in that ordered run; if all array elements at these bit array indices are 1, it is determined that the target composite key exists in that ordered run.

[0041] When querying any ordered run at any persistent level based on a target composite key, the corresponding learned index file can be obtained. Then, starting from the top-level piecewise linear approximation model of this learned index file, the linear segment corresponding to the target composite key in the current layer's piecewise linear approximation model is determined. Based on this linear segment, the predicted position p of the target composite key in the current layer is predicted. Based on this predicted position p and a preset error parameter ε, the corresponding linear segment of the target composite key in the next layer's piecewise linear approximation model is determined. This process is repeated until the predicted position of the target composite key at the bottom layer is obtained. Based on the predicted position of the target composite key at the bottom layer, the corresponding state value is obtained from the value file of the ordered run and returned as the state value corresponding to the target composite key to the requester. When the state query request requires verification, the Merkle file of the ordered run that successfully retrieved the state value corresponding to the target composite key can be obtained, and the corresponding Merkle path can be generated as verification proof and returned to the requester.

[0042] In some embodiments, in response to a tracing query request, the state address to be traced and the block height range [blk_l, blk_u] in the tracing query request are obtained. A first boundary composite key is generated based on the lower bound blk_l of the state address to be traced and the block height range, and a second boundary composite key is generated based on the upper bound blk_u of the state address to be traced and the block height range. Subsequently, composite key-value pairs within the query range defined by the first and second boundary composite keys are queried in the memory hierarchy. If a matching composite key-value pair exists in the memory hierarchy, the Merkle path corresponding to the matching composite key-value pair is obtained. Next, ordered runs in each persistent hierarchy are queried sequentially to determine whether the composite key range of the corresponding ordered run intersects with the above query range. The composite key range of the ordered run consists of the smallest and largest composite keys among them. For ordered operations where the range of composite keys intersects with the query range, the predicted positions of the first boundary composite key and the second boundary composite key at the lowest level are predicted based on the learning index file corresponding to the ordered operation. The specific prediction method is the same as the prediction method used in the processing of the status query request, and will not be repeated here.

[0043] Based on the preset error parameter and the predicted positions of the first and second boundary composite keys at the lowest level, the candidate ranges corresponding to the first and second boundary composite keys are obtained. Assuming the preset error parameter is ε and the predicted position of the first boundary composite key at the lowest level is pred, then the candidate range corresponding to the first boundary composite key can be [pred...]. Similarly, the candidate range corresponding to the second boundary composite key can be obtained from the candidate ranges corresponding to the first and second boundary composite keys. The upper and lower bounds of the index are obtained from these candidate ranges. The upper bound is the first composite key greater than or equal to the first boundary composite key in the candidate ranges corresponding to both the first and second boundary composite keys (i.e., the smallest composite key greater than or equal to the first boundary composite key among all composite keys in both candidate ranges). The lower bound is the last composite key less than or equal to the second boundary composite key in the candidate ranges corresponding to both the first and second boundary composite keys (i.e., the largest composite key less than or equal to the second boundary composite key among all composite keys in both candidate ranges). Composite key value pairs whose composite keys are located between the upper and lower bounds in the corresponding ordered execution are read as the hit composite key value pairs, and the Merkel path corresponding to the hit composite key value pairs is generated based on the Merkel file corresponding to the corresponding ordered execution.

[0044] In addition, the currently valid root hash list referenced ordered runs are obtained as a set, and ordered runs that have been queried and generated Merkel paths are removed from this set. The remaining ordered runs in the set are those that were not searched but participated in the root hash calculation. These are ordered runs that are currently referenced in the root hash list, whose root hashes participate in the block state root hash calculation, but were not expanded and read in this query because the composite key range does not intersect with the query range. The root hashes of the remaining ordered runs in the set are obtained. The state values ​​of all hit composite key-value pairs, the Merkel paths corresponding to the hit composite key-value pairs, and the root hashes of the remaining ordered runs in the set are combined to form a verification proof, and the state values ​​of all hit composite key-value pairs and the verification proof are returned to the requester.

[0045] In other embodiments, after the blockchain system crashes and restarts, the hierarchical files of the memory hierarchy and each persistent hierarchy, as well as the root hash list, are reopened. The hierarchical files of the memory hierarchy and each persistent hierarchy, as well as the root hash list, are stored in persistent memory. These hierarchical files include the value file, learning index file, Merkle file, and log file for the corresponding hierarchy. In some embodiments, the log file includes information such as operation type, target hierarchy, target ordered run identifier, composite key range, transaction status, and verification information. Checkpoints save the hierarchy state, root hash list state, and processed transaction positions when the memory hierarchy refresh is complete. Based on the most recently valid root hash list, the most recent checkpoint, and the log files of each hierarchy, incomplete transactions since the most recent checkpoint are identified. Incomplete transactions include persistent hierarchy merging transactions involving new ordered runs of Merkle file root hashes that are not referenced by the most recently valid root hash list. The most recent checkpoint mentioned above was recorded after the ordered execution of composite key-value pairs in the memory hierarchy had been written to the first persistent hierarchy, and the corresponding value file, learned index file, and Merkle file had been constructed, and the root hash list had been updated. Subsequently, all incomplete transactions since the most recent checkpoint are replayed.

[0046] During replay, the system can query the currently incomplete transactions that need to be replayed, identify the composite key-value pairs involved, and then check whether these composite key-value pairs already exist in the reopened memory-level Merkle tree and the ordered runs of each persistent level referenced by the most recently valid root hash list. This query method is consistent with the status query request processing method described above: first, query the memory-level Merkle tree; if the memory level is not found, access each ordered run of each persistent level in hierarchical order. If the composite key is found and the status value matches the status value to be replayed, it means that the data already exists, and the insertion operation of the corresponding composite key-value pair is skipped; if it does not exist, the insertion operation of the corresponding composite key-value pair is performed.

[0047] In other embodiments, the node loading, node storage, and batch read operations for the Merkle tree can be encapsulated as tasks and submitted to a thread pool. The thread pool distributes these tasks to multiple worker threads for execution. When multiple worker threads access Merkle tree nodes, concurrent access is controlled through shared ownership and read-write locks. After the persistent level merging operation is completed, the deletion operation of related files from different old ordered processes can be encapsulated as an asynchronous task, which is then executed and its execution result is recorded. The thread pool size can be determined based on the processor's parallelism capabilities or system configuration parameters. The task queue is used to cache node loading, node storage, and batch read tasks; when the deletion of related files from old ordered processes fails, the failure result is recorded, and the corresponding old file is marked as a file to be cleaned up.

[0048] In other embodiments, if file errors, data loss, permission errors, or other anomalies occur during system operation, error messages can be received, and it can be determined whether the error message corresponds to a first type of error or a second type of error. First type of errors include data loss that can be recovered from redundant fields, loss of non-critical statistical fields, and failure to clean up old files. Second type of errors include the absence of critical status data, inaccessibility of necessary hierarchical files, failure to read the root hash list, and failure to write necessary data due to insufficient permissions. When the error message corresponds to a first type of error, skip processing is performed, and the error message is written to the error log. When the error message corresponds to a second type of error, the current processing flow is interrupted, error messages are fed back, and the error messages are written to the error log.

[0049] When the system receives a request to delete state data, it adds a deletion marker to the state data to be deleted. This deletion marker is then entered into the corresponding layer along with the corresponding composite key-value pair. When the persistent layer containing the composite key-value pair corresponding to the deletion marker triggers a layer merge operation, all composite key-value pairs carrying deletion markers in that persistent layer will be processed uniformly. Specifically, when all ordered composite key-value pairs in that persistent layer are written into the merge queue in ascending order of their composite keys, if a composite key-value pair carrying a deletion marker is encountered, it is skipped and not written into the merge queue. This delays the deletion operation to the layer merge stage by using the marker. It should be noted that the above method of handling deletion markers applies to blockchain systems that do not undergo forks.

[0050] In summary, the method provided by this invention obtains the status address, status value, and block height from the write request, generates a composite key based on the status address and block height, and combines the composite key with the status value to form a composite key-value pair. Furthermore, by encoding the composite key with the status address in the high-order bits and the block height in the low-order bits, multiple historical versions of the same status address can be indexed in a unified key space, thereby reducing historical index redundancy caused by path node replication. Secondly, by first writing the composite key-value pair to be written into the Merkle tree at the memory level, and then, when the memory level meets the preset flushing conditions, organizing the composite key-value pairs stored in the Merkle tree into an ordered sequence and writing them into the first persistent level, or, when any persistent level meets the preset merging conditions, merging and sorting all the ordered sequences in that persistent level, and writing the merged ordered sequence into the next persistent level, the state data writing can be performed in a batch and hierarchical manner, thereby reducing the direct disturbance to the underlying index structure caused by frequent writes. Furthermore, after a new ordered run is generated during the hierarchical flushing and merging operations, a value file, a learned index file, and a Merkle file are constructed for it. After the value file, learned index file, and Merkle file are constructed, the root hash list is updated. On the one hand, this enables the multi-level learned index structure to generate block state root hashes and verification proofs. More importantly, updating the root hash list after the value file, learned index file, and Merkle file of the new ordered run are all constructed ensures that the root hash list does not reference incomplete new hierarchical data when a crash occurs, thereby guaranteeing the accuracy of crash recovery.

[0051] Furthermore, by generating a learning index file for ordered runs and using a piecewise linear approximation model to predict the position of composite keys in the value file, the query and tracing processes can quickly and accurately locate state values ​​by utilizing the distribution patterns of composite keys. Specifically, by first using a Bloom filter in the state query to determine whether the target composite key exists in the ordered run, ordered runs without the target composite key can be skipped, thereby reducing invalid hierarchical accesses. In addition, by generating boundary composite keys and composite key-value pairs between query boundary composite keys in the tracing query, and collecting Merkel paths and root hashes of unsearched ordered runs, the historical state value query results can be completely and accurately verified.

[0052] The following describes the blockchain state data access and crash recovery system based on learning indexes provided by this invention. The blockchain state data access and crash recovery system based on learning indexes described below can be referred to in correspondence with the blockchain state data access and crash recovery method based on learning indexes described above.

[0053] This invention provides a blockchain state data access and crash recovery system based on a learning index, used to execute the blockchain state data access and crash recovery method based on a learning index provided in any of the above embodiments.

[0054] This invention provides an electronic device, including: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the above embodiments.

[0055] This invention provides a computer-readable storage medium storing computer instructions that cause a processor to perform the method described in any of the above embodiments.

[0056] This invention provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method described in any of the above embodiments.

[0057] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for accessing and recovering blockchain state data based on a learning index, characterized in that, include: Receive a blockchain state data write request and obtain the state address, state value, and block height from the write request; A composite key is generated based on the status address and the block height, and the composite key is combined with the status value to form a composite key-value pair; wherein, the composite key is encoded by placing the status address in the high bit and the block height in the low bit; Write the composite key-value pairs into a Merkle tree at the memory level; When the memory level meets the preset flushing condition, the composite key-value pairs stored in the Merkle tree of the memory level are organized into ordered runs and written to the first persistent level; when any persistent level meets the preset merging condition, all ordered runs in any persistent level are merged and sorted, and the merged ordered runs are written to the next persistent level. If a new ordered run is generated, a value file, a learning index file, and a Merkle file are built for the new ordered run, and the root hash list is updated after the value file, learning index file, and Merkle file are built.

2. The blockchain state data access and crash recovery method based on learning indexes as described in claim 1, characterized in that, When the merging condition is met at any persistent level, all ordered runs in that persistent level are merged and sorted, and the merged ordered runs are written to the next persistent level. Specifically, this includes: All ordered composite key-value pairs in any persistent level are written into the merge queue in ascending order of composite key; wherein, composite keys are compared in order of priority of state address, followed by block height; if the same composite key is found, the most recently written composite key-value pair containing that composite key is written into the merge queue in the order it was written into any persistent level. When all ordered composite key-value pairs in any persistent level have been traversed, the composite key-value pairs in the merge queue are organized into ordered pairs and written to the next persistent level.

3. The blockchain state data access and crash recovery method based on learning indexes as described in claim 1, characterized in that, The new ordered learning index file includes a multi-level piecewise linear approximation model; any level of piecewise linear approximation model is constructed based on the following: If the current piecewise linear approximation model is the lowest level, then obtain the new ordered composite key-value pairs to construct the current key-value pair set; otherwise, obtain the starting key-value pairs of each linear segment in the previous piecewise linear approximation model to construct the current key-value pair set. Based on the optimal segmentation algorithm, each key-value pair in the current key-value pair set is mapped to a corresponding linear segment to obtain a piecewise linear approximation model containing multiple linear segments; wherein, for any linear segment, the error between the predicted position and the state value of the composite key in the key-value pair mapped to that linear segment is within a preset error range.

4. The blockchain state data access and crash recovery method based on learning indexes as described in claim 3, characterized in that, The method further includes: In response to a status query request, a target composite key is generated based on the status address to be queried and the target block height in the status query request; Query the Merkle tree in the memory level based on the target composite key; If the target composite key is matched in the memory level, the status value of the matched composite key-value pair is retrieved and returned to the requester; otherwise, based on the target composite key, the ordered execution in each persistent level is queried in hierarchical order. Specifically, based on the target composite key, a query is performed in any ordered run at any persistent level to obtain the learning index file corresponding to that ordered run. Starting from the top-level piecewise linear approximation model of the learning index file, the linear segment corresponding to the target composite key in the current layer's piecewise linear approximation model is determined. Based on the linear segment, the predicted position of the target composite key in the current layer is predicted. Based on the predicted position and a preset error parameter, the linear segment corresponding to the target composite key in the next layer's piecewise linear approximation model is determined until the predicted position of the target composite key at the bottom layer is obtained. Based on the predicted position of the target composite key at the bottom layer, the state value corresponding to the target composite key is obtained from the value file of any ordered run and returned to the requester.

5. The blockchain state data access and crash recovery method based on learning indexes as described in claim 4, characterized in that, The system queries the ordered executions in each persistent level in hierarchical order, specifically including: Get the currently ordered execution in the current persistent hierarchy; Based on the Bloom filter corresponding to the current ordered operation, determine whether the target composite bond exists in the current ordered operation; If the Bloom filter determines that the target composite key is not in the current ordered run, it skips the current ordered run; otherwise, it queries the current ordered run based on the target composite key.

6. The blockchain state data access and crash recovery method based on learning indexes as described in claim 5, characterized in that, The Bloom filter corresponding to any ordered run is constructed based on the following method: Determine the target false positive rate corresponding to any of the ordered operations; The length of the bit array and the number of hash functions are determined based on the target false positive rate and the number of composite key-value pairs in any ordered operation. Initialize the array elements in the bit array to 0, and obtain the corresponding number of hash functions; Traverse any ordered composite key-value pair, and for the current composite key-value pair, use each hash function to calculate the hash value corresponding to the composite key in the current composite key-value pair, and use the hash value corresponding to the composite key as the index of the bit array, and set the array element at the corresponding index to 1; The bit array, the parameters of the corresponding number of hash functions, the number of composite key-value pairs in any ordered run, and the target false positive rate are saved as the metadata of the Bloom filter corresponding to any ordered run.

7. The blockchain state data access and crash recovery method based on learning indexes as described in claim 6, characterized in that, Determining whether the target composite bond exists in the current ordered process based on the Bloom filter corresponding to the current ordered process specifically includes: The hash value of the target composite key is calculated based on multiple hash functions of the Bloom filter corresponding to the current ordered operation, resulting in multiple bit array indices; Retrieve the array elements at the multiple bit array indices; If the array element at any array index is 0, then the target composite key is determined not to be in the current ordered process; If all the array elements at the multiple bit array indices are 1, then the target composite key is determined to exist in the current ordered process.

8. The blockchain state data access and crash recovery method based on learning indexes as described in claim 3, characterized in that, The method further includes: In response to the source tracing query request, obtain the source status address and block height range to be traced; A first boundary composite key is generated based on the state address to be traced and the lower bound of the block height interval, and a second boundary composite key is generated based on the state address to be traced and the upper bound of the block height interval. In the memory level, query composite key value pairs whose composite keys are located within the query range defined by the first boundary composite key and the second boundary composite key; If a matching composite key-value pair exists in the memory level, then obtain the Merkel path corresponding to the matching composite key-value pair. The system sequentially queries ordered runs in each persistent level to determine whether the composite key range of the corresponding ordered run intersects with the query range. For ordered runs where the composite key range intersects with the query range, it predicts the predicted positions of the first boundary composite key and the second boundary composite key at the lowest level based on the learning index file corresponding to the corresponding ordered run. Based on a preset error parameter and the predicted positions of the first boundary composite key and the second boundary composite key at the lowest level, it obtains the candidate ranges corresponding to the first boundary composite key and the second boundary composite key, and obtains the upper and lower bounds of the index from the candidate ranges corresponding to the first boundary composite key and the second boundary composite key. It reads the composite key value pairs in the corresponding ordered run whose composite keys are located between the upper and lower bounds of the index as the hit composite key value pairs, and generates the Merkle path corresponding to the hit composite key value pairs based on the Merkle file corresponding to the corresponding ordered run. The upper bound of the index is the first composite key greater than or equal to the first boundary composite key in the candidate ranges corresponding to the first boundary composite key and the second boundary composite key, and the lower bound of the index is the last composite key less than or equal to the second boundary composite key in the candidate ranges corresponding to the first boundary composite key and the second boundary composite key. Obtain the ordered runs of the currently valid root hash list references as a total set, remove the ordered runs that have been queried and generated Merkel paths from the total set, and obtain the root hashes of the remaining ordered runs in the total set; The verification proof is composed of the state values ​​of all the hit composite key-value pairs, the Merkel paths corresponding to the hit composite key-value pairs, and the remaining ordered root hashes in the total set. The state values ​​of all the hit composite key-value pairs and the verification proof are then returned to the requester.

9. The blockchain state data access and crash recovery method based on learning indexes as described in claim 1, characterized in that, The method further includes: When the blockchain system crashes and restarts, the hierarchical files of the memory level and each persistent level, as well as the root hash list, are reopened; the hierarchical files of the memory level and each persistent level, as well as the root hash list, are stored in persistent memory; the hierarchical files include the value file, the learned index file, the Merkle file, and the log file of the corresponding level; Based on the most recently valid root hash list, the most recently checked point, and the log files of each level, incomplete transactions since the most recently checked point are identified. These incomplete transactions include persistent level merge transactions involving new ordered runs of Merkle file root hashes that are not referenced by the most recently valid root hash list. The most recently checked point is recorded after the ordered runs of composite key-value pairs in the memory level have been written to the first persistent level, and the corresponding value file, learned index file, and Merkle file have been constructed, and the root hash list has been updated. Replay all unfinished transactions since the last checkpoint.

10. A blockchain state data access and crash recovery system based on a learning index, characterized in that, The system is used to execute the blockchain state data access and crash recovery method based on a learning index as described in any one of claims 1 to 9.

Images