Data reading and writing method of key-value database

Abstract

The application relates to the field of databases and discloses a data read-write method of a key-value database. The application introduces the concept of entity nodes and virtual nodes in a binary hash tree. For a parent node, if any child node of the parent node is a virtual node, the parent node is a virtual node, and the value of the parent node is equal to the value of the non-empty child node, that is, the hash value of the parent node as a virtual node is not calculated or stored. The application also adopts a three-level memory architecture to store different levels of nodes of the binary hash tree and designs various hardware acceleration structures. The application significantly reduces the number of hash calculations and the amount of verification data, improves the data writing and reading efficiency, and guarantees the data integrity.

Description

Technical Field

[0001] This application relates to the field of databases, and in particular to read and write technologies for key-value databases. Background Technology

[0002] This section is intended to provide background or context for understanding the implementation of this application and is for reference only. It should not be construed as an admission by the applicant that this section pertains to prior art that was disclosed before the filing date of this application.

[0003] With the continuous expansion of data volume, key-value databases have been widely used in large-scale distributed systems due to their simple and efficient data model and excellent performance. A key-value database is a type of NoSQL database where data is stored and accessed in the form of key-value pairs.

[0004] In key-value database systems, data integrity verification is a crucial issue. Traditional data integrity verification methods typically employ hashing techniques, which involve performing hash operations on the data content to generate a fixed-length hash value. By comparing the hash values ​​of the data, it is possible to detect whether the data has been tampered with. However, when verifying large amounts of data, calculating and storing the hash value of each individual record incurs significant storage overhead and computational burden.

[0005] To optimize the efficiency of data integrity verification, the industry commonly uses data structures such as binary hash trees or Merkle trees. A binary hash tree is a tree-like data structure where leaf nodes store the hash value of a data block, while non-leaf nodes store the combined hash value of the hash values ​​of their child nodes. This hierarchical structure enables systems to quickly verify the integrity of large amounts of data.

[0006] However, existing data verification schemes based on binary hash trees have the following technical problems: when data is continuously written, the entire hash tree needs to be updated frequently, resulting in high computational overhead.

[0007] Therefore, how to reduce the maintenance cost of hash trees and optimize verification efficiency while ensuring the reliability of data integrity verification is a technical problem that urgently needs to be solved in current key-value database systems. Summary of the Invention

[0008] The purpose of this application is to provide a data reading and writing method for key-value databases, which can reduce the maintenance cost of hash trees and optimize verification efficiency while ensuring the reliability of data integrity verification.

[0009] This application discloses a method for writing data to a key-value database, including:

[0010] The processor receives a write request, which contains key-value pair data;

[0011] The processor writes the key-value pair data as a data payload into the first memory;

[0012] The processor calculates the hash value of the data payload and writes the hash value as the highest leaf node with the largest current sequence number into the binary hash tree stored in the fourth memory. In the binary hash tree, the sequence numbers of each leaf node are sequentially increased. Leaf nodes with sequence numbers not greater than the highest leaf node are physical nodes, and their values ​​are the hash values ​​of the corresponding data payload. Leaf nodes with sequence numbers greater than the highest leaf node are virtual nodes, and their values ​​are preset values.

[0013] The processor determines whether the parent nodes at each level corresponding to the highest leaf node are located within a complete entity subtree in the direction from the leaf node to the root node, wherein the complete entity subtree is a subtree that does not contain virtual nodes;

[0014] In response to determining that the parent node is located within a complete entity subtree, the parent node is treated as an entity node, the hash value of the parent node is calculated based on the hash values ​​of the two child nodes of the parent node, and the hash value of the parent node is stored in the fourth memory;

[0015] In response to determining that the parent node is not located in the complete entity subtree, the parent node is treated as a virtual node. When it is necessary to obtain the hash value of the parent node, the value of the child node of the parent node that is not a preset value is directly used as the hash value of the parent node.

[0016] In a preferred embodiment, it also includes:

[0017] The binary hash tree forms multiple complete entity subtrees from left to right, and the processor adds the root node of each complete entity subtree to the root set;

[0018] The processor obtains the global root hash value by calculating the hash value of each node in the root set from right to left.

[0019] The processor calculates the verification hash set, specifically including: starting from the leaf node, in order from low level to high level, if the current node is on the left, it searches for the hash value of its right-side paired node; if the current node is on the right, it searches for the hash value of its left-side paired node; and adds the hash value of the found paired node to the verification hash set.

[0020] The processor returns data to the requesting device for verifying the write operation. This data includes the global root hash value, the root hash set, and the verification hash set.

[0021] In a preferred embodiment, the requesting device verifies the correctness of the write by:

[0022] A verification target node is calculated based on the verification hash set and the hash value of the leaf node, and the verification target node should be an element in the root set; and

[0023] The global root hash value can be obtained by calculating the hash value of each element in the root set level by level.

[0024] In a preferred embodiment, the fourth memory includes a second memory and a third memory; the first memory and the second memory are non-volatile memories; the access speeds of the first memory, the second memory, and the third memory increase sequentially.

[0025] The nodes of the binary hash tree from level 0 to level N are stored in the second memory, wherein the nodes from level 0 to level N are combined into at least one complete subtree, and each complete subtree is stored in a page of the second memory, the storage space of the complete subtree being the same as the size of the page; the nodes of the binary hash tree from level N+1 and above are stored in the third memory; wherein N is a preset positive integer.

[0026] In a preferred embodiment, in each page of the second memory, nodes of each level of a complete subtree are stored layer by layer from level 0 to level N; the index of the leaf node is directly used as its storage location in level 0; and a one-to-one correspondence between node index and storage location is maintained in each level.

[0027] The processor includes multiple parallel long SHA256 calculation circuits and multiple parallel short SHA256 calculation circuits. The long SHA256 calculation circuits are used to calculate the hash value of each leaf node in a page in parallel according to the data payload. The short SHA256 calculation circuits are used to perform pairwise calculations on each leaf node in a page to obtain the parent node of the first layer. Then, through multi-path parallel calculations layer by layer from bottom to top, the calculations are finally converged into a single path to obtain the hash value of the root node of the complete subtree.

[0028] In a preferred embodiment, it also includes:

[0029] For a complete subtree that goes from empty to half full, use virtual nodes to calculate the virtual root hash value of the complete subtree;

[0030] For a complete subtree that goes from half-full to full, after completing it into a full tree, discard the original virtual nodes and recalculate the hash value from the root of the entire complete subtree.

[0031] In a preferred embodiment, when writing key-value pairs of data in bulk, the following pipeline approach is used for processing:

[0032] In the first pipeline stage, 64 key-value pairs are read to form a complete subtree;

[0033] In the second pipeline stage, 64 parallel SHA256 computing circuits are used to compute the hash values ​​of the 64 key-value pairs as leaf nodes;

[0034] In the third pipeline stage, 32 parallel SHA256 computing circuits are used to perform pairwise calculations on the leaf nodes, and the hash values ​​of each level node of the complete subtree are obtained by sequentially performing 16-way parallel computing, 8-way parallel computing, 4-way parallel computing, 2-way parallel computing and single-way computing.

[0035] In the fourth pipeline stage, the node hash values ​​of the complete subtree are written into a 4KB page of the second memory.

[0036] This application also discloses a data reading method for a key-value database. Key-value pairs are stored as data payloads with incrementing sequence numbers in a first memory, and a fourth memory stores a binary hash tree. In this binary hash tree, each leaf position corresponds to a sequence number. Leaf nodes with sequence numbers not greater than the largest sequence number of the data payload in the first memory are physical nodes, and their values ​​are the hash values ​​of the corresponding data payloads. Leaf nodes with larger sequence numbers are virtual nodes, and their values ​​are preset values. For a parent node, if any of its child nodes are virtual nodes, then the parent node is a virtual node, and its value is equal to the value of its child nodes that do not have preset values. Otherwise, the parent node is a physical node, and its value is the hash result of its two child nodes. The method includes:

[0037] The processor receives a read request from the requesting device and reads the requested data payload from the first memory. The leaf node corresponding to the data payload is the target leaf node.

[0038] The processor generates a verification hash set for the target leaf node, wherein the elements in the verification hash set are paired nodes at each level of the path from the target leaf node to the verification target node in the binary hash tree; the verification target node is the root node of the smallest complete entity subtree containing the target leaf node; the complete entity subtree is a subtree that does not contain virtual nodes;

[0039] The processor returns the requested data payload and verification hash set to the requesting device.

[0040] In a preferred embodiment, the requesting device verifies the correctness of the read data by:

[0041] Obtain the global root hash value and the root set, wherein the global root hash value is the hash value of the root node of the binary hash tree, and the root set includes the root nodes of multiple complete entity subtrees in the binary hash tree from left to right;

[0042] The target node for verification is calculated based on the hash value of the verification hash set and the target leaf node. The target node for verification should be an element in the root set.

[0043] The global root hash value can be obtained by calculating the hash value of each element in the root set level by level.

[0044] In a preferred embodiment, the processor determines the target verification node through the following steps:

[0045] Starting from the most significant bit of the index of the highest leaf node in the binary hash tree, check its corresponding complete entity subtree;

[0046] If the index of the target leaf node is less than or equal to the total number of nodes in the current complete entity subtree, then the root node of the current complete entity subtree is used as the verification target node.

[0047] If the index of the target leaf node is greater than the total number of nodes in the current complete entity subtree, then the total number of nodes in the current complete entity subtree is subtracted from the index of the target leaf node, and the next complete entity subtree corresponding to the valid bit is checked until a complete entity subtree containing the target leaf node is found.

[0048] In a preferred embodiment, the step of the processor generating the verification hash set includes:

[0049] Convert the index of the target leaf node into binary;

[0050] Establish a hierarchical mapping, with the least significant bit corresponding to level 0;

[0051] Starting from the target leaf node, process layer by layer: detect whether the current node is an odd-numbered left subtree or an even-numbered right subtree. If it is an odd-numbered left subtree, calculate the corresponding even-numbered right subtree index and add it to the verification hash set. If it is an even-numbered right subtree, calculate the corresponding odd-numbered left subtree index and add it to the verification hash set, until the layer where the verification target node is located is reached.

[0052] In a preferred embodiment, the fourth memory includes a second memory and a third memory; the first memory and the second memory are non-volatile memories; the access speeds of the first memory, the second memory, and the third memory increase sequentially.

[0053] The nodes of the binary hash tree from level 0 to level N are stored in the second memory, wherein the nodes from level 0 to level N are combined into at least one complete subtree, and each complete subtree is stored in a page of the second memory, the storage space of the complete subtree being the same as the size of the page; the nodes of the binary hash tree from level N+1 and above are stored in the third memory; wherein N is a preset positive integer.

[0054] In a preferred embodiment, the processor determines the path from the target leaf node to the verification target node through the following index-based hardware computation:

[0055] Convert the target leaf index to binary to serve as the index for level 0;

[0056] In the processing of each layer:

[0057] The node type is determined by checking the least significant bit. If the least significant bit is 0, it is a left node; if it is 1, it is a right node.

[0058] For a left node, find its right-side matching node by incrementing by 1; for a right node, find its left-side matching node by decrementing by 1.

[0059] Obtain the index of the previous level by right shifting by 1 bit;

[0060] Repeat the above process until the layer containing the target node is reached.

[0061] In a preferred embodiment, the processor employs a hardware pipeline architecture to process read requests for multiple leaf nodes in parallel, including:

[0062] The first processing resource computes the indexes of multiple target leaf nodes at each level in parallel using bitwise operations;

[0063] The second processing resource is used to read nodes from level 0 to level 6 in multiple 4KB pages from the second memory in parallel and extract nodes from the verification hash set;

[0064] The third processing resource is used to read nodes from the verification hash set of multiple target leaf nodes from the third memory in parallel, from the 7th layer to the root node.

[0065] The fourth processing resource includes multiple parallel SHA256 computation circuits for parallel computation of hash values ​​for multiple verification paths and returning a complete set of verification hashes.

[0066] In a preferred embodiment, the processor includes:

[0067] Hardware acceleration circuitry for calculating SHA256 hash values ​​includes:

[0068] Multiple parallel long SHA256 computing circuits are used to calculate the hash value of the data payload;

[0069] Multiple parallel short SHA256 computation circuits are used to compute the node hash values ​​on the verification path;

[0070] The processor supports parallel processing using multiple SHA256 computing circuits during the verification process.

[0071] This application also discloses a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the method described above.

[0072] This application also discloses a computer program product including computer-executable instructions that, when executed by a processor, implement the steps in the method described above.

[0073] In the embodiments of this application, by introducing the concepts of physical nodes and virtual nodes into the binary hash tree, nodes with indices no greater than the highest leaf node are designated as physical nodes, while those with larger indices are designated as virtual nodes. The hash values ​​of the parent nodes of virtual nodes are neither calculated nor stored, significantly reducing the number of hash calculations. This innovation avoids the problem of traditional Merkle trees requiring hash calculations for all nodes, improving data writing efficiency while ensuring the reliability of data verification.

[0074] Furthermore, by identifying multiple complete entity subtrees from left to right and adding their root nodes to the root set, and then calculating the hash values ​​of the nodes in the root set level by level from right to left to obtain the global root hash value, while generating a verification hash set based on paired nodes, a rapid verification process for data writing is achieved. This hierarchical verification mechanism not only reduces the computational complexity of verification but also provides fine-grained data integrity guarantees.

[0075] Furthermore, by designing a dual verification mechanism (verifying that the target node must exist in the root set and that the global root hash value can be calculated from the root set), a strict correctness guarantee is provided for data write operations. This verification method can effectively prevent data tampering, while the verification process involves low computational cost and high efficiency.

[0076] Furthermore, by adopting a hierarchical storage strategy, nodes from level 0 to level N of the binary hash tree are stored in the second memory, while higher-level nodes are stored in the faster third memory. The complete subtree is then mapped to storage page sizes within the second memory, achieving a rational allocation of storage resources. This hierarchical storage architecture fully considers the access characteristics of nodes at different levels, optimizes storage space utilization, and improves data access performance.

[0077] Furthermore, by designing a processor architecture that includes multiple parallel long SHA256 computation circuits (for calculating leaf node hash values) and short SHA256 computation circuits (for calculating parent node hash values), combined with a layered node storage structure, efficient parallel computation of hash values ​​was achieved. This design significantly improves hash computation performance and reduces data write latency.

[0078] Furthermore, by employing a differentiated processing strategy for complete subtrees with varying fill levels (using virtual nodes to calculate the virtual root hash value when empty to half-full, and recalculating the hash value of the entire subtree when half-full to fully full), a good balance is achieved between computational overhead and storage efficiency. This dynamic adjustment strategy avoids frequent hash recalculations while ensuring data integrity.

[0079] Furthermore, by designing a four-stage pipeline and employing varying degrees of parallel processing at each stage, efficient batch read and write operations were achieved. This pipeline architecture fully utilizes hardware parallel computing resources, significantly improving system throughput.

[0080] Furthermore, by optimizing the generation strategy for verification data, only the set of verification hashes along the path from the target leaf node to the root node of the smallest complete entity subtree is returned, significantly reducing the amount of data required for verification. This approach reduces network transmission overhead, improves data reading efficiency, and ensures verification reliability.

[0081] The various technical features disclosed in the above-described invention, the various technical features disclosed in the following embodiments and examples, and the various technical features disclosed in the accompanying drawings can be freely combined to form various new technical solutions (all of which should be considered as having been recorded in this specification), unless such a combination of technical features is technically infeasible. For example, in one example, feature A+B+C is disclosed, and in another example, feature A+B+D+E is disclosed. Features C and D are equivalent technical means that serve the same function, and technically only one needs to be used; it is impossible to use both simultaneously. Feature E can be technically combined with feature C. Therefore, the solution A+B+C+D should not be considered as having been recorded because it is technically infeasible, while the solution A+B+C+E should be considered as having been recorded. Attached Figure Description

[0082] Figure 1 This is an example of an improved hash tree according to one embodiment of this application;

[0083] Figure 2 This is an example of adding an H13 node to a hash tree that already has 12 elements, according to one embodiment of this application;

[0084] Figure 3This is an example of adding an H14 node to a hash tree that already has 13 elements, according to one embodiment of this application;

[0085] Figure 4 This is an example of reading nodes H7 and H12 in a hash tree with 13 elements according to one embodiment of this application;

[0086] Figure 5 This is an example of reading nodes H7 and H10 in a hash tree with 16 elements according to one embodiment of this application;

[0087] Figure 6 This is a schematic diagram of parallel processing in a hardware pipeline according to an embodiment of this application;

[0088] Figure 7 This is an example table showing the correspondence between different leaf nodes and the fully real tree at each level according to an embodiment of this application;

[0089] Figure 8 This is an example of parallelized insertion of multiple leaves according to an embodiment of this application;

[0090] Figure 9 This is a schematic diagram of two 64-trees computed in parallel within a single PE (processing unit) in a pipeline manner according to an embodiment of this application;

[0091] Figure 10 This is a schematic diagram of the overall database design according to an embodiment of this application. Detailed Implementation

[0092] In the following description, many technical details are presented to help the reader better understand this application. However, those skilled in the art will understand that the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0093] Explanation of some concepts:

[0094] DDR (Double Data Rate) is a memory technology that enables data transmission on both the rising and falling edges of the clock signal, thus achieving twice the data transfer rate at the same clock frequency. This technology is widely used in computer system memory and graphics cards.

[0095] QLC (Quad-Level Cell) is a flash memory technology where each cell can store 4 bits of data (i.e., 16 different voltage states). Compared to traditional SLC, MLC, and TLC storage technologies, it has higher storage density and lower cost, but it also faces challenges in terms of durability and lower write speed.

[0096] SCM (Storage Class Memory) is a new type of storage technology that combines the characteristics of memory and storage. It combines the high-speed access characteristics of DRAM with the non-volatile characteristics of flash memory, and can retain data when power is lost and provide access speeds close to those of memory, bridging the performance gap between traditional memory and storage devices.

[0097] LSB (Least Significant Bit) refers to the rightmost bit of a binary number, which represents the smallest unit of value.

[0098] TPS (Transactions Per Second) is a performance metric that measures the processing capacity of a system. It represents the number of transactions that a system can complete per second and is often used to evaluate the performance of transaction-intensive applications such as databases and payment systems.

[0099] The following is a summary of some of the innovative aspects of the implementation methods of this application.

[0100] This invention introduces the concepts of entity nodes and virtual nodes into a binary hash tree. In this binary hash tree, each leaf position corresponds to a sequence number. Leaf nodes with sequence numbers no greater than the maximum sequence number of the data payload in the key-value database are entity nodes, and their values ​​are the hash values ​​of the corresponding data payloads. Leaf nodes with larger sequence numbers are virtual nodes, and their values ​​are preset values. Preset values ​​can generally be empty or other specific values, as long as they can be distinguished from normal hash values. For a parent node, if any of its child nodes are virtual nodes, then the parent node is a virtual node, and its value is equal to the value of its non-empty child node; otherwise, the parent node is an entity node, and its value is the hash result of its two child nodes. The hash value of the parent node that is a virtual node is neither calculated nor stored. That is, only parent nodes within a complete entity subtree need to calculate a new hash value; parent nodes within a non-complete entity subtree can directly copy the values ​​of their non-empty child nodes. This mechanism significantly reduces the number of hash calculations and improves write efficiency.

[0101] This invention also proposes an efficient data verification scheme: a mechanism is designed to form multiple complete entity subtrees from left to right and establish a root set; the amount of verification data is reduced by generating a minimum verification path (a path containing only the target leaf node to the root node of the minimum complete entity subtree); and a dual verification mechanism is adopted (verifying that the target node must be an element of the root set and that the global root hash value can be calculated from the root set). These designs reduce verification overhead while ensuring data integrity.

[0102] The embodiments of the present invention also propose an optimized storage architecture: a three-level memory architecture is adopted, which allocates nodes of different levels to memory of different speeds according to access characteristics; the complete subtree is mapped to the storage page size of the solid-state drive to optimize storage space utilization; this hierarchical storage strategy improves data access efficiency.

[0103] This invention also proposes a high-performance hardware acceleration design, including: designing multiple parallel long and short SHA256 calculation circuits; employing a four-stage pipeline architecture to handle batch reads and writes; and implementing hardware-based fast index calculations, etc. These hardware optimizations significantly improve system performance.

[0104] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0105] In one embodiment, an implementation of a verifiable key-value database includes:

[0106] An improved Merkle Shrubs hash tree structure is implemented in hardware, achieving maximum storage bandwidth and lifetime based on sequential writes and random reads. Data payloads are written to the first memory in a journal / log format, and to the second memory in a transaction-with-ordinal format. A portion of the Merkle Shrubs is written to the second memory, and the remainder to the third memory. Optionally, the first memory can use QLC flash memory, the second memory can use SCM, and the third memory can use DDR. Other types of memory can also be used for the first, second, and third memories, provided the following conditions are met:

[0107] The first memory is non-volatile memory, the second memory is non-volatile memory, and the third memory can be volatile memory (or non-volatile memory).

[0108] In terms of access speed, the first memory < the second memory < the third memory.

[0109] The design of the above memory types takes into account the best trade-off between performance and cost.

[0110] Real-time verification of write transactions is achieved as follows: After key-value data is written, the database returns a root hash value and a verification hash value. The verification method is to concatenate the hash value of the written data with the verification hash value, and calculate the root hash value using a publicly available fixed algorithm. Subsequent read operations will not change the root hash value returned by the database; it will remain the same as the root hash value at the time the write transaction was completed. All read operations can be verified using this root hash value.

[0111] With the support of a Trusted Execution Environment (TEE) and certified firmware, the verification of a write transaction can be simplified to: the TEE returns a signed verification packet containing a verified timestamp, a monotonically increasing sequence number, a transaction hash, and a transaction verification hash.

[0112] Real-time verification of read transactions is achieved as follows: Before a read operation, the client first obtains the latest root hash value pushed by the database. If there is no write operation, this root hash value remains unchanged. After reading the data, the database returns the data content and a verification hash value. The verification method is to concatenate the data's hash value with the verification hash value, and calculate the root hash value using a publicly available fixed algorithm.

[0113] This solution addresses a core problem in traditional approaches: during database read / write transactions, the search and calculation speed for hash value verification is significantly slower than the read / write speed of the data payload. This solution provides a parallel computing approach that frees the database's transaction throughput (TPS) from storage bandwidth limitations, allowing it to increase linearly with the increase of SHA256 parallel computing units.

[0114] Meanwhile, this solution provides a cost-effective TEE solution: it includes a TEE computing core with certified firmware, protected DDR memory, and protected non-volatile memory (NVM). Large data payloads are stored in a low-cost NVM outside the TEE.

[0115] This implementation scheme is mathematically very secure: it is difficult for a malicious attacker to forge a set of verification hashes and a root hash to pass the verification of the publicly available fixed algorithm without actually writing transactions. Similarly, in read operations, it is also difficult to forge a string of verification hashes for specific returned data to obtain the correct root hash value through the publicly available fixed algorithm.

[0116] This embodiment employs an improved Merkle Shrubs hash tree structure, which requires fewer intermediate nodes for computation and storage than a traditional Merkle Tree, thereby improving computational efficiency. (Reference) Figure 1 The specific implementation is as follows:

[0117] In this hash tree structure, the leaf node Hx represents the SHA256 hash value of the data payload of the transaction with sequence number x. Insert and Update transactions correspond to the hash value Hx of the standard key-value pair {K:V} payload, while Delete and Hide transactions correspond to the hash value of the specially crafted {K:empty} payload. This structural design ensures that the entire improved Merkle Shrubs tree only supports two basic operations: append and read.

[0118] The transaction sequence number follows a monotonically increasing pattern, accumulating continuously as transactions in the database increase. Here, 'x' in Hx represents the transaction sequence number. Intermediate nodes are represented as H1-2, which represents the hash value obtained by concatenating H1 and H2 and calculating their SHA256 hash.

[0119] For ease of expression, the following two basic operators are defined in this application:

[0120] The "→" operator indicates a fixed algorithm that concatenates the elements in the SHA256 set on the left in order and calculates the SHA256 value step by step, resulting in the value on the right.

[0121] The "+{}" operator means adding the elements on the left to the sorted set on the right in sequence.

[0122] When writing transaction number x, the user writes data Data(x), and the database returns the global root hash value GlobalRoot(x), the root hash set RootSet(x), and the verification hash set ProofSet(x). The user verifies Hx + {ProofSet(x)} → W, where W should be an element in RootSet(x). To improve security, it is also necessary to verify RootSet(x) → GlobalRoot(x), especially when x is odd, because Hx itself is an element of RootSet(x) in this case. In this embodiment, the user is the requesting device. The requesting device is generally a computer device containing a processor (connected to memory, hard disk, and communication interface via a bus), used to issue requests, obtain feedback results from the processor, and verify the correctness of the feedback results.

[0123] When reading a transaction with serial number y (y < x), the user holds GlobalRoot(x) and RootSet(x), and the database returns Data(y) and ProofSet(y). The user verifies Hy + {ProofSet(y)} → Z through calculation, and Z should be an element in RootSet(x). It is also necessary to verify the calculation process of RootSet(x) → GlobalRoot(x), especially when y is relatively close to x (such as log2(y - x) < 10), because the anti-interference ability is relatively low at this time. The global root hash value GlobalRoot(x) and the root set RootSet(x) used for data reading verification can be obtained from reliable sources. For example, the global root hash value and the root set feedback during the previous write (it is necessary to ensure that the data read this time has been in the key-value database during this write), or they can be obtained from other reliable third parties, or they can be provided by the key-value database during reading.

[0124] This embodiment adopts a full-tree design from left to right. For the convenience of hardware acceleration implementation, it is required to maintain an independent one-dimensional array ArrayIndex (array index) for each layer. In specific implementation, for the 64-node full tree from layer L0 to L6, SCM storage units with a size of 4KB are used. This storage method is convenient for directly moving data to DDR without displacement operations.

[0125] The following further describes the implementation of the write operation of the improved Merkle Shrubs hash tree. The following is illustrated by specific examples:

[0126] As Figure 2 shown, when appending and writing H13:

[0127] The system generates a virtual global root hash GlobalRoot(13), that is, the virtual H1-16. The corresponding RootSet(13) contains three elements: H1-8, H9-12, and H13. ProofSet(13in13) is its own RootSet. In the embodiments of this application, (a in b) represents the a-th node in the hash tree with a total of b nodes. For quick verification, it is only necessary to confirm that H13 exists in RootSet(13); for complete verification, it is necessary to calculate SHA256(H1-8, SHA256(H9-12, H13)) and verify that it is equal to GlobalRoot(13). Since 13 is an odd serial number, its RootSet is composed of a series of the highest concrete tree roots from left to right and the node itself.

[0128] As Figure 3 shown, when appending and writing H14:

[0129] The system generates a virtual global root hash GlobalRoot(14), which is a virtual H1-16. The corresponding RootSet(14) contains: H1-8, H9-12, and H13-14. Its ProofSet(14in14) contains only H13. For fast verification, SHA256(H13,H14) is calculated and verified to be equal to H13-14; for full verification, SHA256(H1-8,SHA256(H9-12,SHA256(H13,H14))) is calculated and verified to be equal to GlobalRoot(14). Since 14 is an even number, its RootSet consists of a series of the highest entity tree roots from left to right, while its ProofSet is the RootSet corresponding to the number minus one.

[0130] This embodiment introduces the concept of virtual nodes and replaces the SHA256 calculation for virtual nodes with a simple copy operation. This is fundamentally different from the traditional Merkle Tree implementation: in the traditional implementation, each upper-level node is a concrete node. For example, when writing H13, H13-14 must exist as concrete nodes and complete the actual copy storage, and the values ​​of H13-14 need to be overwritten when writing H14. By introducing virtual nodes, the overwrite operation on the tail node of each level can be avoided.

[0131] After the append operation is completed, several power-law entity trees with base 2 are formed from left to right. The root nodes of these entity trees are combined in order from right to left to form a virtual root tree. This design significantly reduces storage and computational overhead.

[0132] The following further explains the implementation of the improved Merkle Shrubs hash tree read operation:

[0133] refer to Figure 4 (The light-colored nodes (such as H14 to H18, H13-14, H15-16, H17-18, H13-16, H9-16) are virtual nodes, and H1-16 is the virtual root of the tree.) The process of reading H7 when the highest node is H13 is as follows:

[0134] GlobalRoot(13) is obtained by calculating SHA256(H1-8, SHA256(H9-12, H13)), and the corresponding RootSet(13) contains H1-8, H9-12, and H13. The system determines ProofTarget(7in13) as H1-8 and generates ProofSet(7in13) containing H1-4, H5-6, H7, and H8. During fast verification, SHA256(H1-4, SHA256(H5-6, SHA256(H7, H8))) is calculated and verified to be equal to H1-8. During full verification, based on fast verification, it is further verified whether GlobalRoot(13) can be calculated from RootSet(13).

[0135] refer to Figure 4 When the highest node is H13, the process of reading H12 is as follows:

[0136] GlobalRoot(13) is obtained by calculating SHA256(H1-8, SHA256(H9-12, H13)), and the corresponding RootSet(13) contains H1-8, H9-12, and H13. The system determines ProofTarget(12in13) as H9-12 and generates ProofSet(12in13) containing H9-10, H11, and H12. During fast verification, SHA256(H9-10, SHA256(H11, H12)) is calculated and verified to be equal to H9-12. During full verification, based on fast verification, it is further verified whether GlobalRoot(13) can be calculated from RootSet(13).

[0137] The generation rule for the verification hash set (ProofSet) is as follows: starting from the target leaf node, search for paired nodes in each level in order from the lowest to the highest level. If the current node is on the left, search for its paired node to the right; if it is on the right, search for its paired node to the left.

[0138] The ProofTarget is defined as the root node of the complete entity tree that the target reads leaves from, which is located in the RootSet with the highest appended leaves.

[0139] The verification process (Proof) proceeds as follows: starting from the leaf node read, the system builds upwards layer by layer. At each layer, a matching node is found based on the node's position (left finds right, or right finds left). The matching nodes are then concatenated in order, and their SHA256 hash is calculated to obtain the node for the next layer. This process is repeated until the root node of a complete entity tree is reached.

[0140] The verification mechanism for virtual nodes is explained further below:

[0141] Taking the case where the tree has accumulated to H13 and H7 needs to be verified as an example: When the tree has accumulated to H13, the GlobalRoot of the entire tree is a virtual H1-16. It should be noted that the real H1-16 will only appear when H16 is also written into the tree. Since H14 to H16 have not yet been written, the RootSet corresponding to H13 at this time contains three elements: H1-8, H9-12, and H13.

[0142] The calculation process for virtual root node H1-16 adopts a hierarchical virtual node processing method:

[0143] First, the virtual H1-16 is calculated as SHA256(H1-8 + virtual H9-16); where virtual H9-16 is calculated using SHA256(H9-12 + virtual H13-16); virtual H13-16 is calculated using SHA256(virtual H13-14 + virtual H15-16); virtual H13-14 is equal to SHA256(H13 + virtual H14), since H14 is empty, so H13-14 is directly equal to H13; virtual H15-16 is calculated using SHA256(virtual H15 + virtual H16), since both H15 and H16 are empty, so H15-16 is also empty.

[0144] In summary, the final calculation of the virtual H1-16 simplifies to: SHA256(H1-8+SHA256(H9-12+H13)). This calculation process actually reflects the SHA256 calculation of the path from the rightmost node H13 to the virtual root. Compared to the original Merkle Tree structure, this design significantly reduces the number of SHA256 calculations and improves computational efficiency.

[0145] The calculation process of the verification hash set (ProofSet) is further explained below. Two examples are used to illustrate this: when the highest node is H16, H7 and H10 are read.

[0146] refer to Figure 5 When reading H7: At this time, GlobalRoot(16) is the entity's H1-16, RootSet(16) only contains H1-16, and ProofTarget(7in16) is H1-16. The system-generated ProofSet(7in16) contains H1-4, H5-6, H7, H8 and H9-16.

[0147] refer to Figure 5When reading H10: GlobalRoot(16) and RootSet(16) remain unchanged, and ProofTarget(10in16) is H1-16. The system-generated ProofSet(10in16) contains H9, H10, H11-12, H13-16 and H1-8.

[0148] To facilitate hardware implementation, this embodiment adopts a node representation method based on each level's index. Taking the calculation of the ProofSet of the target leaf H7 as an example, the index calculation process from left to right is explained in detail:

[0149] The target leaf index x = 7 (binary 111b) has an index of 111b in the L0 layer. The specific calculation steps are as follows:

[0150] Step 1: In layer L0, 111b is the left-left node, and we need to find the right node. 111b + 1 = 1000b = 8. Therefore, the found right node is H8. Push H8 into the ProofSet. (The method to determine if a node is left is that its LSB is 0, and the method to determine if a node is right is that its LSB is 1. Finding the right node from left is an increment operation, and finding the left node from right is a decrement operation. H8 is represented by its index of 8 in layer L0. This design is for hardware implementation convenience.)

[0151] Step 2: In layer L0, the left and right nodes are concatenated to calculate the SHA256, and then the position of this calculation result in layer L1 is found (using the right node). Shifting 1000b one bit to the right gives 100b = 4. (The method for calculating the index of the corresponding intermediate node in the upper layer from the index of the right node of this pair of leaf nodes in layer L0 is simply shifting it one bit to the right. This design is also for the convenience of hardware implementation. The calculation result is H7-8, represented by its index = 4 in layer L1.)

[0152] Step 3: In level L1, 100b is the right node, so we need to find the left node. 100b - 1 = 11b. Therefore, we find the left node H5-6. Push H5-6 into the ProofSet. (Similarly, H5-6 is represented by its index in level L1, which is 11b = 3.)

[0153] Step 4: In layer L1, concatenate the left and right nodes to calculate the SHA256, then find the position of this calculation result in layer L2 (using the right node). Shifting 100b one bit to the right yields 10b.

[0154] Step 5: In L2, 10b is the right node, so we need to find the left node. 10b-1=1b. Push H1-4 into ProofSet (H1-4 is represented by its index = 1b = 1 in L2).

[0155] Step 6: In layer L2, concatenate the left and right nodes to calculate SHA256, and then find the position of this calculation result in layer L3 (using the right node). Shift 10b one bit to the right to get 1b.

[0156] Step 7: In level L3, 1b is the left node, so we need to find the right node. 1b + 1 = 10b. Push H9-16 into a ProofSet (H9-16 is represented by its index in level L3, which is 10b = 2).

[0157] Step 8: In layer L3, concatenate the left and right nodes to calculate SHA256, and then find the position of this calculation result in layer L4 (using the right node). Shift 10b one bit to the right to get 1b.

[0158] Step 9: Level L4 has been reached. This is a real root of the tree at the highest leaf H16, meaning that an element in the RootSet of the highest leaf has been reached, thus successfully pushing the elements into the ProofSet.

[0159] This index-based design significantly simplifies hardware implementation, as all operations can be performed using simple bitwise operations: if the LSB (least significant bit) is 0, it can be determined as the left node (left child) of the bottom-level binary tree; if the LSB is 1, it can be determined as the right node (right child) of the bottom-level binary tree. To find the right node from the left node, use the increment operation; to find the left node from the right node, use the decrement operation; and to perform level transformations, use the right shift operation.

[0160] The performance characteristics of the above embodiments are analyzed as follows:

[0161] Read transaction performance characteristics:

[0162] The complete hash tree from L0 to the root node is calculated and stored in SCM and DDR during data writing. Therefore, the ProofSet returned by a read transaction is read directly from the existing tree structure without recalculation. The number of operations per second (IOPS) of a read transaction is mainly limited by the speed of retrieving the L0 leaf hash values ​​and their corresponding ProofSets from SCM and DDR. Analysis and verification show that the processing volume (TPS) of reading a ProofSet of a transaction is approximately equal to the IOPS of random reads of 4KB SCM pages.

[0163] Write transaction performance characteristics:

[0164] The write process requires calculating the L0 leaf hash value corresponding to the data, as well as the entire tree up to the virtual (or physical) root hash. Specifically: for 10 9 A tree with 10 leaf nodes has a height ranging from L0 to L30; for 10 7A tree with leaf nodes has a height ranging from L0 to L24. In the best case, only one hash value needs to be calculated (the rest are virtual copies of virtual nodes). In the worst case, it needs to be calculated 30 / 24 times from the leaf to the root. The best to worst cases are evenly distributed, and on average, half the number of hash values ​​of the tree height need to be calculated.

[0165] The limiting factors for write transaction TPS (transactions per second) include:

[0166] 1. Parallel processing capability of long SHA256 calculation: Calculates the hash value of a 32-byte leaf node from 1KB of data.

[0167] 2. Parallel processing capability for short SHA256 computation with an average height of half in a tree structure from leaf node to root node (reducing computation by 50% compared to traditional Merkle Tree).

[0168] 3. Data is written using the sequential APPEND method of NAND flash memory.

[0169] 4. Each 64-node hash tree from L0 to L6 is written sequentially to SCM 4KB pages using APPEND.

[0170] 5. The hash tree L7 to the root node is temporarily stored in DDR (bandwidth is significantly higher than the aforementioned factors, so it does not currently constitute a bottleneck). Overall write performance is determined by the minimum bandwidth among the above factors. To achieve K TPS write performance, the system needs:

[0171] -K TPS long SHA256 computing power

[0172] Short SHA256 computing power of -15*K TPS

[0173] This shows that the TPS of write transactions is linearly related to the processing power of the SHA256 parallel computing engine.

[0174] In one embodiment, a hardware pipeline structure that supports parallel reading of multiple leaf nodes is used to process ProofSet retrieval operations of multiple leaf nodes simultaneously.

[0175] The verification process for each leaf node follows a unified pattern:

[0176] First, obtain the node hash value;

[0177] - Read the hash value of the paired node at the same level;

[0178] - Concatenate the two hash values ​​and calculate the hash value of the upper-level node;

[0179] - Repeat the above "read-compute" dependency graph until the predefined root node is reached.

[0180] Based on the storage hierarchy characteristics, the system divides the read operation into two parts that are executed in parallel: the 64 nodes of the L0 to L6 level are stored in 4KB pages of the SCM; the nodes of the L7 level and above are stored in DDR.

[0181] Parallel processing architecture of hardware pipelines, such as Figure 6 As shown, Figure 6 Each column represents a time slice.

[0182] The processing of each leaf node is divided into four processing steps, and each processing step is processed within a single processing resource. That is:

[0183] Resource processing 1: Quickly calculate the indexes at each level (step 1);

[0184] Processing Resource 2: Read the relevant pages from layers L0-L6 of SCM and extract the ProofSet nodes (Step 2);

[0185] Processing resource 3: Simultaneously read L7 from DDR to the ProofSet node of the root node (step 3);

[0186] Process resource 4: Return the L0 value and the complete ProofSet of the leaf node (step 4).

[0187] Different leaf nodes (e.g.) Figure 6 Leaf 1 through Leaf 3 in the process must be processed sequentially by processing resources 1 through 4, but in a staggered order. For example, when processing resource 3 is processing step 3 of Leaf 1, resource 2 is processing step 2 of Leaf 2, and resource 1 is processing step 1 of Leaf 1.

[0188] Preferably, since the data is read from different memories (SCM and DDR), processing resource 2 and processing resource 3 can be further parallelized, that is, steps 2 and 3 can be processed simultaneously.

[0189] The performance of this pipeline design is primarily limited by the random read IOPS of 4KB pages in the SCM, requiring no additional SHA256 computation overhead. Through parallel processing mechanisms, the system can simultaneously handle ProofSet retrieval requests from multiple leaf nodes, significantly improving overall read performance.

[0190] The following describes a hardware computation acceleration scheme for the RootSet of the highest leaf node in one embodiment. The key points of this scheme are as follows:

[0191] Node addressing scheme: Nodes in each layer are numbered starting from 1; the sequence number of a leaf node directly corresponds to its storage location in layer L0; each layer maintains a one-to-one correspondence between sequence number and storage location.

[0192] Take node number 11 (H11) as an example. Since number 11 is less than 16, the 16-node tree is not filled; since number 11 is greater than 8, H1 to H8 have filled the 8-node tree; H9 to H10 have filled the 2-node tree; H11 alone has filled the 1-node tree; finally, the 8-node tree, 2-node tree, and 1-node tree structures are formed from left to right.

[0193] This structure corresponds to the bit pattern of decimal 11 (binary 1011b). The bits from the most significant bit to the least significant bit correspond to full trees of different sizes; the root node of each full tree is located at the odd-numbered position of the corresponding layer; L3, L1, and L0 are layers of bits with values ​​(the layers corresponding to 1 in binary 1011b).

[0194] The root of the entire tree is located at L3@1 (the first unit of L3 level); the root of the entire tree is located at L1@5 (the fifth unit of L1 level); and the root of the entire tree is located at L0@11 (the eleventh unit of L0 level). The index of each level is a truncated sequence that starts from the highest-valued bit and increases by one bit at a time.

[0195] The L2 layer is a bit-free layer (the layer corresponding to 0 in binary 1011b), and the result after truncation is (10)b. This is not the root of a complete tree.

[0196] The RootSet of the highest leaf node with index x is calculated as follows:

[0197] 1. Convert the target index x to its binary representation.

[0198] 2. Processing starts from the most significant bit (MSB):

[0199] - Determine the corresponding floor number based on location (see the one-to-one correspondence method). Figure 7 );

[0200] - Check the value of this bit (1 or 0);

[0201] - If it is 1, then the element at index=1 in this layer is a RootSet element; if it is 0, then there is no RootSet element in this layer.

[0202] 3. Handling the second highest position:

[0203] -Retrieve the MSB-1 and MSB bits;

[0204] - Determine the layer number based on the MSB-1 location (see the one-to-one correspondence method). Figure 7 );

[0205] - If the MSB-1 bit is 1, then the element whose index is equal to the value formed by these two bits is the RootSet element; if the MSB-1 bit is 0, then there is no RootSet element in this layer.

[0206] 4. Repeat the above process until all bits have been processed.

[0207] The above calculation method can be processed in parallel, that is, it can process all bits from MSB to LSB at the same time, thereby improving the calculation efficiency.

[0208] The following describes a method for calculating ProofTarget in one embodiment, which aims to determine the ProofTarget of a target leaf node from the RootSet of the highest leaf node.

[0209] Calculation principles:

[0210] - Check the entire tree set (RootSet) of the highest leaf node in left-to-right (highest-lowest) order.

[0211] - Start the search from the most significant bit of the highest leaf node index.

[0212] The calculation process can be divided into two cases:

[0213] Case 1: The target leaf node is located within the current full entity tree.

[0214] - Judgment condition: The index of the target leaf node is less than or equal to the total number of nodes in the current full entity tree.

[0215] - Processing method: Directly use the root node of the current full entity tree as the ProofTarget.

[0216] - Example (13in 16): The target leaf node index is 13, and the highest leaf node index is 16 (binary 10000b); the 4th bit is 1, corresponding to the 16-node full tree; 13 is less than 16, so the root node (L4@1) of the 16-node full tree is ProofTarget.

[0217] Case 2: The target leaf node index exceeds the current range of the entire entity tree.

[0218] - Judgment condition: The index of the target leaf node is greater than the total number of nodes in the current full entity tree.

[0219] -Solution:

[0220] - Subtract the total number of nodes in the current full entity tree from the target leaf node index;

[0221] - Continue checking the full entity tree corresponding to the next significant bit.

[0222] -Example (13 in 14):

[0223] 1. First, check the most significant bit L3 of the highest leaf node 14 (for the entire 8-node tree);

[0224] -13 is greater than 8, so 13-8=5 is executed.

[0225] 2. Check the next valid bit L2 (4-node full tree);

[0226] -5 is greater than 4, so 5-4=1 is executed.

[0227] 3. Check the next valid bit L1 (2 nodes in the entire tree);

[0228] -1 is less than 2, so determine the entire tree of 2 nodes with ProofTarget of 14;

[0229] -The root of the tree has an index value of 111b at level L1;

[0230] - Index value calculation method: Take the position of the highest leaf node 14 at the corresponding level (L1) of the entire tree and the binary value composed of all bits above it.

[0231] This design approach supports efficient hardware implementation, enabling the rapid determination of the ProofTarget for any target leaf node through layer-by-layer comparison and computation.

[0232] In one embodiment, when ProofTarget is known, the method for generating the instruction sequence for calculating ProofSet is as follows. Taking ProofTarget (13 in 16) as an example:

[0233] Initialization process:

[0234] - Convert the leaf index to binary (13 = 1011b);

[0235] - Confirm that the target ProofTarget is located at L4@1;

[0236] - Establish a hierarchical mapping: the least significant bit corresponds to L0, and so on upwards to L4.

[0237] ProofSet construction steps:

[0238] 1. Initialization: Insert the target leaf node {L0@13}.

[0239] 2. L0 layer processing:

[0240] - 1101b was detected as an odd-numbered left subtree;

[0241] - Calculate the corresponding even-numbered right subtree index 1110b(14);

[0242] - Insert the right node to form {L0@13,L0@14};

[0243] - Shift right by 1110b to get 111b as the L1 layer number.

[0244] 3. L1 layer processing:

[0245] - 111b was detected as an odd-numbered left subtree;

[0246] - Calculate the corresponding even-numbered right subtree index 1000b(8);

[0247] - Insert the right node to form {{L0@13,L0@14},L1@8};

[0248] - Shift 1000b to the right to get 100b as the L2 layer number.

[0249] 4. L2 layer processing:

[0250] - 100b was detected as an even-numbered right subtree;

[0251] - Calculate the corresponding odd-numbered left subtree index 11b(3);

[0252] - Insert the left node to form {L2@3,{{L0@13,L0@14},L1@8}};

[0253] - Shift 100b to the right to get 10b as the L3 layer number.

[0254] 5. L3 layer processing:

[0255] - 10b was detected as an even-numbered right subtree;

[0256] - Calculate the corresponding odd-numbered left subtree index 1b;

[0257] - Insert the left node to form {L3@1,L2@3,{{L0@13,L0@14},L1@8}}}; - Shift right by 1b to get 0b.

[0258] 6. Reach layer L4 (the layer where ProofTarget is located) and complete the calculation.

[0259] Instruction sequence optimization:

[0260] 1. Basic instruction sequence:

[0261] {L0@13 is pushed onto the stack,

[0262] After L0@14 is pushed onto the stack, the right union calculation is performed.

[0263] After L1@8 is pushed onto the stack, the right union calculation is performed.

[0264] L2@3 is pushed onto the stack and then left-joined for calculation.

[0265] L3@1 is pushed onto the stack and then left-joined for calculation.

[0266] Comparison of stack top with ProofTarget}

[0267] 2. Simplified instruction sequence:

[0268] {Start:Data(L0@13),

[0269] Right:Data(L0@14),

[0270] Right:Data(L1@8),

[0271] Left:Data(L2@3),

[0272] Left:Data(L3@1),

[0273] Compare:Data(L4@1)}

[0274] Optimization features:

[0275] -Supports reading only the SHA256 value, deferring the merging calculation to the client.

[0276] -Utilize the parity of node indexes to directly correspond to left and right merge operations.

[0277] - Suitable for 64-bit full-tree hierarchical design, the hierarchical scheduling during instruction execution follows a predetermined scheme.

[0278] - Supports flexible instruction sequence generation, such as ProofSet(14in16) which can generate corresponding optimized instruction sequences.

[0279] This design approach achieves efficient hardware acceleration and instruction optimization while ensuring correctness.

[0280] In one embodiment, ProofSet is calculated using a short computation when ProofTarget is known. For example, ProofSet(13) is calculated when ProofTarget(13 in 14) is known:

[0281] The target leaf index is converted to binary (13 = 1011b), and it is confirmed that ProofTarget is located at L1@7. The hierarchical mapping is established: the least significant bit corresponds to L0, and the mapping extends upward to L1 (the layer where ProofTarget is located).

[0282] ProofSet construction steps:

[0283] 1. Insert the target leaf node {L0@13}

[0284] 2. L0 layer processing:

[0285] - 1101b was detected as an odd-numbered left subtree.

[0286] - Calculate the corresponding even-numbered right subtree index 1110b(14)

[0287] - Insert the right node to form {L0@13,L0@14}

[0288] - Shift right by 1110b to get 111b as the L1 layer number.

[0289] 3. L1 layer processing:

[0290] - The layer containing ProofTarget has been detected.

[0291] -Verify that 111b(7) equals the ProofTarget serial number.

[0292] -Complete calculation

[0293] Simplified instruction sequence:

[0294] {Start:Data(L0@13),

[0295] Right:Data(L0@14),

[0296] Compare:Data(L1@7)}

[0297] In one embodiment, the virtual GlobalRoot is calculated level by level from right to left. Specifically, starting from the rightmost two elements of the RootSet, the SHA256 hash value is calculated level by level until all elements have been processed.

[0298] Using a reverse processing order aligns with the previous algorithm design because the RootSet is constructed by pushing data from the top level of the tree down to the bottom level (e.g., using a stack). This order maintains computational continuity and data locality, ensuring the correct combination order of virtual nodes. This design approach is consistent with the overall system's data flow, effectively supporting the virtual node processing mechanism while maintaining computational efficiency.

[0299] In one embodiment, this is further optimized to parallel insertion of multi-leaf nodes. Parallel insertion can omit unnecessary computation of VirtualGlobalRoot.

[0300] Figure 8 An example is shown where H12 is ready and H13 to H18 are written.

[0301] Corresponding to the 64-level full tree hierarchical structure in SCM, the following design is available:

[0302] From L0 to L6:

[0303] • When the entire tree goes from empty to full, the calculation of the entire tree is first completed in DDR, and then stored in SCM.

[0304] • When the entire tree goes from empty to half full, DDR still uses a virtual node design to calculate the virtual root of the tree. Unless there is an emergency such as a power failure, it will not be written to disk.

[0305] When a 64-bit tree goes from half-full to full, it is padded to a full tree in DDR, the original virtual nodes are discarded, and the entire 64-bit tree to the root is recalculated.

[0306] From L7 layer to Lroot layer: GlobalRoot is calculated based on virtual nodes.

[0307] Compared to inserting only one leaf node, inserting multiple leaf nodes means that multiple upper-level nodes have changed from virtual nodes (copy-type) to nodes that need to be actually computed.

[0308] Generally, the length of the data payload is greater than the length of the SHA256 value. Therefore, the computation of the L0 layer is slower than that of the L1 layer and the layers above it. So an intuitive scheduling is to first read the tail of L0 and then complete the computation of the entire L0 layer (compute pairs from left to right, then append and write). Then read the tail of L1 and complete the computation of the entire L1 layer. Then move up. The computation and throughput pattern of each layer in L1 is (optionally) read the tail, compute pairs, and append the result to the previous layer. Compute the next pair, append the result to the previous layer, until there are no new units in the current layer. Compared with the above pattern, the computation and throughput pattern of the L0 layer is to first compute each new unit of the current layer (L0) from the data payload, and then process it in the same way as in the previous L1 layer. This embodiment can write multiple back-to-back results in a layer at once, and since multiple SHA256 results are needed to form a NAND page, this scheme can better utilize the write advantage of NAND.

[0309] The following example will be used to analyze the hardware bottleneck problem when writing transactions in a database.

[0310] According to an embodiment of this application, L0 to L6 are in 4KB pages of SCM and L7 to Lroot (the layer where the tree root is located) are in DDR, at which point storage bandwidth is no longer a bottleneck.

[0311] The computational cost of computing a 64-bit full tree involves 64 short SHA256 hashes, calculated sequentially as follows: 32, 16, 8, 4, 2, and 1. Here, long SHA256 hashes refer to the hashes calculated from the relatively long original data, while short SHA256 hashes refer to the hashes calculated from the SHA256 hashes of lower-level nodes to the SHA256 hashes of higher-level nodes. Theoretically, with sufficient computational resources for 64 long SHA256 hashes and 63 short SHA256 hashes in parallel, two 64-bit full trees can be computed in a pipelined parallel manner.

[0312] The system can be configured with several processing units, each containing 64 parallel long SHA256 calculation circuits and 32 parallel short SHA256 calculation circuits. These calculation circuits can be hardware circuits that implement SHA256 calculations. There are already many types of such hardware circuits in the prior art, which will not be described in detail in this application. The 64 parallel long SHA256 calculation circuits calculate 64 SHA256 values ​​as L0 leaf nodes based on the 64 input records. Starting from the 32 parallel short SHA256 calculation circuits, the calculation results are combined pairwise, sequentially processed through 16-way parallel calculation, 8-way parallel calculation, 4-way parallel calculation, and 2-way parallel calculation, finally converging into a single-path calculation to obtain the final calculation result.

[0313] like Figure 9 As shown, the pipelined processing involves four steps: reading data until a 64-bit full tree is filled; calculating 64 parallel long SHA256 hashes for the LO layer of this 64-bit full tree; serially calculating 32, 16, 8, 42, and 1 short SHA256 hashes; and writing the tree into the SCM. Within a processing unit, two 64-bit full trees are calculated in parallel, and L7 to Lroot are finally calculated from the roots of these 64-bit full trees, with a lazy commit for L7 to Lroot. The horizontal axis in the diagram represents time and is for illustrative purposes only; it does not precisely represent the actual duration of each step.

[0314] Taking 1 million records as an example, 14 layers from L7 to L24 need to be calculated. The widest width required is 2. 14 = 16K short SHA256.

[0315] Through the above design, this application essentially shifts the bottleneck of database TPS from storage bandwidth to the bandwidth of SHA256 PE (processing unit).

[0316] The following analysis examines the performance bottleneck when reading transactions in a database: parallel reading of the ProofSets of multi-leaf databases.

[0317] System parameter settings: database capacity is 1 billion transactions; single transaction data payload is 1KB; total data payload space is 1TB; total hash tree space is 64GB; parallel read scale is 128 leaf nodes.

[0318] Storage hierarchy design:

[0319] 1. SCM storage (approximately 63.5GB):

[0320] - Each 4KB page stores a 64-node full tree from L0 to L6;

[0321] -Page structure:

[0322] (L0-1,...,L0-64,

[0323] L1-1,...,L1-32,

[0324] L2-1,...,L2-16,

[0325] L3-1,...,L3-8,

[0326] L4-1,...,L4-4,

[0327] L5-1, L5-2,

[0328] L6-1)

[0329] - The SCM page structure is the same as that in DDR, supporting direct data migration;

[0330] The conversion between layer number and in-layer sequence number to page address uses a fixed formula and supports hardware implementation.

[0331] 2. DDR storage (512MB): Stores L7 and higher level nodes.

[0332] Parallel read optimization: Parallel reading of the ProofSet of 128 randomly distributed leaves is equivalent to reading the entire tree of 64 nodes from 4KB pages at 128 random addresses; then, extracting the corresponding 7 elements from levels L0 to L6 from each full tree. Because it is a completely random 32-byte read, the read amplification is reduced from 128 times to 128 / 7 times.

[0333] When writing, the case where the rightmost 64-bit tree is incomplete must be handled (in memory, ensure that SCM is flushed before power failure).

[0334] Special case handling: When writing, the case where the rightmost 64 nodes are not a complete tree must be handled. The incomplete tree is kept in DDR. Before the DDR is powered off, it must be ensured that SCM is written.

[0335] Key performance optimization points: Leveraging the characteristics of NAND page size; fully utilizing data locality; hardware-based address translation calculations, thereby improving speed; minimizing random read overhead.

[0336] This design significantly reduces read amplification and improves the overall read performance of the system through optimized storage structure and parallel read strategy.

[0337] In one embodiment, a verifiable key-value database system comprises: a visible layer in the host computer's main memory, a middle layer on the device side, and an innermost layer. The overall design of this database is as follows: Figure 10 As shown.

[0338] Multiple visual windows are set up in the visual layer of the host computer's main memory. Each window externally presents the mapping relationship from primary key to data payload, and internally implements the mapping relationship from primary key to transaction handle. Each window corresponds to a snapshot of a specific version of the database. The visual layer utilizes the computing and storage resources of the host computer to provide flexible multi-primary key construction and query functions, and can be reconstructed from the middle layer and the innermost layer. To optimize performance, the visual layer supports multi-level caching strategies, including a safe mode without caching, a first-level acceleration mode that caches the data payload address handle after obfuscation hashing, and a second-level acceleration mode that directly caches the data payload.

[0339] The device-side middleware layer implements serialized transaction log management, assigning a unique sequence number and external handle to each transaction. The log records the sequence number, data payload index, data payload hash value, and related signature information. The middleware layer employs an improved log-based hash tree structure, designed based on a transaction sequence numbering mechanism. The transaction handle's security is enhanced through device-specific obfuscated hashing, which remains unchanged throughout the device's lifecycle.

[0340] The innermost layer on the device maintains a data payload log, establishing a one-to-one correspondence between transaction actions and payload units. Each payload unit contains a complete key-value pair or data record. The middle layer uses SCM storage media, while the data payload log uses QLC storage media to balance performance and cost. The system supports parallel use of storage bandwidth, can simultaneously acquire data payloads and hash proofs, and simplifies the design structure through a strict sequential write-random read mode. Although the transaction action log must be acquired before accessing the data payload, the performance penalty is effectively mitigated by using the SCM storage middle layer.

[0341] The system constructs a version chain using leading transaction sequence numbers, supporting data version control, facilitating external review, and reserving expansion space to support future business needs. The host computer CPU is responsible for operations such as primary key search and transaction type determination. This layered architecture ensures data verifiability while achieving a balance between performance and security through multi-level caching and optimized storage strategies.

[0342] In summary, this application provides a verifiable key-value database system for data centers, supporting secure hosting services for multi-party data collaboration. This system provides real-time verifiability assurance for data hosting through trusted computing circuits built into disks or arrays, ensuring data confidentiality, consistency, integrity, and non-repudiation. The system can provide customers with real-time proof that the hosted data has not been corrupted by the hosting provider or malicious third parties. Simultaneously, the system can prove that the full database obtained by the customer after payment is identical to the database selected before payment (CherryPick).

[0343] The technical advantage of this application's embodiments lies in the significant reduction of system costs through optimized storage and memory bandwidth utilization and parallel computation of SHA256. Compared to equivalent performance systems employing server-based trusted computing solutions, this application's embodiments offer a clear cost advantage.

[0344] Currently, the field of data cooperation and transaction matching is in the exploratory stage regarding legal authority and value assessment. Large internet companies and authorized innovative enterprises are exploring verifiable database products based on data centers. As products based on server-based trusted computing solutions gradually form a competitive landscape in terms of performance and price, the system described in this application embodiment will have broad market application potential.

[0345] Accordingly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the various method embodiments of this application. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media do not include transient computer-readable media, such as modulated data signals and carrier waves.

[0346] Furthermore, embodiments of this application also provide a data read / write system for a key-value database, including a memory for storing computer-executable instructions and a processor; the processor is used to implement the steps in the above-described method embodiments when executing the computer-executable instructions in the memory. The processor may be a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP), Microcontroller Unit (MCU), Neural Processing Unit (NPU), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), or other programmable logic devices. The aforementioned memory may be read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or solid-state drive, etc. The steps of the methods disclosed in the embodiments of this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0347] Furthermore, embodiments of this application also provide a computer program product, including computer-executable instructions that, when executed by a processor, implement the steps in the above-described method embodiments.

[0348] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this application, if it refers to performing an action according to an element, it means performing the action at least according to that element, including two cases: performing the action only according to that element, and performing the action according to that element and other elements. Expressions such as "multiple," "repeatedly," and "various" include two, two times, two kinds, and more than two, more than two times, and more than two kinds.

[0349] The numbering used in describing the steps of a method does not itself impose any restrictions on the order of these steps. For example, a step with a larger number does not necessarily have to be executed after a step with a smaller number; the step with the larger number can be executed first and then the step with the smaller number; or they can be executed in parallel, as long as such an execution order is reasonable to a person skilled in the art.

[0350] This specification includes combinations of various embodiments described herein. Individual references to embodiments are made (e.g., "one embodiment," "some embodiments," or "preferred embodiments"); however, these embodiments are not mutually exclusive unless indicated to be mutually exclusive or are readily apparent to those skilled in the art. It should be noted that the word "or" is used in a non-exclusive sense throughout this specification unless the context explicitly indicates or requires it.

[0351] All references to this specification are considered to be incorporated integrally into the disclosure of this application so that they can serve as the basis for modifications if necessary. Furthermore, it should be understood that the above descriptions are merely preferred embodiments of this specification and are not intended to limit the scope of protection of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the scope of protection of one or more embodiments of this specification.

[0352] In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for writing data to a key-value database, characterized in that, include: The processor receives a write request, which contains key-value pair data; The processor writes the key-value pair data as a data payload into the first memory; The processor calculates the hash value of the data payload and writes the hash value as the highest leaf node with the largest current sequence number into the binary hash tree stored in the fourth memory. In this binary hash tree, the index of each leaf node is sequentially increased. Leaf nodes whose index is not greater than the highest leaf node are physical nodes, and their values ​​are the hash values ​​of the corresponding data payloads. Leaf nodes whose index is greater than the highest leaf node are virtual nodes, and their values ​​are preset values. The processor determines whether the parent nodes at each level corresponding to the highest leaf node are located within a complete entity subtree in the direction from the leaf node to the root node, wherein the complete entity subtree is a subtree that does not contain virtual nodes; In response to determining that the parent node is located within a complete entity subtree, the parent node is treated as an entity node, the hash value of the parent node is calculated based on the hash values ​​of the two child nodes of the parent node, and the hash value of the parent node is stored in the fourth memory; In response to determining that the parent node is not located in the complete entity subtree, the parent node is treated as a virtual node. When it is necessary to obtain the hash value of the parent node, the value of the child node of the parent node that is not a preset value is directly used as the hash value of the parent node.

2. The data writing method for a key-value database as described in claim 1, characterized in that, Also includes: The binary hash tree forms multiple complete entity subtrees from left to right, and the processor adds the root node of each complete entity subtree to the root set; The processor obtains the global root hash value by calculating the hash value of each node in the root set from right to left. The processor calculates the verification hash set, specifically including: starting from the leaf node, in order from low level to high level, if the current node is on the left, it searches for the hash value of its right-side paired node; if the current node is on the right, it searches for the hash value of its left-side paired node; and adds the hash value of the found paired node to the verification hash set. The processor returns data to the requesting device for verifying the write operation. This data includes the global root hash value, the root set, and the verification hash set.

3. The data writing method for a key-value database as described in claim 2, characterized in that, The requesting device verifies the correctness of the write in the following way: A verification target node is calculated based on the verification hash set and the hash value of the leaf node, and the verification target node should be an element in the root set; and The global root hash value can be obtained by calculating the hash value of each element in the root set level by level.

4. The data writing method for a key-value database as described in claim 1, characterized in that, The fourth memory includes a second memory and a third memory; the first memory and the second memory are non-volatile memories; the access speeds of the first memory, the second memory, and the third memory increase sequentially. The nodes of the binary hash tree from level 0 to level N are stored in the second memory, wherein the nodes from level 0 to level N are combined into at least one complete subtree, and each complete subtree is stored in a page of the second memory, the storage space of the complete subtree being the same as the size of the page; the nodes of the binary hash tree from level N+1 and above are stored in the third memory; wherein N is a preset positive integer.

5. The data writing method for a key-value database as described in claim 4, characterized in that, In each page of the second memory, nodes of each level of a complete subtree are stored layer by layer from level 0 to level N; the index of the leaf node is directly used as its storage location in level 0; and a one-to-one correspondence between node index and storage location is maintained in each level. The processor includes multiple parallel long SHA256 calculation circuits and multiple parallel short SHA256 calculation circuits. The long SHA256 calculation circuits are used to calculate the hash value of each leaf node in a page in parallel according to the data payload. The short SHA256 calculation circuits are used to perform pairwise calculations on each leaf node in a page to obtain the parent node of the first layer. Then, through multi-path parallel calculations layer by layer from bottom to top, the calculations are finally converged into a single path to obtain the hash value of the root node of the complete subtree.

6. The data writing method for a key-value database as described in claim 5, characterized in that, Also includes: For a complete subtree that goes from empty to half full, use virtual nodes to calculate the virtual root hash value of the complete subtree; For a complete subtree that goes from half-full to full, after completing it into a full tree, discard the original virtual nodes and recalculate the hash value from the root of the entire complete subtree.

7. The data writing method for a key-value database as described in claim 5, characterized in that, When writing key-value pairs of data in batches, the following pipeline method is used for processing: In the first pipeline stage, 64 key-value pairs are read to form a complete subtree; In the second pipeline stage, 64 parallel SHA256 computing circuits are used to compute the hash values ​​of the 64 key-value pairs as leaf nodes; In the third pipeline stage, 32 parallel SHA256 computing circuits are used to perform pairwise calculations on the leaf nodes, and the hash values ​​of each level node of the complete subtree are obtained by sequentially performing 16-way parallel computing, 8-way parallel computing, 4-way parallel computing, 2-way parallel computing and single-way computing. In the fourth pipeline stage, the node hash values ​​of the complete subtree are written into a 4KB page of the second memory.

8. A method for reading data from a key-value database, characterized in that, Key-value pairs are stored as data payloads with incrementing sequence numbers in the first memory, and a binary hash tree is stored in the fourth memory. In this binary hash tree, each leaf position corresponds to a sequence number. Leaf nodes with sequence numbers no greater than the largest sequence number of the data payload in the first memory are physical nodes, and their values ​​are the hash values ​​of the corresponding data payloads. Leaf nodes with larger sequence numbers are virtual nodes, and their values ​​are preset values. For a parent node, if any of its child nodes are virtual nodes, then the parent node is a virtual node, and its value is equal to the value of its child nodes that do not have preset values; otherwise, the parent node is a physical node, and its value is the hash result of its two child nodes. The method includes: The processor receives a read request from the requesting device and reads the requested data payload from the first memory. The leaf node corresponding to the data payload is the target leaf node. The processor generates a verification hash set for the target leaf node, wherein the elements in the verification hash set are paired nodes at each level of the path from the target leaf node to the verification target node in the binary hash tree; the verification target node is the root node of the smallest complete entity subtree containing the target leaf node; the complete entity subtree is a subtree that does not contain virtual nodes; The processor returns the requested data payload and verification hash set to the requesting device.

9. The data reading method for a key-value database as described in claim 8, characterized in that, The requesting device verifies the correctness of the read data in the following ways: Obtain the global root hash value and the root set, wherein the global root hash value is the hash value of the root node of the binary hash tree, and the root set includes the root nodes of multiple complete entity subtrees in the binary hash tree from left to right; The target node for verification is calculated based on the hash value of the verification hash set and the target leaf node. The target node for verification should be an element in the root set. The global root hash value can be obtained by calculating the hash value of each element in the root set level by level.

10. The data reading method for a key-value database as described in claim 8, characterized in that, The processor determines the target node for verification through the following steps: Starting from the most significant bit of the index of the highest leaf node in the binary hash tree, check its corresponding complete entity subtree; If the index of the target leaf node is less than or equal to the total number of nodes in the current complete entity subtree, then the root node of the current complete entity subtree is used as the verification target node. If the index of the target leaf node is greater than the total number of nodes in the current complete entity subtree, then the total number of nodes in the current complete entity subtree is subtracted from the index of the target leaf node, and the next complete entity subtree corresponding to the valid bit is checked until a complete entity subtree containing the target leaf node is found.

11. The data reading method for a key-value database as described in claim 8, characterized in that, The step of the processor generating the verification hash set includes: Convert the index of the target leaf node into binary; Establish a hierarchical mapping, with the least significant bit corresponding to level 0; Starting from the target leaf node, process layer by layer: detect whether the current node is an odd-numbered left subtree or an even-numbered right subtree. If it is an odd-numbered left subtree, calculate the corresponding even-numbered right subtree index and add it to the verification hash set. If it is an even-numbered right subtree, calculate the corresponding odd-numbered left subtree index and add it to the verification hash set, until the layer where the verification target node is located is reached.

12. The data reading method for a key-value database as described in claim 8, characterized in that, The fourth memory includes a second memory and a third memory; the first memory and the second memory are non-volatile memories; the access speeds of the first memory, the second memory, and the third memory increase sequentially. The nodes of the binary hash tree from level 0 to level N are stored in the second memory, wherein the nodes from level 0 to level N are combined into at least one complete subtree, and each complete subtree is stored in a page of the second memory, the storage space of the complete subtree being the same as the size of the page; the nodes of the binary hash tree from level N+1 and above are stored in the third memory; wherein N is a preset positive integer.

13. The data reading method for a key-value database as described in claim 11, characterized in that, The processor determines the path from the target leaf node to the verification target node using the following index-based hardware computation method: Convert the target leaf index to binary to serve as the index for level 0; In the processing of each layer: The node type is determined by checking the least significant bit. If the least significant bit is 0, it is a left node; if it is 1, it is a right node. For a left node, find its right-side matching node by incrementing by 1; for a right node, find its left-side matching node by decrementing by 1. Obtain the index of the previous level by right shifting by 1 bit; Repeat the above process until the layer containing the target node is reached.

14. The data reading method for a key-value database as described in claim 12, characterized in that, The processor employs a hardware pipeline architecture to process read requests from multiple leaf nodes in parallel, including: The first processing resource computes the indexes of multiple target leaf nodes at each level in parallel using bitwise operations; The second processing resource is used to read nodes from level 0 to level 6 in multiple 4KB pages from the second memory in parallel and extract nodes from the verification hash set; The third processing resource is used to read nodes from the verification hash set of multiple target leaf nodes from the third memory in parallel, from the 7th layer to the root node. The fourth processing resource includes multiple parallel SHA256 computation circuits for parallel computation of hash values ​​for multiple verification paths and returning a complete set of verification hashes.

15. The data reading method for a key-value database as described in claim 14, characterized in that, The processor includes: Hardware acceleration circuitry for calculating SHA256 hash values ​​includes: Multiple parallel long SHA256 computing circuits are used to calculate the hash value of the data payload; Multiple parallel short SHA256 computation circuits are used to compute the node hash values ​​on the verification path; The processor supports parallel processing using multiple SHA256 computing circuits during the verification process.

16. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 15.

17. A computer program product comprising computer-executable instructions, characterized in that, When executed by a processor, the computer-executable instructions implement the steps of the method according to any one of claims 1 to 15.

Images