Distributed verifiable ledger database

Abstract

A verifiable ledger database is disclosed, the verifiable ledger database configured to manage business of a ledger, the verifiable ledger database comprising a plurality of slices, each slice being formed by dividing business data. Each slice includes: a ledger memory configured to provide access to business data and support a plurality of attestation; the service manager is configured to execute each service according to the corresponding service request; and a verifier configured to verify the traffic by returning an attestation in accordance with the verification request.

Description

Technical Field

[0001] Generally speaking, the present invention relates to distributed verifiable ledger databases. Background Art

[0002] Verifiable databases protect the integrity of user data and queries executed on untrusted database providers. Until recently, the focus has been on protecting the integrity of query execution. In this context, users upload data to an untrusted provider, which executes queries and returns proofs confirming the correctness of the results. However, such OLAP-style verifiable databases rely on complex cryptographic primitives that limit the performance or range of possible queries.

[0003] With the renewed interest in verifiable databases, recent focus has been placed on OnLine Transactional Processing (OLTP) style systems. In particular, a new class of systems has emerged, known as verifiable ledger databases, which aim to protect the integrity of the history of data. In particular, the data is maintained by an untrusted provider that executes read and update queries. The provider generates integrity proofs about the content of the data and the entire evolution history of the data content.

[0004] An example of a verifiable ledger database is a blockchain. Another example is a certificate transparency log, where a centralized server maintains a tamper-proof, append-only log of public key certificates. A third example is Amazon's Quantum Ledger Database (QLDB) service, which maintains an append-only log similar to Certificate Transparency's append-only log. QLDB uses a log to record data operations, which are then applied to another backend database.

[0005] Among these systems, the first challenge is the lack of a unified framework for comparing verifiable ledger databases. In particular, we note that the three systems above originate from three different fields of computer science: blockchain comes from distributed computing, certificate transparency comes from security, and QLDB comes from database technology. As a result, there is no framework within which the three systems above can be fairly compared. The second challenge is the lack of database abstraction, i.e., operations. The transparency log used in existing systems exposes a key-value interface. For example, blockchain maintains state in a key-value storage space, and certificate transparency and QLDB store certificates and hashes of operations in the log, respectively. This key-value model is too low-level to support general OLTP database workloads. The third challenge is how to achieve high performance while retaining security. For example, blockchain suffers from poor performance due to consensus bottlenecks. Certificate transparency has low performance due to expensive disk-based operations, while QLDB generates inefficient integrity proofs for verifying the latest data.

[0006] It would be desirable to overcome all or at least one of the above-mentioned problems, or at least provide a useful alternative. Summary of the invention

[0007] The present invention discloses a verifiable ledger database configured to manage the business of the ledger, the verifiable ledger database includes a plurality of shards, the plurality of shards are formed by dividing the business data and include:

[0008] A ledger storage configured to provide access to business data and support multiple attestations;

[0009] A service manager configured to execute each service according to a corresponding service request; and

[0010] A validator is configured to verify a transaction by returning a proof in response to a verification request.

[0011] In some embodiments, the ledger storage includes an upper-level POS tree and a lower-level POS tree, each of the upper-level POS tree and the lower-level POS tree includes multiple nodes, the multiple nodes include at least one root node and at least one leaf node, and wherein the ledger storage includes a hash chain sequence of blocks, and each leaf node of the upper-level POS tree includes a block number of a corresponding block.

[0012] In some embodiments, the lower-level POS tree is built on the state of the verifiable ledger database, and the lower-level POS tree includes one or more root nodes, each of which contains a corresponding root hash and a corresponding said block number.

[0013] In some embodiments, the root node of the lower-level POS tree is stored as a leaf node of the upper-level POS tree.

[0014] In some embodiments, the verifiable ledger database is configured to retrieve business data from a given block number by locating a corresponding leaf node of an upper POS tree using the given block number and then traversing the lower POS tree to locate the business data.

[0015] In some embodiments, the verifiable ledger database is configured to update the business data by creating new nodes at the lower POS tree and the upper POS tree using copy-on-write.

[0016] In some embodiments, the business data is modeled as multiple keys, and the verifiable ledger database is configured to partition the keys into shards based on a hash of the keys.

[0017] In some embodiments, the verifiable ledger database is configured to use a two-phase commit (2PC) protocol to ensure the atomicity of the transaction.

[0018] In some embodiments, the transaction manager is configured to log each transaction and respond with a commit or abort based on a concurrency control algorithm.

[0019] In some embodiments, the service manager includes:

[0020] Business queue, used to buffer business requests;

[0021] Multiple service threads for receiving service requests buffered in a service queue;

[0022] Shared memory for storing prepared and committed business data; and

[0023] The persistence thread is used to asynchronously persist the submitted business data to the ledger storage.

[0024] "Persist" in the verb form as used herein refers to building or updating the authentication data structure (two-level POS tree) in the ledger storage (on disk). The word "persist" has been chosen to distinguish this process from the "commit" action, which refers to inserting / updating the submitted data in memory during the commit phase.

[0025] In some embodiments, the service manager is configured to execute services by:

[0026] When the service queue is not full, dispatching the service requests buffered in the service queue to the service threads in sequence; and

[0027] When the service queue is full, the service is terminated.

[0028] In some embodiments, the business manager is configured to asynchronously update the ledger stored in the verifiable ledger database by:

[0029] When the service manager receives the commit message, it allows the shared memory to store the service data in the committed data map;

[0030] Write to the Write-Ahead-Log (WAL);

[0031] Allowing the persistence thread to persist the business data in the submitted data map into the ledger storage to generate persistent data; and

[0032] Removes persisted data from the committed data map.

[0033] In some embodiments, the transaction manager is configured to batch pending transactions before updating the ledger by:

[0034] Collecting corresponding data from each business to be submitted into data blocks; and

[0035] Append the data blocks created within the time window to the ledger storage.

[0036] In some embodiments, collecting corresponding data from each business to be submitted includes: selecting corresponding data from the submitted data map version by version.

[0037] In some embodiments, wherein the proof includes one or more of: an inclusion proof, a current value proof, and an append-only proof.

[0038] In some embodiments, the validator is configured to batch process certification of business data.

[0039] In some embodiments, the validator is configured to immediately return proof of the persisted data during transaction processing.

[0040] In some embodiments, the validator is configured to validate the transaction within a time window after the transaction is processed.

[0041] In some embodiments, the validator is configured to validate the read set and the write set of each transaction.

[0042] In some embodiments, the verifiable ledger database includes one or more auditors, the one or more auditors being configured to ensure proper execution of the verifiable ledger database server, wherein each auditor is configured to:

[0043] checking whether the verifiable ledger database server forked the history log by checking whether users of the verifiable ledger database received digests corresponding to the linear history; and

[0044] Re-execute the transaction to ensure that the current state of the verifiable ledger database is correct.

[0045] In some embodiments, the verifiable ledger database is configured to recover the one or more nodes after the one or more nodes crash and restart. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Embodiments of the invention will now be described by way of non-limiting examples with reference to the accompanying drawings, in which:

[0047] Figure 1 The proposed design (hereinafter interchangeably referred to as "LedgeBase" or "GlassDB", "LedgeBase" or "GlassDB" respectively refer to the same system) is shown;

[0048] Figure 2 A Merkle tree-based transparency log is shown;

[0049] Figure 3 shows a delay failure at the server;

[0050] Figure 4 Delay failures with different persistence intervals are shown;

[0051] Figure 5 shows the effect of latency at the client;

[0052] Figure 6 Shows the impact of latency on overall performance;

[0053] Figure 7 shows the comparison of server and client costs with other baselines;

[0054] Figure 8 The performance of the TPC-C workload is shown;

[0055] Fig. 9 A workload-X with 16 nodes is shown;

[0056] Fig.10 shows workload - X on a single node; and

[0057] Fig.11 Audit performance is shown. DETAILED DESCRIPTION

[0058] Verifiable ledger databases protect data history from malicious tampering. Existing systems, such as blockchain and certificate transparency, are based on transparency logs - simple abstractions that allow users to verify that logs maintained by untrusted servers are append-only. Existing systems expose simple key-value interfaces. On the other hand, building practical databases from transparency logs remains a challenge.

[0059] A distributed database is now disclosed that addresses these limitations under a practical threat model, referred to herein as "LEDGEBASE" or "GlassDB," while also addressing the lack of business support and lower efficiency of existing verifiable ledger databases.

[0060] Figure 1 The design of GlassDB 100 is shown. The distributed database 100 is configured to manage the business of a ledger by partitioning the business data into a plurality of shards 102. The business data is modeled as a plurality of keys or key-value tuples.

[0061] Partitioning can be based on a hash of a key, and a two-phase commit (2PC) protocol is used to ensure the atomicity of cross-shard transactions.

[0062] Each shard 102 has three main components: a business manager 104, a validator 106, and a ledger storage 108. Business requests are forwarded to the business manager 104, which is configured to execute each business according to the corresponding business request - which can be executed using a thread pool with optimistic concurrency control to achieve serializability. Verification requests are forwarded to the validator 106, which is configured to verify the business by returning a proof according to the verification request. The ledger storage 108 is configured to provide access to business data and support multiple proofs. Therefore, the ledger storage 108 maintains core data structures that provide efficient data access and proof generation.

[0063] Each shard 102 maintains a personal ledger based on committed records. The client keeps track of the key-to-shard mapping and caches a summary of the shard's ledger. GlassDB 100 can use a Write-Ahead-Log (WAL) to handle application failures. In some embodiments, a replication layer is added on top of the ledger storage 108. The replication layer enables GlassDB to handle node failures. In some such embodiments, GlassDB tolerates permanent node failures by replicating nodes using a crash-tolerant protocol, such as Raft.

[0064] GlassDB 100 overcomes some limitations of existing distributed database designs. GlassDB 100 supports general business and can be widely used in supporting current and developing applications. GlassDB 100 adopts the same threat model as QLDB and LedgerDB, which assumes that the database server is untrusted and there is a set of one or more auditors who gossip with each other and are trusted to ensure the correct execution of the verifiable ledger database server. GlassDB achieves high throughput and small verification cost. Table 1 shows how the system fits into the design space.

[0065]

[0066] Table 1: GlassDB versus other verifiable ledger databases. N is the number of transactions, m is the number of keys, and B is the number of blocks, where m ≥ N ≥ B. All systems except Forkbase support inclusion proofs of the same size as append-only proofs.

[0067] The ledger storage 108 adopts a hash-protected index structure. This enables GlassDB 100 to comprehensively and efficiently protect the index, thereby avoiding security issues and high verification overhead or one or both of the previous distributed ledgers. The ledger storage 108 is also built on the state of the data, rather than on the business. This structure enables more efficient retrieval and current value proof generation, while generating a smaller data structure than, for example, the Merkle tree of existing systems. The structure also grows more slowly by using batch updates from multiple businesses. Similarly, high throughput is achieved by partitioning data across multiple nodes as discussed below. Finally, GlassDB 100 uses business batching, asynchronous persistence, and deferred verification to speed up business processing and verification.

[0068] The ledger storage 108 can be constructed as a two-layer pattern-oriented split tree (Pattern-Oriented Split tree, POS-tree) (or two-layer POS tree) - a Merkle variant. The POS tree is an instance of a structurally invariant and reusable index (SIRI), which combines a Merkle tree and a balanced search tree and is formed by nodes. The tree is built from a global ordered sequence of data, where a parent node stores the encrypted hash of its child nodes, and the root node therefore contains a summary of the entire tree. Data search is performed by traversing the tree. The data is split into leaf nodes using content-defined blocks, and new nodes are created in the content-defined blocks when a pattern is matched. The encrypted hash values ​​of the nodes in a level form a byte sequence of the upper layer. The byte sequence is split into index nodes using a similar content-defined block approach. The POS tree is optimized for high deduplication rates due to its content-defined blocks. Finally, the POS tree cannot be changed, that is, when a node is updated, a new tree is created using copy-on-write.

[0069] Further references Figure 1 , the ledger storage 108 may include an upper POS tree 110 and a lower POS tree 112. Each tree 110, 112 includes a plurality of nodes, including at least one root node and at least one leaf node. The lower POS tree 112 is built on the database state and includes one or more root nodes, each of which is referred to as a corresponding "data block" (one of the one or more root nodes is represented by the number 124), and the corresponding "data block" contains a corresponding root hash and other block meta information (e.g., the block number in which the previous version of the data resides), such as a block number. The data block 124 is stored as a leaf of the upper POS tree 110. The POS tree is used as an index on the data block 124, and the root of the POS tree is a summary of the entire ledger. In order to retrieve a key from a given block number, the block number is obtained from the upper POS tree 110, and then the lower POS tree 112 is traversed to locate the data.

[0070] When the key is updated, new nodes are created at both levels using copy-on-write. Thus, in addition to inclusion and append-only proofs, this authentication data structure also provides efficient current value proofs. Since each data block represents a snapshot of the database state, the latest value always appears in the last block, so the current value proof can be verified using only one block.

[0071] QLDB and LedgerDB use Merkle trees to ensure the integrity of data. However, QLDB and LedgerDB provide weak security and incur huge verification overhead. The ledger storage 108 protects the integrity of the index - the database index and the clue index are not protected in QLDB and LedgerDB. An embodiment of a two-layer POS tree protects data, indexes, and history. In particular, the upper layer protects the lineage of states, while the lower layer protects data and serves as an index. Therefore, the proof in GlassDB 100 includes relevant nodes from the leaves of the lower POS tree 112 to the root of the upper POS tree 110. Since both levels are tamper-proof, it is impossible for the server to modify the data or skip some versions without being detected. In addition, the client can check the data block 124 from which the data was obtained, thereby verifying that the data is up to date.

[0072] GlassDB 100 divides the key into shards 102 based on the hash value of the key. Each client 105 is a coordinator. Each client 105 generates a read set and a write set for the business, and then sends a "prepare" message to the shard 102. The business manager 104 at each shard 102 records the business and responds with a commit or abort based on the concurrency control algorithm. Regarding concurrency control, the read set and write set of the concurrent business are verified to check for read-write and write-write conflicts. If there is no conflict, the shard 102 returns "commit", otherwise it returns "abort". The client 105 waits for responses from all shards 102 involved in the business, and the client 105 resends the message after a timeout period. If all shards 102 return commit, the client 105 sends a commit message to the shard 102, otherwise the client 105 sends an abort. Then, each shard 102 commits or aborts the business accordingly and returns a confirmation to the client 105.

[0073] At each shard 102, the business is processed by the business manager 104. Incoming requests are buffered in the business queue 114. Requests wait in the queue 114 to be dispatched to an available business thread 116. If the business queue 114 is not full, the business requests buffered in the business queue 114 are dispatched to the business thread 116 in sequence. If the queue 114 is full, the business is aborted. The business thread 116 stores the prepared business 118 and the submitted data in the shared memory 120. The persistence thread 122 asynchronously persists the submitted data to the ledger storage 108.

[0074] To reduce overhead due to contention and execution time, GlassDB 100 updates the ledger asynchronously. When a commit message is received, the business manager 104 stores the business data in a "committed data map" in memory.<key,ver,val> The ver parameter refers to the submitted data map, which may be a multi-version submitted data map. The business manager 104 then writes to the WAL to ensure persistence and recovery. After a timeout, the background thread persists the data in the map to the ledger storage 108. The persisted data is then removed from the submitted data map to keep memory consumption low.

[0075] Business latency is reduced by moving ledger updates out of the critical path for business execution, although the cost of updating and persisting authentication data structures is still large - both levels of the POS tree need to be updated and written to disk. GlassDB 100 can reduce costs by batching multiple submitted businesses before updating the ledger. Batch processing can be performed by collecting independent data from recently submitted businesses into data blocks 124. At least some and preferably all of the blocks 124 created within the time window are appended to the ledger memory. To form block 124, the GlassDB100 server selects data from the submitted data map version by version. For a given data version, GlassDB 100 calculates the sequence number of block 124 by adding the current block sequence to the version sequence in the data map, and the data will be submitted at block 124.

[0076] By batching multiple transactions, GlassDB provides smaller Merkle trees than LedgerDB and QLDB and is therefore more efficient.

[0077] Verification services require checking both read sets and write sets with the validator 106. The proofs involved in verification may include one or more of inclusion proofs, current value proofs, and append-only proofs. To verify the read set, the client 105 checks that the data is correct and up to date (e.g., for the default Get(.) operation - discussed below). The server therefore generates a current value proof. To verify the write set, the client 105 checks that the new ledger is append-only, and checks that the data written to the ledger is correct. The server therefore generates an append-only proof and an inclusion proof.

[0078] The inclusion proof and current value proof in GlassDB 100 can be included in the hash of the nodes along the path from leaf to root, through a specific block and the latest block in the two-layer POS tree 110, respectively. Only the additional proof can contain the hash of the node along the path where the old root resides. If the old root node does not exist in the new Merkle tree (i.e., because the old Merkle tree is not a complete tree), a proof of the left child node of the new Merkle tree can be generated. In order to verify the proof, the client 105 calculates the digest and compares the digest with the locally saved digest. Verification requires obtaining proofs from all participating shards 102. There is no coordination overhead because the ledger 100 is able to be changed using copy-on-write, which means that read operations can run concurrently with other businesses.

[0079] Business verification can occur within a time window, rather than immediately - that is, deferred verification, in which proofs of business data are batched. This strategy is suitable for applications that require high performance and can tolerate temporary violations of data integrity. For these applications, a commitment is sent from the server to the client 105, which contains the future block sequence number, business ID, current summary, key and value in which the data will be submitted. The client 105 can verify the business by sending a verification request with the commitment as a parameter after the block is available. Upon receiving the verification request, the server will check whether the block has been persisted. If the check passes, the server generates a proof containing proof and only attaching proof, and returns the proof and the new summary to the client 105. Then, the client 105 can verify the integrity of the data as mentioned above. The two-layer POS tree 110 allows the server (currently the authenticator 106) to batch the proof of multiple keys. This is especially true when the keys are packaged in the same data block 124. In addition, obtaining data and proof can be completed simultaneously by traversing the tree 110, which means that when obtaining data during business processing, proof generation can be completed at a low cost. Proofs for persisted data are returned immediately during business processing, and proofs for data that will be persisted in future blocks will be generated in deferred verification requests within batches.

[0080] Once the committed block number appears in the ledger 100, any malicious interaction with the ledger 100 will be detected. GlassDB 100 allows the client 105 to specify a customized delay time for verification to find a suitable tradeoff between security guarantees and performance according to the needs of the client 105. Specifically, zero delay time means immediate verification. In this case, the transaction is synchronously persisted to the ledger storage 110 during the commit phase. This strategy is suitable for applications that cannot afford even temporary violations of data integrity.

[0081] Therefore, GlassDB 100 inherits the verifiability of the transparency log while supporting business and delivering high performance. The transparency log provides two important security properties. First, users can verify that the log is append-only, that is, any successful update operation cannot be reverted. Second, users can verify that the log is linear, that is, there are no forks in the history.

[0082] A unified framework is needed for comparing verifiable ledger databases. Currently, the design space we build consists of three dimensions: an abstraction dimension that captures the interface exposed to the user, which can be a key value or a general business; a threat model dimension that includes different security assumptions; and a performance dimension that includes design choices that affect the size of integrity proofs and overall throughput. The benchmarks used for comparison extend traditional database benchmarks, namely YCSB and TPC-C, with additional workloads that contain verification requests for recent or historical data.

[0083] GlassDB 100 supports distributed transactions and is designed to enable database abstraction and high performance while preserving security. GlassDB supports distributed transactions and has efficient proof size. GlassDB relies on auditing and user gossip for security. GlassDB achieves high throughput by building on top of a novel data structure: a two-layer Merkle-like tree. Each node of GlassDB has multiple threads and a single thread, multiple threads are used to process transactions in parallel and generate proofs, and a single thread is used to update the ledger storage.

[0084] As mentioned above, the transparency log is an append-only log accompanied by a Merkle tree. Figure 2An example of transparency logs 100, 102 at two different times is shown: one transparency log 100 with 3 elements and another transparency log (102) with 6 elements. Each leaf 104 represents an operation, such as updating a key value tuple. The proof of the append operation is a new Merkle root tree. Regarding the run proof: the inclusion proof is the Merkle path from the corresponding leaf to the root. The cost of this proof is O(log(N)), where N is the size of the log. In the example, the proof that element 2 exists in the log consists of the hashes of nodes 1 and b. Regarding the append-only proof, the proof consists of the nodes used to rebuild the two trees, which has a complexity of O(log(N)). In the example, to prove that c is included in l, the proof includes the hashes of nodes e, 3, 4, k. The first three are sufficient to calculate c, and all four are sufficient to calculate l. Regarding the current value proof, the proof includes all leaves of the tree, which has a complexity of O(N). In our example, it is assumed that the latest value of key k is set at node 3. Given l and a tuple, the user has to get all 6 elements and check nodes 4, 5, 6 without updating the tuple.

[0085] The security of the transparency log depends on auditing. Specifically, the user broadcasts the signed Merkle root to several auditors, who check that there is a single log without forks. The check is done by requesting and verifying an append-only proof from the database provider.

[0086] Table 1 above compares existing verifiable ledger databases that are built on top of the transparency log based on the current design space and proof cost. QLDB and LedgerDB are mentioned above. Forkbase is a versioned key-value storage system that implements transparency mapping. Blockchain systems assume a majority of trusted providers in a decentralized setting. CreDB assumes that the server can create a trusted execution environment supported by trusted hardware. About the public key transparency log: Trillian combines the transparency log and mapping to implement a new primitive called verifiable log-based mapping. ECT and Merkle are similar to Trillian in the current design space, but improve on Trillian by adding support for privacy, revocation (non-inclusion proofs), and reducing audit costs.

[0087] Now that the structure and behavior of the GlassDB database have been described, the lifecycle of a transaction at the server will be described. The lifecycle is divided into four phases: prepare, commit, persist, and get proof. The prepare phase checks for conflicts between concurrent transactions before making a commit or abort decision. The commit phase stores the write set in memory and appends the transaction to the WAL for durability and recovery. The persist phase appends the committed in-memory data to the ledger storage and updates the authentication data structure for future verification. The get proof phase generates the requested proof for the client. In GlassDB, the persist and get proof phases are executed asynchronously and in parallel with the other two phases.

[0088] There are several APIs that can be used to interact with the GlassDB server and one or more auditors. The APIs can include:

[0089] Init(sk): Initialize the client with the private key sk to sign the transaction and send the public key pk to the auditor for verification.

[0090] BeginTxn(): Start a transaction. BeginTxn() returns a unique transaction ID tid based on the client ID and timestamp.

[0091] Get(tid,key,(timestamp|block_no)): Returns the most recent value of a given key (the default option), or the value of a key before a given timestamp or block number.

[0092] Put(tid, key, value): Buffers the writing of ongoing services.

[0093] Commit(tid): Sign and send the transaction to the server for commit. Commit(tid) returns a commitment.

[0094] Verify(promise): Requests a proof from the server corresponding to a given promise, then verifies the proof.

[0095] Audit(digest,block_no): Sends a digest of the given block to the auditor.

[0096] One or more auditors use the following APIs to ensure that the database server is working correctly.

[0097] VerifyBlock(digest, block_no): Requests the server for the block at block_no, the block's proof, and the signed block transaction. VerifyBlock(digest, block_no) verifies that all keys in the transaction are included in the ledger.

[0098] VerifyDigest(digest,block_no): Verifies that the given digest and the current digest correspond to the linear history by asking the server to generate an append-only proof. If the given block number is greater than the current block number, VerifyDigest(digest,block_no) uses Verify-block to verify all blocks between the given block number and the current block number.

[0099] Gossip(digest,block_no): broadcasts the current digest and block number to other auditors.

[0100] User authentication ensures that the user's own business is executed correctly. GlassDB ensures the correct execution of the database server across multiple users. In particular, LaedgeBase relies on a collection of auditors, some of which are honest, to ensure that different users see a consistent view of the database.

[0101] Each auditor performs two tasks. The first task involves checking that the server does not have a forked history log. This can be accomplished in a variety of ways. In one embodiment, the first task involves checking that the user receives a digest that corresponds to a linear history. The auditor maintains a current digest d and block number b that corresponds to the longest history the auditor has seen so far. When the auditor receives a digest d' from a user, the auditor asks the server for an append-only proof showing that d and d' belong to a linear history.

[0102] The second task performed by the auditor is to re-execute the transactions to ensure that the current database state is correct. This prevents the server from arbitrarily adding unauthorized transactions that tamper with the state. The second task also defends against undetected tampering when some users do not verify - for example because some users are offline or due to resource constraints. The auditor starts from the same initial state as the initial state at the server. For each digest d and the corresponding block number b, the auditor requests the signed transactions included in the block, as well as the proof of the block and the proof of the transaction. The auditor then verifies the signature on the transaction, performs the signature on the auditor's local state, calculates a new digest, and verifies the new digest based on d.

[0103] When an auditor receives a digest corresponding to a block number b' that is greater than the current block number b, the auditor first requests and verifies an append-only proof from the server. Next, for each block between b and b', the auditor requests services and verifies that the state is correctly updated. Afterwards, the auditor updates the current digest and block number to d' and b', respectively. Finally, after a predetermined interval, the auditor broadcasts its current digest and block number to other auditors (i.e., auditor gossip).

[0104] Finally, some embodiments of GlassDB do not tolerate permanent node failures, but instead support recovery after node crashes and restarts. In particular, if a node fails before the commit phase, the client 105 aborts the transaction after a timeout. Otherwise, the client 105 continues to submit transactions. When the failed node recovers, the failed node:

[0105] - Query the client 105 about the status of the service and then decide whether to abort or submit.

[0106] - Check the WAL for updates that have not yet been persisted to the ledger storage 108 and update the updates accordingly.

[0107] If client 105 fails, the node must wait for client 105 to recover because the 2PC protocol is blocking - the delay caused by this waiting can be alleviated by replacing 2PC with a non-blocking atomic commit protocol, such as Three-Phase Commit (3PC).

[0108] In other embodiments, GlassDB is extended to tolerate permanent node failures by replicating nodes using crash-tolerant protocols such as Paxos and Raft.

[0109] Similar to other verifiable databases, GlassDB incurs additional costs compared to conventional databases to maintain authentication data structures and generate verification proofs. For completeness, first consider a Get operation that returns the latest value of a given key at a given digest (other Get variants are similar): the user checks that the returned proof π is a valid inclusion proof corresponding to the latest value of the key in the POS tree, the root of the POS tree is the digest. Since the POS tree is a Merkle tree, integrity is maintained because the proofs for different values ​​will not correspond to the Merkle path to the latest value, which causes verification failure. Next, consider a Put operation that updates the key. The user verifies that the new value is included in the updated digest as the latest value of the key. By the properties of the POS tree, the result cannot be changed without causing verification failure (for example, by updating a different key or updating a given key with a different value to change the result).

[0110] For append-only, the auditor maintains the latest digest corresponding to the history log H. S,H When the auditor receives a digest value from the user S′,H′ When the auditor asks the server to generate an append-only proof π←ProveAppend(digest S′,H′ digest S,H ). Since our POS tree is a Merkle tree, the upper layer of the Merkle tree grows in an append-only manner. If H ′ is not a prefix of H (assuming |H ′ |<|H|), the server cannot generate a valid π. Therefore, the implementation only appends the attributes.

[0111] In GlassDB, each individual user has the latest digest from the server. Due to deferred authentication, users will see the digest during authentication. Sent with the server's promise. When in the server summary Latest summary and history log at H g Correspondingly, |H g |>|H l |, the server also generates a proof π←ProveAppend( digest ) and prove π←ProveAppend( digest ) in the response to the user. This way, the user can detect any local forks in the user's view of the database. After an interval, the user sends the user's latest summary to the auditor, and the auditor uses the user's latest summary to detect global forks.

[0112] In some embodiments, a delay parameter can be used to specify a delay for making a proof. In the case where the delay parameter is 0, the proof is made immediately. In the case where the delay parameter is greater than 0, multiple operations can be batched in the same proof, thereby improving performance. Make the operation VerifiedPut(k,v,delay): returns a promise. The user then calls GetProof(promise) after delay seconds to retrieve the proof. VerifiedGetLatest(k,fromDigest,delay): returns the latest value of k. The user only sees the history up to headDigest, which may be far behind the latest history. For example, the last time a user interacted with the database, the historical digest of the database was fromDigest. After a while, the history was updated to another digest latestDigest. This query allows the user to specify the last history seen. The integrity proof for this query includes an append-only proof that shows a linear history from fromDigest to latestDigest. Relatedly, VerifiedGetAt(k,atDigest,fromDigest): returns a value when the database history is at atDigest. fromDigest is the last history seen by the user. The integrity proof for this query consists of an append-only proof from fromDigest to atDigest.

[0113] Figure 3 (a) shows the latency of different phases while varying the number of operations per transaction (or transaction size). We observe that as transactions become larger, the latency of the prepare and commit phases increases due to more expensive conflict checking and data operations. Figure 3 (b) shows the latency under different workloads. As the workload moves from read-dominant to write-dominant, the latency of the prepare phase also increases slightly because the larger write set results in more writes and write-read conflict checks. Conversely, the commit latency of the read-dominant workload is much higher than that of the write-dominant workload because read operations are more expensive than write operations in GlassDB. Figure 3 (c) shows the latency breakdown for varying number of nodes. The latency of the prepare and commit phases decreases as the number of nodes increases, since having more shards means fewer keys to be processed per node.

[0114] It is worth noting that for Figure 3In the three experiments in Figure 2, persistence and obtaining attestation latencies remain almost constant. This is because the latency of obtaining attestation and persistence depends only on the number of keys batched. The results were measured at peak throughput, and it was observed that the number of keys included for persistence and attestation generation per node is similar for different numbers of nodes. Therefore, the latency of both operations is constant.

[0115] Figure 4 Figure 1 shows the impact of varying latency on server costs. As latency increases, the server processes validation requests and ledger persistence less frequently. As a result, there is less resource contention for other phases, which results in lower latency for the prepare and commit phases. For the persistence phase, higher latency means larger batches and therefore lower persistence latency per key.

[0116] The cost at the client can also be quantified in terms of each key verification delay and proof size (proof size is proportional to the network cost), e.g. Figure 5 When the latency is higher, the client batches more keys for verification, which results in larger proofs, such as Figure 5 (b), and thus increases the verification latency by 5(a). We note that the cost per key decreases with higher latency, indicating that batching is efficient.

[0117] By fixing the client validation delay - a specified delay before validation occurs, it is possible to compare the impact of the persistence interval on the overall performance to benefit from batching - while varying the persistence interval (time period). Figure 6 (a) shows the performance of read-dominant, balanced, and write-dominant workloads. It can be seen that longer intervals lead to higher throughput across all workloads. This is because infrequent updates to core data structures help reduce contention and increase the effectiveness of batching.

[0118] The impact of verification delay can also be evaluated by fixing the persistence interval and varying the delay. Figure 6 (b) shows that Figure 6 In (b), the throughput increases with greater latency. However, for write-dominant and balanced workloads, the throughput drops after the peaks at 160ms and 320ms, respectively. This is because greater latency results in more keys to be verified, which increases contention with business execution. For write-dominant workloads, the number of keys to be verified increases the fastest as there are more updates.

[0119] For the purpose of the experiment, GlassDB was benchmarked against simulated LedgerDB and simulated QLDB. GlassDB has comparable latency to simulated LedgerDB and has lower latency than simulated QLDB in most phases. This is due to the asynchronous persistence of the authentication data structure. GlassDB has the lowest commit latency because GlassDB only persists the write-ahead log when the transaction is committed. In addition, GlassDB has lower latency in the persistence and get-attestation phases because the size of the data structure is smaller.

[0120] Figure 7 (b) and Figure 7 (c) Comparison of verification latency and per-key proof size for different systems. The simulated QLDB has the smallest proof size. Since the authentication data structure in GlassDB is smaller, GlassDB has a smaller latency than the simulated LedgerDB, resulting in Figure 8 (b) shows a smaller verification time, and GlassDB has more keys per block, thus reducing each key proof.

[0121] Figure 7 (d) shows that as the batch size increases, GlassDB consumes less storage space because there are fewer snapshots to save. The space efficiency is highest when the batch size exceeds 100 milliseconds latency. Since the authentication data structure of the simulated LedgerDB is larger than that of GlassDB, the simulated LedgerDB consumes more storage space.

[0122] Figure 8 (a) and Figure 8 (c) shows the throughput of increasing the number of client and server nodes, respectively, under a mixed workload containing various business types, including new orders and payment transactions. GlassDB outperforms the simulated LedgerDB and simulated QLDB. The average latency of the system is Figure 8 The results are shown in (b). The results are consistent with the throughput performance. Figure 8 (d) shows the latency breakdown at peak throughput for each service type; GlassDB consistently has the lowest latency across all service types.

[0123] Fig. 9 (a) shows the throughput for a specific workload with 16 nodes and increasing number of clients. GlassDB achieves higher throughput than simulated QLDB and simulated LedgerDB. Without deferred validation, the throughput is lower than simulated LedgerDB. Fig. 9(b) shows the latency of each operation. GlassDB outperforms other systems in both read and write latency due to GlassDB’s efficient proof (smaller proof size) and efficient persistence phase.

[0124] Under the same workload in a single-node system, GlassDB outperforms the simulated QLDB and Trillian, as shown in Fig.10 Reflected in.

[0125] Finally, the cost of audit processing is estimated using 8 servers and 64 clients, running a balanced business workload: After an interval, the auditor sends a VerifyBlock(.) request to the server and verifies all new blocks created during the interval. Fig.11 The audit cost is shown for the case where the interval is varied from 20ms to 100ms. Both the delay for verifying a new block and the number of new blocks grow almost linearly with the audit interval. This is because more blocks are created during longer intervals, and verifying each block takes roughly constant time.

[0126] In summary, embodiments of GlassDB address many of the limitations of existing systems. GlassDB supports business, with efficient proof and high performance. A new benchmark that supports verification workloads is used to evaluate GlassDB against three baselines. The results show that GlassDB significantly outperforms the baselines.

[0127] It will be appreciated that many further modifications and permutations of various aspects of the described embodiments are possible. Accordingly, the described aspects are intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims.

[0128] Throughout this specification and the claims that follow, unless the context requires otherwise, the word "comprise" and variations such as "comprising" and "containing", will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.

[0129] Reference in this specification to any previous publication (or information derived from any previous publication) or to any matter that is already known is not, and should not be taken as, an acknowledgement or acknowledgment or any form of suggestion that the previous publication (or information derived from that previous publication) or the matter already known forms part of the common general knowledge in the field to which the present specification is endeavored to relate.

Claims

1. A verifiable ledger database configured to manage ledger services, the verifiable ledger database comprising a plurality of shards, each shard being formed by dividing service data and comprising: a ledger storage configured to provide access to the business data and to support a plurality of attestations; A service manager configured to execute each service according to a corresponding service request; as well as A verifier is configured to verify the transaction by returning the proof according to the verification request.

2. The verifiable ledger database of claim 1, wherein: The ledger storage includes an upper-level POS tree and a lower-level POS tree, each of which includes multiple nodes, including at least one root node and at least one leaf node, and wherein the ledger storage includes a hash chain sequence of blocks, and each leaf node of the upper-level POS tree includes a block number of a corresponding block.

3. The verifiable ledger database of claim 2, wherein: The lower-level POS tree is built on the state of the verifiable ledger database, and the lower-level POS tree includes one or more root nodes, each of which contains a corresponding root hash and the corresponding block number.

4. The verifiable ledger database of claim 3, wherein: The root node of the lower-level POS tree is stored as a leaf node of the upper-level POS tree.

5. A verifiable ledger database according to claim 4, wherein the verifiable ledger database is configured to retrieve the business data from the given block number by locating the corresponding leaf node of the upper POS tree using the given block number and then traversing the lower POS tree to locate the business data.

6. The verifiable ledger database of claim 4, wherein: The verifiable ledger database is configured to update the business data by creating new nodes at the lower POS tree and the upper POS tree using copy-on-write.

7. A verifiable ledger database according to any one of claims 1 to 6, wherein: The business data is modeled as a plurality of keys, and the verifiable ledger database is configured to partition the keys into the shards based on a hash of the keys.

8. The verifiable ledger database according to claim 7, wherein the verifiable ledger database is configured to use a two-phase commit (2PC) protocol to ensure the atomicity of the transaction.

9. A verifiable ledger database according to any one of claims 1 to 8, wherein: The service manager is configured to record each service and respond with a commit or abort based on a concurrency control algorithm.

10. A verifiable ledger database according to any one of claims 1 to 9, wherein: The service manager comprises: A service queue, used for buffering the service request; A plurality of service threads, used for receiving the service requests buffered in the service queue; Shared memory for storing prepared and committed business data; and A persistence thread is used to asynchronously persist the submitted business data to the ledger storage.

11. The verifiable ledger database of claim 10, wherein: The service manager is configured to execute the service in the following manner: If the service queue is not full, dispatching the service requests buffered in the service queue to the service threads in sequence; and When the service queue is full, the service is terminated.

12. A verifiable ledger database according to claim 10 or 11, wherein: The business manager is configured to asynchronously update a ledger stored in the verifiable ledger database by: When the service manager receives the submission message, allowing the shared memory to store the service data in the submitted data map; Write to the write-ahead log WAL; Allowing the persistence thread to persist the business data in the submitted data mapping into the ledger storage to generate persistent data; as well as Remove the persisted data from the committed data map.

13. The verifiable ledger database of claim 12, wherein: The transaction manager is configured to batch transactions to be submitted before updating the ledger by: Collect the corresponding data from each business to be submitted into data blocks; as well as The data blocks created within the time window are appended to the ledger storage.

14. The verifiable ledger database of claim 13, wherein: Collecting the corresponding data from each business to be submitted includes: selecting the corresponding data from the submitted data map version by version.

15. A verifiable ledger database according to any one of claims 1 to 14, wherein: The proof includes one or more of: an inclusion proof, a current value proof, and an append-only proof.

16. A verifiable ledger database according to any one of claims 1 to 15, wherein: The validator is configured to batch process the attestations of the business data.

17. A verifiable ledger database according to any one of claims 12 to 16, wherein: The validator is configured to immediately return the proof of the persisted data during transaction processing.

18. A verifiable ledger database according to any one of claims 1 to 17, wherein: The validator is configured to validate the transaction within a time window after the transaction is processed.

19. A verifiable ledger database according to any one of claims 1 to 17, wherein: The validator is configured to validate the read set and the write set of each transaction.

20. A verifiable ledger database according to any one of claims 1 to 19, comprising one or more auditors for ensuring the correct execution of the verifiable ledger database server, wherein: Each auditor is configured to: checking whether the verifiable ledger database server forked the history log by checking whether a user of the verifiable ledger database received a digest corresponding to a linear history; as well as Re-execute the transaction to ensure that the current state of the verifiable ledger database is correct.

21. A verifiable ledger database according to any one of claims 2 to 20, configured to recover the one or more nodes after the one or more nodes crash and restart.

Images