A verifiable index construction, update and verification method for high read-write contention workloads

Abstract

The present application relates to the field of computer database and the field of verifiable computing, more particularly to a verifiable index construction, update and verification method for high read-write contention workloads, comprising a verifiable index structure and construction method based on Merkle parallel tree, a concurrent update method of verifiable index, a query verification method based on verifiable index; while supporting data verifiability, the present application solves the lock-free concurrent control of read-write conflict, batch parallel can bring very good performance in terms of query and batch update, and the characteristics of multi-version index enable historical versions to be retained, thus supporting verification of historical version data.

Description

Technical Field

[0001] The present invention relates to the fields of computer databases and verifiable computing, and more particularly to a verifiable index construction, update and verification method for high read-write contention workloads. Background Art

[0002] In big data scenarios, it's often necessary to outsource storage and computing resources to third-party service platforms. These third-party platforms store data, receive and process user requests, perform relevant computations, and return results to users. However, third-party platforms are not always trustworthy, as data owners lack direct control over the data. They can maliciously modify or lose data, or even return incorrect results to users. To address these issues, verifiable query technology has been extensively researched, with verifiable indexes currently receiving significant attention. Similar to blockchain's tamper-proofing technology, verifiable indexes play a significant role in verifying data accuracy and integrity.

[0003] However, modern data-driven applications require data storage systems that support both fast writes and fast analytical queries. Current verifiable indexing solutions still suffer from high storage overhead, low query verification efficiency, lock dependency during updates, and a lack of support for historical data traceability. Summary of the Invention

[0004] The technical problem solved by the present invention is to overcome the shortcomings of the existing technology and propose a verifiable index construction, update, and verification method for workloads with high read and write contention. The method uses the Merkle parallel tree (MP tree) structure to implement snapshot isolation and multi-version concurrency control for the verifiable index. The core of the MP tree is based on a pure (immutable) data structure that uses path replication for update to achieve fast multi-version control. The multi-version verifiable index structure can serve as an authentication data structure while supporting efficient read and write concurrent operations. The corresponding concurrent update algorithm and query verification algorithm are designed for this structure.

[0005] The technical solutions of the present invention are as follows:

[0006] A verifiable index construction, update, and verification method for workloads with high read and write contention. The method includes three parts: verifiable index construction, data update, and query verification.

[0007] The “high contention” in the high read-write contention workload means that the difference between the proportion of read transactions and write transactions in the workload does not exceed 25% of the total workload.

[0008] A verifiable index building method for workloads with high read and write contention includes the following steps:

[0009] S11, the data owner requests data storage service from the cloud service provider and provides initial data for building a verifiable index;

[0010] S12: The cloud service provider receives the initial data provided by the data owner and constructs a verifiable index structure MP tree (Merkle Parallel Tree) based on the received initial data. The specific steps are as follows:

[0011] S121, sort the initial data by key and remove duplicate data;

[0012] S122: Take out the middle data, construct a root node with key and value, and divide the remaining data into two parts.

[0013] S123, recursively construct the left and right subtrees of the root node according to S122 using the remaining data;

[0014] S124: After the left and right subtrees are constructed, the hash of the root node is calculated to complete the construction of a verifiable index for workloads with high read and write contention. The resulting index is the initial version database snapshot.

[0015] S13, the data owner saves the hash value of the index root node as verification data for the initial version of the database snapshot;

[0016] The update method for verifiable indexes for workloads with high read and write contention includes the following steps:

[0017] S21, the cloud server receives the update request and caches all data to be inserted;

[0018] S22, the cloud server sorts the data to be inserted according to the key and removes duplicate data. The resulting data is called new data;

[0019] S23: The new data is appended to the original index structure in a path-copying manner to form the latest version of the database index snapshot and generate the corresponding index root hash;

[0020] S24: The data owner saves the latest version index root hash and updates the latest version database snapshot metadata.

[0021] The verification method for verifiable indexes for workloads with high read and write contention includes the following steps:

[0022] S31, the user sends a query request to the cloud server;

[0023] S32, the cloud server receives and executes the query request, and adds the nodes that meet the query conditions to the query result set;

[0024] S33, the cloud server generates verification information based on the query result set;

[0025] S34, the cloud server returns the query results and verification information to the user;

[0026] S35: The user requests the index root hash of the specified version of the database snapshot from the data owner.

[0027] S36: The user reconstructs the index structure based on the query results and verification information, calculates the root hash, and compares it with the root hash provided by the data owner to complete the data verification.

[0028] Beneficial effects

[0029] (1) The present invention proposes a verifiable indexing method for workloads with high read-write contention. This method uses the concept of copy-on-write to avoid read-write conflicts, thereby achieving lock-free concurrency for the entire index structure. This method combines path replication and parallel batch operations and applies them to the verifiable index structure. While supporting tree verifiability, it also provides lock-free concurrency control to resolve read-write conflicts. Batch parallelization can achieve excellent performance in querying and batch updating. At the same time, the multi-version index feature allows historical versions to be retained, thus supporting verification of historical version data.

[0030] (2) The present invention discloses an efficient verifiable indexing method for workloads with high read and write contention. The present invention includes a verifiable index structure and construction method based on a Merkel parallel tree, a concurrent update method for the verifiable index, and a query verification method based on the verifiable index. While supporting data verifiability, the present invention solves lock-free concurrency control for read and write conflicts. Batch parallelism can bring very good performance in query and batch update. The characteristics of multi-version indexing enable historical versions to be retained, thereby supporting verification of historical version data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a diagram of the cloud storage service system architecture of the present invention;

[0032] Figure 2 This is the structure and update process of the Merkle parallel tree of the present invention. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0034] The method for building a verifiable index for workloads with high read and write contention includes the following steps:

[0035] S11, the data owner requests data storage service from the cloud service provider and provides initial data for building a verifiable index;

[0036] S12: The cloud service provider receives the data and builds a verifiable index structure MP tree (Merkle Parallel Tree) based on it. The specific steps are as follows:

[0037] S121, sort the initial data by key and remove duplicate data;

[0038] S122: Take out the middle data, construct a root node with key and value, and divide the remaining data into two parts.

[0039] S123, recursively construct the left and right subtrees of the root node according to S122 using the remaining data;

[0040] S124, after the left and right subtrees are constructed, the hash of the root node is calculated;

[0041] S125: The index is constructed. The index constructed in the above steps is the initial version database snapshot.

[0042] S13, the data owner saves the hash value of the index root node as verification data for the initial version of the database snapshot;

[0043] The data updating method of the verifiable index includes the following steps:

[0044] S21, the cloud server receives the update request and caches all data to be inserted;

[0045] S22, the cloud server sorts the data to be inserted according to the key and removes duplicate data;

[0046] S23: The new data is appended to the original index structure in a path-copying manner to form the latest version of the database index snapshot and generate the corresponding index root hash;

[0047] S24: The data owner saves the latest version index root hash and updates the latest version database snapshot metadata.

[0048] The data query verification method of the verifiable index includes the following steps:

[0049] S31, the user sends a query request to the cloud server;

[0050] S32, the cloud server receives and executes the query request, and adds the nodes that meet the query conditions to the query result set;

[0051] S33, the cloud server generates verification information based on the query result set;

[0052] S34, the cloud server returns the query results and verification information to the user;

[0053] S35: The user requests the index root hash of the specified version of the database snapshot from the data owner.

[0054] S36: The user reconstructs the index structure based on the query results and verification information, calculates the root hash, and compares it with the root hash provided by the data owner to complete the data verification;

[0055] Example

[0056] The method for constructing and verifying a verifiable index with high read / write contention load provided in this embodiment has a system architecture for application scenarios such as Figure 1 As shown, the system consists of three parts: the data owner (DO), the cloud service provider (CSP), and the query client (Client). The data owner signs all outsourced data and uploads the data and signature to the cloud service provider's server. The cloud storage server is responsible for storing the data and maintaining a verifiable index structure, providing fast and convenient query support while also supporting data verification.

[0057] Generally, a database has only one owner, who entrusts the database to a cloud service provider for storage and management. Only the owner can modify the data. There are usually multiple querying parties, who access the owner's database through the cloud service provider. From the perspective of data security, the cloud service provider is untrustworthy and may tamper with the database content for some purpose. When a user makes a query, he first sends a query request to the cloud service provider. The cloud service provider first executes the query request, obtains the query results, generates verification information based on the query results, and finally returns it to the user. The user then requests verification information for the specified version of the database snapshot from the data owner, and finally performs verification based on the query results and verification information.

[0058] The verifiable Merkle parallel tree MP tree in the present invention is defined as a special data structure that can be split into multiple Merkle balanced binary search trees. Each search tree corresponds to an index snapshot of a specific version of the database. For a single search tree, it maintains a set of sorted key-value pairs, where the key and value are of any type, such as integers and strings. Each node in the tree is a data container used to store the data information and data block hash information corresponding to the data block, and its structure is shown in the figure. The search tree satisfies the following properties: (i) the difference in height between the left and right subtrees of any node does not exceed 1 (ii) the number of nodes in the lth layer does not exceed 2^(l-1). The verifiable Merkle parallel tree is a combination of search trees corresponding to multiple versions.

[0059] The structure of the MP tree is as follows Figure 2 As shown: Each node N stores a piece of data value and hash value hash, where the hash value of the leaf node N.hash = h1(h2(N.value)), where h1 and h2 are one-way hash functions, and the hash value of the non-leaf node N.hash = h1(h2(N.value)|L.hash|R.hash), where L and R represent the left and right child nodes, and | represents the connection operation.

[0060] The following process involves two important atomic operations:

[0061] concat(L,N,R) or concat(L,R), where L and R are the root nodes of a balanced tree and N is a single node, concat connects the balanced trees containing L and R (or adds node N) into a single binary tree. This tree is then adjusted to become a balanced binary tree again. Adjusting a balanced binary tree involves not only modifying the pointer fields but also recalculating the hash values. Finally, the root node of the newly balanced binary tree is returned.

[0062] copy(N): Constructs a new node N', whose attributes are exactly the same as N, and finally returns N'.

[0063] The construction process of the verifiable index MP tree is as follows:

[0064] 1. Sort the initial data by key and remove duplicate data;

[0065] 2. If the number of keywords is even and non-zero, the data set is divided into two parts, and two balanced search subtrees T1 and T2 are recursively constructed respectively. The two subtrees are connected using the primitive concat(T1, T2) to form a complete binary balanced search tree.

[0066] 3. If the number of keywords is odd, remove the middle data and construct node N accordingly. Use the remaining data to construct subtrees T1 and T2 according to the method in step 2. Use the primitive concat(T1,N,T2) to connect the node and the two subtrees to form a complete binary balanced search tree.

[0067] 4. If the number of keywords is zero, it returns directly to the upper layer call;

[0068] 5. Recursively perform steps 2 to 4;

[0069] 6. Finally generate the initial version of the database index snapshot;

[0070] The update process of the verifiable index MP tree is as follows:

[0071] 1. The cloud server receives and caches the data to be inserted, sorts it by key, and removes duplicate data;

[0072] 2. Traverse from the root node N from top to bottom. If the number of data to be inserted is not zero, clone the root node through the primitive copy(N), denoted as N'; otherwise, return directly to the upper layer to call;

[0073] 3. If the updated data involves N, then modify the value of N' (N will not be modified in the whole process);

[0074] 4. Using the key of N as the benchmark, split the data into two parts and process the left and right subtrees L and R respectively. The processing of the left and right subtrees is concurrent.

[0075] 5. Get the updated left and right subtree root nodes L' and R', connect the node N' and the subtree through concat(L',N',R') and adjust them into a balanced binary search tree;

[0076] 6. Recursively perform steps 2 to 5;

[0077] 7. Finally, a new version of the balanced Merkle tree is generated, with the root node being root'. Physically, it shares the same unchanged nodes as the previous version of the balanced Merkle tree and forms the verifiable index structure MP tree of the entire database.

[0078] 8. The data owner saves the most recent root's hash value and updates the latest version of the database snapshot metadata;

[0079] The query data contains the target keyword data and the verifiable retrieval method is as follows:

[0080] 1. The user sends a query request query=<v,q> , where v represents the snapshot version and q represents the query condition;

[0081] 2. The cloud server executes the query request, finds the index root node corresponding to version v, traverses and searches for all nodes that meet condition q, and adds them to the query result set result; generates verification information VOsp, which includes the hash values ​​of the sibling nodes of the node in the query result set (if it is not part of the query result) and the values ​​of all ancestor nodes;

[0082] 3. The cloud service provider will use the result and VO SP Return to the user together;

[0083] 4. The user requests the database index snapshot root hash VO of version v from the data owner. DO ;

[0084] 5. The user constructs a Merkle balanced tree structure from bottom to top based on the query result set and verification information, calculates the hash value of the index root node, and then compares it with the VO DO Compare and verify the correctness of the results.

[0085] This article uses specific embodiments to illustrate the principles and implementation methods of this article. The description of the above embodiments is only used to help understand the methods and core ideas of this article. At the same time, for those skilled in the art, based on the ideas of this article, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation to this article.

Claims

1. A verifiable index construction, update and verification method for high read and write contention workloads, characterized by The method comprises three parts: a verifiable index construction method for high read-write contention workloads, a verifiable index update method for high read-write contention workloads, and a verifiable index verification method for high read-write contention workloads; The verifiable index construction method for high read-write contention workloads includes the following steps: S11, the data owner requests data storage service from the cloud service provider and provides initial data for building a verifiable index; S12, the cloud service provider receives the initial data provided by the data owner and builds a verifiable index structure MP tree merkle parallel tree based on the received initial data; In step S12, a verifiable index structure MP tree merkle paralleltree is constructed based on the received initial data. The specific steps are as follows: S121, sort the initial data by key and remove duplicate data; S122: Take out the middle data, construct a root node with key and value, and divide the remaining data into two parts. S123, recursively construct the left and right subtrees of the root node according to S122 using the remaining data; S124: After the left and right subtrees are constructed, the hash of the root node is calculated to complete the construction of a verifiable index for workloads with high read and write contention. The resulting index is the initial version database snapshot. The method for updating a verifiable index for a workload with high read / write contention comprises the following steps: S21, the cloud server receives the update request and caches all data to be inserted; S22, the cloud server sorts the data to be inserted according to the key and removes duplicate data. The resulting data is called new data; S23: The new data is appended to the original index structure in a path-copying manner to form the latest version of the database index snapshot and generate the corresponding index root hash; S24: The data owner saves the latest version index root hash and updates the latest version database snapshot metadata. The verification method for verifiable indexes for workloads with high read and write contention includes the following steps: S31, the user sends a query request to the cloud server; S32, the cloud server receives and executes the query request, and adds the nodes that meet the query conditions to the query result set; S33, the cloud server generates verification information based on the query result set; S34, the cloud server returns the query results and verification information to the user; S35: The user requests the index root hash of the specified version of the database snapshot from the data owner. S36: The user reconstructs the index structure based on the query results and verification information, calculates the root hash, and compares it with the root hash provided by the data owner to complete the data verification.

2. A verifiable index construction, update, and verification method for high read / write contention workloads according to claim 1, characterized in that: In the process of building the verifiable index structure MP tree merkle parallel tree, the hash value of the index root node can be obtained.

3. A verifiable index construction, update, and verification method for high read / write contention workloads according to claim 2, characterized in that: The data owner saves the hash value of the index root node as verification data for the initial version of the database snapshot.

Images