A Verifiable Approximate Nearest Neighbor Query Method and System Based on Consortium Blockchain

Abstract

This invention discloses a verifiable approximate nearest neighbor query method and system based on a consortium blockchain. The verifiable approximate nearest neighbor query system includes: a data provider, a query service provider, a user, a blockchain sorting cluster, and a smart contract. The data provider outsources data to the query service provider. The query service provider provides the user with the outsourced query results and corresponding cryptographic proofs. The user initiates a verifiable approximate nearest neighbor query to the query service provider and verifies the correctness of the query results via a VO (Value Object). The smart contract maintains an immutable, trusted data structure, a Merkle HNSW node tree. The blockchain sorting cluster records the Merkle root of the contract execution result (ADS) on the blockchain. This method solves the problem of users lacking a method to verify the correctness of query results in outsourced query scenarios targeting approximate nearest neighbor query semantics.

Description

Technical Field

[0001] This invention relates to the field of blockchain application technology, and in particular to a verifiable approximate nearest neighbor query method and system based on consortium blockchain. Background Technology

[0002] Blockchain technology is an innovative transaction recording technology. A blockchain network is a decentralized network with multiple nodes. Nodes achieve consistency in transaction order, content, and execution results through a consensus protocol, thus ensuring the uniqueness of transaction data. Furthermore, cryptographic hashing technology is used to guarantee the integrity of the block data containing the transaction. Therefore, transactions on the blockchain are unique and immutable.

[0003] There are currently two types of blockchains: consortium blockchains and public blockchains. Compared to public blockchains, consortium blockchains have the following advantages: (1) better information protection, (2) faster transaction speeds, and (3) lower transaction fees.

[0004] Information Protection: Consortium blockchains employ certificate-based access control, meaning each node in a consortium blockchain network requires a valid, authorized identity certificate to join. In contrast, nodes in public blockchains can freely join or leave the network. Therefore, consortium blockchains are more suitable than public blockchains for enterprise-level applications requiring information protection, such as supply chain management, medical data storage, and electronic receipts.

[0005] Faster transaction speeds: Because any node can participate in a public blockchain, every transaction needs to be verified by a majority of nodes through a consensus protocol to prevent double-spending and fraud. This consensus process consumes significant computational resources and time. While ensuring the security of the consensus, it also limits transaction speed. In a consortium blockchain network, participating nodes are certified entities, allowing the use of more efficient consensus protocols, such as Byzantine Fault Tolerance (PBFT), thereby shortening the consensus time and accelerating transaction speeds.

[0006] Low transaction fees: To avoid smart contracts getting stuck in infinite loops, public blockchains employ a gas-based transaction fee mechanism. Executing complex logic in public blockchain smart contracts consumes significant gas transaction fees, increasing transaction costs. In consortium blockchains, if a smart contract gets stuck in an infinite loop, consortium members can quickly reach a consensus and immediately request the node to stop contract execution. Transaction fees in consortium blockchains are typically very low because the operation and security of the consortium blockchain are maintained by its members, with relatively fixed and shared costs. Therefore, consortium blockchains can support the execution of complex smart contract logic without worrying about high costs.

[0007] With the development of Data as a Service (DAS) technology, more and more cloud service providers are offering outsourced query services, enabling users to access large-scale datasets across various fields. Users no longer need to download and store the entire dataset; they can simply delegate the query to the cloud service provider to obtain the results. K-Approximate Nearest Neighbors Search (K-ANNS) is a widely used query semantic in scenarios such as similar music retrieval, personalized services, and stock recommendations. This semantic finds the K nearest neighbors (K-ANNS) in the dataset that are closest to the given query data, given a distance function between vector data. Service providers build hierarchical navigable small world graphs (HNSWs) locally based on the dataset, thus efficiently supporting users' approximate nearest neighbor query requests.

[0008] In traditional near nearest neighbor outsourced query systems, users can only choose to trust cloud service providers and lack the means to verify the correctness of the K-ANNS query results provided by the service providers. Summary of the Invention

[0009] To address the lack of a method for verifying the correctness of outsourced query results in scenarios involving approximate nearest neighbor (APN) queries, this invention proposes a verifiable APN query method and system based on a consortium blockchain. By designing a trusted data structure based on a Merkle HNSW node tree within the consortium blockchain smart contract, this invention ensures that users can verify the authenticity of the outsourced query results provided by the service provider, thereby preventing malicious actions such as service providers tampering with the original dataset or providing users with incorrect query results.

[0010] This invention first provides a verifiable approximate nearest neighbor query method based on consortium blockchains, comprising the following steps:

[0011] 1) Identify the original data provider and query service provider in the consortium blockchain, and deploy smart contracts capable of building ADS in the query service provider of each consortium blockchain based on the construction parameters, thereby building a verifiable approximate nearest neighbor query platform based on the use of consortium blockchain smart contracts; the construction parameters of the consortium blockchain include the construction parameters of the HNSW graph index and the construction parameters of the Merkle HNSW node tree.

[0012] 2) The original data provider sends the original dataset to the query service provider; the query service provider builds an HNSW graph index locally based on the original dataset;

[0013] 3) The original data provider calls the smart contract to generate an outsourced query result verification process based on the Merkle HNSW node tree;

[0014] 4) The user initiates a near nearest neighbor query request to the outsourced query service provider, receives and verifies the query results.

[0015] As a preferred embodiment of the present invention, step 1) specifically comprises:

[0016] 1.1) Identify the original data provider and the query service provider in the consortium blockchain, and issue a certificate for calling smart contracts to the original data provider and a certificate for deploying smart contracts to the query service provider;

[0017] 1.2) Construction parameters for member coordination HNSW graph index and Merkle HNSW node tree in consortium blockchains;

[0018] 1.3) The query service provider is used as a consortium blockchain network node, and a smart contract is deployed in the consortium blockchain network node; the original data provider calls the smart contract to record the construction parameters of the HNSW graph index and the construction parameters of the Merkle HNSW node tree on the consortium blockchain, thus completing the construction of a verifiable near nearest neighbor query platform.

[0019] As a preferred embodiment of the present invention, step 4) specifically comprises:

[0020] 4.1) The user initiates a near nearest neighbor search query request to the query service provider;

[0021] 4.2) The query service provider queries the HNSW index based on the approximate nearest neighbor search query request from the user to obtain the query results and generates a cryptographic proof VO for the query service provider. 查询服务提供端 The query service provider will combine the query results with the VO 查询服务提供端 Send to the user client;

[0022] 4.3) The user client queries the cryptographic proof (VO) recorded on the consortium blockchain. blockchain Utilizing VO 查询服务提供端 and VO blockchain Verify the correctness of the query results. If the user determines that the query results are correct, the query ends. If the user determines that the query results are incorrect, the user initiates an approximate nearest neighbor search query request to other query service providers in the consortium blockchain.

[0023] The present invention also provides a system for implementing the above-described verifiable approximate nearest neighbor query method, comprising a raw data provider, a query service provider, a consortium blockchain construction parameter coordination module, and a smart contract deployment module;

[0024] The original data provider is used to send the original dataset to the query service provider and to call the smart contract to generate an outsourced query result verification process based on the Merkle HNSW node tree;

[0025] The query service provider is used to deploy smart contracts and build an HNSW graph index locally based on the original dataset;

[0026] The consortium blockchain construction parameter coordination module is used to coordinate the construction parameters of the HNSW graph index and the construction parameters of the Merkle HNSW node tree in the consortium blockchain.

[0027] The smart contract deployment module is used to deploy smart contracts in each query service provider.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] 1) An organizational architecture for a verifiable approximate nearest neighbor (ADS) query system using consortium blockchain smart contracts is presented in an outsourced query scenario. Multiple query service providers act as nodes in the consortium blockchain network, deploying smart contracts to build ADS, thereby maintaining an immutable and trustworthy data structure, the Merkle HNSW node tree. Using this trustworthy data structure, the user end can verify the correctness of the outsourced query results.

[0030] 2) A dataset outsourcing and contract deployment process based on a consortium blockchain is proposed. By negotiating the key parameters for building the index and ADS and recording them on the blockchain, it is ensured that honest query service providers build a consistent query index when receiving a unified outsourced dataset; and build the same ADS when deploying smart contracts with consistent algorithms.

[0031] 3) In the process of using smart contracts, a trusted data structure generation scheme based on four stages—"transaction invocation - contract execution - result collection - result recording"—was implemented. When the data provider can collect a majority of consistent execution results, the transaction execution result is trusted, and the corresponding Merkle root is stored on the blockchain to ensure the immutability of ADS.

[0032] 4) A verification process for outsourced query results based on the Merkle HNSW node tree is proposed. After the user initiates a query, the query service provider generates the query result R, and generates the corresponding cryptographic proof VO based on ADS. 查询服务提供端 This is then returned to the user. The user retrieves the publicly available Merkle root VO from the blockchain. blockchain Using VO 查询服务提供端 and VO blockchain The client executes a verification algorithm locally to confirm the correctness of the approximate nearest neighbor query results. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the organizational structure of a verifiable approximate nearest neighbor query system;

[0034] Figure 2 A flowchart for initializing participating entities and determining key parameters in a consortium blockchain;

[0035] Figure 3 A flowchart illustrating the process of sending outsourced datasets to the data provider and building query indexes for the query service provider;

[0036] Figure 4 A flowchart illustrating the process of calling smart contracts and storing Merkle roots on the blockchain;

[0037] Figure 5 A schematic diagram of a Merkle HNSW node tree containing 8 nodes;

[0038] Figure 6 A flowchart for completing the approximate nearest neighbor query request and verification for the user. Detailed Implementation

[0039] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0040] Consortium blockchains provide a secure and reliable smart contract execution environment, and Merkle hash trees are a typical Authenticated Data Structure (ADS) that supports verifiable range query semantics. In outsourced query scenarios, utilizing consortium blockchain smart contracts to implement a trusted data structure, thereby supporting verifiable near-nearest neighbor query methods on the user end, is an effective solution to this problem and improves the security of outsourced query services.

[0041] like Figure 1 As shown, the participating entities of a verifiable approximate nearest neighbor query system using consortium blockchain smart contracts according to the present invention include an original data provider, a query service provider, a user, a smart contract for constructing ADS, and a blockchain sorting cluster.

[0042] The original data provider is the source of the dataset that the user needs to query. However, the data provider lacks sufficient computing resources and therefore cannot directly provide query services to the user. The data provider outsources the dataset to a query service provider, who then provides the query service to the user.

[0043] Outsourced query service providers offer query results and corresponding VOs to users. Multiple query service providers, upon receiving datasets from data providers, construct hierarchical, navigable small-world graph indexes locally based on the datasets. When a query service provider receives an approximate nearest neighbor query from a user, it efficiently retrieves the query results from the graph index and returns the results and cryptographic proofs to the user. Each query service provider is also a node in the consortium blockchain network, with smart contracts deployed on each node.

[0044] The user initiates a near nearest neighbor query request to the outsourced query service provider, and receives and verifies the query results.

[0045] By executing smart contracts, a Merkle HNSW node tree can be built based on the local graph index of the query service provider, and the resulting transaction execution result is the Merkle root. When the data provider collects more than 50% of the consistent contract execution results, it sends them to the block sorting cluster.

[0046] The block sorting cluster is responsible for collecting blockchain transactions, checking their validity, discarding invalid transactions, packaging valid transactions into blocks, and linking them to the blockchain ledger.

[0047] This invention provides a verifiable approximate nearest neighbor query method based on consortium blockchains, comprising the following steps:

[0048] 1) Identify the original data provider and query service provider in the consortium blockchain, coordinate the construction parameters of the consortium blockchain, and deploy smart contracts capable of building ADS in the query service provider of each consortium blockchain, thereby building a verifiable approximate nearest neighbor query platform based on the use of consortium blockchain smart contracts; the construction parameters of the consortium blockchain include the construction parameters of the HNSW graph index and the construction parameters of the Merkle HNSW node tree.

[0049] 2) The original data provider sends the original dataset to the query service provider; the query service provider builds an HNSW graph index locally based on the original dataset;

[0050] 3) The original data provider calls the smart contract to generate an outsourced query result verification process based on the Merkle HNSW node tree;

[0051] 4) The user initiates a near nearest neighbor query request to the outsourced query service provider, receives and verifies the query results.

[0052] like Figure 2 As shown, step 1) specifically includes:

[0053] 1-1) Determine the identities of members in the consortium blockchain. Issue certificates to the original data provider for invoking smart contracts. Issue certificates to the query service provider for deploying smart contracts.

[0054] 1-2) Members in the consortium blockchain coordinate the key construction parameters of the query index HNSW, including: the maximum number of connections M between nodes in the index, the length of the dynamic candidate list efConstruction used in the index construction phase, the seed Seed of the graph index level where the generated node is located, and the length of the dynamic candidate list efSearch used in the index query phase.

[0055] 1-3) Members in the consortium blockchain agree on the construction parameters of the trusted data structure Merkle HNSW node tree, including: the maximum number of nodes L in each sharded subtree, and the fan-out F of the tree (the number of child nodes of the intermediate nodes).

[0056] 1-4) Deploy a smart contract on the query service provider node. The smart contract contains the algorithmic logic for constructing a trusted data structure, the Merkle HNSW node tree, based on the HNSW graph. The original data provider calls the smart contract to record the construction parameters of HNSW and the trusted data structure on the blockchain.

[0057] like Figure 3 As shown, step 2) specifically includes:

[0058] 2-1) The original data provider sends dataset D to multiple query service providers. Dataset D contains N vectors with dimension W. The data provider signs the dataset and also signs the data features within it. Prove that the data was sent by this provider.

[0059] 2-2) Upon receiving the dataset, each query service provider first checks its integrity, verifying that the dataset has been signed by the data provider and that the data characteristics within the dataset match the received dataset. If the dataset is lost during transmission, causing signature verification to fail, the query service provider re-requests the data from the data provider.

[0060] 2-3) Each query service provider uses the HNSW index insertion algorithm to build an HNSW graph index locally based on the dataset. The completed graph index contains N nodes and C levels. Each node contains a node number, vector data, and the node's neighbor relationships. Given consistent index building parameters and the use of the same dataset, multiple query service providers can build a consistent query index. After all N vector data points in dataset D have been inserted into the HNSW query index, the query index will contain N nodes. Node N id The definition is as shown in formula (1), including: node number id, node vector data value v.id The neighbor relationship of the node is eConn.

[0061] N id ={id,v id ,eConn} (1)

[0062] 2-4) The query service provider returns a notification to the data provider that the index construction is complete.

[0063] By completing the invocation of smart contracts and recording them on the Merkle root on the blockchain, a trusted data structure generation scheme based on the four stages of "transaction invocation - contract execution - result collection - result recording" is realized, such as... Figure 4 As shown, step 3) specifically includes:

[0064] 3-1) Transaction Invocation: The data provider invokes the ADS construction contracts deployed at multiple query service providers to initiate an ADS construction transaction.

[0065] 3-2) Contract Execution: After receiving the call transaction, the smart contract deployed at each query service provider reads the node information of the HNSW query index and constructs the Merkle HNSW node tree. The definition is as shown in formula (1). Figure 5 As shown, if there are 8 nodes in the query index, the nodes are hashed and then used as leaf nodes of the Merkle tree in order, with the root being... The smart contract returns the execution result of the ADS-constructed transaction, i.e., the Merkle root, to the data provider.

[0066] 3-3) Result Collection: The data provider collects the smart contract execution results from multiple query service providers. If more than 50% of the query service providers return consistent results, the data provider sends the result to the sorting cluster of the blockchain; otherwise, it continues to collect the smart contract execution results.

[0067] 3-4) Result Recording: After the sorting cluster verifies the validity of the transaction, it stores the ADS-constructed transaction and the corresponding execution result Merkle root on the blockchain. Subsequently, the user can query the value of the Merkle root through a public blockchain explorer.

[0068] 4) Complete the request and verification of the approximate nearest neighbor query on the user side, such as... Figure 6 As shown, it includes:

[0069] 4-1) The user initiates an approximate nearest neighbor query request Q to the query service provider.

[0070] 4-2) Upon receiving Q, the query service provider queries its local HNSW index to obtain the query results. The query service provider needs to search the index portion and its corresponding data, represented by I.q If the query service provider will use I q Send it to the user terminal, and the user terminal can then use it based on I q A local search algorithm is executed to obtain the same query results as the query service provider, ultimately verifying the correctness of the query results. Therefore, ADS is needed to ensure I... q The integrity of the data is guaranteed. A Merkle hash tree is a typical example of an Adjacent Node Tree (ADS). By improving the Merkle hash tree and implementing a Merkle HNSW node tree, the integrity of the data can be guaranteed. q integrity

[0071] In a specific embodiment of the present invention, I q It contains 2 nodes I q ={N5,N6}, then based on as Figure 5 The trusted data structure Merkle HNSW node tree shown is generated This represents the hash value of node N5. The hash value of intermediate nodes is calculated from the hash values ​​of their two child nodes. The query service provider will use VO... 查询服务提供端 Both the query result R and the query result I are sent to the user's client. R contains the query result and I... q .

[0072] 4-3) To verify I q For integrity, the user client can use and I q ={N5,N6} are recalculated to obtain the Merkle root. Since the Merkle root (VO) of ADS... blockchain It has already been stored on the blockchain, and users can obtain it through a blockchain explorer. Therefore, the user can access the recalculated data. The query is compared with the publicly accessible Merkle root on the blockchain. If they match, the client can confirm the I sent by the query service provider. q It has not been tampered with.

[0073] Meanwhile, the user terminal is based on I q The calculated query results are compared with the query results sent by the query service provider. If they match, the correctness of the approximate nearest neighbor outsourcing query can be confirmed.

[0074] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A verifiable approximate nearest neighbor query method based on consortium blockchains, characterized in that, Includes the following steps: 1) Identify the original data provider and query service provider in the consortium blockchain. Based on the consortium blockchain construction parameters, deploy smart contracts capable of building a trusted data structure (ADS) in each query service provider of the consortium blockchain, thereby building a verifiable approximate nearest neighbor query platform based on the use of consortium blockchain smart contracts. The construction parameters of the consortium blockchain include the construction parameters of the HNSW graph index and the construction parameters of the Merkle HNSW node tree. 2) The original data provider sends the original dataset to the query service provider; the query service provider builds an HNSW graph index locally based on the original dataset; 3) The original data provider calls the smart contract in the consortium blockchain to generate a Merkle HNSW node tree to support the query result verification process; 4) The user can log in to the approximate nearest neighbor query platform and initiate an approximate nearest neighbor query request to the query service provider. After receiving the approximate nearest neighbor query request, the query service provider obtains the query result according to the HNSW graph index and sends the query result to the user. The user receives the query result and verifies the query result through the Merkle HNSW node tree in the consortium blockchain.

2. The verifiable approximate nearest neighbor query method according to claim 1, characterized in that, Step 1) specifically includes the following steps: 1.1) Identify the original data provider and the query service provider in the consortium blockchain, and issue a certificate for calling smart contracts to the original data provider and a certificate for deploying smart contracts to the query service provider; 1.2) Construction parameters for member coordination HNSW graph index and Merkle HNSW node tree in consortium blockchains; 1.3) The query service provider is used as a consortium blockchain network node, and a smart contract is deployed in the consortium blockchain network node; the original data provider calls the smart contract to record the construction parameters of the HNSW graph index and the construction parameters of the Merkle HNSW node tree on the consortium blockchain, thus completing the construction of a verifiable near nearest neighbor query platform.

3. The verifiable approximate nearest neighbor query method according to claim 1, characterized in that, Step 2) specifically includes the following steps: 2.1) The original data provider sends the original dataset to each query service provider. The original data provider signs the original dataset and also signs the data features in the dataset. 2.2) After receiving the original dataset, each query service provider first performs signature verification to check the integrity of the original dataset, that is, to verify that the original dataset has been signed by the original data provider and to check whether the data characteristics in the dataset conform to the data characteristic signature; if the original dataset is lost during transmission, causing the signature verification to fail, the query service provider will request the original dataset again from the original data provider. 2.3) Each query service provider uses the HNSW index insertion algorithm to build an HNSW graph index locally based on the validated dataset. After the HNSW graph index is built, the query service provider returns a notification to the original data provider that the HNSW graph index has been built.

4. The verifiable approximate nearest neighbor query method according to claim 3, characterized in that, In step 2.3), if the construction parameters of the HNSW graph index are determined and the same original dataset is used, all query service providers can construct a consistent HNSW graph index.

5. The verifiable approximate nearest neighbor query method according to claim 1, characterized in that, In step 3), the outsourced query result verification process includes transaction invocation, contract execution, result collection, and result recording. Specifically, the transaction call involves the original data provider calling smart contracts deployed at multiple query service providers to initiate an ADS transaction. The contract execution specifically involves the smart contract deployed at the end of each query service receiving a call to ADS to construct a transaction, reading the node information of the HNSW query index, and constructing a Merkle HNSW node tree. Specifically, the result collection involves the original data provider collecting smart contract execution results from multiple query service providers. If more than 50% of the query service providers return consistent results, the original data provider sends the result to the block sorting cluster in the consortium blockchain to verify the validity of the transaction; otherwise, it continues to collect smart contract execution results. The result record specifically involves the blockchain sorting cluster verifying the validity of the transaction, and then constructing a consortium blockchain cryptographic proof (VO) of the transaction and its corresponding execution result using ADS. blockchain Stored on the consortium blockchain; the user client queries the consortium blockchain cryptographic proof (VO) through a publicly available consortium blockchain explorer. blockchain .

6. The verifiable approximate nearest neighbor query method according to claim 1, characterized in that, Step 4) specifically involves: 4.1) Upon logging in, the user can verify the approximate nearest neighbor query platform and initiate an approximate nearest neighbor search query request to the query service provider; 4.2) The query service provider queries the HNSW index based on the approximate nearest neighbor search query request to obtain the query results and generates a cryptographic proof VO for the query service provider. 查询服务提供端 The query service provider will combine the query results with the VO 查询服务提供端 Send to the user client; 4.3) The user client queries the cryptographic proof (VO) recorded on the consortium blockchain. blockchain Utilizing VO 查询服务提供端 and VO blockchain Verify the correctness of the query results. If the user determines that the query results are correct, the query ends. If the user determines that the query results are incorrect, the user initiates an approximate nearest neighbor search query request to other query service providers in the consortium blockchain.

7. The verifiable approximate nearest neighbor query method according to claim 6, characterized in that, The client executes a local search algorithm based on the HNSW index required to obtain the query results, and obtains the local query results.

8. A system for implementing the verifiable approximate nearest neighbor query method of claim 1, characterized in that, It includes the user end, the raw data provider end, the query service provider end, the consortium blockchain construction parameter coordination module, and the smart contract deployment module; The user terminal is used to initiate an approximate nearest neighbor query request to the query service provider and to verify the received query results; The raw data provider is used to send the raw dataset to the query service provider and call the smart contract to generate a trusted data structure Merkle HNSW node tree, thereby supporting the verification process of outsourced query results. The query service provider is used to deploy smart contracts and build an HNSW graph index locally based on the original dataset, and provide query results to the user after the user initiates an approximate nearest neighbor query request. The consortium blockchain construction parameter coordination module is used to coordinate the construction parameters of the HNSW graph index and the construction parameters of the MerkleHNSW node tree in the consortium blockchain. The smart contract deployment module is used to deploy smart contracts in each query service provider.

Images