Vector database retrieval acceleration method and system based on quantum Grover-Merkle Tree algorithm

Abstract

The invention provides a vector database retrieval acceleration method and system based on a quantum Grover-Merkle Tree algorithm, and the method comprises the steps: obtaining query text data, carrying out the vectorization of the query text data, and intelligently scheduling different computing resources under a heterogeneous architecture, so as to match the similarity between a query vector and a candidate vector in a database. Classical computing resources are subjected to conventional processing by being mapped to a traditional index structure, and quantum computing resources execute efficient retrieval on an index constructed by Merkle Tree by using a Grover retrieval algorithm. Through dynamic scheduling and switching of resources, the system can be flexibly switched between classic computing and quantum computing, the retrieval efficiency of a vector database is remarkably improved, and the system is particularly suitable for large-scale data sets. Meanwhile, the system also integrates a quantum resistance hash function, so that the security and integrity of the data are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a vector database retrieval acceleration method and system based on a quantum Grover-Merkle Tree algorithm. Background Art

[0002] Vector Database is a database system specially designed for storing, indexing, querying and calculating vector data. With the rapid development of big data and artificial intelligence technology, vector data is increasingly used in many fields, such as image recognition, natural language processing and recommendation systems. Vector database, with its unique design and efficient processing capabilities, has become an ideal choice for processing such data.

[0003] Vector databases use vectorized query execution engines that can process multiple data at once, greatly reducing computational complexity and increasing processing speed. Compared with traditional relational databases, vector databases have significant advantages in processing large-scale vector data. Vector databases use high-dimensional indexing technologies, such as KD-Tree and LSH, to divide the vector space into multiple hyperplanes and establish index tables to quickly locate and retrieve high-dimensional vector data. This technology enables vector databases to support efficient similarity queries and range queries.

[0004] However, the retrieval process of the vector database in the prior art consumes a lot of computing time, resulting in low efficiency. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a vector database retrieval acceleration method based on the quantum Grover-Merkle Tree algorithm to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present invention provides a vector database retrieval acceleration method based on a quantum Grover-Merkle Tree algorithm, the method comprising the steps of:

[0007] Obtain database text data and convert it into candidate vectors;

[0008] Building a heterogeneous architecture of a quantum vector database based on the candidate vectors, the heterogeneous architecture of the quantum vector database comprising a first hash index tree structure and a second hash index tree structure, classical computing is based on the first hash index tree structure, and quantum computing is based on the second hash index tree structure;

[0009] In the step that the heterogeneous architecture of the quantum vector database includes a first hash index tree structure and a second hash index tree structure, a hash value of a candidate vector is generated by a classic KD tree or a BallTree to construct the first hash index tree structure; a hash value of the candidate vector is generated by a quantum resistant hash function to construct the second hash index tree structure;

[0010] Obtain query text data, convert the query text data into a query vector, and obtain the current state of the quantum computer;

[0011] If the quantum computer is in standby mode, the quantum computer is started to flexibly switch between classical and quantum computing; and based on the query vector and the candidate vector in the second hash index tree structure, the state vector similarity QSS or cosine similarity calculation is performed and the Grover retrieval algorithm is performed through quantum computing to output the database text data corresponding to the matching candidate vector;

[0012] If the quantum computer is not in the standby mode, cosine similarity calculation is performed based on the query vector and the candidate vectors in the first hash index tree structure, and a classical search is performed to output the database text data corresponding to the matching candidate vectors.

[0013] The above scheme is adopted. This scheme first constructs the text data in the database into vector data, and then constructs two tree data based on the vector data. If the quantum computer is in standby mode, quantum computing can be used. The query vector and each candidate vector are calculated similarity using the Grover retrieval algorithm. The similarity is used to determine whether the two match. If they match, the text corresponding to the candidate vector is output.

[0014] In some embodiments of the present invention, in the step of constructing a heterogeneous architecture of a quantum vector database based on a candidate vector, the candidate vector is used as the root node of a first hash index tree structure or a second hash index tree structure, and a quantum-resistant hash function is used to divide the candidate vector multiple times to construct a quantum-resistant hash Merkle Tree structure.

[0015] In some embodiments of the present invention, in the step of using a quantum-resistant hash function to divide the candidate vector multiple times to construct a quantum-resistant hash Merkle Tree structure, in the calculation process of each division, the quantum random number provided by the quantum random number generator is combined with the original data and then hashed to obtain a hash value.

[0016] In some embodiments of the present invention, in the step of performing a state vector similarity QSS or cosine similarity calculation based on the query vector and the candidate vectors in the second hash index tree structure and performing a Grover retrieval algorithm retrieval through quantum computing, and outputting the database text data corresponding to the matching candidate vector, the query vector and each candidate vector are searched by the Grover retrieval algorithm, the state vector similarity QSS or cosine similarity calculation between the query vector and each candidate vector is calculated, the candidate vector matching the query vector is determined based on the similarity, and the integrity of the result is verified by the quantum resistant hash index.

[0017] In some embodiments of the present invention, in the step of verifying the integrity of the result through a quantum-resistant hash index, a determination is made layer by layer from any leaf node in the second hash index tree structure up to the root node to confirm whether the hash value of the final root node matches the overall hash value of the Merkle Tree.

[0018] In some embodiments of the present invention, in the step of performing cosine similarity calculation based on the query vector and the candidate vectors in the first hash index tree structure and executing classical retrieval, and outputting the database text data corresponding to the matching candidate vector, retrieval is performed based on the candidate vectors in the first hash index tree structure through classical calculation, the cosine similarity between the query vector and each candidate vector is calculated, and the best matching candidate vector is determined based on the similarity value.

[0019] In some embodiments of the present invention, in the step of calculating the cosine similarity between the query vector and each candidate vector, the cosine similarity is calculated based on the following formula:

[0020]

[0021] Among them, cos represents cosine similarity, u represents query vector, and v represents candidate vector.<u,v> represents the calculation of the inner product of the query vector and the candidate vector, ‖u‖ represents the norm of the query vector, and ‖v‖ represents the norm of the candidate vector.

[0022] In the specific implementation process, ‖u‖ and ‖v‖ are the norms of vectors u and v, respectively, and are defined as:

[0023]

[0024] ‖v‖ is calculated in the same way.

[0025] In some embodiments of the present invention, during the quantum computing process, the corresponding index of the candidate vector is generated by a quantum circuit, and the efficiency of the index retrieval is improved by the Grover retrieval algorithm, and finally the matching candidate vector and the corresponding database text data are found through the index.

[0026] In some embodiments of the present invention, in the step of using a quantum-resistant hash function to divide the candidate vector multiple times and constructing a quantum-resistant hash Merkle Tree structure, a hash index is constructed using all hash values. When searching using the Grover retrieval algorithm, a quantum circuit is used to determine the hash value of each candidate vector, and the integrity of the data is verified based on the path of the hash index, thereby ensuring the reliability of the retrieval results and the immutability of the data.

[0027] The present invention also provides a vector database retrieval acceleration system based on the quantum Grover-Merkle Tree algorithm, the system comprising a computer device, the computer device comprising a processor and a memory, the memory storing computer instructions, the processor being used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0028] The third aspect of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps implemented by the aforementioned vector database retrieval acceleration method based on the quantum Grover-Merkle Tree algorithm.

[0029] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention can be specifically pointed out and obtained in the specification and the accompanying drawings.

[0030] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0032] Figure 1 A schematic diagram of an implementation of a vector database retrieval acceleration method based on a quantum Grover-Merkle Tree algorithm of the present invention;

[0033] Figure 2 Schematic diagram of the processing steps to verify the data integrity of the Merkle Tree;

[0034] Figure 3 This is a schematic diagram of the processing flow of the Grover retrieval algorithm;

[0035] Figure 4 is a schematic diagram of a quantum circuit diagram;

[0036] Figure 5 for Figure 4 Schematic diagram of the Oracle circuit;

[0037] Figure 6 This is a schematic diagram of the overall framework of this program. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0039] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0040] like Figure 1 and 6 As shown, the present invention proposes a vector database retrieval acceleration method based on the quantum Grover-Merkle Tree algorithm, the steps of the method include:

[0041] Step S100, obtaining database text data and converting it into candidate vectors;

[0042] Specifically, obtaining database text data for matching the query text data, and converting the database text data into candidate vectors;

[0043] In the specific implementation process, the database text data is used to construct a vector database. In the step of obtaining the database text data for matching the query text data, the database text data can be obtained when the vector database is initially constructed, or it can be a step of adding the candidate vectors corresponding to the database text data to the vector database when the vector database is expanded.

[0044] In the specific implementation process, in the step of converting the database text data into candidate vectors, the text can be converted into vectors using bag-of-words model, TF-IDF, Word2Vec and BERT.

[0045] Specifically, the process is:

[0046] 1. Data preparation: Prepare a set of text data or relational database data as the data set to be processed.

[0047] 2. Vectorization:

[0048] Use the TF-IDF (Term Frequency-Inverse Document Frequency) method to convert text data into numerical vector representation.

[0049] TF-IDF calculation formula: TF-IDF(t,d)=TF(t,d)×IDF(t);

[0050] Where TF(t,d) represents the word frequency of text or relational database data t in document d, and IDF(t) is the inverse document frequency of word t, which is used to measure the prevalence of the word in all documents.

[0051] Step S200, constructing a heterogeneous architecture of a quantum vector database based on the candidate vectors, the heterogeneous architecture of the quantum vector database comprising a first hash index tree structure and a second hash index tree structure, classical computing is based on the first hash index tree structure, and quantum computing is based on the second hash index tree structure;

[0052] In the step that the heterogeneous architecture of the quantum vector database includes a first hash index tree structure and a second hash index tree structure, a hash value of a candidate vector is generated by a classic KD tree or a BallTree to construct the first hash index tree structure; a hash value of the candidate vector is generated by a quantum resistant hash function to construct the second hash index tree structure;

[0053] In the specific implementation process, this solution adopts the heterogeneous architecture of the quantum vector database and can adopt different query methods in different situations. On the one hand, it can ensure the efficiency of retrieval queries, and on the other hand, it can ensure the feasibility of processing by setting up multiple processing schemes.

[0054] Step S300, obtaining query text data, converting the query text data into a query vector, and obtaining the current state of the quantum computer;

[0055] In the specific implementation process, the query text data input by the user end will be converted into a query vector, and the conversion method is the same as that of the candidate vector.

[0056] Step S410, if the quantum computer is in standby mode, start the quantum computer, and perform state vector similarity QSS or cosine similarity calculation based on the query vector and the candidate vector in the second hash index tree structure, and perform Grover retrieval algorithm retrieval through quantum computing, and output the database text data corresponding to the matching candidate vector;

[0057] In the specific implementation process, the quantum computing is performed using a quantum computer.

[0058] Step S420: If the quantum computer is not in the standby mode, a cosine similarity calculation is performed based on the query vector and the candidate vectors in the first hash index tree structure, and a classical search is performed to output the database text data corresponding to the matching candidate vectors.

[0059] This solution provides a feasible path for quantum computing, which can improve the efficiency of query processing through quantum computing.

[0060] In the specific implementation process, the classical calculation is performed using a computer.

[0061] The above scheme is adopted. This scheme first constructs the text data in the database into vector data, and then constructs two tree data based on the vector data. If the quantum computer is in standby mode, quantum computing can be used. The query vector and each candidate vector are calculated similarity using the Grover retrieval algorithm. The similarity is used to determine whether the two match. If they match, the text corresponding to the candidate vector is output.

[0062] In the specific implementation process, the Grover retrieval algorithm is a quantum algorithm used to find specific items in an unsorted database, which has the advantage of quadratic acceleration.

[0063] Time complexity: The classic algorithm requires O(N) time, while the Grover retrieval algorithm only requires O(√N) time.

[0064] The flowchart of Grover retrieval algorithm is as follows Figure 3 As shown, the main steps of the Grover retrieval algorithm are:

[0065] 1. Initialize the superposition state: Initialize the quantum bit to a uniform superposition state of all possible states.

[0066] 2. Oracle application: Mark the target state, usually by applying a phase reversal to the target state.

[0067] 3. Diffuser application: enhance the probability amplitude of the target state and reduce the probability of other states.

[0068] 4. Iteration: Repeat the application of Oracle and Diffuser several times until the probability of the target state is close to 1.

[0069] 5. Measurement: Measure the quantum state and obtain the target state.

[0070] Process and mathematical formula:

[0071] 1. Initialize the super-superposition state: Initialize all possible states to uniform super-superposition states:

[0072]

[0073] 2. Oracle operation: Build an Oracle, mark the target state and flip its phase:

[0074]

[0075] 3. Grover diffusion operation: Perform the Grover diffusion operation to amplify the probability amplitude of the target state:

[0076] D=2|ψ><ψ|-I

[0077] 4 Complete Grover iteration: Repeat steps 2 and 3

[0078] 5 Apply k Grover iterations:

[0079] |ψ final 〉=G k |ψ>

[0080] Among them, the choice of k maximizes the search probability, which is generally:

[0081]

[0082] The flowchart of Grover retrieval algorithm is as follows Figure 3 As shown, Figure 3 The "Whether to match the target vector" in the calculation is to calculate whether the cosine similarity value of the query vector matches the candidate vector. In this case, Quantum State Similarity is not used as the cosine similarity of the query vector and the candidate vector for comparison because the number of qubits in the current quantum computer is limited and is not enough to support the calculation of Quantum State Similarity for big data. Here, this solution reserves an interface to use Quantum State Similarity instead of cosine_similarity when the error rate of the quantum computer is reduced and the number of qubits is sufficient. This is because the calculation speed of Quantum State Similarity is higher than the calculation speed of the classic cosine_similarity.

[0083] In some embodiments of the present invention, in the step of constructing a heterogeneous architecture of a quantum vector database based on a candidate vector, the candidate vector is used as the root node of a first hash index tree structure or a second hash index tree structure, and a quantum-resistant hash function is used to divide the candidate vector multiple times to construct a quantum-resistant hash Merkle Tree structure.

[0084] In the specific implementation process, for any non-leaf node, the hash value of the parent node is composed of the hash values ​​of its two child nodes: Hparent = SHA3-256 (Hleft + Hright). The final generated root hash value (MerkleRoot) represents the integrity summary of the entire data set.

[0085] In the specific implementation process, the quantum-resistant hash Merkle Tree structure is a tree-like data structure that is mainly used to verify data integrity and consistency.

[0086] In some embodiments of the present invention, in the step of using a quantum-resistant hash function to divide the candidate vector multiple times to construct a quantum-resistant hash Merkle Tree structure, in the calculation process of each division, the quantum random number provided by the quantum random number generator is combined with the original data and then hashed to obtain a hash value.

[0087] The implementation process of quantum resistant hash function:

[0088] With the development of quantum computing technology, traditional hash algorithms may face certain security threats. First, in order to enhance the anti-quantum computing ability of hash functions, quantum random numbers provided by quantum random number generators (QRNG) are introduced into hash calculations. The unpredictability of quantum random numbers can significantly improve the security of data encryption and hash operations. Then, a hash function with quantum randomness is used: SHA3-256 is a widely used hash function whose security performs well in a classical computer environment.

[0089] Process Description:

[0090] 1. Quantum random number generation: First, a random bit sequence is generated by a quantum random number generator. This bit sequence is generated using the superposition and uncertainty of quantum states to ensure that it is unpredictable and unreproducible. Function name: generate_quantum_random_bits

[0091] Formula: Qrand = QRNG(n);

[0092] Among them, Qrand represents the generated quantum random bit sequence, and n is the length of the bit sequence.

[0093] Combining quantum random numbers with data: The generated quantum random bit sequence is combined with the original data to form a new input data. In this way, the randomness and unpredictability of the hash calculation are increased.

[0094] Formula: Dcombined=D+Qrand;

[0095] Among them, D represents the original data, Qrand is the quantum random bit sequence, and Dcombined is the combined data.

[0096] 2. Calculate the quantum-resistant hash value: Input the input data Dcombined combined with the quantum random number into the SHA-3 hash function to generate the final quantum-resistant hash value.

[0097] Function name: quantum_resistant_SHA3-256_with_qrng

[0098] Formula: H(Dcombined)=SHA3-256(Dcombined);

[0099] Among them, H (Dcombined) represents the hash value generated after combining the quantum random number.

[0100] In the specific implementation process, in addition to Grover Merkle tree, Grover hash index can also be other quantum-resistant hash indexes of Grover such as Grover Kyber, Grover Dilithium and Grover Falcon.

[0101] In the specific implementation process, in addition to the Merkle tree used in this case, other post-quantum algorithms can also be used to construct hash indexes for quantum-resistant hashes, such as CRYSTALS-Kyber, CRYSTALS-Dilithium, Falcon, and SPHINCS+.

[0102] In the specific implementation process, the hash function with quantum randomness: SHA-3 is a widely used hash function, and its security performs well in the classical computer environment. However, with the development of quantum computing technology, traditional hash algorithms may face certain security threats. In order to enhance the anti-quantum computing ability of hash functions, the quantum randomness provided by the quantum random number generator (QRNG) can be introduced into hash calculations. The unpredictability of quantum random numbers can significantly improve the security of data encryption and hash operations.

[0103] First, a random bit sequence is generated through a quantum random number generator. This bit sequence is generated using the superposition and uncertainty of quantum states to ensure that it is unpredictable and unreproducible;

[0104] The generated quantum random bit sequence is combined with the original data to form a new input data. In this way, the randomness and unpredictability of the hash calculation are increased;

[0105] The input data combined with the quantum random number is input into the SHA-3 hash function to generate the final quantum-resistant hash value.

[0106] By adopting the above scheme, the technical effects include:

[0107] Security improvement: Due to the unpredictability of quantum random numbers, the input data of the hash function becomes more complex and difficult to predict. This significantly improves the security of the hash value. Even if the attacker understands the working mechanism of the hash function, it is difficult to reproduce or crack the generated hash value.

[0108] Anti-quantum computing attack: In the era of quantum computing, traditional hash functions may face attack threats. However, the introduction of quantum random numbers makes the hash value generation process uncertain and random, making it difficult for quantum computers to effectively attack it, thereby enhancing the hash function's ability to resist quantum computing.

[0109] Enhanced irreversibility: The hash function itself is irreversible, but the introduction of quantum random numbers further enhances this irreversibility. The participation of quantum random numbers makes each hash calculation dependent on unpredictable quantum states, making cracking and reverse calculation more difficult.

[0110] By introducing quantum random numbers into the SHA-3 hash function, we have implemented a quantum random hash function. This function takes advantage of the unpredictability of quantum computing and improves the security and collision resistance of hash values. This approach has important application prospects in data encryption and authentication scenarios that require high security. With these enhancements, the quantum random hash function can still maintain a high level of security in the face of quantum computing attacks.

[0111] In the specific implementation process, in the step of dividing the data in the upper node into two parts, the data in the upper node is evenly divided into two parts, and a quantum-resistant hash function is used to calculate each part of the data.

[0112] In some embodiments of the present invention, in the step of performing state vector similarity QSS or cosine similarity calculation based on the query vector and the candidate vectors in the second hash index tree structure and performing Grover retrieval algorithm retrieval through quantum computing, and outputting database text data corresponding to the matching candidate vector, the Grover retrieval algorithm is performed on the query vector and each candidate vector, the state vector similarity QSS or cosine similarity calculation between the query vector and each candidate vector is calculated, the candidate vector matching the query vector is determined based on the similarity, and the integrity of the result is verified through the quantum resistant hash index.

[0113] Using the above scheme, the Grover retrieval algorithm is a quantum algorithm, which is mainly used for fast searching in unsorted databases and has a significant quantum acceleration effect.

[0114] like Figure 2 As shown in the figure, in the specific implementation process, the realization of the Grover retrieval algorithm involves the design and execution of quantum circuits, and finally the search results are obtained through quantum measurements. Tools such as matplotlib can be used to intuitively display the structure and operation steps of quantum circuits. In this step, this solution uses visualization tools to display the structure of quantum circuits in order to better understand and analyze the working principle of quantum algorithms.

[0115] Specifically, first, calculate the number of incoming query vectors. This step is to determine how many qubits are needed to represent these query vectors; use binary logarithm to calculate the number of qubits required and round up. This step determines the number of qubits required in the quantum circuit; create a quantum circuit containing the calculated qubits, which will be used to perform the Grover search algorithm; apply the Hadamard gate to all qubits to put them in a superposition state, which means that each qubit is in the |0> and |1> states at the same time, ready for the search operation; build an Oracle circuit for marking the target state, and determine the marking state by checking the similarity of each query vector with the candidate vector. The Oracle circuit marks the query vector that best matches the candidate vector. This loop traverses all query vectors and checks whether each query vector is close to the candidate vector (implemented by the np.allclose function). If it matches, the operation is applied to the qubit to mark this state; use the Oracle circuit to create a Grover operation. This step builds a special quantum gate to enhance the most likely result, that is, maximize its amplitude; apply the Grover operation to the entire quantum circuit. This step performs reflection and expansion operations on the quantum state in order to increase the measurement probability of the most likely result; adds measurement operations on all quantum bits. This "collapses" the quantum state into a classical bit state in order to obtain the measurement result; runs the quantum circuit using a quantum simulator and executes it multiple times to obtain statistical results. The final measurement result is obtained by simulating the operation of the quantum computer; finds the result with the most occurrences from the measurement results, which is the most likely query vector found by the Grover algorithm; finally, returns the index of the most likely query vector, indicating the query vector closest to the candidate vector.

[0116] Adopting the above scheme, this scheme combines Merkle Tree with Grover retrieval algorithm. Merkle Tree is a hash-based binary tree structure that can efficiently verify the integrity of large-scale data. Traditionally, Merkle Tree queries need to traverse multiple nodes, and as the amount of data increases, the query overhead will also increase. By applying the Grover retrieval algorithm to the leaf nodes of Merkle Tree, the search process of these hash indexes can be accelerated. This scheme combines the computational advantages of quantum algorithms and the security of Merkle Tree, and can improve the efficiency of query and verification while maintaining data integrity. However, its practical application is still constrained by the development of quantum computing resources and hardware, and needs to be further optimized and expanded in the future.

[0117] In some embodiments of the present invention, in the step of verifying the integrity of the result through a quantum-resistant hash index, a determination is made layer by layer from any leaf node in the second hash index tree structure up to the root node to confirm whether the hash value of the final root node matches the overall hash value of the Merkle Tree.

[0118] In the specific implementation process, the verify_proof function is used to recalculate the hash value of the parent node at each layer by combining the target_hash and the hash value of the sibling nodes in the path layer by layer.

[0119] Finally, the calculated root node hash value is compared with the root node hash value stored in the Merkle Tree.

[0120] 3. Verify data integrity:

[0121] The verification process is to combine the hash values ​​of the sibling nodes level by level until the root hash is calculated:

[0122] Hcurrent=SHA3-256(Hdata+Hsibling) The finally calculated root hash should match the root hash value of the Merkle Tree.

[0123] like Figure 2 As shown in the figure, in the specific implementation process, Merkle Tree ensures the integrity of data through its structure. The hash value of each node depends on the hash value of its child node, which means that if the data of any leaf node changes, its impact will be transmitted to the root node.

[0124] For each non-leaf node, its hash value is generated by combining the hash values ​​of the left and right child nodes. That is, if any leaf node changes, the hash values ​​of all its parent nodes will change, eventually resulting in a different root hash value.

[0125] In the specific implementation process, during the data integrity verification process, a hash path (Proof of Inclusion) is constructed to confirm whether specific data belongs to a given Merkle Tree. The verification process verifies the integrity of the data by calculating the hash path of the given data and gradually combining and calculating the value that matches the root hash. During the verification process, the hash values ​​of the sibling nodes are combined step by step until the root hash is calculated.

[0126] Using the above scheme, this scheme can efficiently verify whether a data block is in the set through Merkle Tree without checking the entire data set. To verify a data block, you only need to check the hash value on the path from the data block to the root, based on the Proof of Inclusion principle: given the hash value of the target data and the hash values ​​of all sibling nodes from the target node to the root node, you can verify whether the target data is in the Merkle Tree.

[0127] In the specific implementation process, Merkle Tree achieves quantum resistance in the following ways:

[0128] 1. Quantum-resistant hash function: ensures that the hash value is difficult to be cracked by quantum computers;

[0129] 2. Structural tamper-proof: Any data modification will cause the hash value of the entire tree to change, preventing data tampering;

[0130] 3. Efficient verification: Quickly verify the inclusion of data and ensure data integrity.

[0131] In summary, Merkle Tree achieves security in the era of quantum computing by using quantum-resistant hash functions and its structural characteristics, and can ensure the integrity and immutability of data under the threats that quantum computers may bring.

[0132] In some embodiments of the present invention, the second hash index tree structure in this solution may also adopt Faiss, Annoy, LSH and KD trees;

[0133] 1. Faiss: It performs well when processing large-scale data, especially when using optimized indexes. Query time usually grows logarithmically with the size of the data, but may be close to linear at very large scales.

[0134] 2.Annoy: Suitable for approximate nearest neighbor search of high-dimensional data. As the number of trees increases, the accuracy and time will change accordingly. The performance is relatively stable.

[0135] 3.LSH: Accelerates high-dimensional data search through hash functions. The query time depends on the design of the hash function and the parameter ρ\rhoρ. It performs well under large-scale data.

[0136] 4. KD tree: It performs well in low-dimensional data, but its efficiency drops significantly in high-dimensional data, especially when the data scale is very large.

[0137] The Merkle Tree is mainly used to verify the integrity of data. The query time grows logarithmically with the depth of the tree, and the growth rate is slow, so the query effect is better.

[0138] The comparison of the above algorithms is shown in Table 1 below:

[0139] Table 1

[0140] algorithm 10,000Tokens 100,000Tokens 1,000,000Tokens 10,000,000Tokens Faiss ～0.001-0.01s ～0.01-0.1s ～0.1-1.0s ～1.0-10.0s Annoy ～0.001-0.01s ～0.01-0.05s ～0.05-0.5s ～0.5-5.0s LSH ～0.01-0.05s ～0.05-0.1s ～0.1-0.5s ～0.5-2.0s KD Tree ～0.001-0.01s ～0.01-0.1s ～0.1-1.0s ～1.0-10.0s Merkle Tree ～0.001-0.01s ～0.01-0.05s ～0.05-0.1s ～0.1-0.5s

[0141] As can be seen from Table 1, the query time of the quantum-resistant hash function index-Merkle Tree is very stable under different data volumes, and as the amount of data increases, the query time increases relatively little, that is, not only the query is stable but also the query speed is fast, which is suitable for large-scale vector databases; whether the data volume is 10,000 or 10,000,000 tokens, the query time of Merkle Tree is between ~0.001-0.5 seconds. This stability stems from the logarithmic growth characteristics of MerkleTree. Even if the amount of data increases, the query time complexity remains at O(logn), and MerkleTree maintains a low growth rate, that is, Merkle Tree will not increase the query time due to the increase in the amount of query data, which is suitable for large-scale vector databases. The design of Merkle Tree is suitable for data integrity and tamper-proof verification. It can not only provide efficient verification operations and but also is suitable for vector retrieval.

[0142] Comparison with other algorithms:

[0143] 1. Faiss and KD trees: The query time is similar when the data size is small, but as the data size increases, the query time may increase nearly linearly.

[0144] 2. Annoy and LSH: These two algorithms perform well for small and medium-sized data, but the query time increases significantly for large-scale data, while Merkle Tree maintains a lower growth rate, that is, Merkle Tree does not increase query time due to the increase in the amount of query data, and is suitable for large-scale vector databases.

[0145] Applicable scenarios:

[0146] 1. Data integrity verification: The original intention of the Merkle Tree design is to verify data integrity and anti-tampering. In this scenario, it can not only provide efficient verification operations but also be suitable for vector retrieval.

[0147] Combined with other indexing algorithms: Merkle Tree can also be used in combination with other indexing algorithms, such as using Faiss or Annoy for vector retrieval, and then using Merkle Tree to verify the integrity of the retrieval results.

[0148] In some embodiments of the present invention, in the step of performing cosine similarity calculation based on the query vector and the candidate vectors in the first hash index tree structure and executing classical retrieval, and outputting the database text data corresponding to the matching candidate vector, retrieval is performed based on the candidate vectors in the first hash index tree structure through classical calculation, the cosine similarity between the query vector and each candidate vector is calculated, and the best matching candidate vector is determined based on the similarity value.

[0149] In some embodiments of the present invention, in the step of calculating the cosine similarity between the query vector and each candidate vector, the cosine similarity is calculated based on the following formula:

[0150]

[0151] Among them, cos represents cosine similarity, u represents query vector, and v represents candidate vector.<u,v> represents the calculation of the inner product of the query vector and the candidate vector, ‖u‖ represents the norm of the query vector, and ‖v‖ represents the norm of the candidate vector.

[0152] In some embodiments of the present invention, during the quantum computing process, the corresponding index of the candidate vector is generated by a quantum circuit, and the efficiency of the index retrieval is improved by the Grover retrieval algorithm, and finally the matching candidate vector and the corresponding database text data are found through the index.

[0153] like Figure 4 and 5 As shown, in some embodiments of the present invention, in the step of using a quantum-resistant hash function to divide the candidate vector multiple times and constructing a quantum-resistant hash Merkle Tree structure, a hash index is constructed using all hash values. When searching using the Grover retrieval algorithm, a quantum circuit is used to determine the hash value of each candidate vector, and the integrity of the data is verified based on the path of the hash index, thereby ensuring the reliability of the retrieval results and the immutability of the data.

[0154] Circuit analysis:

[0155] Hadamard Gate (H Gate):

[0156] The Hadamard gate on q0,q1,q2 puts all qubits into a uniform superposition state. The first step of the Grover algorithm is to create a uniform superposition of all possible states.

[0157] Oracle: (circuit in dark grey)

[0158] Figure 4 The role of the Oracle is to flip the target state, that is, it will apply a phase reversal to the target state. In this figure, the oracle is represented as a single gate and is labeled Q. Usually this step includes a series of X gates and multi-controlled X gates.

[0159] Grover Diffuser (Light Gray Circuit):

[0160] In the figure, the gray gate on the right represents the Grover diffuser, which amplifies the amplitude probability of the target state. This usually includes:

[0161] Applying the Hadamard gate

[0162] Application X-gate

[0163] Apply a controlled multi-controlled Z-gate to n-1 qubits (usually implemented via a controlled-X gate, MCX gate)

[0164] Then apply X gate

[0165] Finally, the Hadamard gate is applied

[0166] Measurement:

[0167] The last part of the circuit is the measurement gate, which projects the measurement result of the quantum state onto the classical bit.

[0168] Oracle: The purple gate block in the provided circuit diagram is marked as Q, which represents the oracle part of Grover's algorithm. This part implements the marking of the target state and distinguishes it by inverting the phase of the target state.

[0169] Grover Diffuser: The portion before the measurement gate in the circuit diagram most likely represents a Grover Diffuser, which aims to amplify the amplitude probability of the target state previously marked by the oracle.

[0170] The specific Oracle circuit (dark gray part of the circuit) is expanded as shown below Figure 5 As shown;

[0171] The internal operation of the quantum circuit is as follows:

[0172] 1. Convert the index of the candidate vector into binary format to determine the qubits that need to be operated.

[0173] 2. For each quantum bit, if the corresponding binary bit is 0, apply the X gate (NOT gate) to flip the quantum bit state.

[0174] 3. Use a multi-control-bit Toffoli gate, which flips the target qubit only when all control qubits are 1.

[0175] 4. Apply the X-gate again to restore the quantum state.

[0176] After the final measurement, the binary form corresponding to the index is obtained, and the corresponding relationship between the index and the binary representation (that is, "encoding matching index" in the flowchart)

[0177] Index 0 -> binary "000"

[0178] Index 1 -> binary "001"

[0179] Index 2 -> binary "010"

[0180] Index 3 -> binary "011"

[0181] Index 4 -> binary "100"

[0182] Index 5 -> binary "101"

[0183] Index 6 -> binary "110"

[0184] Index......->Binary"......"

[0185] The overall framework diagram of StateVector of the quantum vector database:

[0186] Quantum inner product similarity formula:

[0187] For two quantum states |ψ1> and |ψ2>, their similarity calculation can be expressed as:

[0188]

[0189] in:

[0190] <ψ1|ψ2> is the inner product of two quantum states;

[0191] ||ψ1|| and ||ψ2|| are the norms (or lengths) of the respective quantum states.

[0192] The form of the cosine similarity formula is very similar:

[0193]

[0194] Formula Example

[0195] The formula for the quantum inner product can be written as follows to express similarity, which can also be called: the similarity of the state vector QSS:

[0196]

[0197] Formula Explanation

[0198] <ψ1|ψ2> represents the inner product of quantum states |ψ1> and |ψ2>;

[0199] ||ψ1|| represents the norm of the quantum state |ψ1>, which is calculated as

[0200] ||ψ2|| represents the norm of the quantum state |ψ2>, which is calculated as

[0201] The beneficial effects of this program include:

[0202] 1. The quantum Grover-Merkle Tree algorithm accelerates the search speed of the vector database. It can not only determine the location of the target hash value more quickly and reduce the traversal steps required in the classic Merkle Tree query process, but also is more suitable for large-scale vector databases.

[0203] 2. The construction of quantum-resistant hash index not only reduces conflict processing and query time, but more importantly, it can encrypt the data entering the vector database. It is a quantum-resistant hash function that kills two birds with one stone.

[0204] 3. The hybrid system architecture design should not only seamlessly switch or run classical and quantum computing resources simultaneously, but also meet the algorithm scheduling system to dynamically allocate tasks to classical or quantum computing nodes according to the nature of the tasks and the availability of computing resources.

[0205] 4. Quantum random hash functions enhance irreversibility, making cracking and reverse calculation more difficult.

[0206] An embodiment of the present invention also provides a vector database retrieval acceleration system based on the quantum Grover-Merkle Tree algorithm, the system comprising a computer device, the computer device comprising a processor and a memory, the memory storing computer instructions, the processor being used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0207] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps implemented by the aforementioned vector database retrieval acceleration method based on the quantum Grover-Merkle Tree algorithm are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0208] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0209] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0210] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0211] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A vector database retrieval acceleration method based on quantum Grover-Merkle Tree algorithm, characterized in that: The steps of the method include: Obtain database text data and convert it into candidate vectors; Building a heterogeneous architecture of a quantum vector database based on the candidate vectors, the heterogeneous architecture of the quantum vector database comprising a first hash index tree structure and a second hash index tree structure, classical computing is based on the first hash index tree structure, and quantum computing is based on the second hash index tree structure; In the step that the heterogeneous architecture of the quantum vector database includes a first hash index tree structure and a second hash index tree structure, a hash value of a candidate vector is generated by a classic KD tree or a BallTree to construct the first hash index tree structure; a hash value of the candidate vector is generated by a quantum resistant hash function to construct the second hash index tree structure; Obtain query text data, convert the query text data into a query vector, and obtain the current state of the quantum computer; If the quantum computer is in standby mode, the quantum computer is started, and a state vector similarity QSS or cosine similarity calculation is performed based on the query vector and the candidate vector in the second hash index tree structure, and a Grover retrieval algorithm is performed through quantum computing to output the database text data corresponding to the matching candidate vector; If the quantum computer is not in the standby mode, cosine similarity calculation is performed based on the query vector and the candidate vectors in the first hash index tree structure, and a classical search is performed to output the database text data corresponding to the matching candidate vectors.

2. The vector database retrieval acceleration method based on the quantum Grover-Merkle Tree algorithm according to claim 1 is characterized in that: In the step of constructing a heterogeneous architecture of a quantum vector database based on a candidate vector, the candidate vector is used as a root node of a first hash index tree structure or a second hash index tree structure, and a quantum-resistant hash function is used to divide the candidate vector multiple times to construct a quantum-resistant hash Merkle Tree structure.

3. The vector database retrieval acceleration method based on the quantum Grover-Merkle Tree algorithm according to claim 2 is characterized in that: In the step of using a quantum-resistant hash function to divide the candidate vector multiple times and constructing a quantum-resistant hash Merkle Tree structure, in the calculation process of each division, the quantum random number provided by the quantum random number generator is combined with the original data and then hashed to obtain a hash value.

4. The vector database retrieval acceleration method based on the quantum Grover-Merkle Tree algorithm according to claim 1 is characterized in that: In the step of performing state vector similarity QSS or cosine similarity calculation based on the query vector and the candidate vectors in the second hash index tree structure and performing Grover retrieval algorithm retrieval through quantum computing, and outputting database text data corresponding to the matching candidate vector, Grover retrieval algorithm retrieval is performed on the query vector and each candidate vector, the state vector similarity QSS or cosine similarity calculation between the query vector and each candidate vector is calculated, the candidate vector matching the query vector is determined based on the similarity, and the integrity of the result is verified through the quantum resistant hash index.

5. The vector database retrieval acceleration method based on the quantum Grover-Merkle Tree algorithm according to claim 4 is characterized in that: In the step of verifying the integrity of the result through the quantum-resistant hash index, a determination is made layer by layer from any leaf node in the second hash index tree structure to the root node to confirm whether the hash value of the final root node matches the overall hash value of the Merkle Tree.

6. The vector database retrieval acceleration method based on the quantum Grover-Merkle Tree algorithm according to claim 1, characterized in that: In the step of performing cosine similarity calculation based on the query vector and the candidate vectors in the first hash index tree structure and executing classical retrieval, and outputting the database text data corresponding to the matching candidate vector, retrieval is performed based on the candidate vectors in the first hash index tree structure through classical calculation, the cosine similarity between the query vector and each candidate vector is calculated, and the best matching candidate vector is determined based on the similarity value.

7. The vector database retrieval acceleration method based on the quantum Grover-Merkle Tree algorithm according to claim 6 is characterized in that: In the step of calculating the cosine similarity between the query vector and each candidate vector, the cosine similarity is calculated based on the following formula: Among them, cos represents cosine similarity, u represents query vector, and v represents candidate vector.<u,v> represents the calculation of the inner product of the query vector and the candidate vector, ‖u‖ represents the norm of the query vector, and ‖v‖ represents the norm of the candidate vector.

8. The vector database retrieval acceleration method based on quantum Grover-Merkle Tree algorithm according to claim 1, characterized in that: During the quantum computing process, the corresponding index of the candidate vector is generated through the quantum circuit, and the efficiency of index retrieval is improved through the Grover retrieval algorithm. Finally, the matching candidate vector and the corresponding database text data are found through the index.

9. The vector database retrieval acceleration method based on the quantum Grover-Merkle Tree algorithm according to claim 2, characterized in that: In the step of using quantum-resistant hash functions to divide candidate vectors multiple times and constructing a quantum-resistant hash Merkle Tree structure, a hash index is constructed using all hash values. When searching using the Grover retrieval algorithm, a quantum circuit is used to determine the hash value of each candidate vector, and the integrity of the data is verified based on the path of the hash index, thereby ensuring the reliability of the retrieval results and the immutability of the data.

10. A vector database retrieval acceleration system based on quantum Grover-Merkle Tree algorithm, characterized in that: The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions. The processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method as described in any one of claims 1 to 9.

Images