A vector database retrieval method and system

Abstract

The present invention proposes a vector database retrieval method and system, which relates to the technical field of big data processing and intelligent information retrieval. The method includes preprocessing high-dimensional multimodal data to obtain a dimension reduction feature vector; constructing a cluster center based on the dimension reduction feature vector; establishing an inverted file and an intra-cluster plane index based on the cluster center, compressing the inverted file and the intra-cluster plane index to obtain a compressed inverted file and an intra-cluster plane index; obtaining a user's query vector, using the inverted file to screen candidate clusters for the query vector, introducing a hierarchical navigable small-world graph in the intra-cluster plane index, performing an intra-cluster search on the cluster cluster, and obtaining a vector database retrieval result. The present invention effectively takes into account the efficiency and accuracy of high-dimensional data retrieval, and improves the retrieval efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing and intelligent information retrieval, and in particular to a vector database retrieval method and system. Background Art

[0002] As an important means of data processing and information retrieval, vector databases have been widely used in fields such as image retrieval, recommendation systems, and semantic analysis. However, existing vector database retrieval methods still have significant problems when facing large-scale, multimodal, and high-dimensional data, and further optimization is urgently needed.

[0003] At present, vector database retrieval methods mainly rely on brute force search or simple index structures, such as inverted file index (IVF) and hierarchical navigable small world graph (HNSW). Although brute force search can provide high retrieval accuracy, the query time grows exponentially when processing large-scale data, resulting in high retrieval latency and high storage cost, which makes it difficult to meet real-time requirements. Index-based retrieval methods, such as IVF, reduce the search space through clustering to improve efficiency, but in high-dimensional scenarios, the retrieval effect may decrease due to the limited accuracy of clustering division. On the other hand, although HNSW can improve accuracy and speed through multi-layer graph structures, its index construction time is long, and memory consumption increases sharply when the data scale grows, which limits its application in dynamic update scenarios. In addition, the conventional process of existing vector database retrieval includes key steps such as data collection and preprocessing, feature extraction and index construction; then, the index structure is generated through clustering algorithm to improve retrieval efficiency; after efficient optimization, these models can quickly complete similarity search of large-scale data, thereby achieving efficient and accurate retrieval goals; however, when faced with high-dimensional data and dynamic update scenarios, due to the computational complexity and storage requirements of high-dimensional data, it is impossible to take into account both retrieval efficiency and accuracy. For example, clustering indexes are not flexible enough when processing large-scale dynamic data, resulting in reduced retrieval efficiency or decreased accuracy. Summary of the invention

[0004] In order to solve the problem that the above-mentioned prior art cannot take into account both the efficiency and accuracy of high-dimensional data retrieval, the present invention proposes a vector database retrieval method and system, which effectively takes into account both the efficiency and accuracy of high-dimensional data retrieval and improves the retrieval efficiency and accuracy.

[0005] In order to achieve the above technical effects, the technical solution of the present invention is as follows: A vector database retrieval method comprises the following steps: S1. Preprocess high-dimensional multimodal data to obtain dimension-reduced feature vectors; S2. constructing cluster centers based on the dimension-reduced feature vector; S3. Based on the cluster center, an inverted file and an intra-cluster plane index are established respectively, and the inverted file and the intra-cluster plane index are compressed to obtain a compressed inverted file and an intra-cluster plane index; S4. Obtain the user's query vector, use the inverted file to screen candidate clusters for the query vector, introduce a hierarchical navigable small-world graph into the intra-cluster plane index, perform intra-cluster search on the clusters, and obtain a vector database search result.

[0006] Preferably, the preprocessing of high-dimensional multimodal data includes: S101. Cleaning the high-dimensional multimodal data to obtain cleaned data; S102. Performing feature extraction on the cleaned data to obtain a high-dimensional multimodal feature vector; S103. performing vector quantization, encoding and normalization on the high-dimensional multimodal feature vector in sequence to obtain a normalized high-dimensional vector; S104. Perform high-dimensional vector dimensionality reduction on the normalized high-dimensional vector to obtain a reduced-dimensional feature vector.

[0007] Preferably, a K-means clustering algorithm is used to construct cluster centers as inverted file nodes in the vector database.

[0008] Preferably, the step of establishing an inverted file and an intra-cluster flat index based on the cluster center includes: S301. Establishing the inverted file by assigning all the reduced-dimensional feature vectors to the clusters corresponding to the cluster centers; S302. Store all vectors in the cluster in a data structure in a preset order, and establish a plane index within the cluster.

[0009] Preferably, the obtaining of the user's query vector includes: receiving query request data of the user, performing feature extraction and vectorization processing on the query request data to obtain the query vector.

[0010] Preferably, the using the inverted file to screen candidate clusters for the query vector includes: S401. Calculate the distance between the query vector and each cluster center to obtain a distance calculation result; S402. According to the distance calculation result, select the first n clusters closest to the query vector as candidate clusters.

[0011] Preferably, the step of introducing a hierarchical navigable small-world graph into the intra-cluster plane index and performing intra-cluster retrieval on the cluster clusters comprises: S411. Perform nearest neighbor retrieval acceleration on all vectors within the cluster using the intra-cluster plane index introduced into the hierarchical navigable small-world graph; S412. After the nearest neighbor search is accelerated, the similarity between the query vector and the vectors in each cluster is calculated to obtain a similarity score; S413. Based on the similarity score, sort the vectors in each cluster to obtain vector similarity sorting results, and select the top k query results that best meet the query requirements from the vector similarity sorting results; S414. Filter and post-process the query results to obtain the final vector database retrieval results.

[0012] Preferably, the filtering and post-processing of the query results includes: removing noise vectors or vectors with similarity scores lower than a preset threshold in the query results.

[0013] The present invention also proposes a vector database retrieval system, comprising: A vector database construction module is used to preprocess high-dimensional multimodal data to obtain a reduced-dimensionality feature vector; construct a cluster center based on the reduced-dimensionality feature vector; and respectively establish an inverted file and an intra-cluster plane index based on the cluster center, and compress the inverted file and the intra-cluster plane index to obtain a compressed inverted file and an intra-cluster plane index; The vector database query module is used to obtain the user's query vector, use the inverted file to screen candidate clusters for the query vector, introduce a hierarchical navigable small-world graph into the intra-cluster plane index, perform intra-cluster search on the cluster, and obtain a vector database search result.

[0014] The present invention also proposes a computer device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations such as the vector database retrieval method.

[0015] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: The present invention proposes a vector database retrieval method and system. Firstly, by constructing a cluster index based on an inverted file and an intra-cluster plane index, the scale of the data to be retrieved is effectively reduced, thereby reducing the computational complexity and resource consumption in the retrieval process; secondly, a hierarchical navigable small-world graph is introduced into the intra-cluster plane index. This graph structure can efficiently approximate the nearest neighbor search, realize accelerated calculation of vector searches within clusters, and significantly improve the speed and accuracy of vector searches within clusters, thereby obtaining vector database retrieval results, so as to further optimize the retrieval efficiency and accuracy, thereby improving the retrieval performance and system response speed of the vector database in high-dimensional data scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a vector database retrieval method proposed in an embodiment of the present invention is shown; Figure 2 A flowchart of preprocessing high-dimensional multimodal data proposed in an embodiment of the present invention is shown; Figure 3 A flowchart of establishing an inverted file and a plane index within a cluster proposed in an embodiment of the present invention is shown; Figure 4 A flowchart of clustering process for selecting candidate clusters for the query vector proposed in an embodiment of the present invention is shown; Figure 5 A flowchart of performing intra-cluster search on the cluster clusters proposed in an embodiment of the present invention is shown; Figure 6 A flowchart showing a vector database search method according to an embodiment of the present invention; Figure 7 A structural block diagram of a vector database retrieval system proposed in an embodiment of the present invention is shown; Figure 8 A block diagram showing the structure of a computer device proposed in an embodiment of the present invention; 801. Processor; 802. Memory; 803. Communication interface; 804. Communication bus; 805. Executable instructions. DETAILED DESCRIPTION

[0017] The drawings are for illustrative purposes only and should not be construed as limiting the present patent; It is understandable to those skilled in the art that some well-known contents may be omitted in the drawings; The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0018] Example 1 like Figure 1 As shown, this embodiment proposes a vector database retrieval method, comprising the following steps: S1. Preprocess high-dimensional multimodal data to obtain dimension-reduced feature vectors; In S1, see Figure 2 , the preprocessing of high-dimensional multimodal data includes: S101. Cleaning the high-dimensional multimodal data to obtain cleaned data; In S101, the high-dimensional multimodal data is collected from a data source, the high-dimensional multimodal data includes image data, video data, audio data and text data, and the high-dimensional multimodal data is cleaned to remove invalid and noisy data, remove stop words and punctuation marks from text data, remove noise from audio data, and remove fuzzy samples from image data and video data, so as to ensure that the high-dimensional multimodal data is high-quality data for subsequent vectorization.

[0019] S102. Performing feature extraction on the cleaned data to obtain a high-dimensional multimodal feature vector; In S102, natural language processing models (such as BERT) are used for audio data and text data, and convolutional neural networks (such as ResNet-50) and CLIP are used for image data and video data to extract features to obtain high-dimensional multimodal feature vectors; the goal of feature extraction is to map complex input data into a high-dimensional space of fixed dimensionality for subsequent indexing and retrieval operations.

[0020] S103. performing vector quantization, encoding and normalization on the high-dimensional multimodal feature vector in sequence to obtain a normalized high-dimensional vector; In S103, the high-dimensional multimodal feature vector is a feature vector of different types, and vector quantization, encoding and normalization are sequentially performed on the high-dimensional multimodal feature vector, specifically including: The high-dimensional multimodal feature vector is vectorized and encoded, specifically: the extracted feature vectors of different types (such as picture feature vectors, video feature vectors, audio feature vectors, and text feature vectors, etc.) are concatenated to form a complete high-dimensional vector description; then, the high-dimensional vector is encoded using the locality sensitive hashing (LSH) method and mapped into a feature vector of a binary signature. , LSH can compress the vector into a shorter binary code while maintaining the similarity relationship in the original vector space; finally, the product quantization method (PQ) is used to quantize the feature vector Perform quantization processing to divide the original high-precision vector into multiple sub-vectors; Normalize the vector, specifically: normalize the concatenated feature vector Use L2 normalization to make the modulus of the feature vector equal to 1. This step is mainly to make different feature vectors have similar scales in high-dimensional space, thereby improving the accuracy of similarity calculation. For example, given the feature vector , the normalized eigenvector It can be expressed as:

[0021] in, Represents the feature vector Model; S104. Perform high-dimensional vector dimensionality reduction on the normalized high-dimensional vector to obtain a reduced-dimensional feature vector.

[0022] In S104, principal component analysis (PCA) or random projection is used to reduce the dimension of the normalized high-dimensional vector to reduce computational overhead and improve retrieval speed. The goal of dimensionality reduction is to compress redundant information in the original high-dimensional space, reduce computational complexity, and retain key features of the vector as much as possible.

[0023] S2. constructing cluster centers based on the dimension-reduced feature vector; In S2, the K-means clustering algorithm is used to construct cluster centers as inverted file nodes in the vector database to create a cluster index of the inverted file. These cluster centers will be used as inverted file nodes in the vector database to reduce the number of candidate sets to be retrieved during vector retrieval.

[0024] S3. Based on the cluster center, an inverted file (IVF) and an intra-cluster flat index (FLAT) are respectively established, and the inverted file and the intra-cluster flat index are compressed to obtain compressed inverted file and intra-cluster flat index; In S3, see Figure 3 , based on the cluster center, establishing an inverted file and an intra-cluster plane index respectively, including: S301. Establishing the inverted file by assigning all the reduced-dimensional feature vectors to the clusters corresponding to the cluster centers; In S301, the inverted file is used to record the feature vectors under each cluster center to speed up the preliminary screening operation in the query process.

[0025] S302. All vectors in the cluster are stored in a data structure in a preset order, and a plane index within the cluster is established; In S302, a flat index (FLAT index) is constructed for the vectors in each cluster to ensure that the vectors in the cluster can be quickly located. When searching within the cluster, a point-by-point comparison method is used for accurate matching, and a simple method such as brute force search can be used here.

[0026] The inverted file and the intra-cluster flat index are compressed, and quantization methods such as PQ (Product Quantization) are used to reduce the storage requirements of the vector from floating point numbers to discrete quantized values, so as to reduce storage space and speed up query speed while minimizing information loss.

[0027] After obtaining the compressed inverted file and the in-cluster flat index, a database update mechanism is established: the cluster center and inverted file index are updated regularly to adapt to dynamic changes in data. For new vectors, they are assigned to appropriate clusters through incremental clustering and the inverted file is updated; for expired vectors, a mark-and-delete strategy is used to ensure that the vectors in the database maintain high quality and efficient retrieval characteristics.

[0028] S4. Obtain the user's query vector, use the inverted file to screen candidate clusters for the query vector, introduce a hierarchical navigable small-world graph (HNSW) into the intra-cluster plane index, perform intra-cluster search on the cluster, and obtain a vector database search result.

[0029] In S4, the step of obtaining the user's query vector includes: receiving the user's query request data, performing feature extraction and vectorization processing on the query request data to ensure that it has the same feature space and dimension as the vector in the database, and obtaining the query vector; and performing L2 normalization on the query vector to ensure that it has the same scale as the vector in the database, so that the similarity calculation is not affected by the vector length, thereby improving the accuracy and stability of the query.

[0030] See also Figure 4 , the step of using the inverted file to screen candidate clusters for the query vector includes: S401. Calculate the distance between the query vector and each cluster center to obtain a distance calculation result; In S401, similarity calculation is performed on the cluster centers using an inverted index file (IVF), by calculating the distance between the query vector and each cluster center; S402. According to the distance calculation result, the first n clusters closest to the query vector are selected as candidate clusters; this step can effectively reduce the amount of data to be processed later, narrow the query scope, and thus improve query efficiency.

[0031] See also Figure 5 The step of introducing a hierarchical navigable small-world graph into the intra-cluster plane index and performing intra-cluster retrieval on the cluster clusters comprises: S411. Perform nearest neighbor retrieval acceleration on all vectors within the cluster using the intra-cluster plane index introduced into the hierarchical navigable small-world graph; In S411, in the initially screened clusters, the intra-cluster flat index (FLAT) is used to perform a detailed nearest neighbor search on all vectors in the cluster, and then the similarity between the query vector and each intra-cluster vector is calculated through brute force search or other precise methods to determine the most matching vector. This step ensures the accuracy of the search to ensure that the results returned to the user are highly relevant. In order to further improve the query speed within the cluster, the hierarchical navigable small world graph (HNSW) is introduced on the basis of FLAT search for acceleration. HNSW achieves fast approximate nearest neighbor search by constructing a graph structure, effectively shortening the search time and achieving better query performance in high-dimensional scenarios.

[0032] S412. After the nearest neighbor search is accelerated, the similarity between the query vector and the vectors in each cluster is calculated to obtain a similarity score; S413. Based on the similarity score, sort the vectors in each cluster to obtain vector similarity sorting results, and select the top k query results that best meet the query requirements from the vector similarity sorting results; In S413, the search results from different clusters are summarized, and all candidate vectors in each cluster are sorted based on the similarity score; the sorting uses cosine similarity or Euclidean distance as a metric to ensure that the most similar vectors are ranked in front, and finally the top k query results that best meet the query requirements are selected from the vector similarity sorting results; this step ensures that the results returned to the user are highly accurate and relevant.

[0033] S414. Filter and post-process the query results to obtain the final vector database retrieval results.

[0034] In S414, the query results are filtered and post-processed, including: removing noise vectors or vectors with similarity scores below a preset threshold in the query results to ensure that the returned results meet expectations. At the same time, the similarity score results are aggregated to remove duplicates to improve user experience and the effectiveness of the results; finally, the final filtered vector database retrieval results are returned to the user, and the query and the queried vector database retrieval results are cached for rapid response to subsequent identical or similar queries. The cache uses strategies such as least recently used (LRU) to prioritize the retention of recently frequently queried data, thereby reducing repeated calculations and improving the overall system response speed and user experience.

[0035] It should be noted that existing methods such as brute force search have high accuracy, but the computational cost is high and cannot meet real-time requirements; while index-based retrieval methods (such as IVF and HNSW) show limitations when processing high-dimensional or dynamically updated data, such as reduced index accuracy, low construction and update efficiency, etc. This embodiment proposes a vector database retrieval method, aiming to design a new fusion method that can not only improve retrieval accuracy, but also significantly optimize efficiency, and has the ability to process dynamic data, thereby meeting the actual needs of high-dimensional data in application scenarios such as image retrieval, recommendation systems, and semantic analysis.

[0036] The method of the present invention realizes efficient retrieval of vector databases in large-scale high-dimensional data scenarios by integrating the retrieval architecture of inverted files, intra-cluster flat index (IVF_FLAT) and hierarchical navigable small-world graph (HNSW). Compared with traditional methods, the method of the present invention shows significant advantages in retrieval efficiency, accuracy and dynamic adaptability. Specifically, the clustering index of IVF_FLAT significantly reduces the scale of candidate data during retrieval and reduces the overall computational complexity; HNSW uses its efficient graph search algorithm to further optimize the nearest neighbor retrieval within the cluster, thereby greatly improving the retrieval efficiency while maintaining a high retrieval accuracy.

[0037] In addition, the support capability of the present invention for dynamic data is significantly better than that of the prior art. Through the incremental clustering and mark deletion mechanism, the clustering index can be quickly updated to avoid the high cost of rebuilding the entire database index, which is particularly suitable for real-time update application scenarios (such as recommendation systems, video searches, etc.). Compared with the traditional index construction method that requires full update, the incremental update scheme of the present invention further enhances the flexibility and practicality of the system. The present invention also demonstrates superior performance in the processing of high-dimensional multimodal data and the optimization of system resources. By combining deep learning models (such as BERT, ResNet-50, CLIP) to extract high-dimensional feature vectors, and quantizing, normalizing and high-dimensional dimension reduction processing, not only the storage requirements are reduced, but also the retrieval efficiency is improved. In addition, the intelligent caching mechanism (such as the least recently used strategy, LRU) effectively accelerates the response speed of repeated queries and the overall response efficiency of the system. In summary, the present invention is superior to the prior art in terms of improving retrieval efficiency and accuracy, optimizing dynamic data support capabilities, and reducing storage and computing costs, and has important theoretical value and practical application significance.

[0038] Example 2 See also Figure 6This embodiment further illustrates a vector database retrieval method proposed in the above embodiment, which is used for an efficient retrieval system for high-dimensional multimodal data (such as vectors, texts, pictures, and videos). First, a deep learning model (such as BERT, ResNet-50, or CLIP) is used to extract the high-dimensional multimodal feature vector of the input data, and normalize and quantize it to optimize storage and computing efficiency.

[0039] In the index construction process, the K-means algorithm is used to generate cluster centers, and a cluster index based on inverted files and intra-cluster flat index (IVF_FLAT) is constructed, which significantly reduces the size of the data to be retrieved. At the same time, a hierarchical navigable small-world graph (HNSW) is introduced in each cluster for detailed intra-cluster retrieval, realizing efficient nearest neighbor search.

[0040] When querying, the vector input by the user is quickly screened out as a candidate cluster through the IVF structure, and HNSW is used within the cluster to complete high-precision matching. Finally, the system sorts the search results by similarity score and returns them to the user. With this method, when processing 1 billion-level high-dimensional data, the query efficiency is improved by more than 40%, the search accuracy is improved by 15%, and the storage requirements and query latency are significantly reduced.

[0041] The core innovation of the method of the present invention is: (1) A new and efficient vector database retrieval architecture is proposed by integrating inverted files and intra-cluster flat indexes (IVF_FLAT) with hierarchical navigable small-world graphs (HNSW). The initial screening of cluster indexes is achieved through IVF_FLAT, which quickly narrows the search scope and reduces the processing scale of high-dimensional multimodal data. At the same time, the HNSW structure is introduced into each cluster, and its efficient graph search algorithm is used to perform fine intra-cluster search, which significantly improves the retrieval accuracy and efficiency. This combined method takes into account the high efficiency of IVF and the high precision of HNSW, providing a superior solution for high-dimensional multimodal data retrieval.

[0042] (2) In order to adapt to high-dimensional multimodal data processing and dynamic data updates, the present invention designs a set of efficient feature vectorization and update mechanisms. Through deep learning models (such as BERT, ResNet-50 or CLIP), high-dimensional multimodal data such as text, pictures and videos are extracted, and combined with quantization and normalization processing, the generated high-dimensional multimodal feature vectors can not only accurately express the original data, but also improve efficiency in the retrieval process. In addition, for dynamic data scenarios, incremental clustering and mark deletion strategies are adopted to achieve rapid update of indexes, ensuring the stable performance and high retrieval accuracy of the database in high-frequency data changes.

[0043] (3) The sorting and caching mechanism of search results is further optimized. By comprehensively sorting the candidate cluster results, the most relevant results are screened out based on similarity scores (such as cosine similarity or Euclidean distance), and intelligent caching strategies (such as the least recently used strategy, LRU) are combined to accelerate the response speed of repeated queries. Combined with high-dimensional dimensionality reduction techniques (such as principal component analysis PCA or random projection), the present invention reduces the computational complexity while maintaining feature integrity, significantly improving the overall performance of large-scale high-dimensional data retrieval.

[0044] Example 3 See also Figure 7 , this embodiment proposes a vector database retrieval system, including: A vector database construction module is used to preprocess high-dimensional multimodal data to obtain a reduced-dimensionality feature vector; construct a cluster center based on the reduced-dimensionality feature vector; and respectively establish an inverted file and an intra-cluster plane index based on the cluster center, and compress the inverted file and the intra-cluster plane index to obtain a compressed inverted file and an intra-cluster plane index; The vector database query module is used to obtain the user's query vector, use the inverted file to screen candidate clusters for the query vector, introduce a hierarchical navigable small-world graph into the intra-cluster plane index, perform intra-cluster search on the cluster, and obtain a vector database search result.

[0045] In this embodiment, firstly, by constructing a cluster index based on an inverted file and an intra-cluster plane index, the scale of the data to be retrieved is effectively reduced, thereby reducing the computational complexity and resource consumption in the retrieval process; secondly, a hierarchical navigable small-world graph is introduced in the intra-cluster plane index. This graph structure can efficiently approximate the nearest neighbor search, thereby accelerating the calculation of vector searches within clusters, significantly improving the speed and accuracy of vector searches within clusters, and thus obtaining vector database retrieval results, so as to further optimize the retrieval efficiency and accuracy, thereby improving the retrieval performance and system response speed of the vector database in high-dimensional data scenarios.

[0046] Example 4 See also Figure 8 This embodiment also proposes a computer device, see Figure 8 , comprising: a processor 801, a memory 802, a communication interface 803 and a communication bus 804, wherein the processor 801, the memory 802 and the communication interface 803 communicate with each other through the communication bus 804; Wherein: the processor 801, the memory 802 and the communication interface 803 communicate with each other through the communication bus 804. The communication interface 803 is used for network communication with other devices such as a client or other servers. The processor 801 is used to execute the executable instruction 805, which can specifically execute the relevant steps in the above-mentioned vector database retrieval method embodiment.

[0047] Specifically, the executable instruction 805 may include program code. The processor 801 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0048] The memory 802 is used to store executable instructions 805. The memory 802 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0049] The executable instruction 805 may be specifically called by the processor 801 to enable the computer device to perform the following operations: S1. Preprocess high-dimensional multimodal data to obtain dimension-reduced feature vectors; S2. constructing cluster centers based on the reduced dimension feature vector; S3. Based on the cluster center, an inverted file and an intra-cluster plane index are established respectively, and the inverted file and the intra-cluster plane index are compressed to obtain a compressed inverted file and an intra-cluster plane index; S4. Obtain the user's query vector, use the inverted file to screen candidate clusters for the query vector, introduce a hierarchical navigable small-world graph into the intra-cluster plane index, perform intra-cluster search on the clusters, and obtain a vector database search result.

[0050] In this embodiment, firstly, by constructing a cluster index based on an inverted file and an intra-cluster plane index, the scale of the data to be retrieved is effectively reduced, thereby reducing the computational complexity and resource consumption in the retrieval process; secondly, a hierarchical navigable small-world graph is introduced in the intra-cluster plane index. This graph structure can efficiently approximate the nearest neighbor search, thereby accelerating the calculation of vector searches within clusters, significantly improving the speed and accuracy of vector searches within clusters, and thus obtaining vector database retrieval results, so as to further optimize the retrieval efficiency and accuracy, thereby improving the retrieval performance and system response speed of the vector database in high-dimensional data scenarios.

[0051] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the implementation methods of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all implementation methods here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A vector database retrieval method, characterized in that: The following steps are involved: S1. Preprocess high-dimensional multimodal data to obtain dimension-reduced feature vectors; S2. constructing cluster centers based on the dimension-reduced feature vector; S3. Based on the cluster center, an inverted file and an intra-cluster plane index are established respectively, and the inverted file and the intra-cluster plane index are compressed to obtain a compressed inverted file and an intra-cluster plane index; S4. Obtain the user's query vector, use the inverted file to screen candidate clusters for the query vector, introduce a hierarchical navigable small-world graph into the intra-cluster plane index, perform intra-cluster search on the clusters, and obtain a vector database search result.

2. The vector database retrieval method according to claim 1, characterized in that: The preprocessing of high-dimensional multimodal data includes: S101. Cleaning the high-dimensional multimodal data to obtain cleaned data; S102. Performing feature extraction on the cleaned data to obtain a high-dimensional multimodal feature vector; S103. performing vector quantization, encoding and normalization on the high-dimensional multimodal feature vector in sequence to obtain a normalized high-dimensional vector; S104. Perform high-dimensional vector dimensionality reduction on the normalized high-dimensional vector to obtain a reduced-dimensional feature vector.

3. The vector database retrieval method according to claim 1, characterized in that: The K-means clustering algorithm is used to construct the cluster centers of the inverted file nodes in the vector database.

4. The vector database retrieval method according to claim 1, characterized in that: The step of establishing an inverted file and an intra-cluster plane index based on the cluster center includes: S301. Establishing the inverted file by assigning all the reduced-dimensional feature vectors to the clusters corresponding to the cluster centers; S302. Store all vectors in the cluster in a data structure in a preset order, and establish a plane index within the cluster.

5. The vector database retrieval method according to claim 1, characterized in that: The obtaining of the user's query vector includes: receiving the user's query request data, performing feature extraction and vectorization processing on the query request data, and obtaining the query vector.

6. The vector database retrieval method according to claim 1, characterized in that: The using the inverted file to screen candidate clusters for the query vector includes: S401. Calculate the distance between the query vector and each cluster center to obtain a distance calculation result; S402. According to the distance calculation result, select the first n clusters closest to the query vector as candidate clusters.

7. The vector database retrieval method according to claim 1, characterized in that: The step of introducing a hierarchical navigable small-world graph into the intra-cluster plane index and performing intra-cluster retrieval on the cluster clusters comprises: S411. Perform nearest neighbor retrieval acceleration on all vectors within the cluster using the intra-cluster plane index introduced into the hierarchical navigable small-world graph; S412. After the nearest neighbor search is accelerated, the similarity between the query vector and the vectors in each cluster is calculated to obtain a similarity score; S413. Based on the similarity score, sort the vectors in each cluster to obtain vector similarity sorting results, and select the top k query results that best meet the query requirements from the vector similarity sorting results; S414. Filter and post-process the query results to obtain the final vector database retrieval results.

8. The vector database retrieval method according to claim 7, characterized in that: The filtering and post-processing of the query results includes: removing noise vectors or vectors with similarity scores lower than a preset threshold in the query results.

9. A vector database retrieval system, characterized in that: include: A vector database construction module is used to preprocess high-dimensional multimodal data to obtain a reduced-dimensionality feature vector; construct a cluster center based on the reduced-dimensionality feature vector; and respectively establish an inverted file and an intra-cluster plane index based on the cluster center, and compress the inverted file and the intra-cluster plane index to obtain a compressed inverted file and an intra-cluster plane index; The vector database query module is used to obtain the user's query vector, use the inverted file to screen candidate clusters for the query vector, introduce a hierarchical navigable small-world graph into the intra-cluster plane index, perform intra-cluster search on the cluster, and obtain a vector database search result.

10. A computer device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation of the vector database retrieval method as described in any one of claims 1-8.

Images