Vector library pre-merging method for improving vector retrieval speed

Abstract

The invention discloses a method for improving the vector retrieval speed through vector library pre-merging. The method comprises the following steps of 1, data collection and preprocessing; 2, real-time vector library division: dividing the preprocessed vector data into a plurality of sub-vector libraries according to a certain rule; step 3, sub-vector library index construction: constructing an inverted index by taking the three-level sub-vector library as granularity; step 4, performing vector library pre-merging according to the performing retrieval through an inverted index according to a business demand expression provided by an upper-layer actual business application; 5, monitoring and updating pre-merging trigger condition judgment; and step 6, performing optimization processing after pre-merging. According to the method, by pre-merging and optimizing the vector library, the dispersion and redundancy of the data are reduced, the indexing efficiency is improved, and the retrieval time is greatly shortened.

Description

Technical Field

[0001] The invention belongs to the technical field of data processing, and in particular relates to a method for pre-merging a real-time vector library to improve vector retrieval speed. Background Art

[0002] In today's AI era, the retrieval and processing of vector data has become increasingly important. As the amount of data continues to grow, traditional vector retrieval methods are gradually unable to meet actual needs in terms of speed and efficiency. For example, in common application scenarios, the application system needs to divide vector data according to certain business dimensions to form different vector database instances to achieve data and permission isolation (such as user attributes, business attributes, time dimensions, etc.). When implementing vector retrieval, it is often necessary to perform multi-database retrieval across multiple dimensions to meet business needs.

[0003] In the current commonly used multi-database vector search business process, in order to extract the 40 most relevant data to the user's question, the system needs to provide TOP40 data from each vector library (or shard) and submit it to the rerank relevance ranking model for re-ranking, such as Figure 1 As shown in the figure, the actual data queried in a single query is 40*4, totaling 160 data items, and at least 4 vector query IO operations are performed. This will increase the number of queries, resulting in additional IO and network overhead, and the speed is often slower. In addition, the accuracy of the acquired data is greatly affected, affecting the overall performance of the system and user experience. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for pre-merging vector libraries to improve vector retrieval speed by pre-merging and optimizing vector libraries, reducing data dispersion and redundancy, improving indexing efficiency, and thus greatly shortening retrieval time.

[0005] The object of the present invention is achieved through the following technical scheme: A method for pre-merging a vector library to improve vector retrieval speed comprises the following steps:

[0006] Step 1: Data collection and preprocessing: Collect the original vector data and perform preprocessing operations such as cleaning and normalization;

[0007] Step 2: Real-time vector library division: The preprocessed vector data is divided into multiple sub-vector libraries according to certain rules. The rules are as follows:

[0008] Step 2.1, according to the source system of the data, divide the preprocessed data into several first-level sub-vector libraries;

[0009] Step 2.2: Divide the first-level sub-vector library into a second-level sub-vector library according to the mode of the data, where:

[0010] For text data, first use TextSplitter to split long texts, and then use the pre-trained Sentence Embedding model to convert the entire sentence or paragraph into a vector representation;

[0011] For audio data, use pre-trained Audio Embedding to convert the audio data into vector representation;

[0012] For image and video data, first convert the video data into image data by key frames, and then use the pre-trained Multimodal Embedding model to convert the image data into vector representation;

[0013] Step 2.3, using time series as the segmentation, divide the second-level sub-vector library into three-level sub-vector libraries by year and month, and finally form a vector library group with a three-level structure of business source, mode, and time series;

[0014] Step 3: Sub-vector library index construction: Using the three-level sub-vector library as the granularity, construct an inverted index;

[0015] Step 4: Pre-merge vector libraries according to business: According to the business requirement description provided by the upper-layer actual business application, search through the inverted index to determine which sub-vector libraries need to be selected from the vector library cluster established in step 2 for pre-merging;

[0016] Step 5: Monitor and update the pre-merge trigger condition judgment: regularly monitor the changes of each sub-vector library, query the vector library that needs to be updated and merged according to the pre-merge rule formed in step 4, and merge according to the pre-merge rule; the monitoring and triggering conditions include:

[0017] (a) When the data in the original vector library is detected to have changed, the vector data is triggered and the pre-merge operation is updated;

[0018] (b) When the structure and layering of the original vector library are detected to have changed, the pre-merge rule is triggered again according to the operation in step 4, and the library is rebuilt;

[0019] (c) When the upper-layer application creates or modifies a multi-span data query service, the pre-merge rule update is triggered again according to the operation in step 4, and the database is rebuilt;

[0020] Step 6, optimization processing after pre-merging: Optimize the new sub-vector library after pre-merging, use the HNSW graph-based approximate nearest neighbor search algorithm, query the distance vector library for duplicate vectors with a distance threshold less than a specified value, and perform deduplication pre-merging.

[0021] The vector library pre-merging method of the present invention has the following significant advantages:

[0022] 1. Significantly improved vector retrieval speed: By pre-merging and optimizing the vector library, data dispersion and redundancy are reduced, indexing efficiency is improved, and retrieval time is greatly shortened.

[0023] 2. Improved system scalability: It can adapt to the growing amount of data and changing business needs, and ensure the performance of the system in a large-scale data environment through flexible sub-vector library division and pre-merge strategies.

[0024] 3. Improved resource utilization, optimized the use of storage space and computing resources, and reduced the hardware cost and operating cost of the system.

[0025] 4. Enhanced user experience: Fast and accurate vector retrieval can provide users with more timely and satisfactory services, and improve user satisfaction and loyalty to related applications.

[0026] In summary, the method and application of pre-merging a vector library to improve vector retrieval speed of the present invention have important practical value and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a commonly used comprehensive search method across multiple vector libraries;

[0028] Figure 2 A flow chart of a method for pre-merging vector libraries to improve vector retrieval speed according to the present invention;

[0029] Figure 3 A comprehensive search method using the vector pre-merge of the present invention. DETAILED DESCRIPTION

[0030] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0031] like Figure 2 As shown, a method for pre-merging a vector library to improve vector retrieval speed of the present invention comprises the following steps:

[0032] Step 1: Data collection and preprocessing: Collect raw vector data and perform preprocessing operations such as cleaning and normalization to improve data quality and the accuracy of subsequent processing;

[0033] Step 2: Real-time vector library division: The preprocessed vector data is divided into multiple sub-vector libraries according to certain rules. The rules are as follows:

[0034] Step 2.1, according to the source system of the data, divide the preprocessed data into several first-level sub-vector libraries;

[0035] Step 2.2: According to the data mode (such as text, audio, image, etc.), divide the first-level sub-vector library into a second-level sub-vector library, where:

[0036] For text data, first use TextSplitter to split long texts, and then use the pre-trained Sentence Embedding model to convert the entire sentence or paragraph into a vector representation;

[0037] For audio data, use pre-trained Audio Embedding to convert the audio data into vector representation;

[0038] For image and video data, first convert the video data into image data by key frames, and then use the pre-trained Multimodal Embedding model to convert the image data into vector representation;

[0039] Step 2.3: Based on the time series (minimum span is hours), divide the second-level sub-vector library into three-level sub-vector libraries by year and month, and finally form a vector library group with a three-level structure of business source, modality, and time series.

[0040] Step 3: Sub-vector library index construction: Using the three-level sub-vector library as the granularity, construct an inverted index. The inverted index consists of a vocabulary and an inverted list. The vocabulary contains the keywords that appear in the basic information of each level of the vector library, and each keyword has a pointer to its inverted list. The inverted list records the vector library number containing the keyword and the weight of the vocabulary in the vector library. The inverted index is used to quickly find the vector library containing the keyword. The construction process is as follows:

[0041] Step 3.1, keyword annotation: perform keyword annotation on the meta information of each sub-library in the three-level vector library set;

[0042] Step 3.2, record vector library information: After extracting keywords, for each word, record the vector library number and weight where it appears;

[0043] Step 3.3, build vocabulary and inverted list: All extracted keywords are organized into a vocabulary, and an inverted list index is organized in the form of a B+ tree index.

[0044] Step 4: Pre-merge vector libraries according to business: According to the business requirement description provided by the upper-level actual business application, search through the inverted index to determine which sub-vector libraries need to be selected from the vector library cluster established in step 2 for pre-merging. A multi-way merge algorithm based on sorting (k-way Merge Sort) is used to form a new sub-vector library merge rule based on business pre-merger. The execution process of the merge algorithm is as follows: Use a minimum heap to assist in merging. First, put the first element of the k vector data sets into the minimum heap. Then, each time the smallest element is taken from the minimum heap, it is put into the merged result array, and the next element of the data set where the taken element is located is put into the minimum heap. This process continues until the elements of all vector data sets are merged into the result array. For example, there are three sorted arrays A = [1, 4, 7], B = [2, 5, 8], and C = [3, 6, 9]. First, put A[0]=1, B[0]=2, and C[0]=3 into the minimum heap, take out the smallest 1, put it into the result array, and then put A[1]=4 into the minimum heap. Then take out the smallest 2, put it into the result array, put B[1]=5 into the minimum heap, and so on, and finally get the merged ordered array.

[0045] Step 5, monitor and update pre-merge trigger condition judgment: regularly monitor the changes of each sub-vector library, such as data changes, upper-layer business changes, etc., query the involved vectors that need to be updated and merged according to the pre-merge rules formed in step 4, merge the vector library according to the pre-merge rules, and update each new sub-vector library in step 4. Its monitoring and triggering conditions include:

[0046] (a) When the data in the original vector library is detected to have changed (such as timestamp), the vector data update pre-merge operation is triggered;

[0047] (b) When the structure and layering of the original vector library are monitored to change (such as the increase or decrease of the mode, or the change of quantity), the pre-merge rule is triggered again according to the operation in step 4, and the library is rebuilt;

[0048] (c) When the upper-layer application creates or modifies a multi-span data query service, the pre-merge rule update is triggered again according to the operation in step 4, and the database is rebuilt.

[0049] Step 6, post-merger optimization processing: Optimize the new sub-vector library after pre-merger, use the HNSW (Hierarchical Navigable Small World) graph-based approximate nearest neighbor search algorithm, query the distance vector library for duplicate vectors with a distance threshold less than a specified value, and perform deduplication pre-merge to reduce the size of the merged vector library and improve query efficiency.

[0050] The comprehensive search method using the vector pre-merge of the present invention is as follows Figure 3 As shown in the figure. After the improvement, the business process using the vector pre-merging technology, because the system asynchronously completes the pre-merging of the vector library according to different business domain scenarios in advance, forming a temporary sub-vector library, at this time, in order to extract the 40 most relevant data to the user's question, only the TOP40 data needs to be extracted from the pre-merged sub-vector library and submitted to the rerank relevance sorting model for re-ranking. In this process of this diagram, only 40 data are actually queried, and only one vector query IO operation is performed.

[0051] Depending on the concurrent volume of the business and the number of vector libraries (or shards), vector query using vector pre-merge technology can reduce system overhead exponentially and greatly improve query performance.

[0052] Query performance test data of the merged vector library: In order to verify the effectiveness of the method of the present invention, a series of query performance tests were conducted. The following is part of the test data:

[0053]

[0054]

[0055] From the above test data, it can be seen that after the vector library is pre-merged, the query speed is significantly improved in different application scenarios.

[0056] The vector library pre-merging method of the present invention can be widely applied to the following fields:

[0057] 1. Image retrieval system: quickly retrieve similar images in a large-scale image database. By pre-merging the vector library, it can respond to user query requests more quickly and improve the efficiency and accuracy of image retrieval. For example, in the product image search of an e-commerce platform, when a user enters a product image, the system can quickly find similar products from a massive product image library, providing users with more accurate shopping recommendations.

[0058] 2. Natural language processing: plays a role in text vector representation and retrieval. Pre-merging text vector libraries helps to quickly search for relevant text information, such as quickly finding answers similar to user questions in intelligent customer service. For example, when a user asks a complex technical question, the intelligent customer service system can quickly retrieve the most relevant solution from the huge knowledge base, improving service quality and response speed.

[0059] 3. Recommendation system: Generate vectors based on user behavior and preferences. By pre-merging the vector library, it can more accurately recommend personalized content to users. Taking video recommendation as an example, the system generates user vectors and video vectors based on the user's viewing history, likes, comments and other behaviors. By pre-merging and optimizing the vector library, it recommends videos that are more in line with the user's interests.

[0060] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for pre-merging vector libraries to improve vector retrieval speed, characterized in that: The following steps are involved: Step 1: Data collection and preprocessing: Collect the original vector data and perform preprocessing operations such as cleaning and normalization; Step 2: Real-time vector library division: The preprocessed vector data is divided into multiple sub-vector libraries according to certain rules. The rules are as follows: Step 2.1, according to the source system of the data, divide the preprocessed data into several first-level sub-vector libraries; Step 2.2: Divide the first-level sub-vector library into a second-level sub-vector library according to the mode of the data, where: For text data, first use TextSplitter to split long texts, and then use the pre-trained SentenceEmbedding model to convert the entire sentence or paragraph into a vector representation; For audio data, use pre-trained Audio Embedding to convert the audio data into vector representation; For image and video data, first convert the video data into image data by key frames, and then use the pre-trained Multimodal Embedding model to convert the image data into vector representation; Step 2.3, using time series as the segmentation, divide the second-level sub-vector library into three-level sub-vector libraries by year and month, and finally form a vector library group with a three-level structure of business source, mode, and time series; Step 3: Sub-vector library index construction: Using the three-level sub-vector library as the granularity, construct an inverted index; Step 4: Pre-merge vector libraries according to business: According to the business requirement description provided by the upper-layer actual business application, search through the inverted index to determine which sub-vector libraries need to be selected from the vector library cluster established in step 2 for pre-merging; Step 5: Monitor and update the pre-merge trigger condition judgment: regularly monitor the changes of each sub-vector library, query the vector library that needs to be updated and merged according to the pre-merge rule formed in step 4, and merge according to the pre-merge rule; the monitoring and triggering conditions include: (a) When the data in the original vector library is detected to have changed, the vector data is triggered and the pre-merge operation is updated; (b) When the structure and layering of the original vector library are detected to have changed, the pre-merge rule is triggered again according to the operation in step 4, and the library is rebuilt; (c) When the upper-layer application creates or modifies a multi-span data query service, the pre-merge rule update is triggered again according to the operation in step 4, and the database is rebuilt; Step 6, optimization processing after pre-merging: Optimize the new sub-vector library after pre-merging, use the HNSW graph-based approximate nearest neighbor search algorithm, query the distance vector library for duplicate vectors with a distance threshold less than a specified value, and perform deduplication pre-merging.

Images