Clustering-based sparse and dense retrieval enhancement generation system and method
By adopting cluster-based sparse and dense search methods in the search enhancement generation system, the efficiency and accuracy problems of dense vector search when dealing with inaccurate matching queries are solved, and efficient and accurate search results are achieved.
Patent Information
- Application Number
- CN202411589033.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-08
- Publication Date
- 2025-04-04
AI Technical Summary
In the existing search enhancement generation system, intensive vector search cannot effectively handle the problem of inaccurate matching of user queries and stored data, and every time a new query is processed, all documents need to be sorted in similarity, resulting in wasted time.
The cluster-based sparse and dense retrieval enhancement generation system is adopted. The document is split into fixed-size chunks through the construction module of the intensive knowledge base and vectorized, and clustered using the K-means algorithm; then the vectors in clusters with high similarity to the user query are filtered out through the dense knowledge base line vector filter module and converted into sparse vectors to achieve precise character-level retrieval.
The search time is shortened through clustering, and the problems of retrieval accuracy and efficiency are solved by combining sparse and dense vectors, achieving efficient and accurate search results.
Smart Images
Figure CN119760134A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a clustering-based sparse and dense retrieval enhancement generation system and a method thereof, and belongs to the field of artificial intelligence. Background Art
[0002] Retrieval-augmented generation (RAG) is a mechanism that helps large language models (LLMs) generate better answers. It solves the problem that pre-trained LLMs cannot easily expand or modify their memory, encounter hallucinations, outdated knowledge, and opaque and untraceable reasoning processes. Its overall working mechanism can be divided into three main steps: indexing, retrieval, and generation.
[0003] Indexing step: Load the raw data from the external knowledge database and divide the input data into several chunks, which are then embedded as vectors and stored in the index database. Retrieval step: Embed the query as a vector using the same encoding model. Retrieve the top K most similar data chunks by calculating the cosine similarity between the query and the documents in the knowledge database. These chunks are then added to the input prompt.
[0004] Generation step: The query and the retrieved data chunks are synthesized into a new prompt. Based on this new prompt, LLMs will generate more accurate and robust answers because they are supported by the information provided in the data chunks.
[0005] As more and more people realize the importance of RAG to the implementation of large models, the retrieval module has also developed rapidly. The retrieval module often uses vector search, usually dense vectors, but this cannot always produce satisfactory results. First, this limitation becomes apparent when the user's specific query keywords do not exactly match the stored data. Because the vector itself cannot represent precise semantic information: a vector can represent a word, a sentence, a paragraph, or even an entire article. Although used to convey the "meaning" of the query, this representation is the probability of the query occurring simultaneously with other text in the context window. Second, every time a new query is processed and related documents are retrieved, the similarity of all documents needs to be sorted, which wastes a lot of time. Summary of the invention
[0006] In order to overcome the defects of the prior art, the present invention provides a clustering-based sparse and dense retrieval enhancement generation system and method thereof. The technical solution of the present invention is:
[0007] A clustering-based sparse and dense retrieval enhancement generation system, comprising:
[0008] The building block of the dense knowledge base is used to split documents into fixed-size chunks, vectorize the chunks, and cluster the vectorized chunks using the K-means algorithm;
[0009] The dense knowledge base line vector screening module is used to screen out vectors in the cluster whose similarity with the query is higher than a preset threshold according to the user query;
[0010] The sparse vector introduction module is used to convert the filtered dense vectors into sparse vectors to achieve accurate retrieval at the character level.
[0011] The steps of constructing the dense knowledge base construction module are as follows:
[0012] 1.1 First, split the document into fixed-size chunks and vectorize the chunks. The vector dimensions are 768 and 1024. After vectorization, use K-means to cluster them.
[0013] 1.2 Select the value of K. Refer to the value of K in this article, classify all documents, and adjust the value of K according to the number of categories;
[0014] 1.3 After the K value is selected, the centroid of each category is randomly initialized, that is, vectors in different categories are randomly selected as the centroid, followed by distance calculation, using Euclidean distance:
[0015]
[0016] Among them, E1 is the vector in a cluster, CE is the vector of the centroid, and the final calculated centroid represents the entire vector representation:
[0017]
[0018] Among them, c i represents the class that is closest to the sample Ei among the K tears, c i The value of is one from 1 to K, and then the centroid is updated and the loop is iterated until convergence, completing the clustering of dense vectors.
[0019] The screening steps of the dense knowledge base line vector screening module include:
[0020] A threshold is set for the cluster. According to the user's question, the question is first vectorized and the similarity is calculated with the centroid of each cluster. If it is greater than the threshold, the similarity with each vector in the cluster is calculated. Otherwise, it is filtered out, and finally the TOP-K dense vectors are obtained.
[0021] The specific introduction steps of the sparse vector introduction module are:
[0022] 2.1 Use the SPLADE model to process these dense vectors to generate sparse vectors; 2.2 The SPLADE model uses BERT's encoding capabilities to convert text into BERT vectors after initial Tokenization; Apply the masked language model MLM to all token positions and calculate the corresponding probability of each token to each word in the BERT vocabulary;
[0023] 2.3SPLADE aggregates these probabilities for each word and promotes the sparsity of the vector by applying the normalization method of log saturation effect. The obtained weight reflects the relevance of each word to the input token, generating a learned sparse vector.
[0024] 2.4 Generate Token-to-Weight mapping.
[0025] A clustering-based sparse and dense retrieval enhancement generation method comprises the following steps:
[0026] S1 builds a dense knowledge base, splits documents into fixed-size chunks, vectorizes the chunks, and clusters the vectorized chunks using the K-means algorithm; S2 uses the dense knowledge base line vector screening module to filter out vectors in clusters whose similarity to the query is higher than a preset threshold based on the user query;
[0027] S3 converts the filtered dense vectors into sparse vectors through the sparse vector introduction module to achieve accurate retrieval at the character level.
[0028] The advantages of the present invention are:
[0029] The retrieval time is shortened by clustering dense vectors, and a more accurate and complete RAG system is realized by combining sparse and dense vectors. It has the following advantages:
[0030] Efficiency: Based on the user input, the dense vectors are clustered and similarity is calculated with the centroid of the cluster. The clusters with similarity to the user's question are filtered out, and similarity is only calculated for the vectors in the remaining clusters, which can reduce the retrieval delay.
[0031] Completeness: This solves the problem that dense vectors cannot retrieve characters. By introducing sparse vectors, a complete retrieval method is formed to meet the needs of different retrieval methods.
[0032] Flexibility: The RAG system based on long text does not impose any restrictions on individual components. As long as they meet the requirements, they can be replaced. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 It is the main structural block diagram of the system of the present invention.
[0034] Figure 2 It is a schematic diagram of the workflow of the present invention.
[0035] Figure 3 It is a flow chart of creating a dense knowledge base of the present invention.
[0036] Figure 4 It is a dense vector screening flow chart of the present invention.
[0037] Figure 5 It is a flow chart of creating a sparse knowledge base of the present invention. DETAILED DESCRIPTION
[0038] The present invention will be further described below in conjunction with specific embodiments, and the advantages and features of the present invention will become clearer as the description proceeds. However, these embodiments are exemplary only and do not constitute any limitation to the scope of the present invention. It should be understood by those skilled in the art that the details and forms of the technical solution of the present invention may be modified or replaced without departing from the spirit and scope of the present invention, but these modifications and replacements all fall within the scope of protection of the present invention.
[0039] See also Figures 1 to 5 The present invention relates to a clustering-based sparse and dense retrieval enhancement generation system, comprising:
[0040] Building block 1 of the dense knowledge base is used to split documents into fixed-size chunks, vectorize the chunks, and cluster the vectorized chunks using the K-means algorithm;
[0041] Dense knowledge base line vector screening module 2, used to screen out vectors in clusters whose similarity to the query is higher than a preset threshold according to the user query;
[0042] The sparse vector introduction module 3 is used to convert the screened dense vector into a sparse vector to achieve accurate retrieval at the character level.
[0043] The steps of constructing the dense knowledge base construction module are as follows:
[0044] 1.1 First, split the document into fixed-size chunks and vectorize the chunks. The vector dimensions are 768 or 1024. Common vectorization models include Bce and Bge. After vectorization, use K-means to cluster them.
[0045] 1.2 It should be noted that the value of K should be selected based on experience. Refer to the value of K in this article, classify all documents, and adjust the value of K according to the number of categories;
[0046] 1.3 After the K value is selected, the centroid of each category is randomly initialized, that is, vectors in different categories are randomly selected as the centroid, followed by distance calculation, using Euclidean distance:
[0047]
[0048] Among them, E1 is the vector in a cluster, CE is the vector of the centroid, and the final calculated centroid represents the entire vector representation:
[0049]
[0050] Among them, c i represents the class that is closest to the sample Ei among the K tears, c i The value of is one from 1 to K, and then the centroid is updated and the loop is iterated until convergence, completing the clustering of dense vectors.
[0051] The screening steps of the dense knowledge base line vector screening module include:
[0052] A threshold is set for the cluster. According to the user's question, the question is first vectorized and the similarity is calculated with the centroid of each cluster. If it is greater than the threshold, the similarity with each vector in the cluster is calculated. Otherwise, it is filtered out, and finally the TOP-K dense vectors are obtained.
[0053] The specific introduction steps of the sparse vector introduction module are:
[0054] 2.1 Use the SPLADE model to process these dense vectors to generate sparse vectors; 2.2 The SPLADE model uses BERT's encoding capabilities to convert text into BERT vectors after initial Tokenization; Apply the masked language model MLM to all token positions and calculate the corresponding probability of each token to each word in the BERT vocabulary;
[0055] 2.3SPLADE aggregates these probabilities for each word and promotes the sparsity of the vector by applying the normalization method of log saturation effect. The obtained weight reflects the relevance of each word to the input token, generating a learned sparse vector.
[0056] 2.4 Generate Token-to-Weight mapping.
[0057] The present invention also relates to a clustering-based sparse and dense retrieval enhancement generation method, comprising the following steps:
[0058] S1: Build a dense knowledge base, split documents into fixed-size chunks, vectorize the chunks, and cluster the vectorized chunks using the K-means algorithm;
[0059] S2: Through the dense knowledge base line vector screening module, according to the user query, the vectors in the cluster whose similarity with the query is higher than the preset threshold are screened out;
[0060] S3: Through the sparse vector introduction module, the filtered dense vectors are converted into sparse vectors to achieve accurate retrieval at the character level.
[0061] The present invention first introduces sparse vectors and combines them with dense vectors for hybrid search. Unlike dense vectors, sparse vectors are not simplified representations of semantics, but are an alternative to full-text search, aiming to prune and expand keywords in full-text search and define keyword weights for inverted index vocabulary. This allows a document to be represented by a sparse vector composed of pruned keywords. Then, similar documents are merged together using K-means clustering. When a new query begins, irrelevant documents can be filtered out in blocks, greatly saving retrieval time.
[0062] The advantages of the present invention are mainly reflected in the following aspects:
[0063] Improve retrieval efficiency: By clustering documents using the K-means clustering algorithm, similar documents can be grouped together. This way, when searching, documents that are not relevant to the query can be quickly filtered out, thereby reducing the search scope and improving retrieval efficiency.
[0064] Enhanced retrieval accuracy: By converting dense vectors into sparse vectors, accurate retrieval at the character level can be achieved. As an alternative to full-text search, sparse vectors can trim and expand keywords to improve retrieval accuracy.
[0065] Hybrid search strategy: Combining sparse vectors and dense vectors for hybrid search can simultaneously utilize the accuracy of sparse vectors and the semantic richness of dense vectors, thereby achieving better results during retrieval.
[0066] Dynamically adjust clustering: When building a dense knowledge base, the K value can be dynamically adjusted according to the number of document categories, so that the clustering is more in line with the actual document distribution and the clustering effect is improved.
[0067] Utilize advanced vectorization models: Using advanced vectorization models such as BERT to vectorize documents can better capture the semantic information of the documents and provide more accurate vector representation for subsequent clustering and retrieval.
[0068] Generation of sparse vectors: Sparse vectors are generated through the SPLADE model. By using the encoding capability of BERT and combining it with the masked language model MLM, the generated sparse vectors can reflect the correlation between vocabulary and input tokens, thereby improving the accuracy of retrieval.
[0069] Save storage space: Sparse vectors are more efficient in storage than dense vectors because they only record the position and value of non-zero elements, which can greatly reduce the storage space required.
[0070] Flexible threshold setting: In the screening module, different thresholds can be set for different clusters and dynamically adjusted according to the similarity of user queries to meet different retrieval requirements.
[0071] Scalability: The system design allows expansion in different application scenarios, such as application to different document types and retrieval requirements, with good adaptability and flexibility.
[0072] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A clustering-based sparse and dense retrieval enhancement generation system, characterized in that: include: The building block of the dense knowledge base is used to split documents into fixed-size chunks, vectorize the chunks, and cluster the vectorized chunks using the K-means algorithm; The dense knowledge base line vector screening module is used to screen out vectors in the cluster whose similarity with the query is higher than a preset threshold according to the user query; The sparse vector introduction module is used to convert the filtered dense vectors into sparse vectors to achieve accurate retrieval at the character level.
2. The clustering-based sparse and dense retrieval enhancement generation system according to claim 1, characterized in that: The steps of constructing the dense knowledge base construction module are as follows: 1.
1. First, split the document into fixed-size chunks and vectorize the chunks. The vector dimensions are 768 and 1024. After vectorization, use K-means to cluster them. 1.
2. Select the value of K. Refer to the value of K in this article, classify all documents, and adjust the value of K according to the number of categories; 1.
3. After the K value is selected, the centroid of each category is randomly initialized, that is, vectors in different categories are randomly selected as the centroid, followed by distance calculation, using Euclidean distance: Among them, E1 is the vector in a cluster, CE is the vector of the centroid, and the final calculated centroid represents the entire vector representation: Among them, c i represents the class that is closest to the K tears, c i The value of is one from 1 to K, and then the centroid is updated and the loop is iterated until convergence, completing the clustering of dense vectors.
3. The clustering-based sparse and dense retrieval enhancement generation system according to claim 1, characterized in that: The screening steps of the dense knowledge base line vector screening module include: setting a threshold for the cluster, first vectorizing the question according to the user's question, and then calculating the similarity with the centroid of each cluster. When it is greater than the threshold, continue to calculate the similarity with each vector in the cluster, otherwise filter it out, and finally obtain TOP-K dense vectors.
4. The clustering-based sparse and dense retrieval enhancement generation system according to claim 1, characterized in that: The specific introduction steps of the sparse vector introduction module are: 2.1 Use the SPLADE model to process these dense vectors to generate sparse vectors; 2.2 The SPLADE model uses BERT’s encoding capabilities to convert text into BERT vectors after initial Tokenization. It applies the masked language model MLM to all token positions and calculates the probability of each token corresponding to each word in the BERT vocabulary. 2.3SPLADE aggregates these probabilities for each word and promotes the sparsity of the vector by applying the normalization method of log saturation effect. The obtained weight reflects the relevance of each word to the input token, generating a learned sparse vector. 2.4 Generate Token-to-Weight mapping.
5. A clustering-based sparse and dense retrieval enhancement generation method, characterized in that: The following steps are involved: S1. Build a dense knowledge base, split documents into fixed-size chunks, vectorize the chunks, and cluster the vectorized chunks using the K-means algorithm. S2, through the dense knowledge base line vector screening module, according to the user query, screen out the vectors in the cluster whose similarity with the query is higher than the preset threshold; S3. Through the sparse vector introduction module, the filtered dense vectors are converted into sparse vectors to achieve accurate retrieval at the character level.