Large language model retrieval enhancement method based on multiple indexes
By employing a multi-indexing approach that combines keyword, question, and summary retrieval, the slow retrieval speed and redundant results caused by the single index structure in existing technologies are resolved. This enables multi-dimensional retrieval, improves retrieval efficiency and accuracy, and enhances the response quality of large language models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-04-10
AI Technical Summary
In existing search enhancement generation technologies, the knowledge base index has a single structure, which cannot perform multi-dimensional matching, resulting in slow search speed, high redundancy of results, and difficulty in meeting users' multi-faceted needs.
A multi-indexing approach is adopted, including keyword retrieval, question retrieval, and summary retrieval. A question vector index, summary vector index, and inverted index are generated through a large language model. Multi-dimensional relevance scores are calculated and re-ranked to generate the final answer.
It improves retrieval efficiency and accuracy, enhances system robustness, provides more accurate and comprehensive information, and improves the response quality of large language models.
Smart Images
Figure CN121833874A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method for enhancing retrieval of large language models based on multiple indexes. Background Technology
[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have made groundbreaking progress in the field of natural language processing due to their powerful semantic understanding, generation, and reasoning capabilities. Intelligent question answering and knowledge retrieval systems have been widely applied in education, government affairs, healthcare, finance, and enterprise information scenarios. However, Large Language Models have inherent defects: knowledge lag (the model cannot update dynamic knowledge in real time after training) and knowledge illusion (it is prone to generating seemingly reasonable but factual content when dealing with in-depth questions in specialized domains). To solve these problems, Retrieval-Augmented Generation (RAG) technology has emerged. By retrieving accurate information from external knowledge bases and inputting it into the LLM along with the user query, the model is guided to generate answers based on reliable external knowledge, becoming a mainstream technical solution to improve the accuracy and reliability of LLM applications.
[0003] Currently, the core of mainstream search enhancement generation technology lies in the indexing and retrieval strategies of the knowledge base. The main indexing and retrieval methods can be divided into two categories: keyword retrieval and vector semantic retrieval. However, current search enhancement generation technology has the following shortcomings in its use: Existing knowledge base indexes are mostly single document block indexes or keyword indexes. This flat structure cannot reflect the multi-dimensional attributes of knowledge. Retrieval can only be performed on a single dimension, making it impossible to perform precise matching at multiple granularities and angles. This results in slow retrieval speed, high redundancy of results, and difficulty in meeting users' different needs for knowledge access. Summary of the Invention
[0004] The technical objective of this invention is to address the shortcomings of the prior art by providing a large language model retrieval enhancement method based on multiple indexes, which enables retrieval from different dimensions and improves retrieval efficiency and accuracy.
[0005] The technical solution adopted in this invention is as follows: A method for enhancing large language model retrieval based on multiple indexes, the method comprising the following steps: S1. Obtain the query statement input by the user, and perform keyword retrieval, question retrieval and summary retrieval in the pre-built knowledge base according to the query statement to obtain a keyword retrieval result set including document ID and keyword relevance score, a question retrieval result set including document ID and multiple question relevance scores and a summary retrieval result set including document ID and summary relevance score. The knowledge base includes a question vector index, a summary vector index, an answer vector index and an inverted index. S2. Integrate the keyword search result set, question search result set, and summary search result set according to the document ID to obtain a candidate result set including the document ID and the corresponding keyword relevance score, question relevance score, and summary relevance score; S3. Calculate the final relevance score corresponding to each document ID based on the keyword relevance score, question relevance score, and summary relevance score corresponding to each document ID. Re-sort each document ID according to the final relevance score to obtain the sorting result. Select a predetermined number of document IDs with the highest sorting in the sorting result as the result set. Obtain the text content corresponding to each document ID in the result set and perform content preprocessing to obtain the enhanced context. S4. Input the query statement and the enhanced context into the large language model to generate the final response.
[0006] The above technical measures obtain the query statement input by the user, and perform keyword retrieval, question retrieval, and summary retrieval in a pre-built knowledge base to obtain the relevance score of the query statement in multiple dimensions, thereby realizing multi-dimensional retrieval. By integrating and calculating the relevance scores of multiple dimensions, the final relevance score is obtained, and the results are re-ranked based on the final relevance score. A predetermined number of search result documents with the highest ranking are selected, pre-processed, and used as enhanced context to be input into a large language model along with the query statement. The large language model is then used to generate the final response.
[0007] The aforementioned technical measures employ multi-path parallel retrieval, which effectively compensates for poor performance on one path by allowing other paths to compensate, thus preventing retrieval failure due to the failure of a single path and enhancing system robustness. Simultaneously, by calculating the final relevance score based on the relevance scores of the query statement across multiple dimensions, and re-ranking the results based on this final relevance score, a predetermined number of top-ranked retrieval result documents are preprocessed and used as enhanced context input along with the query statement into the large language model. This provides the large language model with more accurate and comprehensive information, effectively improving retrieval efficiency and accuracy, and ultimately enhancing the quality of the final response.
[0008] Furthermore, the construction of the knowledge base includes the following steps: Step 1: Obtain the data source and perform document preprocessing to obtain a document set containing multiple documents; Step 2: Use the large language model to generate multiple questions associated with each document, and convert them into vectors to construct a question vector index; use the large language model to generate summaries of each document, and convert them into vectors to construct a summary vector index; use the large language model to extract the core content of each document, and convert it into vectors to construct an answer vector index; use the large language model to extract the keywords of the document and the frequency and position of the keywords to construct an inverted index. Step 3: Construct a knowledge base based on the document set, question vector index, summary vector index, answer vector index, and inverted index.
[0009] The above-mentioned technical measures facilitate matching question intent by constructing a question vector index; enable rapid location of knowledge domains by constructing a summary vector index; obtain fine-grained fragments by constructing an answer vector index; ensure entity accuracy by constructing an inverted index; and improve retrieval efficiency and recall quality by constructing multiple indexes.
[0010] Furthermore, the specific steps for keyword retrieval are as follows: The keywords of the query statement are extracted using the large language model, and the inverted index of the knowledge base is queried based on the keywords to obtain a keyword retrieval result set including document ID and keyword relevance score.
[0011] Furthermore, the specific steps for the question retrieval are as follows: The query statement is decomposed into multiple sub-questions and transformed into vectors using the large language model, resulting in multiple sub-question vectors. Based on the sub-question vectors, the question vector index and answer vector index of the knowledge base are queried to obtain a question retrieval result set including document ID and multiple question relevance scores.
[0012] The above-mentioned technical measures enhance multi-level semantic retrieval capabilities by using a large language model to transform the query statement into multiple sub-questions. They can accurately locate knowledge fragments related to each sub-question from the knowledge base, and obtain more accurate answer fragments by querying the answer vector index, providing high-quality support for subsequent operations.
[0013] Furthermore, the specific steps of the abstract retrieval are as follows: The query statement is transformed into a vector using the large language model to obtain a semantic vector. Based on the semantic vector, the summary vector index in the knowledge base is queried to obtain a candidate retrieval result set. Based on the semantic vector, the answer vector index of each document in the candidate retrieval result set is queried to obtain a summary retrieval result set including document ID and summary relevance score.
[0014] The above-mentioned technical measures utilize summary vector indexes to locate documents in the knowledge domain related to the query statement, and then use answer vector indexes for matching again, which can obtain more accurate fragments and effectively improve retrieval accuracy.
[0015] Furthermore, the formula for calculating the final relevance score corresponding to the document ID is as follows: ; Where S is the final relevance score, α is the keyword retrieval weight, X is the keyword relevance score, β is the vector semantic retrieval weight, Y is the average question relevance score, and Z is the summary relevance score.
[0016] The above-mentioned technical measures can quantify the search results of multiple dimensions in a unified manner by setting calculation formulas, thereby improving the accuracy of the search; by setting weights, the contribution of each dimension can be flexibly adjusted according to different situations, which is conducive to improving the accuracy of the search.
[0017] Furthermore, the keyword retrieval weight and vector semantic retrieval weight are obtained as follows: The large language model is used to classify the query statements into scenarios to obtain the scenario type of the query statements. Based on the scenario type, the keyword retrieval weight and vector semantic retrieval weight are determined.
[0018] The above-mentioned technical measures can be flexibly adapted to various application scenarios by establishing the association between scenario type and keyword retrieval weight and vector semantic retrieval weight.
[0019] Furthermore, the average score for the relevance of the problem is obtained as follows: The average relevance score of multiple issues corresponding to the document ID is calculated to obtain the average relevance score of the issue.
[0020] The above-mentioned technical measures, by using average values, can avoid outliers from having a significant impact on the results, thus making the output stable and reliable.
[0021] Furthermore, the predetermined quantity is determined based on the scenario type.
[0022] The above technical measures dynamically adjust the selected predetermined quantity according to the scenario type. Different scenario types have different requirements for the quantity and accuracy of search results, and efficiency can be improved by flexibly adjusting the predetermined quantity.
[0023] Furthermore, the content preprocessing includes formatting and deduplication of the text content corresponding to each document ID in the result set.
[0024] The above-mentioned technical measures can unify the format of questions of different formats into a format suitable for input to a large language model by adjusting the format; through deduplication, duplicate content can be removed, reducing the interference of redundant information on the large language model, while improving the quality and efficiency of the final response generation.
[0025] One or more technical solutions provided by this invention have at least the following technical effects or advantages: This invention obtains the query statement input by the user, performs keyword retrieval, question retrieval, and summary retrieval in a pre-built knowledge base to obtain the relevance score of the query statement in multiple dimensions, and realizes multi-dimensional retrieval. By integrating and calculating the relevance scores of multiple dimensions, a final relevance score is obtained, and the final relevance score is re-ranked. A predetermined number of search result documents with the highest ranking are selected, preprocessed, and used as enhanced context to be input into a large language model along with the query statement. The large language model is then used to generate the final response.
[0026] This invention employs multi-path parallel retrieval, where other paths can effectively compensate for poor performance in one path, preventing retrieval failure due to a single path failure and enhancing system robustness. Simultaneously, by calculating the final relevance score of the query statement across multiple dimensions, and re-ranking based on this final relevance score, a predetermined number of top-ranked retrieval result documents are preprocessed and input into a large language model along with the query statement. This provides the large language model with more accurate and comprehensive information, effectively improving retrieval efficiency and accuracy, and ultimately enhancing the quality of the final response. Attached Figure Description
[0027] The accompanying drawings, which are provided to further illustrate embodiments of the invention and constitute a part of this invention, are not intended to limit the scope of the invention. Figure 1 This is a flowchart illustrating the present invention. Detailed Implementation
[0028] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, where there is no conflict, the embodiments of the present invention and the features thereof can be combined with each other.
[0029] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.
[0030] Reference Figure 1This embodiment provides a method for enhancing large language model retrieval based on multiple indexes. The method includes the following steps: S1. Obtain the query statement input by the user, and perform keyword retrieval, question retrieval and summary retrieval in the pre-built knowledge base according to the query statement to obtain the keyword retrieval result set including document ID and keyword relevance score, the question retrieval result set including document ID and multiple question relevance scores, and the summary retrieval result set including document ID and summary relevance score. The knowledge base includes question vector index, summary vector index, answer vector index and inverted index. The construction of the knowledge base includes the following steps: Step 1: Obtain the data source and perform document preprocessing to obtain a document set containing multiple documents; Document preprocessing includes format conversion, deduplication, and outlier filtering. Format conversion involves converting multiple formats from the data source into a unified text format; deduplication involves removing duplicate content from the data source; and outlier filtering involves removing low-quality content such as garbled characters and missing values from the data source.
[0031] Data sources can be publicly available external data, obtained using technologies such as web scraping; data sources can also be private internal enterprise data, obtained through internal system API interfaces, direct export from databases, or by scraping file systems.
[0032] Step 2: Generate multiple questions associated with each document using a large language model (5-8 questions in total), and convert them into vectors. This vector conversion can be performed using a text embedding model to construct a question vector index. Generate summaries for each document using the large language model (summaries covering core themes, key viewpoints, and conclusions), and convert them into vectors to construct a summary vector index. Extract the core content of each document (key paragraphs, core argument sections, or the entire text) using the large language model, and convert it into vectors to construct an answer vector index. Extract keywords from the documents using the large language model, along with their frequency and location, and construct an inverted index. Step 3: Construct a knowledge base based on the document set, question vector index, summary vector index, answer vector index, and inverted index.
[0033] The specific steps for keyword retrieval are as follows: Keywords are extracted from the query statement using a large language model. Based on the keywords, the inverted index of the knowledge base is queried to obtain a keyword retrieval result set including document ID and keyword relevance score.
[0034] The specific steps for question retrieval are as follows: The query statement is decomposed into multiple sub-questions and transformed into vectors using a large language model, resulting in multiple sub-question vectors. Based on the sub-question vectors, the question vector index and answer vector index of the knowledge base are queried to obtain a question retrieval result set including document ID and multiple question relevance scores.
[0035] The specific steps for abstract retrieval are as follows: The query statement is transformed into a vector using a large language model to obtain a semantic vector. Based on the semantic vector, the summary vector index in the knowledge base is queried to obtain a candidate retrieval result set. Based on the semantic vector, the answer vector index of each document in the candidate retrieval result set is queried to obtain a summary retrieval result set including document ID and summary relevance score.
[0036] S2. Integrate the keyword search result set, question search result set, and summary search result set according to document ID to obtain a candidate result set including document ID and corresponding keyword relevance score, question relevance score, and summary relevance score; S3. Calculate the final relevance score for each document ID based on the keyword relevance score, question relevance score, and summary relevance score. The formula for calculating the final relevance score for each document ID is as follows: ; Where S is the final relevance score, α is the keyword retrieval weight, X is the keyword relevance score, β is the vector semantic retrieval weight, Y is the average question relevance score, and Z is the summary relevance score.
[0037] The keyword retrieval weight and vector semantic retrieval weight are obtained as follows: The query statements are classified into scenarios using a large language model to obtain the scenario type of the query statement. Based on the scenario type, the keyword retrieval weight and vector semantic retrieval weight are determined.
[0038] Specifically, the scenario types include fact query, concept explanation, comparative analysis, causal reasoning, and open discussion.
[0039] The average score for question relevance is obtained as follows: The average relevance score of multiple issues corresponding to a document ID is calculated by averaging the scores.
[0040] Based on the final relevance score, each document ID is re-ranked to obtain a ranking result. A predetermined number of document IDs with the highest ranking in the ranking result are selected as the result set. The predetermined number is determined according to the scenario type. For example, if the scenario type is fact query, the predetermined number is 5-8; if the scenario type is open discussion, the predetermined number is 8-12. The text content corresponding to each document ID in the result set is obtained and preprocessed. The content preprocessing includes format adjustment and deduplication of the text content corresponding to each document ID in the result set to obtain enhanced context. To improve accuracy, a cross-encoder model based on domain retrieval data fine-tuning can be introduced to reorder the document IDs.
[0041] S4. Input the query statement and enhanced context into the large language model to generate the final response.
[0042] When applying this, handle an open-ended concept comparison query: The user enters the query: "Please make a detailed comparison between supervised learning and unsupervised learning in deep learning, and give examples of their application scenarios." The large language model retrieves the query statement and performs keyword retrieval, question retrieval, and summary retrieval respectively: The large language model extracts keywords: "deep learning", "supervised learning", "unsupervised learning" and "application scenarios". Using the above keywords, the inverted index of the knowledge base is queried to obtain the document IDs of documents containing the above keywords and the keyword relevance scores of the keyword search results. The large language model decomposes the query statement into multiple sub-questions: "What is supervised learning?", "What is unsupervised learning?", "What is the difference between supervised learning and unsupervised learning?", "What are the application scenarios of supervised learning?", and "What are the application scenarios of unsupervised learning?". The above sub-questions are vectorized, and the question vector index and answer vector index of the knowledge base are queried to obtain the document IDs of the documents related to the above sub-questions and the question retrieval result set with multiple question relevance scores. The large language model vectorizes the query statement, queries the summary vector index, and then queries the answer vector index to obtain a summary retrieval result set with the document ID and summary relevance score corresponding to the documents about "deep learning algorithm classification" and "machine learning paradigm comparison". The keyword search results set, question search results set, and summary search results set are integrated by document ID, and the final relevance score is calculated and re-ranked. Among them, the large language model determines the scenario type based on the query statement and judges it as "comparative analysis type" and "concept explanation type". The weights are set as follows: keyword search weight is 0.3 and vector semantic search weight is 0.7. The top 5 results are selected as the result set, and the format is adjusted and duplicates are removed to obtain the enhanced context. The enhanced context and query statement are input into the large language model to generate the final answer.
[0043] Processing an exact fact query: A user enters the query: "In which country did AstraZeneca's COVID-19 vaccine Vaxzevria receive emergency use authorization in January 2021?" The large language model retrieves the query statement and performs keyword retrieval, question retrieval, and summary retrieval respectively: The large language model extracts keywords: "AstraZeneca", "COVID-19 vaccine", "Vaxzevria", "January 2021", and "Emergency Use Authorization". Using these keywords, the inverted index of the knowledge base is queried to obtain a set of keyword search results containing the document IDs and keyword relevance scores of the documents containing these keywords. The large language model decomposes the query into multiple sub-questions: "What is a COVID-19 vaccine?", "What is Vaxzevria?", "What vaccines has AstraZeneca developed?", "What happened to AstraZeneca in January 2021?", and "Which countries have granted emergency use authorization to AstraZeneca's vaccines?". These sub-questions are vectorized, and the question vector index and answer vector index of the knowledge base are queried to obtain the document IDs of documents related to the sub-questions and a set of question retrieval results with relevance scores for multiple questions. The large language model vectorizes the query statement, queries the summary vector index, and then queries the answer vector index to obtain a summary retrieval result set with the document IDs and summary relevance scores corresponding to the documents about "COVID-19 vaccine" and "Vaxzevria". The keyword search results set, question search results set, and summary search results set are integrated by document ID, and the final relevance score is calculated and re-ranked. Among them, the large language model determines the scenario type based on the query statement and judges it as "fact query type", so the weight is set as 0.9 for keyword search and 0.1 for vector semantic search. The top 6 results are selected as the result set, and the format is adjusted and duplicates are removed to obtain the enhanced context. The enhanced context and the query statement are input into the large language model to generate the final answer, which is "United Kingdom".
[0044] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.
[0045] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A method for enhancing retrieval in a large language model based on multiple indexes, characterized in that: The large language model retrieval enhancement method includes the following steps: S1. Obtain the query statement input by the user, and perform keyword retrieval, question retrieval and summary retrieval in the pre-built knowledge base according to the query statement to obtain a keyword retrieval result set including document ID and keyword relevance score, a question retrieval result set including document ID and multiple question relevance scores and a summary retrieval result set including document ID and summary relevance score. The knowledge base includes a question vector index, a summary vector index, an answer vector index and an inverted index. S2. Integrate the keyword search result set, question search result set, and summary search result set according to the document ID to obtain a candidate result set including the document ID and the corresponding keyword relevance score, question relevance score, and summary relevance score; S3. Calculate the final relevance score corresponding to each document ID based on the keyword relevance score, question relevance score, and summary relevance score corresponding to each document ID. Re-sort each document ID according to the final relevance score to obtain the sorting result. Select a predetermined number of document IDs with the highest sorting in the sorting result as the result set. Obtain the text content corresponding to each document ID in the result set and perform content preprocessing to obtain the enhanced context. S4. Input the query statement and the enhanced context into the large language model to generate the final response.
2. The method for enhancing large language model retrieval based on multiple indexes according to claim 1, characterized in that: The construction of the knowledge base includes the following steps: Step 1: Obtain the data source and perform document preprocessing to obtain a document set containing multiple documents; Step 2: Use the large language model to generate multiple questions associated with each document, and convert them into vectors to construct a question vector index; use the large language model to generate summaries of each document, and convert them into vectors to construct a summary vector index; use the large language model to extract the core content of each document, and convert it into vectors to construct an answer vector index; use the large language model to extract the keywords of the document and the frequency and position of the keywords to construct an inverted index. Step 3: Construct a knowledge base based on the document set, question vector index, summary vector index, answer vector index, and inverted index.
3. The method for enhancing large language model retrieval based on multiple indexes according to claim 1, characterized in that: The specific steps for keyword retrieval are as follows: The keywords of the query statement are extracted using the large language model, and the inverted index of the knowledge base is queried based on the keywords to obtain a keyword retrieval result set including document ID and keyword relevance score.
4. The method for enhancing large language model retrieval based on multiple indexes according to claim 1, characterized in that: The specific steps for retrieving the question are as follows: The query statement is decomposed into multiple sub-questions and transformed into vectors using the large language model, resulting in multiple sub-question vectors. Based on the sub-question vectors, the question vector index and answer vector index of the knowledge base are queried to obtain a question retrieval result set including document ID and multiple question relevance scores.
5. The method for enhancing large language model retrieval based on multiple indexes according to claim 1, characterized in that: The specific steps for the abstract retrieval are as follows: The query statement is transformed into a vector using the large language model to obtain a semantic vector. Based on the semantic vector, the summary vector index in the knowledge base is queried to obtain a candidate retrieval result set. Based on the semantic vector, the answer vector index of each document in the candidate retrieval result set is queried to obtain a summary retrieval result set including document ID and summary relevance score.
6. The method for enhancing large language model retrieval based on multiple indexes according to claim 1, characterized in that: The formula for calculating the final relevance score corresponding to the document ID is as follows: ； Where S is the final relevance score, α is the keyword retrieval weight, X is the keyword relevance score, β is the vector semantic retrieval weight, Y is the average question relevance score, and Z is the summary relevance score.
7. The method for enhancing large language model retrieval based on multiple indexes according to claim 6, characterized in that: The keyword retrieval weight and vector semantic retrieval weight are obtained as follows: The large language model is used to classify the query statements into scenarios to obtain the scenario type of the query statements. Based on the scenario type, the keyword retrieval weight and vector semantic retrieval weight are determined.
8. The method for enhancing large language model retrieval based on multiple indexes according to claim 6, characterized in that: The average relevance score of the aforementioned issues is obtained as follows: The average relevance score of multiple issues corresponding to the document ID is calculated to obtain the average relevance score of the issue.
9. The method for enhancing large language model retrieval based on multiple indexes according to claim 7, characterized in that: The predetermined quantity is determined based on the scenario type.
10. The method for enhancing large language model retrieval based on multiple indexes according to claim 1, characterized in that: The content preprocessing includes formatting and deduplication of the text content corresponding to each document ID in the result set.