Method for enhancing large model RAG retrieval efficiency

Abstract

The invention provides a method for enhancing large model RAG retrieval efficiency. The method specifically comprises the steps of data index optimization, retrieval query optimization, retrieval result optimization and generation process optimization. During data index optimization, a large language model is adopted to process and mark data before data storage, data density is improved, low information amount and noise are reduced, and the method is used for reducing corresponding cost and improving accuracy. During data processing, a document abstract is created, and first-layer filtering is performed by using the document abstract, so that the retrieval efficiency is improved; a series of hypothetical / possible question and answer pairs are provided for each document, and then the generated questions are used as embedded retrieval blocks to improve the symmetry of retrieval; and then deleting repeated information by using a large language model, and simplifying data indexes and improving the response quality of the RAG system by clustering and merging the repeated information. According to the method, the retrieval efficiency and the retrieval accuracy can be further improved through various optimization schemes in actual use.

Description

Technical Field

[0001] The present invention relates to the technical field of retrieval optimization, and in particular to a method for enhancing the efficiency of large model RAG retrieval. Background Art

[0002] Retrieval-Augmented Generation (RAG) technology is an advanced question-answering system that can process and understand files and databases in various formats to build a comprehensive knowledge base. RAG technology cleverly combines the accuracy of information retrieval with the creativity of content generation, aiming to enhance the comprehensive capabilities of language models by utilizing huge data sets. The core advantage of this technology lies in its organic combination of large-scale language models and efficient retrieval systems. This combination gives the model a dual advantage in generating answers: it can rely on its built-in knowledge base and access external massive information resources, so that it can provide more accurate and comprehensive answers. Traditional large-scale language models usually rely only on their preset knowledge to generate answers.

[0003] The invention patent with announcement number: CN117891838B discloses a large model retrieval enhancement generation method and device. First, a block algorithm is designed to improve the current vector block strategy, vector indexing is performed, two indexes are constructed, and in-depth search is performed; then the post-processor design is entered, and a cross encoder is used to rearrange the search results, and intelligent re-ranking is performed. Finally, response synthesis is performed, and the answers obtained are integrated or refined and summarized to form the final reply. Compared with the prior art, the present invention can provide powerful guiding retrieval capabilities for various downstream tasks, further enhancing the accuracy and efficiency of information retrieval.

[0004] The invention patent with the publication number: CN118467681A discloses a RAG retrieval optimization method and system based on title enhancement and intent recognition, including extracting high-frequency words in the document, combining high-frequency words in the same scene knowledge base, and forming a scene knowledge base index; identifying the title in the document, judging whether the title conforms to the title format, and grading the title; segmenting the document according to the title level, and vectorizing and storing it; vectorizing the question and finding the similarity with the scene knowledge base index, determining the scene knowledge base where the document is located, and retrieving the document blocks and returning them to the large model to generate answers. The present invention improves the problem of ignoring the connection between the title and the text in the conventional segmentation method, making the information stored in the knowledge base richer and the results of the large model answer more diverse.

[0005] Although the above solution can improve the accuracy of retrieval to a certain extent, in actual use, it is still necessary to further improve the retrieval accuracy and shorten the retrieval time. Summary of the invention

[0006] The purpose of the present invention is to provide a method for enhancing the retrieval efficiency of large model RAG, which can further improve the retrieval efficiency and retrieval accuracy through a variety of optimization schemes in actual use.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A method for enhancing the efficiency of large model RAG retrieval, specifically including data index optimization, retrieval query optimization, retrieval result optimization and generation process optimization;

[0009] Among them, when optimizing data index, a large language model is used to process and mark the data before storing it, which improves the density of the data and reduces the amount of low information and noise, so as to reduce the corresponding cost and improve the accuracy; when processing data, a document summary is created, and the document summary is used for the first layer of filtering to improve the retrieval efficiency; a series of hypothetical / possible question and answer pairs are generated for each document, and then the generated questions are used as embedded retrieval blocks to improve the symmetry of the retrieval;

[0010] Then, a large language model is used to remove duplicate information, and the data index is streamlined by clustering and merging duplicate information, thereby improving the response quality of the RAG system.

[0011] Among them, data index optimization also includes testing and chunking optimization steps. The documents are chunked, and the chunk size should be adjusted according to the specific embedding model used to find the best chunking strategy. Sentence-level chunking is adopted, and simplicity and efficiency are balanced while maintaining text semantics. A sliding window mechanism is used to improve the coverage and coherence of information.

[0012] Furthermore, when optimizing retrieval queries, user queries are adjusted and optimized to make them more suitable for specific search systems, thereby improving the relevance of search results. At the same time, queries are decomposed to derive several simple sub-questions from complex questions to assist in retrieval.

[0013] Further qualifying, pseudo-document generation after query decomposition, using the HyDE approach, a hypothetical document is generated based on the user query, and the embedding of the hypothetical answer is used to retrieve similar documents.

[0014] Further optimization,retrieval result optimization includes the step of prioritizing search results,through reranking, DLM reranking or ILDE reranking, placing the most,relevant documents close to the query and improving the accuracy of,the response.

[0015] Among them, the retrieval result optimization also includes a context hint compression step, which is used to compress unnecessary information in the generated hints, thereby reducing the generated noise and improving the performance.

[0016] Furthermore, retrieval result optimization also includes the RAG correction step, using a lightweight model to score and filter the retrieval results to ensure that only the most relevant information is used to generate the final response.

[0017] Among them, the generation process optimization reduces the noise in the generation process through logical reasoning prompts to ensure that the model can focus on relevant information. At the same time, through the self-reflection mechanism, the model can self-evaluate and select the best output during the generation process; the model is fine-tuned for the RAG scenario to enhance the model's ability to ignore irrelevant information, and the natural language inference model is used to filter out irrelevant context to ensure the high accuracy of the generation results.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] In actual use, the present invention performs multi-layer optimization on the retrieval process through data index optimization, retrieval query optimization, retrieval result optimization and generation process optimization steps, deletes duplicate information by using a large language model, and streamlines the data index by clustering and merging duplicate information. Compared with traditional retrieval methods, the present invention has the advantages of faster response, high retrieval efficiency and higher accuracy in actual use. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 It is the overall principle diagram of the present invention. DETAILED DESCRIPTION

[0022] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the embodiments of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature. Embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0023] See also Figure 1,This embodiment discloses a method for enhancing the efficiency of large model RAG retrieval, which specifically includes data index optimization, retrieval query optimization, retrieval result optimization and generation process optimization;

[0024] Among them, when optimizing data index, a large language model is used to process and mark the data before storing it, which improves the density of the data and reduces the amount of low information and noise, so as to reduce the corresponding cost and improve the accuracy; when processing data, a document summary is created, and the document summary is used for the first layer of filtering to improve the retrieval efficiency; a series of hypothetical / possible question and answer pairs are generated for each document, and then the generated questions are used as embedded retrieval blocks to improve the symmetry of the retrieval;

[0025] Then, a large language model is used to remove duplicate information, and the data index is streamlined by clustering and merging duplicate information, thereby improving the response quality of the RAG system.

[0026] In actual use, the GPT-4 model is used to extract relevant information from unstructured data to increase the information density in the block and improve the accuracy of data processing.

[0027] In actual use, the present invention performs multi-layer optimization on the retrieval process through data index optimization, retrieval query optimization, retrieval result optimization and generation process optimization steps, deletes duplicate information by using a large language model, and streamlines the data index by clustering and merging duplicate information. Compared with traditional retrieval methods, the present invention has the advantages of faster response, high retrieval efficiency and higher accuracy in actual use.

[0028] In order to facilitate those skilled in the art to further understand the present invention, the present invention is further described below in conjunction with specific steps.

[0029] In some preferred implementation cases, data index optimization also includes testing and chunking optimization steps, in which documents are chunked, and the chunk size should be adjusted based on the specific embedding model used to find the best chunking strategy; sentence-level chunking is used, and simplicity and efficiency are balanced while maintaining text semantics, and a sliding window mechanism is used to improve information coverage and coherence.

[0030] When performing RAG retrieval, the user asks a natural language question to the RAG system. The retrieval module of the RAG system retrieves the most relevant documents or paragraphs from the external knowledge base based on the user's question. This step usually uses various search engine technologies, such as inverted indexing, vector retrieval, etc., to quickly find relevant paragraphs from a large-scale document collection. The retrieved documents or paragraphs are passed to the generation module, which combines this information with the user's question as input for the subsequent generation process.

[0031] Furthermore, in some preferred implementation cases, retrieval query optimization is performed by adjusting and optimizing user queries to make them more suitable for a specific search system, thereby improving the relevance of search results. At the same time, queries are decomposed to derive several simple sub-questions from complex questions to assist in retrieval.

[0032] Furthermore, after decomposing the query, pseudo documents are generated. Using the HyDE method, a hypothetical document is generated based on the user query, and the embedding of the hypothetical answer is used to retrieve similar documents.

[0033] Furthermore, in actual use, retrieval result optimization includes the step of prioritizing search results by reranking, and placing the most relevant documents close to the query through reranking to improve the accuracy of the response; specifically, through DLM reranking or ILDE reranking, placing the most relevant documents close to the query to improve the accuracy of the response.

[0034] Furthermore, in some preferred implementation cases, the retrieval result optimization also includes a context hint compression step, which is used to compress unnecessary information in the generated hints, thereby reducing the generated noise and improving the performance.

[0035] Furthermore, retrieval result optimization also includes the RAG correction step, using a lightweight model to score and filter the retrieval results to ensure that only the most relevant information is used to generate the final response.

[0036] Furthermore, in some preferred implementation cases, the generation process optimization reduces the noise in the generation process through logical reasoning prompts, ensuring that the model can focus on relevant information. At the same time, through the self-reflection mechanism, the model can self-evaluate and select the best output during the generation process; the model is fine-tuned for RAG scenarios to enhance the model's ability to ignore irrelevant information, and the natural language reasoning model is used to filter out irrelevant context to ensure the high accuracy of the generation results.

[0037] Furthermore, in some preferred implementation cases, the knowledge graph is introduced into the RAG system to enhance the system's ability to process complex information. At the same time, a long-context LLM (large language model) is used. An LLM that supports long context, such as Gemini 1.5 or GPT-4 128k, is used to replace the traditional blocking and retrieval method, thereby improving the efficiency and accuracy of the RAG system.

[0038] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0039] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for enhancing the efficiency of large model RAG retrieval, characterized in that: Specifically, it includes data index optimization, search query optimization, search result optimization and generation process optimization; Among them, when optimizing data index, a large language model is used to process and mark the data before storing it, which improves the density of the data and reduces the amount of low information and noise, so as to reduce the corresponding cost and improve the accuracy; when processing data, a document summary is created, and the document summary is used for the first layer of filtering to improve the retrieval efficiency; a series of hypothetical / possible question and answer pairs are generated for each document, and then the generated questions are used as embedded retrieval blocks to improve the symmetry of the retrieval; Then, a large language model is used to remove duplicate information, and the data index is streamlined by clustering and merging duplicate information, thereby improving the response quality of the RAG system.

2. The method for enhancing the efficiency of large model RAG retrieval according to claim 1, characterized in that: When optimizing data indexes, testing and chunking optimization steps are also included. Documents are chunked, and the chunk size should be adjusted based on the specific embedding model used to find the best chunking strategy. Sentence-level chunking is used, and simplicity and efficiency are balanced while maintaining text semantics. A sliding window mechanism is used to improve information coverage and coherence.

3. The method for enhancing the efficiency of large model RAG retrieval according to claim 1, characterized in that: When optimizing retrieval queries, we adjust and optimize user queries to make them more suitable for specific search systems and improve the relevance of search results. At the same time, we decompose queries and derive several simple sub-questions from complex questions to assist in retrieval.

4. The method for enhancing the efficiency of large model RAG retrieval according to claim 3, characterized in that: After decomposing the query, pseudo documents are generated. Using the HyDE method, a hypothetical document is generated based on the user query, and the embedding of the hypothetical answer is used to retrieve similar documents.

5. The method for enhancing the efficiency of large model RAG retrieval according to claim 1, characterized in that: Retrieval result optimization includes the step of prioritizing search results using reranking, placing the most relevant documents close to the query through DLM reranking or ILDE reranking, and improving the accuracy of the response.

6. The method for enhancing the efficiency of large model RAG retrieval according to claim 5, characterized in that: The retrieval result optimization also includes a contextual hint compression step, which is used to compress unnecessary information in the generated hints, thereby reducing the generated noise and improving the performance.

7. The method for enhancing the efficiency of large model RAG retrieval according to claim 6, characterized in that: Result optimization also includes the correct RAG step, which uses lightweight models to score and filter the results, ensuring that only the most relevant information is used to generate the final response.

8. A method for enhancing large model RAG retrieval efficiency according to any one of claims 1 to 7, characterized in that: The generation process optimization reduces the noise in the generation process through logical reasoning prompts to ensure that the model can focus on relevant information. At the same time, through the self-reflection mechanism, the model self-evaluates and selects the best output during the generation process. The model is fine-tuned for RAG scenarios to enhance the model's ability to ignore irrelevant information. The natural language inference model is used to filter out irrelevant context to ensure the high accuracy of the generation results.

Images