Retrieval enhancement generation method and system based on hybrid retrieval and self-adaptive sorting

Abstract

The invention discloses a retrieval enhancement generation method and system based on hybrid retrieval and adaptive sorting, and relates to the technical field of artificial intelligence and natural language processing. Comprising the following steps: 1, analyzing a query language and providing multi-path retrieval: receiving a natural language query input by a user, and performing semantic analysis and structured processing; starting dense vector retrieval and sparse semantic retrieval in parallel, and respectively obtaining candidate document sets from the knowledge base; 2, candidate mixed result fusion is carried out, wherein duplicate removal and preliminary fusion are carried out on candidate documents obtained through dense retrieval and sparse retrieval, and a unified candidate document pool is formed; all the candidate documents are evaluated according to the query semantic matching degree, document authority and quality, context coherence and generation task type factors, sorting weights are dynamically generated based on all the factors, and a candidate document pool is resorted; and constructing a structured context prompt prompt, inputting the structured context prompt prompt into a pre-trained large model, and generating final response content.

Description

Technical Field

[0001] This invention discloses a retrieval enhancement generation method and system based on hybrid retrieval and adaptive ranking, which relates to the fields of artificial intelligence and natural language processing technology. Background Technology

[0002] With the rapid development of Large Language Models (LLMs), they have demonstrated powerful capabilities in tasks such as text generation, question answering systems, and content creation. However, because the model's knowledge is embedded in the training data, it is difficult to acquire the latest or domain-specific knowledge in real time, leading to problems such as factual errors or information lag in the generated content. To address this, Retrieval-Enhanced Generation (RAG) technology has been proposed. By retrieving relevant information from external knowledge bases before generation and using it as contextual input to the generative model, it enhances the accuracy and reliability of its output.

[0003] Existing RAG systems mostly employ a single retrieval method, such as sparse retrieval based on BM25 or dense retrieval based on vector embedding. The former relies on keyword matching and has weak semantic generalization ability; while the latter can capture deep semantics, it is sensitive to lexical differences and easily affected by embedding bias. In addition, existing systems usually use fixed weights to sort or reorder retrieval results, lacking the ability to dynamically adapt to query intent, document quality, and generation task type, resulting in a mismatch between retrieval results and generation requirements, affecting the quality of the final output. Summary of the Invention

[0004] This invention addresses the problems of existing technologies by providing a retrieval enhancement generation method and system based on hybrid retrieval and adaptive sorting. It overcomes the shortcomings of existing retrieval enhancement generation systems, such as single retrieval methods and rigid sorting strategies. By constructing a hybrid retrieval mechanism that integrates dense and sparse retrieval and designing an adaptive sorting module that can dynamically adjust sorting weights, it achieves accurate screening and optimized sorting of candidate documents, thereby significantly improving the relevance, accuracy, and contextual consistency of the generated content.

[0005] The specific solution proposed in this invention is as follows:

[0006] This invention provides a retrieval enhancement generation method based on hybrid retrieval and adaptive ranking, comprising:

[0007] Step 1: Parse the query language and provide multi-path retrieval: Receive the natural language query input by the user, perform semantic parsing and structured processing; initiate dense vector retrieval and sparse semantic retrieval in parallel, and obtain candidate document sets from the knowledge base respectively;

[0008] Step 2: Perform candidate result fusion: Deduplicate and initially fuse the candidate documents obtained from dense and sparse searches to form a unified candidate document pool;

[0009] Step 3: Perform adaptive ranking: Evaluate each candidate document based on factors such as query semantic matching degree, document authority and quality, contextual coherence, and generation task type, and dynamically generate ranking weights based on each factor to re-rank the candidate document pool.

[0010] Step 4: Build context and generate search content: Select several top-ranked documents, build a structured context prompt, input it into the pre-trained large model, and generate the final response content.

[0011] Furthermore, in step 1 of the retrieval enhancement generation method based on hybrid retrieval and adaptive ranking, when dense vector retrieval is initiated, the query text and document are encoded into vectors based on a pre-trained language model, and relevant candidate documents are obtained through vector similarity matching.

[0012] When sparse semantic retrieval is initiated, a keyword matching method is used to analyze the occurrence of keywords in the query text and the documents to measure the relevance of the documents and obtain candidate documents.

[0013] Furthermore, in step 2 of the retrieval enhancement generation method based on hybrid retrieval and adaptive sorting, duplicate candidate documents are deduplicated: a similarity threshold is set, and when the similarity between two candidate documents exceeds the similarity threshold, they are considered duplicate documents. Based on the ranking of the candidate documents in the retrieval, document length, and source, one candidate document is deleted, while the other candidate document is retained.

[0014] During the initial fusion, candidate documents are fused based on the relevance scores calculated for each candidate document using dense vector retrieval and sparse semantic retrieval. Dense vector retrieval is based on vector similarity, while sparse semantic retrieval is based on keyword matching relevance metrics. During fusion, the relevance scores of the two candidate documents are directly added or weighted to obtain the fused relevance score.

[0015] Furthermore, in step 3 of the retrieval enhancement generation method based on hybrid retrieval and adaptive ranking, initial weights are set for query semantic matching degree, document authority and quality, contextual coherence, and generation task type. The initial weights are set based on experience or prior knowledge, and then a lightweight neural network or ranking learning model, Learning toRank, is used to dynamically allocate ranking weights based on each factor.

[0016] Furthermore, in step 4 of the retrieval enhancement generation method based on hybrid retrieval and adaptive ranking, the structured context prompt is input into the pre-trained large model LlaMA or ChatGLM to generate the final response content.

[0017] This invention also provides a retrieval enhancement generation system based on hybrid retrieval and adaptive ranking, including a query parsing module, a hybrid retrieval fusion module, an adaptive ranking module, a context building module, and a generative model inference engine module.

[0018] The query parsing module parses the query language and provides multi-path retrieval: it receives natural language queries input by users, performs semantic parsing and structured processing; and initiates dense vector retrieval and sparse semantic retrieval in parallel to obtain candidate document sets from the knowledge base, respectively.

[0019] The hybrid retrieval and fusion module performs hybrid fusion of candidate results: it deduplicates and initially merges the candidate documents obtained from dense and sparse retrieval to form a unified candidate document pool.

[0020] The adaptive ranking module performs adaptive ranking: it evaluates each candidate document based on factors such as query semantic matching degree, document authority and quality, contextual coherence, and generation task type, and dynamically generates ranking weights based on each factor to re-rank the candidate document pool.

[0021] The context building module constructs the context, and the generation model inference engine module generates the search content. The context building module selects several top-ranked documents and constructs a structured context prompt, which is then input into the pre-trained large model of the generation model inference engine module to generate the final response content.

[0022] Furthermore, when the query parsing module of the retrieval enhancement generation system based on hybrid retrieval and adaptive ranking starts dense vector retrieval, it encodes the query text and documents into vectors based on a pre-trained language model, and retrieves relevant candidate documents through vector similarity matching.

[0023] When sparse semantic retrieval is initiated, a keyword matching method is used to analyze the occurrence of keywords in the query text and the documents to measure the relevance of the documents and obtain candidate documents.

[0024] Furthermore, the hybrid retrieval fusion module of the retrieval enhancement generation system based on hybrid retrieval and adaptive sorting performs deduplication on candidate documents: a similarity threshold is set, and when the similarity between two candidate documents exceeds the similarity threshold, they are considered duplicate documents. Based on the ranking of the candidate documents in the retrieval, document length, and source, one candidate document is deleted, while the other candidate document is retained.

[0025] During the initial fusion, candidate documents are fused based on the relevance scores calculated for each candidate document using dense vector retrieval and sparse semantic retrieval. Dense vector retrieval is based on vector similarity, while sparse semantic retrieval is based on keyword matching relevance metrics. During fusion, the relevance scores of the two candidate documents are directly added or weighted to obtain the fused relevance score.

[0026] Furthermore, in step 3 of the aforementioned retrieval enhancement generation system based on hybrid retrieval and adaptive ranking, initial weights are set for query semantic matching degree, document authority and quality, contextual coherence, and generation task type. The initial weights are set based on experience or prior knowledge, and then a lightweight neural network or ranking learning model, Learning toRank, is used to dynamically allocate ranking weights based on each factor.

[0027] Furthermore, the context building module of the retrieval enhancement generation system based on hybrid retrieval and adaptive ranking inputs the structured context prompt into the pre-trained large model LlaMA or ChatGLM of the generative model inference engine module to generate the final response content.

[0028] The advantages of this invention are:

[0029] (1) Improve search coverage and accuracy: By integrating dense and sparse search methods, and taking into account semantic understanding and keyword matching capabilities, the recall and accuracy of key information are significantly improved.

[0030] (2) Enhance sorting flexibility and intelligence: The adaptive sorting module can dynamically adjust the sorting strategy according to different query intentions and task types, avoid the bias caused by fixed weights, and improve the matching degree between search results and generation requirements.

[0031] (3) Improve the quality of generated content: The optimized sorted context information is more in line with the generation goal, effectively reducing illusions, contradictions and missing information, and improving the accuracy and readability of the output.

[0032] (4) The system is highly scalable: the modular design supports flexible access to different search engines and generation models, and is suitable for a variety of application scenarios, such as intelligent customer service, knowledge Q&A, automatic report generation, etc. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the system application process of the present invention. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0035] Example 1

[0036] This invention provides a retrieval enhancement generation method based on hybrid retrieval and adaptive ranking, comprising:

[0037] Step 1: Parse the query language and provide multi-path retrieval: Receive the natural language query input by the user, perform semantic parsing and structured processing; initiate dense vector retrieval and sparse semantic retrieval in parallel, and obtain candidate document sets from the knowledge base respectively.

[0038] When dense vector retrieval is initiated, the query text and document are encoded into vectors based on a pre-trained language model, and relevant candidate documents are retrieved through vector similarity matching.

[0039] When sparse semantic retrieval is initiated, a keyword matching method is used to analyze the occurrence of keywords in the query text and the documents to measure the relevance of the documents and obtain candidate documents.

[0040] Step 2: Perform candidate result fusion: Deduplicate and initially fuse the candidate documents obtained from dense and sparse searches to form a unified candidate document pool.

[0041] The process of deduplicating candidate documents involves setting a similarity threshold. If the similarity between two candidate documents exceeds the threshold, they are considered duplicate documents. Based on the candidate documents' ranking in the retrieval, document length, and source, one candidate document is deleted while the other is retained.

[0042] During the initial fusion, candidate documents are fused based on the relevance scores calculated for each candidate document using dense vector retrieval and sparse semantic retrieval. Dense vector retrieval is based on vector similarity, while sparse semantic retrieval is based on keyword matching relevance metrics. During fusion, the relevance scores of the two candidate documents are directly added or weighted to obtain the fused relevance score.

[0043] Step 3: Adaptive Ranking: Evaluate each candidate document based on factors such as query semantic matching degree, document authority and quality, contextual coherence, and generation task type. Then, dynamically generate ranking weights based on each factor and re-rank the candidate document pool. Initial weights are set for query semantic matching degree, document authority and quality, contextual coherence, and generation task type. These initial weights are determined based on experience or prior knowledge. A lightweight neural network or a learning-to-rank model is then used to dynamically allocate ranking weights based on each factor.

[0044] Step 4: Build context and generate search content: Select several top-ranked documents, build a structured context prompt, input it into the pre-trained large model, and generate the final response content.

[0045] Structured context prompts can be input into pre-trained large models such as LlaMA or ChatGLM to generate the final response content.

[0046] Example 2

[0047] This invention also provides a retrieval enhancement generation system based on hybrid retrieval and adaptive ranking, including a query parsing module, a hybrid retrieval fusion module, an adaptive ranking module, a context building module, and a generative model inference engine module.

[0048] The query parsing module parses the query language and provides multi-path retrieval: it receives natural language queries input by users, performs semantic parsing and structured processing; and initiates dense vector retrieval and sparse semantic retrieval in parallel to obtain candidate document sets from the knowledge base, respectively.

[0049] The hybrid retrieval and fusion module performs hybrid fusion of candidate results: it deduplicates and initially merges the candidate documents obtained from dense and sparse retrieval to form a unified candidate document pool.

[0050] The adaptive ranking module performs adaptive ranking: it evaluates each candidate document based on factors such as query semantic matching degree, document authority and quality, contextual coherence, and generation task type, and dynamically generates ranking weights based on each factor to re-rank the candidate document pool.

[0051] The context building module constructs the context, and the generation model inference engine module generates the search content. The context building module selects several top-ranked documents and constructs a structured context prompt, which is then input into the pre-trained large model of the generation model inference engine module to generate the final response content.

[0052] The information interaction and execution process between the modules in the above system are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description in the method embodiment of the present invention, and will not be repeated here.

[0053] Similarly, the advantages of the system of the present invention are:

[0054] (1) Improve search coverage and accuracy: By integrating dense and sparse search methods, and taking into account semantic understanding and keyword matching capabilities, the recall and accuracy of key information are significantly improved.

[0055] (2) Enhance sorting flexibility and intelligence: The adaptive sorting module can dynamically adjust the sorting strategy according to different query intentions and task types, avoid the bias caused by fixed weights, and improve the matching degree between search results and generation requirements.

[0056] (3) Improve the quality of generated content: The optimized sorted context information is more in line with the generation goal, effectively reducing illusions, contradictions and missing information, and improving the accuracy and readability of the output.

[0057] (4) The system is highly scalable: the modular design supports flexible access to different search engines and generation models, and is suitable for a variety of application scenarios, such as intelligent customer service, knowledge Q&A, automatic report generation, etc.

[0058] It should be noted that not all steps and modules in the above processes and system structures are mandatory; some steps or modules can be omitted as needed. The execution order of each step is not fixed and can be adjusted as required. The system structures described in the above embodiments can be physical or logical structures. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0059] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A method for generating retrieval enhancement based on hybrid retrieval and adaptive ranking, characterized in that Comprise: Step 1: parsing the query language and providing multi-path retrieval: receiving user input natural language query, semantic analysis and structured processing; Parallel start dense vector retrieval and sparse semantic retrieval, respectively from the knowledge base to obtain the candidate document set; Step 2: perform candidate hybrid result fusion: the candidate documents obtained by dense retrieval and sparse retrieval are de-duplicated and preliminarily fused to form a unified candidate document pool; Step 3: adaptive ranking: evaluate each candidate document according to query semantic matching degree, document authority and quality, context coherence, and generation task type factors, and dynamically generate ranking weights based on each factor to re-rank the candidate document pool, Step 4: build context and generate retrieval content: select a number of documents ranked at the top to build a structured context prompt, input it into a pre-trained large model to generate the final response content.

2. The method of claim 1, wherein the method further comprises: receiving a query from a user; and generating a result set based on the query. The feature is that in step 1, when starting dense vector retrieval, the query text and the document are encoded into vectors based on the pre-trained language model, and the relevant candidate documents are obtained by vector similarity matching retrieval; When starting sparse semantic retrieval, use keyword matching method to analyze the occurrence of keywords in query text and keywords in document to measure the relevance of document, and obtain candidate documents.

3. The method of claim 1, wherein the method further comprises: In step 2, the candidate documents are de-duplicated: set a similarity threshold, when the similarity of two candidate documents exceeds the similarity threshold, they are duplicate documents, according to the ranking of candidate documents in retrieval, document length and source, select to delete one of the candidate documents, and keep the other one; In the preliminary fusion, the relevance scores of each candidate document calculated by dense vector retrieval and sparse semantic retrieval are used for candidate document fusion, where dense vector retrieval is based on vector similarity, and sparse semantic retrieval is based on keyword matching relevance measurement, and the relevance scores of the two candidate documents are directly added or weighted added to obtain the fused relevance score.

4. The method of claim 1, wherein the method further comprises: In step 3, set the initial weight of query semantic matching degree, document authority and quality, context coherence, and generation task type, and then use lightweight neural network or learning to rank model to dynamically assign ranking weight based on each factor.

5. The retrieval enhancement generation method based on hybrid retrieval and adaptive ranking according to claim 1, wherein in step 4, the structured context prompt is input into the pre-trained large model LlaMA or ChatGLM to generate the final response content.

6. A hybrid search and adaptive ranking based retrieval augmentation generation system, characterized by Comprise query analysis module, hybrid retrieval fusion module, adaptive ranking module, context construction module and generation model inference engine module, The query analysis module parses the query language and provides multi-path retrieval: receiving user input natural language query, semantic analysis and structured processing; Parallel start dense vector retrieval and sparse semantic retrieval, respectively from the knowledge base to obtain the candidate document set; The mixed retrieval fusion module performs mixed result fusion of the candidates: the dense retrieval and the sparse retrieval are performed on the candidate documents to remove duplicates and preliminarily fuse the candidate documents to form a unified candidate document pool; The adaptive ranking module performs adaptive ranking: each candidate document is evaluated according to the query semantic matching degree, the document authority and quality, the context coherence, and the generation task type, and the ranking weight is dynamically generated based on each factor to reorder the candidate document pool, The context construction module constructs the context, and the model inference engine module generates the retrieval content, wherein the context construction module selects a plurality of documents ranked at the top to construct a structured context prompt, and inputs the structured context prompt into a pre-trained large model of the model inference engine module to generate final response content.

7. The retrieval enhancement generation system based on hybrid retrieval and adaptive ranking of claim 6, wherein When the query analysis module starts the dense vector retrieval, the query text and the document are encoded into vectors based on the pre-trained language model, and the relevant candidate documents are obtained through vector similarity matching retrieval; When the sparse semantic retrieval is started, the keyword matching method is used to analyze the occurrence of keywords in the query text and the keywords in the document to measure the relevance of the document, and the candidate documents are obtained.

8. The retrieval enhancement generation system based on hybrid retrieval and adaptive ranking of claim 6, wherein The mixed retrieval fusion module removes duplicates from the candidate documents: a similarity threshold is set, when the similarity of two candidate documents exceeds the similarity threshold, the two candidate documents are duplicates, and one of the two candidate documents is deleted according to the ranking, length and source of the candidate documents in the retrieval, and the other candidate document is retained; During the preliminary fusion, the relevance scores of each candidate document calculated by the dense vector retrieval and the sparse semantic retrieval are used for candidate document fusion, wherein the dense vector retrieval is based on vector similarity, and the sparse semantic retrieval is based on keyword matching relevance measurement, and the relevance scores of the two candidate documents are directly added or weighted added to obtain the fused relevance score.

9. The retrieval enhancement generation system based on hybrid retrieval and adaptive ranking of claim 6, wherein In step 3, initial weights are set for the query semantic matching degree, the document authority and quality, the context coherence, and the generation task type, the initial weights are set according to experience or prior knowledge, and a lightweight neural network or a learning to rank model is used to dynamically assign ranking weights based on each factor.

10. A retrieval enhancement generation system based on hybrid retrieval and adaptive ranking according to claim 6, characterized in that the context... The construction module inputs the structured context prompt into the pre-trained large model LlaMA or ChatGLM of the model inference engine module to generate the final response content.

Images