Retrieval enhancement generation method and system based on iteration self-improvement mechanism

Abstract

The invention discloses a retrieval enhancement generation method and system based on an iteration self-improvement mechanism, and belongs to the retrieval enhancement generation technology, and the method comprises the steps: extracting a core entity, a relationship category and a query intention in input information through an entity relationship recognition model, and generating a standardized query statement; performing retrieval on the standardized generated statement based on the retrieval strategy, and returning a retrieval result; performing dynamic information fusion processing on the retrieval result to generate a reference text; generating a final query result based on the reference text, and performing quality scoring on the query result; and if the quality score result does not reach the preset threshold value, optimizing the retrieval strategy, and performing retrieval again until the integrity score of the query result reaches the set threshold value. According to the method, the retrieval accuracy can be improved.

Description

Technical Field

[0001] This invention relates to the field of retrieval enhancement generation technology, and more specifically to a retrieval enhancement generation method and system based on an iterative self-improvement mechanism. Background Technology

[0002] Retrieval-Augmented Generation (RAG) is a key technology that integrates information retrieval with Large Language Model (LLM) generation capabilities. It aims to solve problems such as knowledge lag, factual errors, and illusion generation in large language models, and is widely used in scenarios such as intelligent question answering, knowledge base question answering, and professional domain consulting.

[0003] Its core foundational technologies consist of two parts: first, information retrieval technology, encompassing keyword-based retrieval (such as inverted index technology), semantic-based retrieval (such as converting text into vectors through models like Sentence-BERT and combining them with algorithms like cosine similarity to achieve semantic matching), and hybrid retrieval strategies. The retrieval targets are typically data from structured databases, unstructured document collections, or vector databases; second, large language model technology, based on the Transformer architecture, which acquires language understanding and generation capabilities through large-scale text data pre-training. It can generate natural language answers based on the input question and retrieved reference information. Typical models include the GPT series and the LLaMA series.

[0004] Traditional retrieval enhancement generation methods mainly include natural language question parsing, single retrieval based on standardized query statements, information fusion of multiple retrieved information, and model training by concatenating standardized questions and fused reference text into fixed-format prompts.

[0005] However, the above-mentioned retrieval enhancement generation methods usually have the following shortcomings: (1) Incomplete information acquisition and weak multi-hop reasoning ability: Existing technologies usually adopt a single retrieval mode, with fixed retrieval strategies and no iterative optimization. When faced with multi-hop questions (such as "Which university did the founder of a drug research and development company graduate from?"), it is necessary to obtain three related information in sequence: "drug-research and development company", "company-founder", and "founder-graduation university". However, a single retrieval can only obtain the first layer of information based on the initial question and cannot generate new retrieval clues based on the information already obtained, resulting in the lack of subsequent related information and logical gaps in the generated answer.

[0006] (2) Poor coordination between retrieval and generation, making it difficult to guarantee accuracy: In the existing technology, retrieval and generation adopt a one-way linear process, and the generation process of question parsing cannot provide feedback on the quality of results to the retrieval process. When the retrieval module in the retrieval process obtains low-relevance information due to a fixed strategy (such as using only keywords to retrieve complex semantic questions), the generation module used for question parsing can only generate inaccurate answers based on incorrect references, and the system cannot autonomously identify and correct retrieval deviations, resulting in errors being transmitted to the final result. Summary of the Invention

[0007] In view of this, the present invention provides a retrieval enhancement generation method and system based on an iterative self-improvement mechanism, which is used to at least solve some of the technical problems in the background art.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: This invention discloses a retrieval enhancement generation method based on an iterative self-improvement mechanism, comprising the following steps: S1. Problem Analysis: Utilize the entity relationship recognition model to extract the core entities, relationship categories, and query intent from the input problem information, and generate standardized query statements; S2. Statement Retrieval: Based on the generated standardized query statement and using retrieval strategies, preliminary retrieval results are obtained from the database; S3: Dynamic Information Fusion Optimization: The preliminary search results are extracted in a structured manner, and a correlation graph is constructed based on the structured extraction results. The preliminary search results are then optimized using the correlation graph to obtain the optimized fused reference text. S4. Generation and Quality Assessment: Generate the final query results based on the fused reference text, and score the quality of the final query results; S5. Iterative Feedback: If the quality score of the final query result does not reach the preset threshold, the retrieval strategy is optimized using preset rules and in combination with the quality score, and steps S2-S4 are repeated until the quality score of the query result reaches the set threshold.

[0009] Furthermore, in S1, the entity relationship recognition model includes an entity relationship recognition model based on the BERT-BiLSTM-CRF architecture.

[0010] Further, in S2, the retrieval strategy includes: The Sentence-BERT model is used to transform standardized query statements into data vectors, and then the cosine similarity between each vector and the data in the database is calculated and sorted to obtain the Top-N semantic retrieval results. By using standardized query keywords to perform keyword retrieval in the database, the top-M keyword search results with the highest keyword matching degree are obtained.

[0011] Furthermore, S3 specifically includes the following steps: Information structure extraction: Entity extraction and relation extraction are performed on the preliminary search results to extract the "entity1-relation-entity2-source-time" five-tuples from the search results; where entity1 represents the execution target in the preliminary search results and entity2 represents the execution subject in the preliminary search results; Association graph construction and optimization: Using entities as nodes and relationships as edges, construct a directed association graph from the five-tuples of the search results, and optimize the constructed directed association graph; Structured text generation: Based on the topological order of the optimized directed graph, nodes and edges are transformed into natural language text to form a fused reference text.

[0012] Furthermore, the constructed directed association graph is optimized, specifically including: Remove duplicate "Entity 1-Relationship-Entity 2" edges from the directed association graph; The credibility score of each edge in the directed association graph is calculated based on the weight and quantity of the information source, and edges with credibility scores below a set threshold are deleted.

[0013] Furthermore, S4 specifically includes: Generate corresponding prompts based on the question type in the input question information; The obtained prompts and fused reference text are input into the large language model LLaMA-13B to obtain the final query results; The completeness, accuracy, and logicality of the final query results are scored separately, and the scores are weighted and summed to calculate a comprehensive quality score for the final query results.

[0014] On the other hand, the present invention also discloses a retrieval enhancement generation system based on an iterative self-improvement mechanism, comprising: Problem Analysis Module: The entity relationship recognition model is used to extract the core entities, relationship categories and query intent from the input information, and to generate standardized query statements. Iterative retrieval module: Based on the generated standardized query statements and using retrieval strategies, preliminary search results are obtained from the database. Dynamic information fusion optimization fusion module: The preliminary search results are extracted in a structured manner, and a correlation graph is constructed based on the structured extraction results. The preliminary search results are then optimized using the correlation graph to obtain the optimized fused reference text. Generation and Quality Assessment: The final query results are generated based on the fused reference text, and a quality score is given to the final query results; Iterative Feedback Module: If the quality score of the final query result does not reach the preset threshold, the retrieval strategy is optimized using preset rules and the quality score result, and steps S2-S4 are repeated until the quality score of the query result reaches the set threshold.

[0015] Preferably, the dynamic information fusion optimization module specifically performs the following steps: Information structure extraction: Entity extraction and relation extraction are performed on the preliminary search results to extract the "entity1-relation-entity2-source-time" five-tuples from the search results; where entity1 represents the execution target in the preliminary search results and entity2 represents the execution subject in the preliminary search results; Association graph construction and optimization: Using entities as nodes and relationships as edges, construct a directed association graph from the five-tuples of the search results, and optimize the constructed directed association graph; Structured text generation: Based on the topological order of the optimized directed graph, nodes and edges are transformed into natural language text to form a fused reference text.

[0016] Prior to this, the iterative feedback module specifically performs the following steps: Generate corresponding prompts based on the question type in the input question information; The obtained prompts and fused reference text are input into the large language model LLaMA-13B to obtain the final query results; The completeness, accuracy, and logicality of the final query results are scored separately, and the scores are weighted and summed to calculate a comprehensive quality score for the final query results.

[0017] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a retrieval enhancement generation method and system based on an iterative self-improvement mechanism, which has the following beneficial effects: To address the issues of incomplete information acquisition and weak multi-hop reasoning capabilities in existing technologies, this invention proposes a dynamic feedback closed-loop architecture for retrieval and generation. Based on a three-dimensional score of the accuracy, completeness, and logicality of the generated answers, the retrieval parameters (similarity threshold, strategy type) are optimized in reverse, and a self-correcting mechanism for the retrieval direction is established.

[0018] To address the issues of poor coordination and difficulty in guaranteeing accuracy in existing retrieval and generation technologies, this invention proposes a retrieval enhancement generation method based on an iterative self-improvement mechanism. Through a multi-round cyclical process of "retrieval-generation-evaluation-feedback-re-retrieval," it achieves the acquisition of multi-source related information. The core lies in the dynamic generation of retrieval clues and the iterative adjustment of retrieval strategies. Simultaneously, a dynamic information fusion method integrating information association graph construction and credibility assessment is proposed. Through entity-relationship modeling and multi-source weight scoring, it achieves the structured integration of multi-round retrieval information, improving the reliability of reference texts. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the overall architecture of the retrieval enhancement generation system based on an iterative self-improvement mechanism provided by the present invention.

[0021] Figure 2 This is a schematic diagram illustrating the problem analysis steps provided by the present invention.

[0022] Figure 3 This is a schematic diagram of the iterative retrieval process provided by the present invention.

[0023] Figure 4 This is a schematic diagram of the dynamic information fusion process provided by the present invention.

[0024] Figure 5 This is a schematic diagram of the generation and quality assessment process provided by the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] This invention discloses a retrieval enhancement generation method based on an iterative self-improvement mechanism, comprising the following steps: S1. Problem Analysis: Utilize the entity relationship recognition model to extract the core entities, relationship categories, and query intent from the input problem information, and generate standardized query statements; S2. Statement Retrieval: Based on the generated standardized query statement and using retrieval strategies, preliminary retrieval results are obtained from the database; S3: Dynamic Information Fusion Optimization: The preliminary search results are extracted in a structured manner, and a correlation graph is constructed based on the structured extraction results. The preliminary search results are then optimized using the correlation graph to obtain the optimized fused reference text. S4. Generation and Quality Assessment: Generate the final query results based on the fused reference text, and score the quality of the final query results; S5. Iterative Feedback: If the quality score of the final query result does not reach the preset threshold, the retrieval strategy is optimized using preset rules and in combination with the quality score, and steps S2-S4 are repeated until the quality score of the query result reaches the set threshold.

[0027] like Figure 1 As shown, the above method can be implemented through an iterative self-improvement-based retrieval enhancement generation system, which includes four core modules: problem analysis module, iterative retrieval module, dynamic information fusion module, and generation and quality assessment module. Each module achieves bidirectional data interaction through a data bus.

[0028] In this system architecture, the problem parsing module is responsible for initial problem processing and retrieval clue generation; the iterative retrieval module performs multiple rounds of retrieval and strategy adjustment; the dynamic information fusion module realizes the association and integration of multi-source information; and the generation and quality assessment module completes answer generation and quality scoring. These four modules work together to form a complete technical solution with "iterative retrieval," "dynamic feedback," and "self-improvement" capabilities. The functions of each system module are described in detail below.

[0029] 1. Problem Analysis Module The question parsing module is used to parse the input question information. It uses an entity relationship recognition model based on the BERT-BiLSTM-CRF architecture to parse the input question information and outputs standardized query statements and new search clues.

[0030] refer to Figure 2 The specific steps for solving the problem are as follows: After receiving the user's original question, the entity relationship recognition model is used to extract the core entities (E), relationship types (R), and query intent (I). For example, after inputting the question "In what year was the research and development company of a certain vaccine established?", the entity relationship recognition model will extract E = "a certain vaccine" "research and development company", R = "research and development relationship", I = "query the year of establishment", and generate a standardized query statement Q = "the research and development relationship of a certain vaccine research and development company and query the year of establishment".

[0031] 2. Iterative Retrieval Module This module is the core of multi-round retrieval. It consists of a retrieval strategy selector, a hybrid retrieval engine, and a parameter adjuster. The input is a standardized query statement and quality assessment feedback, and the output is a set of multi-round retrieval results.

[0032] refer to Figure 3 The specific steps for iterative retrieval are as follows: (1) Initialization of retrieval strategy: The retrieval strategy selector is used to perform different retrieval strategies on the input standardized query statement. Specifically, the standardized query statement is converted into a 768-dimensional vector through the Sentence-BERT model, and the cosine similarity is calculated and sorted with the data in the database one by one to obtain the Top-N semantic retrieval results; the keywords of the standardized query statement are used to perform keyword retrieval in the database to obtain the Top-M keyword retrieval results with the highest keyword matching degree.

[0033] (2) Search execution: The hybrid search engine filters out the same search results based on the obtained Top-N semantic search results and Top-M keyword search results; and uses a weighted summation strategy of "semantic similarity × 0.7 + keyword matching degree × 0.3" to filter the same search results again and return the final candidate results.

[0034] (3) Dynamic adjustment of search parameters: The parameter adjuster receives the quality score (P) from the generation module. When P < the preset threshold, the parameters are adjusted based on the preset rules: If the low score is caused by the redundancy of the results (redundancy > 0.6), the similarity threshold is increased (e.g., from 0.7 to 0.8); if the low score is caused by the low relevance of the results (relevance < 0.5), the search strategy is switched (e.g., from keyword search to hybrid search), and the next round of search is started.

[0035] 3. Dynamic Information Fusion Module This module adopts a fusion mechanism of "association modeling + credibility screening". Its core consists of an information association graph builder and a credibility evaluator. The input is a set of multi-round retrieval results, and the output is a structured fusion reference text.

[0036] refer to Figure 4 The specific steps for implementing dynamic information fusion are as follows: (1) Structured information extraction: Entity extraction (based on spaCy tool) and relation extraction (based on FewRel model) are performed on the results of each round of retrieval. The five-tuple "entity 1 - relation - entity 2 - source - time" is extracted. For example, the result "a certain vaccine was developed by Company A in 2020 (source: official website of the State Drug Administration)" is extracted as (a certain vaccine, development, Company A, official website of the State Drug Administration, 2020).

[0037] (2) Construction and optimization of association graph: The information association graph builder uses entities as nodes and relationships as edges to construct a directed association graph from quintuples; the redundant elimination unit merges duplicate "entity-relationship-entity" edges and retains the record with the most sources; the credibility evaluator calculates the credibility score of each edge based on "source authority (e.g., official website weight 1.0, ordinary blog weight 0.3) + number of sources" (score = Σ source weight), and retains the edges with a score ≥ 0.5.

[0038] (3) Structured text generation: Based on the topological order of the association graph (starting from the initial entity and sorted according to the relationship transmission path), nodes and edges are transformed into natural language text, such as "a certain vaccine was developed by Company A (credibility 1.0, source: official website of the State Drug Administration), Company A was founded in 2010 (credibility 0.8, source: 2 authoritative financial websites)", forming a fusion reference text.

[0039] 4. Generation and Quality Assessment Module This module adopts a "dynamic prompt generation + multi-dimensional quality scoring" design. Its core consists of a prompt word generator (based on GPT-3.5), a large language model (using LLaMA-13B), and a quality evaluator (based on RoBERTa). The input is the fused reference text, and the output is the final answer and quality score, thus forming a dynamic feedback closed-loop architecture for retrieval and generation, which can autonomously correct the retrieval direction.

[0040] refer to Figure 5 The specific steps for generation and quality assessment are as follows: (1) Dynamic prompt generation: The prompt generator generates appropriate prompts based on the question type (fact query / logical reasoning / summary). For example, the prompt for fact query is "Based on the following reference information (credibility has been marked), answer the question accurately: [Question]. Reference information: [Mixed text]. Requirement: Answer only based on the reference information, without adding any extra content."; the prompt for reasoning adds the requirement that "the reasoning process must be clearly shown".

[0041] (2) Response generation: The large language model loads dynamic prompt words and fused reference text, and generates the initial response by using a temperature coefficient of 0.3 (to reduce randomness).

[0042] (3) Multi-dimensional quality assessment: The quality evaluator scores from three dimensions (maximum score 1.0): ① Accuracy (factual consistency between the answer and the reference text, calculated based on text similarity and entity matching); ② Completeness (whether the answer covers all the core requirements of the question, calculated based on intent matching); ③ Logicality (whether the reasoning process is coherent, calculated based on the completeness of relational transmission). The total score P = accuracy × 0.5 + completeness × 0.3 + logicality × 0.2. If P ≥ 0.8, the final answer is output; otherwise, the P value and error type (such as "insufficient accuracy") are fed back to the iterative retrieval module.

[0043] Combining the above modules, the complete implementation steps of the method of the present invention are as follows: (1) Users input the original question through the interactive interface (such as "Which university did the founder of a certain anti-cancer drug research and development company graduate from?").

[0044] (2) The problem analysis module extracts the initial entities “a certain anti-cancer drug” and “research and development company”, generates the initial query Q0 = “a certain anti-cancer drug research and development company”, and sends it to the iterative retrieval module.

[0045] (3) The iterative retrieval module adopts a semantic retrieval strategy and returns Top-8 results. After being processed by the dynamic information fusion module, the reference text "A certain anti-cancer drug was developed by Company B (credibility 1.0)" is generated and pushed to the generation module.

[0046] (4) The generation module generates the answer "A certain anti-cancer drug was developed by Company B". The quality evaluator scores it at P=0.4 (insufficient completeness, no answer regarding the founder and the university graduated). The score and "insufficient completeness" are fed back to the iterative retrieval module.

[0047] (5) The iterative retrieval module switches to a hybrid retrieval strategy based on feedback, obtains and integrates founder information, generates a second round of answers, and continues iterating until it generates "A certain anti-cancer drug was developed by Company B, whose founder C graduated from University X" (P=0.9).

[0048] (6) The system outputs the final answer and stores the case in the data repository.

[0049] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0050] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A retrieval enhancement generation method based on an iterative self-improvement mechanism, characterized in that, Includes the following steps: S1. Problem Analysis: Utilize the entity relationship recognition model to extract the core entities, relationship categories, and query intent from the input problem information, and generate standardized query statements; S2. Statement Retrieval: Based on the generated standardized query statement and using retrieval strategies, preliminary retrieval results are obtained from the database; S3: Dynamic Information Fusion Optimization: The preliminary search results are extracted in a structured manner, and a correlation graph is constructed based on the structured extraction results. The preliminary search results are then optimized using the correlation graph to obtain the optimized fused reference text. S4. Generation and Quality Assessment: Generate the final query results based on the fused reference text, and score the quality of the final query results; S5. Iterative Feedback: If the quality score of the final query result does not reach the preset threshold, the retrieval strategy is optimized using preset rules and in combination with the quality score, and steps S2-S4 are repeated until the quality score of the query result reaches the set threshold.

2. The retrieval enhancement generation method according to claim 1, characterized in that, In S1, the entity relationship recognition model includes an entity relationship recognition model based on the BERT-BiLSTM-CRF architecture.

3. The retrieval enhancement generation method according to claim 1, characterized in that, In S2, the retrieval strategy includes: The Sentence-BERT model is used to transform standardized query statements into data vectors, and then the cosine similarity between each vector and the data in the database is calculated and sorted to obtain the Top-N semantic retrieval results. By using standardized query keywords to perform keyword retrieval in the database, the top-M keyword search results with the highest keyword matching degree are obtained. Based on the obtained Top-N semantic search results and Top-M keyword search results, identical search results are filtered out, and the identical search results are weighted and summed. The weighted sum results are then sorted to obtain preliminary search results.

4. The retrieval enhancement generation method according to claim 1, characterized in that, S3 specifically includes the following steps: Information structure extraction: Entity extraction and relation extraction are performed on the preliminary search results to extract the "entity1-relation-entity2-source-time" five-tuples from the search results; where entity1 represents the execution target in the preliminary search results and entity2 represents the execution subject in the preliminary search results; Association graph construction and optimization: Using entities as nodes and relationships as edges, construct a directed association graph from the five-tuples of the search results, and optimize the constructed directed association graph; Structured text generation: Based on the topological order of the optimized directed graph, nodes and edges are transformed into natural language text to form a fused reference text.

5. The retrieval enhancement generation method according to claim 4, characterized in that, Optimize the constructed directed association graph, specifically including: Remove duplicate "Entity 1-Relationship-Entity 2" edges from the directed association graph; The credibility score of each edge in the directed association graph is calculated based on the weight and quantity of the information source, and edges with credibility scores below a set threshold are deleted.

6. The retrieval enhancement generation method according to claim 1, characterized in that, S4 specifically includes: Generate corresponding prompts based on the question type in the input question information; The obtained prompts and fused reference text are input into the large language model LLaMA-13B to obtain the final query results; The completeness, accuracy, and logicality of the final query results are scored separately, and the scores are weighted and summed to calculate a comprehensive quality score for the final query results.

7. A retrieval enhancement generation system based on an iterative self-improvement mechanism, characterized in that, include: Problem Analysis Module: The entity relationship recognition model is used to extract the core entities, relationship categories and query intent from the input information, and to generate standardized query statements. Iterative retrieval module: Based on the generated standardized query statements and using retrieval strategies, preliminary search results are obtained from the database. Dynamic information fusion optimization fusion module: The preliminary search results are extracted in a structured manner, and a correlation graph is constructed based on the structured extraction results. The preliminary search results are then optimized using the correlation graph to obtain the optimized fused reference text. Generation and Quality Assessment: The final query results are generated based on the fused reference text, and a quality score is given to the final query results; Iterative Feedback Module: If the quality score of the final query result does not reach the preset threshold, the retrieval strategy is optimized using preset rules and the quality score result, and steps S2-S4 are repeated until the quality score of the query result reaches the set threshold.

8. The retrieval enhancement generation system according to claim 7, characterized in that, In the iterative retrieval module, the retrieval strategy includes: The Sentence-BERT model is used to transform standardized query statements into data vectors, and then the cosine similarity between each vector and the data in the database is calculated and sorted to obtain the Top-N semantic retrieval results. By using standardized query keywords to perform keyword retrieval in the database, the top-M keyword search results with the highest keyword matching degree are obtained.

9. The retrieval enhancement generation system according to claim 7, characterized in that, The dynamic information fusion optimization module specifically performs the following steps: Information structure extraction: Entity extraction and relation extraction are performed on the preliminary search results to extract the "entity1-relation-entity2-source-time" five-tuples from the search results; where entity1 represents the execution target in the preliminary search results and entity2 represents the execution subject in the preliminary search results; Association graph construction and optimization: Using entities as nodes and relationships as edges, construct a directed association graph from the five-tuples of the search results, and optimize the constructed directed association graph; Structured text generation: Based on the topological order of the optimized directed graph, nodes and edges are transformed into natural language text to form a fused reference text.

10. The retrieval enhancement generation system according to claim 7, characterized in that, The iterative feedback module specifically performs the following steps: Generate corresponding prompts based on the question type in the input question information; The obtained prompts and fused reference text are input into the large language model LLaMA-13B to obtain the final query results; The completeness, accuracy, and logicality of the final query results are scored separately, and the scores are weighted and summed to calculate a comprehensive quality score for the final query results.

Images