Retrieval joint optimization method for retrieval enhancement generation system

Abstract

The invention relates to the technical field of artificial intelligence, and provides a retrieval joint optimization method for a retrieval enhancement generation system, which comprises the following steps: constructing a knowledge base and a vector library of a large language model, and obtaining an input problem of a user; performing keyword matching and similarity comparison on the input question and a knowledge base and a vector base of the large language model, and extracting text blocks from the knowledge base according to a result to obtain an initial retrieval candidate set; extracting related information in the initial retrieval candidate set and the large language model, and generating an initial external related information block set and an internal related information block set; performing progressive information verification on the initial external related information block set to obtain an external related information block set; the internal related information block set and the external related information block set are integrated, conflict information is removed, and a final retrieval result is generated; and sending the input question of the user and the final retrieval result to the large language model, and generating a corresponding answer by the large language model. The accuracy of large language model output can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically discloses a retrieval joint optimization method for a retrieval augmented generation system. Background Art

[0002] As an important method for large language models (LLMs) to handle knowledge-intensive tasks, the Retrieval-Augmented Generation (RAG) system is widely used to solve the knowledge limitation problem of LLMs. However, recent research has found that the RAG technology has significant defects: due to factors such as corpus quality, retriever reliability, and query complexity, its retrieval results often contain incomplete, irrelevant, or conflicting information, reducing the accuracy of the LLM output. These defects are divided into three categories: one is incomplete retrieval results, which cause the model to be unable to obtain comprehensive data support, affecting the integrity and accuracy of the generated content; the second is irrelevant retrieval results, which make it difficult for the model to distinguish useful information, resulting in information confusion; the third is conflicting retrieval results, which make it difficult for the model to determine the correct information. These defects increase the uncertainty of the model and affect the accuracy and reliability of the generated content.

[0003] Currently, retrieval source optimization, index optimization, and query optimization are common solutions, but existing methods are difficult to balance the integrity and relevance of retrieved information. For example, coarse-grained retrieval units can provide more relevant information, but they may also contain redundant content, reducing relevance. In addition, existing research rarely jointly optimizes the integrity and consistency of retrieved content. Early on, unstructured text materials were the main retrieval sources, and indexes were established through chunking strategies, but this method would cause sentence truncation and it was difficult to balance semantic integrity and context length. Subsequent research proposed optimization methods such as hierarchical retrieval, metadata appendices, small2big, etc., and extended the retrieval sources to semi-structured data such as PDF documents and structured data such as knowledge graphs. Recently, retrieval methods based on structural information have emerged continuously, such as the RAG system based on knowledge graph prompts (KGP), the RAG system based on personalized knowledge bases (KnowledGPT), and the RAG system enhanced by graph neural networks (G-Retriever). In addition, Fei Wang et al. proposed the ASTUTE RAG method to solve the internal and external knowledge conflict problem through source-aware knowledge integration.

[0004] However, there are still many problems with existing methods. On the one hand, most studies only focus on one or two of the integrity, relevance, and consistency of retrieved information, lacking comprehensive optimization research. Simply concatenating existing methods will result in a too long inference time and low efficiency of the RAG system, and the research methods in different dimensions lack synergy, leading to poor optimization effects, uneven information quality, and potential contradictions and conflicts. On the other hand, most retrieval results do not directly contain explicit results, and the implicit information is scattered in the knowledge base in a fragmented form. The existing methods have limited ability to identify implicit information and are difficult to effectively retrieve and integrate, resulting in the generated content may miss key implicit information, and the answers are incomplete or inaccurate. Therefore, there is an urgent need for a new type of retrieval joint optimization method to solve the isolated optimization problem of existing methods through joint optimization, and introduce progressive information verification in the reasoning process of ensuring relevance and consistency to solve the problem of information fragmentation, thereby improving the overall performance of the RAG system and the accuracy and reliability of the generated content. Summary of the Invention

[0005] The present invention aims to solve the problems of lack of comprehensive optimization, limited ability to identify implicit information, and difficulty in effective retrieval and integration in the retrieval augmented generation system, and proposes a retrieval joint optimization method for the retrieval augmented generation system.

[0006] The present invention provides a retrieval joint optimization method for the retrieval augmented generation system, including the following steps:

[0007] S1. Construct the knowledge base and vector library of the large language model, and obtain the input question of the user;

[0008] S2. Perform keyword matching and similarity comparison on the input question obtained in step S1 with the knowledge base and vector library of the large language model, and extract text blocks from the knowledge base according to the results of the similarity comparison and keyword matching to obtain an initial retrieval candidate set;

[0009] S3. Extract the initial retrieval candidate set obtained in step S2 and relevant information inside the large language model to generate an initial external relevant information block set and an internal relevant information block set;

[0010] S4. Perform progressive information verification on the initial external relevant information block set obtained in step S3 to obtain an external relevant information block set;

[0011] S5. Integrate the internal relevant information block set obtained in step S3 and the external relevant information block set obtained in step S4, remove conflicting information, and generate a final retrieval result;

[0012] S6. Send the input question of the user and the final retrieval result obtained in step S5 to the large language model, and the large language model generates a corresponding answer.

[0013] A retrieval joint optimization method for a retrieval-enhanced generation system according to some embodiments of the present application. In step S1, constructing the knowledge base and vector library of the large language model includes: obtaining knowledge base documents, splitting the knowledge base documents into text blocks to obtain a knowledge base composed of text blocks; using an embedding model to vectorize the text blocks to generate vector blocks, and obtaining a vector library composed of vector blocks.

[0014] A retrieval joint optimization method for a retrieval-enhanced generation system according to some embodiments of the present application. The embedding model is bge-m3, and the vector library is constructed using the database FAISS.

[0015] A retrieval joint optimization method for a retrieval-enhanced generation system according to some embodiments of the present application. Step S2 includes the following steps:

[0016] S201. Calculate the similarity between the user's input question and each text block in the knowledge base D through the BM25 algorithm to obtain a keyword matching result, as shown in formula (1):

[0017]

[0018] where r BM25 represents the similarity between the user's input question and the text block in the knowledge base, d ∈ D, d represents the text block, Q represents the question, Q = {q0, q1,..., q M}, M represents the number of words in the question, q i represents the i-th term in the question, i ∈ (1, 2, 3,..., M), f(q i , d) represents the frequency of the term q i appearing in the text block d, len(d) represents the number of words in the text block d, avg_len represents the average number of words in all text blocks in the knowledge base, k1 represents a parameter for normalizing the text block term frequency range, b represents a parameter for adjusting the degree of article length and information volume, IDF(q i ) represents the inverse document frequency of q i in the knowledge base D, as shown in formula (2):

[0019]

[0020] where N represents the number of text blocks in the knowledge base, n(q i ) represents the number of text blocks containing the term q i ;

[0021] S202. Calculate the semantic vector of the user's input question, compare the semantic vector of the user's input question with each vector block in the vector library to obtain a similarity comparison result, as shown in formula (3):

[0022]

[0023] Among them, r represents vs the vector similarity between the user's input question and the vector block in the knowledge base, v q represents the semantic vector of the user's input question, v q ={V q1 , V q2 ,…, V qT}, T represents the length of the semantic vector of the user input question, q ∈ Q, V qt represents the eigenvalue of the user's input question on the t-th dimension, v d represents the vector of text block d, v d ={V d1 , V d2 ,…, V dt}, V dt represents the eigenvalue of text block d on the t-th dimension;

[0024] S203. Combine the keyword matching result and the similarity comparison result through reciprocal rank fusion to obtain a fused similarity result, as shown in formula (4):

[0025]

[0026] Among them, r RRF represents the reciprocal rank fusion operation, R represents the sorting index, R = {r BM25 , r vs}, r ∈ R, c represents the ranking smoothing factor, and the ranking smoothing factor is used to reduce the influence of outlier elements on sorting;

[0027] S204. Sort the text blocks in the text library based on the fused similarity result, and extract the text blocks corresponding to the vector blocks with the highest similarity in the top k fused similarity results as the initial retrieval candidate set.

[0028] According to a retrieval joint optimization method for a retrieval enhanced generation system according to some embodiments of the present application, the parameter k1 for standardizing the text block word frequency range is set to 1.5, the parameter b for adjusting the article length and information amount degree is set to 0.75, and the ranking smoothing factor is set to a fixed value of 60.

[0029] According to a retrieval joint optimization method for a retrieval enhanced generation system according to some embodiments of the present application, the step S3 includes the following steps:

[0030] S301. Use a large language model to extract information related to the initial retrieval candidate set E = [e0, e1, …, e n and generate an initial external relevant information block set E ′ = [e0 ′ , e1 ′ , …, e ′ n , as shown in formula (5):

[0031]

[0032] Among them, e′ j ∈ E′, e j ∈ E, e′ j represents the Jth initial external relevant information block, e j represents the jth initial retrieval result, and I represents the information in the internal knowledge of the large language model related to the user's input question;

[0033] S302. Obtain relevant information blocks of internal knowledge from the large language model, and generate n I relevant information blocks through an adaptive method to obtain an internal relevant information block set, as shown in formula (6):

[0034]

[0035] Among them, I represents the internal relevant information block set, n I ≤ N I , LLM represents the large language model, represents the n I th internal relevant information block, and N I represents generating no more than N I internal relevant information blocks.

[0036] According to a retrieval joint optimization method of a retrieval enhanced generation system according to some embodiments of the present application, the step S4 includes the following steps:

[0037] S401. Perform initial integrity verification on each of the initial external relevant information blocks in the initial external relevant information block set in the front and back directions, splice each of the initial external relevant information blocks with one text block adjacent to the front and back of the corresponding text block respectively to obtain the front extended text block and the back extended text block of each of the initial external relevant information blocks, use the large language model to detect whether the front extended text block and the back extended text block of each of the initial external relevant information blocks introduce relevant information blocks of new information, and expand the relevant information blocks of new information into the corresponding initial external relevant information blocks to obtain extended external relevant information blocks, as shown in formula (7):

[0038]

[0039] Among them, represents the j-th extended external related information block, represents the j-th related information block that introduces new information through the pre-expanded text block, represents the j-th related information block that introduces new information through the post-expanded text block, represents the j-th related information block that introduces new information through the pre-expanded text block and the post-expanded text block, 0 < j < n;

[0040] S402. For the related information blocks introducing the new information, use a large language model to iteratively perform integrity verification in the introduction direction and append related information blocks of new new information until no new related information blocks of new new information are introduced, obtaining an external related information block set

[0041] According to a retrieval joint optimization method for a retrieval-enhanced generation system according to some embodiments of the present application, the step S5 includes the following steps:

[0042] S501. Merge the internal related information block set and the external related information block set to obtain an integrated related information block set, as shown in formula (8):

[0043]

[0044] Among them, A0 represents the integrated related information block set;

[0045] S502. Extract the metadata information set of the integrated related information block set, as shown in formula (9):

[0046]

[0047] Among them, S0 represents the metadata information set corresponding to the integrated related information block set A0, S I represents the metadata information set corresponding to the internal related information block set I in the integrated related information block set A0, a represents a related information block in the integrated related information block set A0, represents the external related information block set in the integrated related information block set A0 corresponding metadata information set;

[0048] S503. Use a large language model to detect conflicting information in the integrated related information blocks, judge the reliability according to the information content of the integrated related information blocks and the metadata information, filter errors and knowledge irrelevant to the user's input question, and perform W iterations to obtain multiple information block groups, as shown in formula (10):

[0049] <A w+1 ,S w+1 > = LLM(Q, <A0, S0>, <A w ,S w ,I itg ) (10)

[0050] Among them, W represents the total number of iterations, and 0 ≤ w ≤ W. <A w+1 ,S w+1 > represents the set of integrated relevant information blocks and metadata information pairs obtained after the (w + 1)-th iteration. A w+1 represents the set of integrated relevant information blocks obtained after the (w + 1)-th iteration. S w+1 represents the set of metadata information obtained after the (w + 1)-th iteration. A w represents the set of integrated relevant information blocks obtained after the w-th iteration. S w represents the set of metadata information obtained after the w-th iteration. I itg represents a hint instruction for integrating conflict information to filter misleading information;

[0051] S504. Take the set of integrated relevant information blocks and metadata information pairs obtained after each w-th iteration in step S503 as the input of the large language model, and respectively obtain the corresponding answers for the set of integrated relevant information blocks and metadata information pairs obtained after the (w + 1)-th iteration. Evaluate the reliability of the generated multiple answers, and take the most reliable answer as the final retrieval result, as shown in formula (11):

[0052] [ans w ,r w = LLM(Q, <A0, S0>, <A w ,S w ,I ans ) (11)

[0053] ans = ans w , if and only if r w = max(r0, …, r W )

[0054] Among them, ans w represents the answer generated according to the set of integrated relevant information blocks after the w-th iteration. r w represents the reliability score of the answer generated according to the set of integrated relevant information blocks after the w-th iteration. ans represents the final answer. r W represents the reliability score of the answer generated according to the set of integrated relevant information blocks after the W-th iteration. I ans represents a hint instruction for generating answers and reliability scores.

[0055] A retrieval joint optimization method for a retrieval enhanced generation system according to some embodiments of the present application. The reliability evaluation includes evaluating using multi-dimensional evaluation indicators, and the evaluation indicators of the dimension include consistency, relevance, and integrity.

[0056] The present invention also provides a retrieval joint optimization system for a retrieval enhanced generation system, using the above-mentioned retrieval joint optimization method for a retrieval enhanced generation system, including:

[0057] An input-output module for obtaining the user's input question and sending the corresponding answer to the user;

[0058] A knowledge base and a vector library for storing a knowledge text set of the required field, storing semantic vector blocks, and calculating the similarity with the user's input question;

[0059] A retriever for retrieving relevant information from the knowledge base related to the user's input and obtaining the final retrieval result through the retrieval joint optimization method;

[0060] A generator that uses a large language model to receive the user's input question and the final retrieval result obtained by the retriever, and generates an answer corresponding to the user's input question.

[0061] A retrieval joint optimization method for a retrieval enhanced generation system proposed by the present invention. Aiming at the fragmented distribution of implicit information, by extracting relevant information, the implicit information is converted into directly relevant information, which can ensure the accurate identification of implicit relevant information. At the same time, in order to better capture the information scattered in the long text, information verification is iteratively performed to ensure the comprehensiveness and integrity of the retrieval, and effectively improve the hit rate and recall rate of the retrieval enhanced generation system. This method takes into account that the authority and timeliness of the knowledge source will affect the correctness of the retrieval result. Unauthoritative and outdated information may introduce errors and knowledge conflicts, resulting in a decrease in the accuracy of the answer generated by the large language model. This method introduces metadata information in the conflict detection, provides the source and release time of the knowledge, provides source awareness and time awareness for the retriever, and avoids introducing low-quality and misleading information. The present invention uses internal relevant information blocks and external relevant information blocks as media, combines progressive information verification and conflict detection to jointly optimize the integrity, relevance, and self-consistency of the retrieval information, and solves the problem of lack of collaborative optimization in retrieval optimization. At the same time, it provides a retrieval result that better fits the user's input question, more meets the user's expectations, enables a more complete retrieval result to be utilized by the large language model, and does not introduce irrelevant and conflicting information, thereby improving the accuracy of the output of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic flowchart of a retrieval joint optimization method for a retrieval enhanced generation system of the present invention. Detailed Implementation Manner

[0063] The following further describes the implementation manner of the present invention in detail with reference to the accompanying drawings and embodiments. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0064] Embodiment 1 provides a retrieval joint optimization method for a retrieval enhanced generation system, as Figure 1 shown, including the following steps:

[0065] S1. Build the knowledge base and vector library of the large language model, and obtain the input question of the user;

[0066] S2. Perform keyword matching and similarity comparison on the input question obtained in step S1 with the knowledge base and vector library of the large language model, and extract text blocks from the knowledge base according to the results of the similarity comparison and keyword matching to obtain an initial retrieval candidate set;

[0067] S3. Extract the initial retrieval candidate set obtained in step S2 and relevant information inside the large language model to generate an initial external relevant information block set and an internal relevant information block set;

[0068] S4. Perform progressive information verification on the initial external relevant information block set obtained in step S3 to obtain an external relevant information block set;

[0069] S5. Integrate the internal relevant information block set obtained in step S3 and the external relevant information block set obtained in step S4, remove conflicting information, and generate a final retrieval result;

[0070] S6. Send the input question of the user and the final retrieval result obtained in step S5 to the large language model, and the large language model generates a corresponding answer.

[0071] Embodiment 2 provides a retrieval joint optimization method for a retrieval enhanced generation system, including the following steps:

[0072] S1. Build the knowledge base and vector library of the large language model, and obtain the input question of the user;

[0073] As a preference of this embodiment, building the knowledge base and vector library of the large language model includes: obtaining knowledge base documents, splitting the knowledge base documents into text blocks to obtain a knowledge base composed of text blocks; using an embedding model to vectorize the text blocks to generate vector blocks to obtain a vector library composed of vector blocks. More preferably, the embedding model is bge-m3, and the vector library is built using the database FAISS.

[0074] S2. Match the input problem obtained in step S1 with the knowledge base and vector library of the large language model for keyword matching and similarity comparison, and extract text blocks from the knowledge base according to the results of the similarity comparison and keyword matching to obtain an initial retrieval candidate set;

[0075] Preferably, in this embodiment, specifically, the following steps are included:

[0076] S201. Calculate the similarity between the user's input problem and each text block in the knowledge base D through the BM25 algorithm to obtain the keyword matching result, as shown in formula (1):

[0077]

[0078] where r BM25 represents the similarity between the user's input problem and the text block in the knowledge base, d ∈ D, d represents the text block, Q represents the problem, Q = {q0, q1,..., q M}, M represents the number of words in the problem, q i represents the i-th term in the problem, i ∈ (1, 2, 3,..., M), f(q i , d) represents the frequency of the term q i appearing in the text block d, len(d) represents the number of words in the text block d, avg_len represents the average number of words in all text blocks in the knowledge base, k1 represents the parameter for normalizing the text block word frequency range, b represents the parameter for adjusting the degree of article length and information volume, IDF(q i ) represents the inverse document frequency of q i in the knowledge base D, as shown in formula (2):

[0079]

[0080] where N represents the number of text blocks in the knowledge base, n(q i ) represents the number of text blocks containing the term q i ;

[0081] S202. Calculate the semantic vector of the user's input problem, and compare the semantic vector of the user's input problem with each vector block in the vector library to obtain the similarity comparison result, as shown in formula (3):

[0082]

[0083] where represents r vs the vector similarity between the user's input problem and the vector block in the knowledge base, v q represents the semantic vector of the user's input problem, v q = {V q1 , V q2,…,V qT}, where T represents the length of the semantic vector of the user's input question, q ∈ Q, V qt represents the eigenvalue of the user's input question on the t-th dimension, v d represents the vector of text block d, v d = {V d1 , V d2 ,…, V dt}, V dt represents the eigenvalue of text block d on the t-th dimension;

[0084] S203. Combine the keyword matching result and the similarity comparison result through reciprocal rank fusion to obtain the fused similarity result, as shown in formula (4):

[0085]

[0086] where r RRF represents the reciprocal rank fusion operation, R represents the sorting index, R = {r BM25 , r vs}, r ∈ R, c represents the ranking smoothing factor, and the ranking smoothing factor is used to mitigate the impact of outlier elements on sorting;

[0087] S204. Sort the text blocks in the text library based on the fused similarity result, and extract the text blocks corresponding to the vector blocks with the highest similarity in the top k fused similarity results as the initial retrieval candidate set.

[0088] More preferably, in this embodiment, the parameter k1 for normalizing the text block word frequency range is set to 1.5, the parameter b for adjusting the article length and information amount is set to 0.75, the ranking smoothing factor is set to a fixed value of 60, and the text blocks corresponding to the vector blocks with the highest similarity in the top 5 - 10 fused similarity results are extracted as the initial retrieval candidate set.

[0089] S3. Extract the initial retrieval candidate set obtained in step S2 and the relevant information inside the large language model to generate the initial external relevant information block set and the internal relevant information block set;

[0090] As a preference of this embodiment, specifically, it includes the following steps:

[0091] S301. Use the large language model to extract the information related to the initial retrieval candidate set E = [e0, e1,…, e n to generate the initial external relevant information block set E ′ = [e0 ′ , e1 ′ ,…, e ′ n, including filtering out information irrelevant to the user's input question, retaining the information relevant to the user's input question in the original initial retrieval candidate set, merging duplicate relevant information in the same text block. At the same time, it is also necessary to ensure that the content in the initial external relevant information block generated by the large language model is included in the original text block, that is, without additional internal knowledge, as shown in formula (5):

[0092]

[0093] where e′ j ∈E′, e j ∈E, e′ j represents the j-th initial external relevant information block, e j represents the j-th initial retrieval result, and I represents the information relevant to the user's input question in the large language model's internal knowledge;

[0094] S302. Obtain the relevant information blocks of the internal knowledge from the large language model, generate n I relevant information blocks through an adaptive method to obtain the internal relevant information block set. This internal knowledge relevant information block set reflects the consensus of extensive pre-training and fine-tuning data and can supplement the information missing in the external retrieval results, as shown in formula (6):

[0095]

[0096] where I represents the internal relevant information block set, n I ≤N I , LLM represents the large language model, represents the n I -th internal relevant information block, and N I represents generating no more than N I internal relevant information blocks.

[0097] S4. Perform progressive information verification on the initial external relevant information block set obtained in step S3 to obtain the external relevant information block set;

[0098] As a preference of this embodiment, specifically, it includes the following steps:

[0099] S401. Perform initial integrity verification in the front and back directions of each initial external relevant information block in the initial external relevant information block set, splice each initial external relevant information block with one text block adjacent to the front and one text block adjacent to the back of the corresponding text block respectively, obtain the front extended text block and the back extended text block of each initial external relevant information block, use the large language model to detect whether the front extended text block and the back extended text block of each initial external relevant information block introduce relevant information blocks of new information, and expand the relevant information blocks of new information into the corresponding initial external relevant information blocks to obtain the extended external relevant information blocks, as shown in formula (7):

[0100]

[0101] Among them, represents the j-th extended external relevant information block, represents the j-th relevant information block that introduces new information through the front extended text block, represents the j-th relevant information block that introduces new information through the back extended text block, represents the j-th relevant information block that introduces new information through the front extended text block and the back extended text block, 0 < j < n;

[0102] S402. For the relevant information blocks that introduce new information, use the large language model to iteratively perform integrity verification in the introduction direction and append new relevant information blocks of new information until no new relevant information blocks of new information are introduced, to obtain the external relevant information block set

[0103] S5. Integrate the internal relevant information block set obtained in step S3 and the external relevant information block set obtained in step S4, remove the conflicting information, and generate the final retrieval result;

[0104] As a preference of this embodiment, specifically, it includes the following steps:

[0105] S501. Merge the internal relevant information block set and the external relevant information block set to obtain the integrated relevant information block set, as shown in formula (8):

[0106]

[0107] Among them, A0 represents the integrated relevant information block set;

[0108] S502. When providing the integrated relevant information chunks to the large language model, the metadata information of the integrated relevant information chunks is provided simultaneously. The metadata information includes the source and time, that is, the integrated relevant information chunk is from a certain file inside or outside the large language model, a certain website, and the release time of the corresponding file or web page. The metadata information can assist the large language model in evaluating the reliability of the corresponding integrated relevant information chunk. The metadata information set of the integrated relevant information chunk set is extracted as shown in formula (9):

[0109]

[0110] Among them, S0 represents the metadata information set corresponding to the integrated relevant information chunk set A0, S I represents the metadata information set corresponding to the internal relevant information chunk set I in the integrated relevant information chunk set A0, a represents a relevant information chunk in the integrated relevant information chunk set A0, represents the external relevant information chunk set in the integrated relevant information chunk set A0 corresponding metadata information set;

[0111] S503. Use the large language model to detect the conflicting information in the integrated relevant information chunks, judge the reliability according to the information content and metadata information of the integrated relevant information chunks, filter out the errors and the knowledge irrelevant to the user's input question, and perform W iterations to obtain multiple information chunk groups. For longer contexts, more iteration times can be allocated to better detect the relevant information, as shown in formula (10):

[0112] <A w+1 ,S w+1 >=LLM(Q,<A0,S0>,<A w ,S w >,I it ) (10)

[0113] Among them, W represents the total number of iterations, and 0 ≤ w ≤ W. <A w+1 ,S w+1 > represents the set of integrated relevant information chunk and metadata information pairs obtained after the (w + 1)-th iteration, A w+1 represents the integrated relevant information chunk set obtained after the (w + 1)-th iteration, S w+1 represents the metadata information set obtained after the (w + 1)-th iteration, A w represents the integrated relevant information chunk set obtained after the w-th iteration, S w represents the metadata information set obtained after the w-th iteration, I it represents the hint instruction for integrating conflicting information to filter misleading information;

[0114] S504. Use the integrated relevant information block obtained after each (w + 1)-th iteration in step S503 and the set of metadata information pairs as the input to the large language model, and respectively obtain the corresponding answers for the integrated relevant information block and the set of metadata information pairs obtained after each (w + 1)-th iteration. Evaluate the reliability of the multiple generated answers, and use the most reliable answer as the final retrieval result, as shown in formula (11):

[0115] [ans w ,r w =LLM(Q,<A0,S0>,<A w ,S w >,I ans ) (11)

[0116] ans=ans w , if and only if r w =max(r0,…,r W )

[0117] where ans w represents the answer generated according to the integrated relevant information block set after the w-th iteration, r w represents the reliability score of the answer generated according to the integrated relevant information block set after the w-th iteration, ans represents the final answer, r W represents the reliability score of the answer generated according to the integrated relevant information block set after the W-th iteration, and I ans represents the prompt instruction used to generate the answer and the reliability score.

[0118] More preferably, in this embodiment, the reliability evaluation includes evaluating using multi-dimensional evaluation indicators, and the evaluation indicators of the dimension include consistency, relevance, and integrity.

[0119] S6. Send the user's input question and the final retrieval result obtained in step S5 to the large language model, and the large language model generates a corresponding answer.

[0120] This embodiment also provides a retrieval joint optimization system for a retrieval-enhanced generation system. Using the above retrieval joint optimization method for the retrieval-enhanced generation system, it includes:

[0121] An input-output module, used to obtain the user's input question and send the corresponding answer to the user;

[0122] A knowledge base and a vector library, used to store the knowledge text set in the required field and store semantic vector blocks, and calculate the similarity with the user's input question;

[0123] A retriever, which retrieves the knowledge base to obtain information related to the user's input, and obtains the final retrieval result through the retrieval joint optimization method;

[0124] A generator that uses a large language model to accept the questions input by the user and the final retrieval results obtained by the retriever to generate answers corresponding to the questions input by the user.

[0125] This embodiment also provides a computer device, including a storage medium, a processing unit, and a computer program persistently stored in the storage medium and executable on the processing unit. When the processing unit executes the computer program, it implements the retrieval optimization method in the first aspect as described above and the large language model retrieval enhanced generation system based on the retrieval optimization method in the second aspect as described above.

[0126] In order to improve the relevance, integrity, and self-consistency of the retrieval method of the existing retrieval enhanced generation system (RAG system), this method designs a novel RAG system retrieval joint optimization method and system, and optimizes the initial retrieval results based on the iterative reasoning of the large language model (LLM) and progressive information verification. In the present invention, it mainly aims at the isolated optimization problem of the retrieval information quality, and also pays attention to the fragmented distribution problem of implicit information. Optimizing retrieval information is a key issue in the RAG system, but isolated optimization in different dimensions will lead to the problem of lack of synergy, which has become a bottleneck in retrieval optimization. In the present invention, an iterative reasoning method is proposed to solve this problem. The LLM is used to extract relevant information in text chunks and vector chunks, and the chunks containing implicit information and direct information are converted into directly relevant information chunks containing only directly relevant information. Then, when optimizing for integrity and self-consistency, directly relevant information chunks are used for efficient reasoning, and the relevance, integrity, and self-consistency are jointly optimized through the directly relevant information chunks. For the problem of fragmented implicit information, this method uses progressive information verification based on directly relevant information chunks to enhance the ability to retrieve implicit information from two directions, which can significantly improve the ability to retrieve implicit information. Extensive experiments conducted on multiple different types of data sets such as single-hop question answering, multi-hop question answering, and domain question answering show the effectiveness of the method proposed in the present invention under different types of questions.

[0127] Embodiment 3, this embodiment provides a retrieval joint optimization method for a retrieval enhanced generation system, including the following steps:

[0128] S1. Pre-collect knowledge texts, pre-establish a knowledge base containing relevant knowledge, and segment the knowledge base documents into text chunks.

[0129] And obtain the questions input by the user;

[0130] Preferably, in this embodiment, specifically, it includes: obtaining knowledge base documents, data preparation: the knowledge base documents include two parts, an open-domain question-answering dataset and a multiple-choice question dataset. Among them, the open-domain dataset uses the NarrativeQA dataset and the HotpotQA dataset. The NarrativeQA dataset contains 1,567 complete stories from books and movie scripts. The dataset is divided into three non-overlapping parts: training, validation, and testing, with a total of 46,765 question-answer pairs, covering various types of documents and stories, and providing detailed metadata. The HotpotQA dataset contains 113,000 Wikipedia-based question-answer pairs. The question content in the HotpotQA dataset is relatively diverse, and the answers to the questions must be based on multiple supporting documents. The multiple-choice question dataset uses the QuALITY dataset, which was proposed by researchers at New York University in 2022. The QuALITY dataset is an innovative multiple-choice question dataset that contains English context paragraphs with an average length of about 5,000 tokens, far exceeding the processing range of most models.

[0131] Data preprocessing: For each document in the knowledge base documents, first identify the punctuation marks in the document and segment the document into a sequence of several sentences. Then, use the tokenizer of the LLaMA3.1-8B language model to convert each sentence into a token sequence. According to the length of the token sequence of the sentence, combine adjacent sentences into text blocks with a total token sequence length not exceeding 300. For a single sentence with a token sequence length exceeding 300, truncate it into multiple sentence blocks with a token sequence length not exceeding 300, and use them as text blocks to obtain a knowledge base composed of text blocks. Use the bge-m3 embedding model to vectorize the text blocks to generate vector blocks; use the database FAISS to build a vector library.

[0132] S2. Perform keyword matching and similarity comparison between the input question and the knowledge base and vector library of the large language model, and extract text blocks from the knowledge base according to the results of the similarity comparison and keyword matching to obtain an initial retrieval candidate set;

[0133] S3. Extract relevant information inside the initial retrieval candidate set and the large language model to generate an initial external relevant information block set and an internal relevant information block set;

[0134] S4. Perform progressive information verification on the initial external relevant information block set to obtain an external relevant information block set;

[0135] S5. Integrate the internal relevant information block set and the external relevant information block set, remove conflicting information, and generate a final retrieval result;

[0136] S6. Send the user's input question and the final retrieval result to the large language model, and the large language model generates a corresponding answer.

[0137] The embodiments of the present invention are given for purposes of illustration and description, and are not exhaustive or limit the invention to the disclosed forms. Many modifications and variations are obvious to those of ordinary skill in the art. The embodiments are chosen and described in order to best explain the principles of the invention and its practical application, and to enable those of ordinary skill in the art to understand the invention and design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A retrieval joint optimization method for a retrieval enhancement generation system, characterized in that: The steps include: S1. Build a knowledge base and vector library of a large language model to obtain user input questions; S2. Perform keyword matching and similarity comparison on the input question obtained in step S1 with the knowledge base and vector base of the large language model, and extract text blocks from the knowledge base according to the results of the similarity comparison and keyword matching to obtain an initial retrieval candidate set; S3. extracting the relevant information of the initial search candidate set obtained in step S2 and the internal large language model to generate an initial external related information block set and an internal related information block set; S4. Perform progressive information verification on the initial external related information block set obtained in step S3 to obtain an external related information block set; S5. Integrate the internal related information block set obtained in step S3 and the external related information block set obtained in step S4, remove conflicting information, and generate a final search result; S6. Send the user's input question and the final search result obtained in step S5 to the large language model, and the large language model generates a corresponding answer.

2. The retrieval joint optimization method for a retrieval enhancement generation system according to claim 1, characterized in that: The construction of the knowledge base and vector library of the large language model in step S1 includes: obtaining knowledge base documents, dividing the knowledge base documents into text blocks, and obtaining a knowledge base composed of text blocks; using an embedding model to vectorize the text blocks, generating vector blocks, and obtaining a vector library composed of vector blocks.

3. The retrieval joint optimization method for a retrieval enhancement generation system according to claim 2, characterized in that: The embedding model is bge-m3, and the database FAISS is used to build the vector library.

4. The retrieval joint optimization method for a retrieval enhancement generation system according to claim 2, characterized in that: The step S2 comprises the following steps: S201. Calculate the similarity between the user's input question and each text block in the knowledge base D by using the BM25 algorithm to obtain a keyword matching result, as shown in formula (1): Among them, r BM25 represents the similarity between the user's input question and the text block in the knowledge base, d∈D, d represents the text block, Q represents the question, Q={q0,q1,…,q M }, M represents the number of words in the question, q i represents the i-th term in the question, i∈(1,2,3,…,M), f(q i ,d) represents the term q i The frequency of words in text block d, len(d) represents the number of words in text block d, avg_len represents the average number of words in all text blocks in the knowledge base, k1 represents the parameter for standardizing the frequency range of text blocks, b represents the parameter for adjusting the length and information content of the article, and IDF(q i ) indicates q i The inverse document frequency in the knowledge base D is shown in formula (2): Where N represents the number of text blocks in the knowledge base, n(q i ) means that it contains the term q i The number of text blocks; S202. Calculate the semantic vector of the user's input question, and compare the semantic vector of the user's input question with each vector block in the vector library for similarity to obtain a similarity comparison result, as shown in formula (3): Among them, it represents r vs The vector similarity between the user's input question and the vector block in the knowledge base, v q The semantic vector representing the user’s input question, v q = {V q1 ,V q2 ,…,V qT }, T represents the length of the semantic vector of the user input question, q∈Q, V qt represents the eigenvalue of the user's input question in the tth dimension, v d The vector representing the text block d, v d = {V d1 ,V d2 ,…,V dt },V dt Represents the feature value of text block d on the tth dimension; S203. The keyword matching result and the similarity comparison result are combined by reciprocal rank fusion to obtain a fused similarity result, as shown in formula (4): Among them, r RRF represents the reciprocal rank fusion operation, R represents the sorting index, R = {r BM25 ,r vs }, r∈R, c represents a ranking smoothing factor, which is used to reduce the impact of outlier elements on the ranking; S204. Sort the text blocks in the text library based on the fusion similarity results, and extract the text blocks corresponding to the vector blocks with the highest similarity in the first k fusion similarity results as the initial retrieval candidate set.

5. The retrieval joint optimization method for a retrieval enhancement generation system according to claim 4, characterized in that: The parameter k1 of the standardized text block word frequency range is set to 1.5, the parameter b for adjusting the article length and information content is set to 0.75, and the ranking smoothing factor is set to a fixed value of 60.

6. The retrieval joint optimization method for a retrieval enhancement generation system according to claim 1, characterized in that: The step S3 comprises the following steps: S301. Use a large language model to extract the initial search candidate set E = [e0, e1, ..., e n ] related information, generate the initial external related information block set E′=[e′0,e′1,…,e′ n ], as shown in formula (5): Among them, e′ j ∈E′,e j ∈E,e′ j represents the jth initial external related information block, e j represents the jth initial retrieval result, and I represents the information related to the user's input question in the internal knowledge of the large language model; S302. Obtain relevant information blocks of internal knowledge from the large language model and generate n I Related information blocks are obtained to obtain the internal related information block set, as shown in formula (6): Where I represents the internal related information block set, n I ≤N I , LLM stands for Large Language Model, Indicates the nth I Internal related information blocks, N I Indicates that no more than N I A block of internal related information.

7. The retrieval joint optimization method for a retrieval enhancement generation system according to claim 6, characterized in that: The step S4 comprises the following steps: S401. Perform initial integrity verification on each of the initial external related information blocks in the initial external related information block set in both the front and rear directions, respectively splice each of the initial external related information blocks with the text blocks immediately before and after the corresponding text block, and obtain the front extended text block and the rear extended text block of each of the initial external related information blocks, use the large language model to detect whether the front extended text block and the rear extended text block of each of the initial external related information blocks introduce the related information blocks of the new information, and expand the related information blocks of the new information into the corresponding initial external related information blocks, and obtain the extended external related information blocks, as shown in formula (7): in, It represents the jth expanded external related information block, represents the jth related information block that introduces new information through the previous expanded text block, represents the jth related information block that introduces new information through the post-expanded text block, represents the jth related information block that introduces new information through the front expansion text block and the back expansion text block, 0 <j<n; S402. For the related information blocks of the introduced new information, use the large language model to iteratively perform integrity verification in the introduction direction and append the related information blocks of the new new information until no new related information blocks of the new information are introduced, and obtain the external related information block set 8. The retrieval joint optimization method for a retrieval enhancement generation system according to claim 7, characterized in that: The step S5 comprises the following steps: S501. Merge the internal related information block set and the external related information block set to obtain an integrated related information block set, as shown in formula (8): Among them, A0 represents the set of integrated related information blocks; S502. Extract the metadata information set of the integrated related information block set, as shown in formula (9): Among them, S0 represents the metadata information set corresponding to the integrated related information block set A0, S I represents the metadata information set corresponding to the internal related information block set I in the integrated related information block set A0, a represents a related information block in the integrated related information block set A0, Indicates the external related information block set in the integrated related information block set A0 The corresponding metadata information set; S503. Use the large language model to detect conflicting information in the integrated related information block, determine the reliability according to the information content of the integrated related information block and the metadata information, filter out errors and knowledge irrelevant to the user's input question, and perform W iterations to obtain multiple information block groups, as shown in formula (10): <A w+1 ,S w+1 >=LLM(Q,<A0,S0>,<A w ,S w >,I it ) (10) Where W represents the total number of iterations, and 0≤w≤W, w+1 ,S w+1 > represents the set of integrated related information blocks and metadata information pairs obtained after the w+1th round of iteration, A w+1 represents the integrated related information block set obtained after the w+1th round of iteration, S w+1 represents the metadata information set obtained after the w+1th round of iteration, A w represents the set of integrated related information blocks obtained after the wth iteration, S w represents the metadata information set obtained after the wth iteration, I it Indicates prompt instructions for integrating conflicting information and filtering misleading information;​ S504. The set of integrated related information blocks and metadata information pairs obtained after each of the w-th iterations obtained in step S503 is used as the input of the large language model, and the answers corresponding to the sets of integrated related information blocks and metadata information pairs obtained after the w+1-th iterations are obtained respectively. The reliability of the generated multiple answers is evaluated, and the most reliable answer is taken as the final retrieval result, as shown in formula (11): [ans w ,r w ]=LLM(Q,<A0,S0> , w ,S w >,I ans ) (11)ans=ans w , if and only if r w =max(r0,…,r W )​ Among them, ans w represents the answer generated based on the integrated related information block set after the wth iteration, r w represents the reliability score of the answer generated based on the integrated related information block set after the wth iteration, ans represents the final answer, and r W represents the reliability score of the answer generated based on the integrated related information block set after the Wth round of iterations, I ans Represents the prompt instructions used to generate answers and reliability scores.

9. The retrieval joint optimization method for a retrieval enhancement generation system according to claim 8, characterized in that: The reliability evaluation includes using multi-dimensional evaluation indicators for evaluation, and the dimensional evaluation indicators include consistency, relevance and integrity.

10. A retrieval joint optimization system for a retrieval enhancement generation system, characterized in that: The retrieval joint optimization method for the retrieval enhancement generation system according to any one of claims 1 to 9 comprises: Input and output module, used to obtain user input questions and send corresponding answers to users; Knowledge base and vector library, used to store knowledge text sets in the required fields and semantic vector blocks, and calculate the similarity with user input questions; The retriever searches the knowledge base to obtain information related to the user input, and obtains the final search results through the search joint optimization method; The generator uses a large language model to accept the questions input by the user and the final search results obtained by the retriever, and generates the answers corresponding to the user's input questions.

Images