Document question and answer method, system and equipment based on RAG and medium
By dynamically adjusting the answer generation plan and selecting the appropriate processing method according to the complexity of the question, the existing question-and-answer system's waste of resources and insufficient accuracy in answering complex questions is solved, and more efficient and accurate question-and-answer processing is achieved.
Patent Information
- Application Number
- CN202411079096.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
AI Technical Summary
Existing question-and-answer systems based on large language models and RAGs waste too much resources when processing simple queries, and insufficient answer accuracy for complex questions, resulting in poor customer experience and degraded answer quality.
By dynamically adjusting the answer generation plan, using preset classifiers to evaluate the complexity of the problem, selecting the appropriate answer generation plan, including directly using a large language model or retrieving external knowledge, combining a large language model to generate answers, and iteratively processing on complex problems Until accuracy requirements are met.
It simplifies the process of handling simple questions, saves computing resources, improves the accuracy of answering complex questions, and improves the overall efficiency and accuracy of the question-and-answer system.
Smart Images

Figure CN119003725A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent question answering technology, and in particular to a RAG-based document question answering method, system, device and medium. Background Art
[0002] Existing document retrieval methods slice documents and store them in a vector database, obtain user questions and match them with documents in the knowledge base, then use a large inference model or a large question conversion model to generate answers, and add questions and answers to the historical conversation set. Existing methods include:
[0003] (1) Obtain the answer to the question and a certain number of recalled documents, guide the large question conversion model to effectively convert the current question through the preset prompt, and generate the answer based on the question and recalled documents by the large language model. If the answer meets the preset question and answer output conditions, the answer is sliced to generate answer slices, and the answer slices are traced back through the recalled documents to determine the traceability results of the answer slices;
[0004] (2) By retrieving document data matching the user's question in the knowledge base, the system generates answers using the Retrieval-Augmented Generation (RAG) reasoning model or the question conversion model, and analyzes the relevance of the current question and historical conversations in combination with the historical conversation set to generate more accurate answers. When processing the first question, the system directly retrieves and generates answers. When processing the second and subsequent questions, it first analyzes the matching relationship with the historical conversations, then performs question preprocessing based on the analysis results, and then generates answers.
[0005] The existing question-answering system based on large language models and RAG does not consider the classification of questions. When processing simple query questions, it also uses RAG, which causes unnecessary overhead and affects customer experience. For very complex questions, it may sometimes be unable to provide sufficient query processing capabilities, resulting in a decrease in the accuracy of the answer. Summary of the invention
[0006] In view of this, the present invention provides a RAG-based document question-and-answer method, system, device and medium to solve the problem of waste of resources in the processing of simple questions and low accuracy of answers to complex questions.
[0007] In a first aspect, the present invention provides a document question-answering method based on RAG, the method comprising:
[0008] Obtain the target question queried by the user, and use a preset classifier to evaluate the complexity of the target question to obtain a complexity evaluation result of the target question;
[0009] An answer generation scheme for the target question is determined based on a complexity evaluation result of the target question. The answer generation scheme includes directly generating an answer using a preset large language model, or using a preset retriever to retrieve preset documents to obtain external knowledge related to the target question, and then generating an answer using a preset large language model in combination with the external knowledge.
[0010] The RAG-based document question-and-answer method provided by the present invention optimizes question-and-answer responses to different types of questions by dynamically adjusting the answer generation scheme, thereby simplifying the processing process of simple questions, saving computing resources, optimizing the question-and-answer process, and improving the accuracy of answers to complex questions.
[0011] In an optional implementation, the training process of the preset classifier includes:
[0012] Obtain multiple questions queried by the user, and answer each question through multiple preset answer generation schemes to obtain multiple answers, each preset answer generation scheme having a different complexity;
[0013] Compare multiple answers with a preset standard answer to determine the similarity between each answer and the preset standard answer;
[0014] Compare the similarity between each answer and the preset standard answer, and take the complexity of the preset answer generation solution corresponding to the answer with the greatest similarity as the complexity of the corresponding question, and the complexity from low to high includes the first complexity, the second complexity, and the third complexity;
[0015] The preset classification model is trained using each problem and its corresponding complexity to obtain a preset classifier.
[0016] The RAG-based document question-answering method provided by the present invention utilizes answer generation schemes with different complexities to answer target questions, determines the complexity of the question by comparing the answers, and then trains a preset classification model based on multiple questions of different complexities to obtain a preset classifier to achieve classification of different questions. Different types of questions use different preset answer generation schemes, which not only avoids wasting resources for simple questions, but also achieves the purpose of improving the accuracy of answers to complex questions.
[0017] In an optional implementation, determining a solution for generating an answer to the target question according to a complexity evaluation result of the target question includes:
[0018] If the complexity evaluation result of the target question is the first complexity, the target question is input into the preset large language model to generate an answer;
[0019] If the complexity evaluation result of the target question is the second complexity, the preset retriever is used to obtain external knowledge, and the external knowledge and the target question are input into the preset large language model to generate an answer;
[0020] If the complexity evaluation result of the target question is the third complexity, the process of generating answers using a preset retriever and a preset large language model will be iterated until the generated answer meets the preset standard evaluation requirements.
[0021] The RAG-based document question-answering method provided by the present invention automatically evaluates the complexity of each question and adopts different answer generation schemes for questions of different complexity, thereby reducing the computing power burden and unnecessary labeling overhead and improving the accuracy of question-answering.
[0022] In an optional implementation, inputting the target question into a preset large language model to generate an answer includes:
[0023] Determine a first tag sequence according to a target problem having a first complexity;
[0024] Inputting the first token sequence into a preset large language model and outputting a second token sequence;
[0025] A first answer to the target question having a first complexity is determined according to the second tag sequence.
[0026] The RAG-based document question-answering method provided by the present invention directly uses a preset large language model to answer target questions of the first complexity, automatically interacts with user questions, quickly locates the answers corresponding to the target questions, and summarizes the answers, which greatly reduces the workload of manual reading and querying and improves the efficiency of question-answering.
[0027] In an optional implementation, a preset retriever is used to obtain external knowledge, and the external knowledge and the target question are input into a preset large language model to generate an answer, including:
[0028] Obtain multiple documents and convert each document into text data;
[0029] Vectorize the text data to obtain vector data, and store the vector data in a vector database;
[0030] According to the target problem with the second complexity, a preset searcher is used to search the vector database to obtain external knowledge;
[0031] The external knowledge and the target question with the second complexity are input into a preset large language model to generate a second answer to the target question with the second complexity.
[0032] The RAG-based document question-answering method provided by the present invention, for target questions of the second complexity, combines the external knowledge retrieved by the retriever to make document information extraction more precise and detailed, thereby obtaining content related to the target question more comprehensively and accurately, thereby improving the accuracy of document query.
[0033] In an optional implementation, the process of generating an answer using a preset retriever and a preset large language model is iterated until the generated answer meets the preset requirements, including:
[0034] According to the target problem with the third complexity, a preset searcher is used to search the vector database to obtain external knowledge;
[0035] Input the external knowledge and the target question with the third complexity into the preset large language model to obtain the third answer;
[0036] Determine whether the third answer meets the preset accuracy evaluation requirement. If not, return to the step of searching the vector database using the preset retriever until the generated third answer meets the preset accuracy evaluation requirement.
[0037] The RAG-based document question-answering method provided by the present invention uses a cyclic iterative approach to answer target questions of the third complexity until the generated third answer meets the preset accuracy evaluation requirements, thereby ensuring the accuracy of answering the target question and improving the question-answering quality.
[0038] In a second aspect, the present invention provides a document question-answering system based on RAG, the system comprising:
[0039] The complexity evaluation module is used to obtain the target question queried by the user and evaluate the complexity of the target question using a preset classifier to obtain a complexity evaluation result of the target question;
[0040] The answer generation scheme determination model is used to determine the answer generation scheme for the target question based on the complexity evaluation result of the target question. The answer generation scheme includes directly generating the answer using a preset large language model, or using a preset retriever to retrieve preset documents to obtain external knowledge related to the target question, and generating the answer using the external knowledge and the preset large language model.
[0041] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0042] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to execute the method of the first aspect or any corresponding embodiment thereof.
[0043] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0045] Figure 1 is a flowchart of a RAG-based document question-answering method according to an embodiment of the present invention;
[0046] Figure 2 is a flowchart of another RAG-based document question-answering method according to an embodiment of the present invention;
[0047] Figure 3 is a schematic diagram of the overall process of the RAG-based document question-answering method according to an embodiment of the present invention;
[0048] Figure 4 is a structural block diagram of a RAG-based document question-answering system according to an embodiment of the present invention;
[0049] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0051] The embodiment of the present invention provides a RAG-based document question-answering method, which optimizes the question-answering responses to different types of questions by dynamically adjusting the answer generation scheme, so as to save computing resources and improve the accuracy of answers to complex questions.
[0052] According to an embodiment of the present invention, a RAG-based document question and answer method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0053] In this embodiment, a document question-answering method based on RAG is provided, which can be used in the above-mentioned computer system. Figure 1 is a flowchart of a RAG-based document question-answering method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:
[0054] Step S101, obtaining a target question queried by a user, and evaluating the complexity of the target question using a preset classifier to obtain a complexity evaluation result of the target question.
[0055] Specifically, the RAG-based document question-and-answer method provided in this embodiment can be widely used in multiple industries, and the application scenarios include but are not limited to internal enterprise knowledge management. Enterprises can use the method provided in this embodiment to build an internal enterprise knowledge base to help employees quickly find the information they need and improve work efficiency.
[0056] The user can input questions through the human-computer interaction interface. The input form of the target question can be voice, text, etc., which is not limited here. The preset classifier is a pre-trained small language model that can automatically evaluate the complexity of the target question and generate the complexity evaluation result of the target question. The classification level and number of levels of complexity can be set as needed, for example, divided into five levels from the first complexity to the fifth complexity, which is only for example, but not limited to this.
[0057] Step S102, determining an answer generation scheme for the target question based on the complexity evaluation result of the target question, the answer generation scheme includes directly generating an answer using a preset large language model, or using a preset retriever to retrieve preset documents to obtain external knowledge related to the target question, and generating an answer using a preset large language model in combination with the external knowledge.
[0058] Specifically, the preset large language model can be any existing mature large language model (Large Language Model, LLM), or it can be a preset large language model obtained by training with relevant documents, data, etc. in combination with actual application scenarios, industries, etc. The training process of the model is a relatively mature technology and will not be repeated here. The most appropriate answer generation scheme is determined according to the complexity evaluation result of the target question, and the answer is generated using the most appropriate answer generation scheme. For example, for simple questions, the preset large language model is directly used to generate answers; for complex questions, the preset retriever is used to retrieve preset documents to obtain external knowledge related to the target question, and the preset large language model is used in combination with the external knowledge to generate answers. Specifically, the database formed by the preset retriever is retrieved by the preset document, and the document fragments related to the target question are obtained through indexing and querying. These document fragments are used as context information for LLM to generate answers; for more complex questions, the RAG method can be adopted, that is, the step of retrieving the preset document by the preset retriever is iterated, multiple retrievals, multiple answers are generated, and each time relevant external knowledge is obtained, and after combining with the target question, the preset large language model is used to generate answers.
[0059] It should be noted that each time an answer is generated, the answer needs to be evaluated. The RAG-based document question answering method provided in this embodiment uses document recall rate and F1 value as fitness criteria to evaluate the quality of the generated answers. The evaluation criteria are only used as examples, but are not limited to this.
[0060] The RAG-based document question-and-answer method provided in this embodiment optimizes the question-and-answer responses to different types of questions by dynamically adjusting the answer generation scheme, thereby simplifying the processing of simple questions, saving computing resources, optimizing the question-and-answer process, and improving the accuracy of answers to complex questions.
[0061] In this embodiment, a document question-answering method based on RAG is provided, which can be used in the above-mentioned computer system. Figure 2 is a flowchart of a RAG-based document question-answering method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:
[0062] Step S201, obtaining a target question queried by a user, and evaluating the complexity of the target question using a preset classifier to obtain a complexity evaluation result of the target question.
[0063] In some optional implementations, the training process of the preset classifier in the above step S201 includes:
[0064] Step S2011, obtaining multiple questions queried by the user, and answering each question through multiple preset answer generation schemes to obtain multiple answers, each preset answer generation scheme having a different complexity.
[0065] Specifically, in this embodiment, the Llamma2-7B model is used as a classifier model, that is, a classifier model, and the Llamma2-7B model is trained to obtain a preset classifier, and the preset classifier is used to evaluate the complexity of the questions raised by the user.
[0066] For each question queried by the user, the multiple preset answer generation schemes mentioned in the previous embodiment are used to answer each question and obtain corresponding multiple answers. It should be noted that the multiple preset answer generation schemes have different complexities. For example, directly using the preset large language model to generate answers is mainly aimed at simple questions, so the complexity of the preset answer generation scheme can be defined as simple; using the preset retriever to retrieve preset documents to obtain external knowledge related to the target question, and combining the external knowledge with the preset large language model to generate answers is mainly aimed at complex questions, so the complexity of the preset answer generation scheme can be defined as complex; the RAG method is mainly aimed at more complex problems, so the complexity of the preset answer generation scheme can be defined as very complex.
[0067] Step S2012, compare the multiple answers with the preset standard answer to determine the similarity between each answer and the preset standard answer.
[0068] Specifically, a preset standard answer corresponding to each question can be manually set, and then multiple answers to each question can be compared with the preset standard answer by means of a pre-trained encoder to obtain the similarity between each answer and the preset standard answer. First, the answer is encoded, and a pre-trained text encoder (such as BERT) is used to convert the preset standard answer and multiple generated answers into vector representations;
[0069] The similarity between the standard answer vector and each generated answer vector is calculated using methods such as cosine similarity.
[0070]
[0071] A similarity threshold may also be determined. An answer having a similarity threshold higher than the threshold is considered similar to the standard answer and is valid; otherwise, the answer is considered invalid.
[0072] In a specific embodiment, the question asked by the user is: What is machine learning?
[0073] Default answer: Machine learning is a branch of artificial intelligence that uses statistical techniques to enable computer systems to "learn" (i.e., gradually improve their performance at specific tasks) without being explicitly programmed.
[0074] The various answers obtained using different answer generation schemes are:
[0075] Answer 1 (easy): Machine learning is a technique that allows computers to learn automatically.
[0076] Answer 2 (Complex): Machine learning is a method that allows computer systems to automatically improve their performance based on data and algorithms without human intervention.
[0077] Answer 3 (very complex): Machine learning is an important branch of artificial intelligence that uses statistical methods to enable computer systems to learn patterns from data and improve their performance on specific tasks without explicit programming instructions. This involves multiple steps such as data preprocessing, feature extraction, model training and evaluation.
[0078] The answer is vectorized using the BERT encoder, and then the cosine similarity with the standard answer is calculated using the above formula (1), resulting in: similarity of answer 1: 0.75, similarity of answer 2: 0.85, similarity of answer 3: 0.95.
[0079] Assuming that the similarity threshold is set to 0.8, answers 2 and 3 are considered similar to the standard answer, while answer 1 is not similar enough.
[0080] Step S2013, compare the similarity between each answer and the preset standard answer, and take the complexity of the preset answer generation solution corresponding to the answer with the greatest similarity as the complexity of the corresponding question, and the complexity from low to high includes the first complexity, the second complexity, and the third complexity.
[0081] Specifically, exemplarily, for a certain question, the answer obtained by using a preset answer generation scheme with a simple complexity has a similarity of 95% with the preset standard answer, the answer obtained by using a preset answer generation scheme with a complex complexity has a similarity of 70% with the preset standard answer, and the answer obtained by using a preset answer generation scheme with a very complex complexity has a similarity of 50% with the preset standard answer, then the complexity of the question is simple, which is only taken as an example, but is not limited to this. For the convenience of recording, the complexity can be marked with 0, 1, and 2 from low to high, respectively, the first complexity is marked as 0, the second complexity is marked as 1, and the third complexity is marked as 2, and the complexity of the question and the preset answer generation scheme is marked accordingly. If the answers obtained by the preset answer generation schemes of different complexities have the same similarity with the preset standard answer, the complexity of the question is set to a relatively low complexity, for example, the answer obtained by using a preset answer generation scheme with a simple complexity has the same similarity with the preset standard answer, and the answer obtained by using a preset answer generation scheme with a complex complexity has the same similarity with the preset standard answer, then the complexity of the answer is simple, which can reduce the calculation time and unnecessary resource consumption.
[0082] Step S2014, using each question and its corresponding complexity to train the preset classification model to obtain a preset classifier.
[0083] Specifically, the labeled multiple questions and the corresponding complexity are used as sample data to train the preset classification model to obtain a preset classifier, and the cross entropy loss function is used to predict the complexity of the target question. The preset classification model can be the Llamma2-7B model, which is only used as an example, but not limited to this. The training process of the model is a relatively mature technology and will not be repeated here.
[0084] The preset classifier formula can be expressed as:
[0085] o=Classifier(q)
[0086] Among them, o represents the complexity level and q represents the question. Once the model is trained, it can be used to classify the complexity of the question. The correctly classified question and answer pairs in the whole process will be recorded and added to the training set of the sample data to enhance the sample data, so as to facilitate the future training of other classifier models.
[0087] The RAG-based document question-answering method provided in this embodiment uses answer generation schemes with different complexities to answer target questions, determines the complexity of the question by comparing the answers, and then trains the preset classification model according to multiple questions of different complexities to obtain a preset classifier to achieve classification of different questions. Different types of questions use different preset answer generation schemes, which not only avoids wasting resources for simple questions, but also achieves the purpose of improving the accuracy of answers to complex questions.
[0088] Step S202, determining an answer generation scheme for the target question based on the complexity evaluation result of the target question, the answer generation scheme includes directly generating an answer using a preset large language model, or using a preset retriever to retrieve preset documents to obtain external knowledge related to the target question, and generating an answer using a preset large language model in combination with the external knowledge.
[0089] Specifically, the above step S202 includes:
[0090] Step S2021: If the complexity evaluation result of the target question is the first complexity, the target question is input into a preset large language model to generate an answer.
[0091] In some optional implementations, the above step S2021 includes:
[0092] Step a1: determining a first tag sequence according to a target problem with a first complexity.
[0093] Specifically, for simple questions, LLM is used to generate answers directly without the need to retrieve external knowledge. The first tag sequence is x = [x1, x2, ..., x n ], define an LLM, accept a token sequence x = [x1, x2, ..., x n ] as input, generate a token sequence y = [y1, y2, ..., y n ] as the output, where x and y correspond to the target question queried by the user and the answer of LLM respectively.
[0094] Step a2: input the first tag sequence into a preset large language model and output a second tag sequence.
[0095] Specifically, the second tag sequence is y=[y1,y2,…,y n ].
[0096] Step a3: determining a first answer to the target question having a first complexity according to the second tag sequence.
[0097] Specifically, the process of generating the first answer using the preset large language model is expressed as follows: in, represents the first answer, and q1 represents the target problem of the first complexity. For simple question q, the answer It should match the correct answer a, indicating that the quality of the answer is good enough. However, if it is a complex question, a question of the second or third complexity, the answer obtained by this method is not accurate enough.
[0098] like Figure 3 As shown, for a target question with a complexity of the first complexity, the preset large language model is directly input to obtain a first answer, and then the accuracy of the first answer is evaluated. Since the question is relatively simple, the first answer obtained by directly using the preset large language model meets the preset accuracy evaluation requirements, and the first answer is directly output. It should be noted that since the complexity of the target question has been determined, the accuracy evaluation of the first answer can be omitted here and it can be directly output, which is only used as an example, but is not limited to this.
[0099] The RAG-based document question-answering method provided in this embodiment directly uses a preset large language model to answer target questions of the first complexity, automatically interacts with user questions, quickly locates the answers corresponding to the target questions, and summarizes the answers, which greatly reduces the workload of manual reading and querying and improves the efficiency of question-answering.
[0100] Step S2022: If the complexity evaluation result of the target question is the second complexity, a preset retriever is used to obtain external knowledge, and the external knowledge and the target question are input into a preset large language model to generate an answer.
[0101] In some optional implementations, the above step S2022 includes:
[0102] Step b1, obtaining multiple documents and converting each document into text data.
[0103] Specifically, the document may be a key document determined according to the query scenario, or an industry-related document of the corresponding industry. The document formats include but are not limited to .txt, .pdf, .md, and .doc. These documents are generally complete documents and cannot be used directly, so they need to be parsed. Document segmentation is a common method, especially in RAG, to avoid files that are too long to be processed. However, in the process of segmenting the document, how to choose the size of the blocks will directly affect the segmentation results. In this embodiment, the document is segmented according to the page boundaries to maintain the integrity of the table information as much as possible, without losing the context, thereby improving the accuracy of retrieval in RAG.
[0104] Step b2, vectorizing the text data to obtain vector data, and storing the vector data in a vector database.
[0105] Specifically, a parser is used to identify the structure of a document, and documents of different formats are converted into text data. The text data is then stored in a vectorized form in a vector database to facilitate subsequent retrieval and processing as external knowledge.
[0106] Step b3, based on the target problem with the second complexity, use a preset retriever to search the vector database to obtain external knowledge.
[0107] Specifically, a target problem with the second complexity requires certain external knowledge support. According to the target problem, a preset retriever is used to search the vector database to obtain external knowledge related to the target problem. The formula is expressed as: d = Retriever (q; D), where d represents the external knowledge related to the target problem, q represents the target problem, and D represents the external knowledge base, that is, the vector database.
[0108] Step b4: input the external knowledge and the target question of the second complexity into a preset large language model to generate a second answer to the target question of the second complexity.
[0109] Specifically, the external knowledge related to the target question is input into the preset large language model together with the target question to generate the second answer to the target question of the second complexity, which is expressed as: in, represents the second answer, and q2 represents the target problem of the second complexity.
[0110] like Figure 3 As shown, for a target question with a complexity of the second complexity, only a single round of retrieval is required. The external knowledge related to the target question obtained after the single round of retrieval is input into the preset large language model together with the target question to obtain a second answer. When the accuracy of the second answer is evaluated, it meets the preset accuracy evaluation requirements and is directly output. It should be noted that since the complexity of the target question has been determined, the accuracy evaluation of the second answer can be omitted here and it can be directly output, which is only an example, but not limited to this.
[0111] The RAG-based document question-answering method provided in this embodiment, for target questions of the second complexity, combines the external knowledge retrieved by the retriever to make document information extraction more precise and detailed, thereby obtaining content related to the target question more comprehensively and accurately, thereby improving the accuracy of document queries.
[0112] Step S2023, if the complexity evaluation result of the target question is the third complexity, the process of generating an answer using a preset retriever and a preset large language model is iterated until the generated answer meets the preset standard evaluation requirements.
[0113] In some optional implementations, the above step S2023 includes:
[0114] Step c1, based on the target problem with the third complexity, use a preset searcher to search the vector database to obtain external knowledge.
[0115] Specifically, for very complex questions, it is necessary to iteratively use a preset retriever to retrieve external knowledge and use a preset large language model to generate answers, and gradually build a complete answer through multiple rounds of interaction. The method of obtaining external knowledge is the same as in the above step S2022, and will not be repeated here.
[0116] Step c2: input the external knowledge and the target question with the third complexity into the preset large language model to obtain a third answer.
[0117] Specifically, the retrieved external knowledge related to the target question is input into the preset large language model together with the target question to obtain the third answer. This process is the same as that in step S2022 and will not be repeated here.
[0118] Step c3, determining whether the third answer meets the preset accuracy assessment requirement. If not, returning to the step of searching the vector database using the preset retriever until the generated third answer meets the preset accuracy assessment requirement.
[0119] Specifically, by acquiring external knowledge related to the target question, and generating answers based on the external knowledge and the target question using a preset large language model, in each retrieval process, new retrieval content is extracted from the vector database to form new external knowledge, and then the new external knowledge and the target question are used to generate answers, and the target question and each round of answers are added to the historical dialogue set. The formula is expressed as follows: in, represents the answer to the retrieval answer in round i, q3 represents the target question of the third complexity, c i represents the context information of the first i-1 rounds of dialogue records, d i represents the external knowledge related to the target problem obtained in the i-th round of retrieval, D represents a vector database. It should be noted that the historical dialogue set is a dialogue set consisting of multiple rounds of dialogue questions and answers of users.
[0120] In this embodiment, the preset accuracy assessment requires the use of three indicators to measure effectiveness:
[0121] (1) The F1 score is the harmonic mean of precision and recall, which is used to measure the overlap between the predicted answer and the true answer:
[0122]
[0123] (2) Exact Match (EM), which measures whether the predicted answer and the true answer are exactly the same. If the predicted answer and the true answer are exactly the same, EM is 1, and if they are different, it is 0.
[0124] (3) Accuracy, which measures the extent to which the true answer is included in the predicted answer. Even if the predicted answer is longer or contains more information than the true answer, as long as the true answer is part of the predicted answer, Acc is 1.
[0125]
[0126] Among them, T P is the number of true positives, that is, the number of answers predicted correctly and true; F N is the number of false negatives, i.e., the number of true answers that were not predicted as the correct answer.
[0127] When all the above three indicators meet the preset requirements, it means that the third answer meets the preset accuracy assessment requirements.
[0128] In a specific embodiment, the user's target question is: What is the greenhouse effect?
[0129] The default standard answer: The greenhouse effect is a phenomenon in which certain gases in the atmosphere (such as carbon dioxide, methane, etc.) absorb and re-radiate the heat emitted by the earth's surface, thereby raising the temperature of the earth's surface. This process is similar to greenhouse insulation, so it is called the greenhouse effect.
[0130] The answers obtained using different answer generation schemes are:
[0131] Answer 1 (simple): The greenhouse effect is a natural phenomenon that causes the earth to warm.
[0132] Answer 2 (Complex): The greenhouse effect is the process by which certain gases in the atmosphere trap the Earth's heat, causing the Earth's surface temperature to rise. The main greenhouse gases include carbon dioxide and methane.
[0133] Answer 3 (very complicated): The greenhouse effect is a natural phenomenon, which refers to the process in which certain gases in the atmosphere (mainly carbon dioxide, methane, water vapor, etc.) absorb and reradiate long-wave radiation emitted by the earth's surface, thereby increasing the temperature of the atmosphere and the earth's surface. This effect keeps the average temperature of the earth at a level suitable for life, but the increase in greenhouse gases caused by human activities is exacerbating this effect and causing global climate change.
[0134] Evaluate the three responses above:
[0135] (1) F1 score: Since the F1 score requires accurate vocabulary matching, it can be roughly estimated:
[0136] Answer 1: F1≈0.3, Answer 2: F1≈0.6, Answer 3: F1≈0.8;
[0137] Exact match (EM): No answer matches the standard answer exactly, so the EM of all three answers is 0;
[0138] Accuracy (Acc): Answer 1: Acc = 1 (contains the core concepts of the standard answer), Answer 2: Acc = 1 (contains the main content of the standard answer), Answer 3: Acc = 1 (not only contains all the contents of the standard answer, but also provides additional information).
[0139] Assuming the evaluation criteria are: F1>0.7, EM>0 (not achievable in this example), Acc>0.9, according to this criterion, only answer 3 meets all the requirements, indicating that more complex and comprehensive answers tend to better meet the accuracy evaluation requirements.
[0140] The RAG-based document question-answering method provided in this embodiment uses a loop iteration method to answer the target question of the third complexity until the generated third answer meets the preset accuracy evaluation requirements, thereby ensuring the accuracy of answering the target question and improving the quality of question-answering. The complexity of each question is automatically evaluated, and different answer generation schemes are used for questions of different complexity, which reduces the computing power burden and unnecessary tagging overhead and improves the accuracy of question-answering.
[0141] In this embodiment, a document question-and-answer system based on RAG is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0142] This embodiment provides a document question-answering system based on RAG, such as Figure 4 As shown, including:
[0143] The complexity evaluation module 401 is used to obtain the target question queried by the user, and evaluate the complexity of the target question using a preset classifier to obtain a complexity evaluation result of the target question.
[0144] The answer generation scheme determination model 402 is used to determine the answer generation scheme for the target question based on the complexity evaluation result of the target question. The answer generation scheme includes directly generating the answer using a preset large language model, or using a preset retriever to retrieve preset documents to obtain external knowledge related to the target question, and generating the answer using the external knowledge and the preset large language model.
[0145] In some optional implementations, the complexity evaluation module 401 includes:
[0146] Multiple question-answering units are used to obtain multiple questions queried by users, and to answer each question through multiple preset answer generation schemes to obtain multiple answers, each preset answer generation scheme having a different complexity.
[0147] The similarity determination unit is used to compare the multiple answers with the preset standard answer and determine the similarity between each answer and the preset standard answer.
[0148] The complexity determination unit is used to compare the similarity between each answer and the preset standard answer, and take the complexity of the preset answer generation scheme corresponding to the answer with the greatest similarity as the complexity of the corresponding question. The complexity includes the first complexity, the second complexity, and the third complexity from low to high.
[0149] The model training unit is used to train the preset classification model using various problems and their corresponding complexities to obtain a preset classifier.
[0150] In some optional implementations, the answer generation solution determination model 402 includes:
[0151] The first answer generating unit is used to input the target question into a preset large language model to generate an answer if the complexity evaluation result of the target question is a first complexity.
[0152] The second answer generating unit is used for obtaining external knowledge by using a preset retriever if the complexity evaluation result of the target question is a second complexity, and inputting the external knowledge and the target question into a preset large language model to generate an answer.
[0153] The third answer generation unit is used to iterate the process of generating answers using a preset retriever and a preset large language model if the complexity evaluation result of the target question is a third complexity, until the generated answer meets the preset standard evaluation requirements.
[0154] In some optional implementations, the first answer generating unit includes:
[0155] The first tag sequence determination subunit is used to determine a first tag sequence according to a target problem with a first complexity.
[0156] The second tag sequence output subunit is used to input the first tag sequence into a preset large language model and output a second tag sequence.
[0157] The first answer generating subunit is used to determine a first answer to the target question with a complexity of a first complexity according to the second tag sequence.
[0158] In some optional implementations, the second answer generating unit includes:
[0159] The document conversion subunit is used to obtain multiple documents and convert each document into text data.
[0160] The data vectorization subunit is used to vectorize the text data to obtain vector data, and store the vector data in a vector database.
[0161] The external knowledge retrieval subunit is used to retrieve the vector database using a preset retriever according to the target problem with the second complexity to obtain external knowledge.
[0162] The second answer generation subunit is used to input external knowledge and a target question of a second complexity into a preset large language model to generate a second answer to the target question of the second complexity.
[0163] In some optional implementations, the second answer generating unit includes:
[0164] The external knowledge determination subunit is used to retrieve the vector database using a preset retriever according to the target problem with the third complexity to obtain external knowledge.
[0165] The third answer generation subunit is used to input the external knowledge and the target question with the third complexity into the preset large language model to obtain the third answer.
[0166] The answer evaluation subunit is used to determine whether the third answer meets the preset accuracy evaluation requirements. If not, it returns to the step of searching the vector database using a preset retriever until the generated third answer meets the preset accuracy evaluation requirements.
[0167] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0168] The RAG-based document question-and-answer system in this embodiment is presented in the form of functional units, where the units refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories that execute one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.
[0169] The embodiment of the present invention also provides a computer device having the above Figure 4 The RAG-based document question answering system shown.
[0170] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.
[0171] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0172] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0173] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0174] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0175] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.
[0176] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0177] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.
[0178] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A document question-answering method based on RAG, It is characterized in that The method comprises: Obtaining a target question queried by a user, and evaluating the complexity of the target question using a preset classifier to obtain a complexity evaluation result of the target question; An answer generation scheme for the target question is determined based on a complexity evaluation result of the target question. The answer generation scheme includes directly generating an answer using a preset large language model, or using a preset retriever to retrieve preset documents to obtain external knowledge related to the target question, and then generating an answer using a preset large language model in combination with the external knowledge.
2. The method according to claim 1, It is characterized in that The training process of the preset classifier includes: Obtain multiple questions from a user, and answer each question using multiple preset answer generation schemes to obtain multiple answers, each preset answer generation scheme having a different complexity; Comparing the multiple answers with a preset standard answer to determine the similarity between each answer and the preset standard answer; Compare the similarity between each answer and the preset standard answer, and take the complexity of the preset answer generation scheme corresponding to the answer with the greatest similarity as the complexity of the corresponding question, wherein the complexity includes the first complexity, the second complexity, and the third complexity from low to high; The preset classification model is trained using each problem and its corresponding complexity to obtain a preset classifier.
3. The method according to claim 2, It is characterized in that Determining an answer generation scheme for the target question according to a complexity evaluation result of the target question includes: If the complexity evaluation result of the target question is the first complexity, inputting the target question into a preset large language model to generate an answer; If the complexity evaluation result of the target question is the second complexity, using a preset retriever to obtain external knowledge, and inputting the external knowledge and the target question into the preset large language model to generate an answer; If the complexity evaluation result of the target question is the third complexity, the process of generating an answer using the preset retriever and the preset large language model will be iterated until the generated answer meets the preset standard evaluation requirements.
4. The method according to claim 3, It is characterized in that Input the target question into a preset large language model to generate an answer, including: Determine a first tag sequence according to a target problem having a first complexity; Inputting the first tag sequence into a preset large language model and outputting a second tag sequence; A first answer to the target question having a first complexity is determined according to the second tag sequence.
5. The method according to claim 3, It is characterized in that Using a preset retriever to obtain external knowledge, and inputting the external knowledge and the target question into the preset large language model to generate an answer, including: Obtain multiple documents and convert each document into text data; Vectorizing the text data to obtain vector data, and storing the vector data in a vector database; According to the target problem of the second complexity, using a preset searcher to search the vector database to obtain external knowledge; The external knowledge and the target question of the second complexity are input into the preset large language model to generate a second answer to the target question of the second complexity.
6. The method according to claim 5, It is characterized in that The process of generating an answer by using the preset retriever and the preset large language model is iterated until the generated answer meets the preset requirements, including: According to the target problem of the third complexity, using a preset searcher to search the vector database to obtain external knowledge; Inputting the external knowledge and the target question with a third complexity into a preset large language model to obtain a third answer; Determine whether the third answer meets the preset accuracy evaluation requirement. If not, return to the step of searching the vector database using a preset retriever until the generated third answer meets the preset accuracy evaluation requirement.
7. A document question-answering system based on RAG, It is characterized in that The system comprises: The complexity evaluation module is used to obtain the target question queried by the user, and evaluate the complexity of the target question using a preset classifier to obtain a complexity evaluation result of the target question; An answer generation scheme determination model is used to determine the answer generation scheme of the target question based on the complexity evaluation result of the target question, wherein the answer generation scheme includes directly generating an answer using a preset large language model, or using a preset retriever to retrieve preset documents to obtain external knowledge related to the target question, and generating an answer using the external knowledge and a preset large language model.
8. A computer device, It is characterized in that include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 6 by executing the computer instructions.
9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 6.
10. A computer program product, It is characterized in that The method comprises computer instructions for causing a computer to execute the method according to any one of claims 1 to 6.
Citation Information
Cited By
Error positioning method for retrieval enhancement generation system, electronic equipment and medium
CN119474276A
Intelligent question and answer method and device and medium
CN119829725A
Self-adaptive retrieval enhancement generation method based on topic filtering and iterative reasoning
CN119961442A
Extraction type retrieval question and answer method and device, equipment and medium
CN120030128A
Method, device and equipment for generating reply, medium and product
CN120804268A
