A method and device for optimizing a RAG-based question answering system

Abstract

The application provides a RAG-based question and answer system optimization method and device, which comprises the following steps: firstly, grouping a test set; then, explicitly defining a first evaluation label for each test sample in a non-related field group, and explicitly defining a second evaluation label for each test sample in a related field group. The first evaluation label is a correct answer label or a prompt word optimization label, and the second evaluation label comprises at least one of a correct answer label, a correct recall label, a prompt word optimization label, a recall optimization label, an information fusion optimization label and a corpus deficiency label; in this way, the answer accuracy and the recall accuracy can be calculated based on the number of test samples, the number of correct answer labels and the number of correct recall labels. In this way, while the evaluation is completed, the test samples carrying the prompt word optimization label, the recall optimization label or the corpus deficiency label can also enable the R&D personnel to explicitly define the reason for the low answer quality, so that targeted optimization can be performed.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus for optimizing a question-answering system based on RAG. Background Technology

[0002] Retrieval-Augmented Generation (RAG) is an artificial intelligence framework that combines information retrieval systems with the generative capabilities of Large Language Models (LLMs). It primarily enhances the knowledge accuracy and contextual understanding of LLMs, ensuring that the generated response text is semantically appropriate within its context and that the knowledge upon which the generated response text relies is verifiable in a knowledge base. Currently, RAG has become a mainstream paradigm for improving the performance of large LLMs on knowledge-intensive tasks, effectively mitigating the "model illusion" problem and enhancing the interpretability of responses.

[0003] However, RAG technology still faces some technical challenges in text storage and retrieval, such as inability to retrieve responses, insufficient relevance in retrieval, and partial truncation of answers due to context token limitations. Therefore, developers need to precisely adapt and optimize it, making performance evaluation of the RAG system particularly crucial. However, most existing methods for evaluating the quality of RAG-generated text determine the accuracy of responses by comparing the generated text with preset answers. This evaluation method is relatively simplistic and does not consider classifying different response scenarios to identify the reasons for low-quality generated text. Therefore, it cannot provide effective solutions to the existing response generation problems, failing to guarantee optimization efficiency or ultimately resolve the response issues. Summary of the Invention

[0004] The purpose of this invention is to provide a method and apparatus for optimizing a question-answering system based on RAG (Research and Analysis of Generic Question Answering), thereby improving the problems existing in the prior art. Embodiments of this invention can be implemented as follows:

[0005] In a first aspect, the present invention provides a method for optimizing a question-answering system based on RAG, comprising:

[0006] Obtain a test set comprising several test samples, wherein the test samples include question text, recall content, and response text;

[0007] Based on whether the question text belongs to the target knowledge domain, all test samples are divided into a relevant domain group and an irrelevant domain group;

[0008] For each test sample in the unrelated domain group, a first evaluation tag for the test sample is determined by judging whether the response text of the test sample conforms to a preset response persona; the first evaluation tag is either a correct response tag or a prompt word optimization tag.

[0009] For each test sample in the relevant domain group, an intent cross-comparison is performed on the question text, recall content, and response text in the test sample, or the second evaluation label of the test sample is determined by judging whether the response text of the test sample conforms to the response persona; wherein, the second evaluation label includes at least one of the following: correct response label, correct recall label, prompt word optimization label, recall optimization label, information fusion optimization label, and corpus lack label;

[0010] Based on the number of test samples, the number of correct response labels, and the number of correct recall labels, the response accuracy and recall accuracy are calculated respectively.

[0011] Based on the first evaluation label or the second evaluation label of each test sample, corresponding optimization processing is performed to obtain the optimized question-answering system.

[0012] Secondly, the present invention provides a RAG-based question-answering system optimization device, comprising:

[0013] The acquisition module is used to acquire a test set including several test samples, wherein the test samples include question text, recall content and response text;

[0014] The grouping module is used to divide all test samples into relevant domain groups and unrelated domain groups based on whether each question text belongs to the target knowledge domain;

[0015] The first evaluation module is used to determine the first evaluation tag of each test sample in the unrelated domain group by judging whether the response text of the test sample conforms to the preset response persona; the first evaluation tag is a correct response tag or a prompt word optimization tag.

[0016] The second evaluation module is used to perform intent cross-comparison on the question text, recall content, and response text in each test sample in the relevant domain group, or to determine the second evaluation label of the test sample by judging whether the response text of the test sample conforms to the response persona; wherein the second evaluation label includes at least one of the following: correct response label, correct recall label, prompt word optimization label, recall optimization label, information fusion optimization label, and corpus lack label;

[0017] The calculation module is used to calculate the response accuracy and recall accuracy based on the number of test samples, the number of correct response tags, and the number of correct recall tags, respectively.

[0018] The optimization module is used to perform corresponding optimization processing based on the first evaluation label or the second evaluation label of each test sample to obtain the optimized question-answering system.

[0019] Compared with existing technologies, this invention provides a method and apparatus for optimizing a question-answering system based on RAG (Related Aspects of Language). First, all test samples in the test set are divided into relevant and irrelevant domain groups. Each test sample in the irrelevant domain group is assigned a first evaluation label, and each test sample in the relevant domain group is assigned a second evaluation label. Since the first evaluation label is either a correct response label or a prompt word optimization label, and the second evaluation label includes at least one of a correct response label, a correct recall label, a prompt word optimization label, a recall optimization label, an information fusion optimization label, and a corpus deficiency label, the response accuracy and recall accuracy can be calculated based on the number of test samples, the number of correct response labels, and the number of correct recall labels, respectively. Finally, based on the first or second evaluation label of each test sample, corresponding optimization processing can be performed. In this way, while completing the evaluation, these test samples carrying prompt word optimization labels, recall optimization labels, or corpus deficiency labels also allow developers to identify the reasons for low response quality, thereby taking targeted measures to optimize the question-answering system. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a question-answering system optimization method based on RAG, provided as an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of a process for determining the first evaluation label for each test sample in an unrelated domain group, provided as an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram of a process for determining a second evaluation label for each test sample in a relevant domain group, provided as an embodiment of the present invention.

[0024] Figure 4 This is a schematic diagram of the process for corpus expansion using optimization measure 1, provided in an embodiment of the present invention.

[0025] Figure 5 This is a schematic diagram illustrating a process for determining new recall content based on an updated recall strategy, as provided in an embodiment of the present invention.

[0026] Figure 6 This is a schematic diagram of a RAG-based question-answering system optimization device provided in an embodiment of the present invention.

[0027] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0029] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0030] Furthermore, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance. It should be noted that, where there is no conflict, features in the embodiments of the present invention can be combined with each other.

[0031] Please see Figure 1 , Figure 1 This invention provides a flowchart illustrating a method for optimizing a question-answering system based on RAG. The execution subject of this method can be a computing device such as a personal laptop, personal computer, or server. Figure 1 The RAG-based question-answering system optimization method includes the following steps S101~S105:

[0032] S101. Obtain a test set that includes several test samples, including question text, recall content, and response text.

[0033] Optionally, multiple test samples can be extracted by obtaining the intelligent question-answering history records, logs, etc. of the question-answering big data model to form a test set as the basis for evaluation.

[0034] S102. Based on whether the question text belongs to the target knowledge domain, all test samples are divided into relevant domain group and unrelated domain group.

[0035] In this embodiment, the question-answering model has a preset target knowledge domain. For a test sample: if the question text belongs to the target knowledge domain, the test sample belongs to the relevant domain evaluation group; if the question text does not belong to the target knowledge domain, the test sample belongs to the irrelevant domain evaluation group. Dividing all test samples into two groups effectively defines the total number of samples required to calculate recall and response precision, improving calculation accuracy. Simultaneously, it facilitates a more detailed diagnosis of problems in the generated text, allowing for targeted optimization measures to be taken, as it helps determine whether the low quality of the generated text is due to limitations in the model's capabilities, poor quality of retrieved information, or because the user asked questions outside the target knowledge domain or invalid questions.

[0036] For example, assuming the question-and-answer model of this invention is a government affairs model, its target knowledge domain is the government affairs-related domain. Question texts involving policy and regulation interpretation, government services, and service guidance belong to the relevant domain assessment group, while non-government affairs questions or meaningless questions belong to the irrelevant domain assessment group. For example, question texts in the relevant domain assessment group could be: "What documents are needed for a child's first ID card application?", "Can Ya'an City bus card top-ups be done online?", "What materials are needed to renew an expired ID card?", etc. Question texts in the irrelevant domain assessment group could be: "Online shopping return and exchange process and rights protection methods", "How to distinguish genuine from counterfeit goods (such as luxury goods, electronic products)", "How to communicate with teenagers during their rebellious phase", etc.

[0037] It should be noted that the above examples are merely illustrations. The target knowledge domain of the question-answering model can also be related to finance, law, or medicine, etc. This invention does not limit the target knowledge domain of the question-answering model or the specific content of the question text in the two evaluation groups.

[0038] S103. For each test sample in the unrelated domain group, determine the first evaluation label of the test sample by judging whether the response text of the test sample conforms to the preset response persona.

[0039] In this embodiment, the first evaluation label can be a correct answer label or a prompt word optimization label. The proportion of the final determined prompt word optimization labels in the irrelevant domain group can reflect whether the prompt words of the question-answering model can guide the model to generate a guiding answer when facing question-answering tasks outside the target knowledge domain. This guiding answer can guide the user to interact with the model within the target knowledge domain.

[0040] S104. For each test sample in the relevant domain group, perform intent cross-comparison on the question text, recall content, and response text in the test sample, or determine the second evaluation label of the test sample by judging whether the response text of the test sample conforms to the response persona.

[0041] In this embodiment, the second evaluation label includes at least one of the following: correct response label, correct recall label, prompt word optimization label, recall optimization label, information fusion optimization label, and corpus deficiency label. Utilizing all test samples from the relevant domain group, the evaluation focuses on whether accurate recall and correct responses can be achieved for questions in the target knowledge domain. Furthermore, within the relevant domain group: if a prompt word optimization label exists, it indicates that the prompt word portion guiding the large model to generate the correct fallback answer (a fallback response conforming to the predetermined persona) urgently needs optimization; if a recall optimization label exists, it indicates that the recall is not accurate enough, and the recall strategy (i.e., the method of determining the recall content based on knowledge base retrieval) urgently needs optimization; if an information fusion optimization label exists, it indicates that the method by which the large model generates response text based on the recall text urgently needs optimization; and if a corpus deficiency label exists, it indicates that the knowledge base lacks relevant corpus for the target knowledge domain and needs to be supplemented.

[0042] S105. Calculate the response accuracy and recall accuracy based on the number of test samples, the number of correct response tags, and the number of correct recall tags.

[0043] In this embodiment, if all test samples are taken as the object of consideration, and the number of test samples is x, the number of correct response tags is y, and the number of correct recall tags is z, then the response accuracy rate obtained in this evaluation is y / x, and the recall accuracy rate is z / x.

[0044] Optionally, the relevant domain group can be used as the object of study. Assuming that the number of test samples in the relevant domain group is x', the number of correct response labels is y', and the number of correct recall labels is z', then for the target knowledge domain, the response accuracy obtained in this evaluation is y' / x', and the recall accuracy is z' / x'.

[0045] Assuming we denote the first evaluation label and the second evaluation label as Label1 and Label2 respectively, and the correct response label as a1 (a corresponds to Accurate, meaning accurate or correct), the correct recall label as a2, the prompt word optimization label as p1 (p corresponds to Prompt, meaning prompt word), the recall optimization label as r1 (r corresponds to recall, meaning recall), the corpus deficiency label as c1 (c corresponds to Corpus, meaning corpus), and the information fusion optimization label as f1 (f1 corresponds to fusion, meaning fusion), the following combines... Figure 2 , Figure 3 The implementation process of steps S103 and S104 is described below.

[0046] S106. Based on the first evaluation label or the second evaluation label of each test sample, perform corresponding optimization processing to obtain the optimized question-answering system.

[0047] In this embodiment, based on the labels of each test sample, the optimization of the question-answering system can be defined as the following four optimization measures:

[0048] Optimization measure 1: Based on the first question text in each first test sample for the second evaluation label including the lack of corpus label, optimization can be achieved by supplementing the knowledge base of the question answering system with the relevant corpus of the first question text;

[0049] Optimization measure 2: For the second question text in each second test sample where the first evaluation label is the prompt word optimization label or the second evaluation label only includes the prompt word optimization label, the developers can optimize the prompt words so that when the question-answering model faces questions that are unrelated to its own focused response domain, it can provide a guiding catch-all answer based on the updated prompt words.

[0050] Optimization measure 3: For the third question text in each third test sample including the recall optimization label in the second evaluation label, it is shown that the original recall strategy of the question answering system is insufficient and easily recalls irrelevant content. The developers can optimize the recall strategy of the question answering system so that the question answering system can effectively recall relevant content of the question based on the updated recall strategy.

[0051] Optimization measure 4: For each fourth test sample in the second evaluation label, including the information fusion optimization label, it indicates that the question answering system is prone to extracting irrelevant content and generating incorrect answers during the large model generation stage. Developers can filter out low-relevance recall texts based on their corresponding similarity values, keywords, etc.; or use other models (such as LLM), sentence-transformer cross-encoders, etc. to rearrange the recall texts and select those with higher relevance, and then use them as reference texts for the large model to generate response texts, or fine-tune the large model to generate response texts, etc. to optimize the system.

[0052] In one optional implementation, in step S103 above, for a test sample from an unrelated domain group, please refer to... Figure 2 The method for determining the first evaluation label of this test sample is as follows:

[0053] S1031. Determine whether the response text in the test sample conforms to the preset response persona. If yes, confirm that the first evaluation label of the test sample is the correct response label (let's say Label1=a1). If no, confirm that the first evaluation label of the test sample is the prompt word optimization label (let's say Label1=p1).

[0054] If a test sample in the unrelated domain group has Label1=p1, it indicates that the prompt content needs optimization. For example, prompt optimization can be achieved by giving the large model a persona that matches the target knowledge domain. For instance, a persona description can be added to the prompt: "Hello, I am 'Pan,' the digital panda developed by XXX, specializing in answering various government service-related questions."

[0055] In this way, by using all test samples from unrelated domain groups, we can focus on examining whether the response text generated by the question-and-answer model is based on the response persona preset by the question-and-answer model to generate the correct fallback answer to guide users to ask questions within the target knowledge domain, thus avoiding situations where prompt words are used to bypass security.

[0056] For example, suppose in a test sample from an unrelated domain group, the question text is "How to communicate with teenagers during their rebellious phase," which is clearly not a government-related domain. If the response text is Response Text Example 1 in Table 1 below, then the persona in the response text is consistent with the preset response persona, indicating that a correct catch-all answer has been generated. If the response text is Response Text Example 2 in Table 1 below, then the response text does not conform to the response persona and is not a catch-all response. In this case, the answer generated in the context of the prompt word jailbreak is unverifiable, has low credibility, and is prone to large model illusion.

[0057] Table 1. Examples of two response texts from unrelated domain groups

[0058]

[0059] It should be noted that the above examples are merely illustrative and are not intended to be limiting.

[0060] In one optional implementation, in step S104 above, taking a test sample from a relevant domain group as an example, please refer to [link to relevant documentation]. Figure 3 The methods for determining the second evaluation label of a test sample based on the question text, recall content, and response text in the test sample include:

[0061] S1041. Based on the question text or a question-answer pair consisting of question text and answer text, perform similarity matching with several knowledge fragments in the knowledge base to obtain the matching results.

[0062] In this embodiment, the knowledge base of the question-answering system includes a vector database, a document database, and a knowledge graph. The vector database contains a large number of knowledge fragments and their knowledge vectors. After vectorizing the question text or question-answer pairs, similarity matching can be performed in the vector database. The matching results can include the similarity of several matched knowledge fragments. This process is existing technology and will not be described in detail here.

[0063] S1042. If the similarity of at least one matching knowledge fragment in the matching results exceeds the preset first threshold, then perform intent cross-comparison on the question text, recall content and response text to determine the second evaluation label of the test sample.

[0064] In this embodiment, if the similarity of at least one matched knowledge fragment in the matching results exceeds a first threshold, it indicates that there is question-related corpus in the knowledge base. Next, it is necessary to determine the intent relevance between each pair of question text, recalled content, and response text to determine whether the recalled content is correctly recalled and whether the response text is a correct response. In an optional example, the first threshold can be 80% or 85%.

[0065] Optionally, the second evaluation label may also include a label for correct fuzzy identification and a label for incorrect fuzzy identification. Before step S1042, which involves "performing an intent cross-comparison of the question text, recall content, and response text to determine the second evaluation label of the test sample," the following steps are also included:

[0066] Determine whether the first intent corresponding to the question text is complete; if complete, execute the step of "cross-comparing the intent of the question text, recall content, and response text to determine the second evaluation label of the test sample"; if incomplete, determine whether the response text corresponding to the question text is a guiding response; if so, determine that the second evaluation label is a correct label for fuzzy recognition, and return to step S1042 based on the user's feedback on the clarification question; if not, determine that the second evaluation label is an incorrect label for fuzzy recognition.

[0067] A response text that includes content guiding the user to actively input their true intent is considered a guiding response. It's understandable that in a test sample, the primary intent corresponding to the question text is incomplete, indicating that the question text is an ambiguous intent question, possibly containing only one keyword (e.g., the question text only contains the single keyword "ID card"), failing to reflect the user's true intent (i.e., the user's intent is ambiguous). Therefore, an ideal response text should include guiding content, such as: "Hello, I'm Pan from Digital Panda, specializing in government service consultations. What aspect of ID cards would you like to inquire about? For example, the process for adults to replace their ID cards in a different location, the age at which one can apply for an ID card, and the materials required for applying for an ID card."

[0068] The robustness of a large model can be examined by analyzing the correct and incorrect fuzzy recognition labels, assessing its fault tolerance and correction capabilities when facing fuzzy intent issues. Assuming the number of correct fuzzy recognition labels in the relevant domain group is m, and the number of incorrect fuzzy recognition labels is n, then the fuzzy recognition accuracy of the large model is m / (m+n), where m+n represents the number of fuzzy intent issues in the relevant domain group. A high fuzzy recognition accuracy indicates that the large model can effectively guide users to input their true intent.

[0069] Optionally, the implementation of "performing intent cross-comparison of question text, recall content, and response text to determine the second evaluation label of the test sample" includes the following sub-steps:

[0070] S10421. Obtain the first intent, second intent, and third intent corresponding to the question text, recall content, and response text, respectively;

[0071] S10422. If the first intent and the second intent are not related, then confirm that the second evaluation label of the test sample is the recall optimization label.

[0072] S10423, If the first intention, the second intention, and the third intention Figure 3 If there is a pairwise correlation between the test samples, then the second evaluation label for the test samples is confirmed to include a correct recall label and a correct response label.

[0073] S10424. If the first intent and the second intent are related and the third intent is not related to either the first intent or the second intent, then confirm that the second evaluation label of the test sample includes the correct recall label and the information fusion optimization label.

[0074] In optional examples, other large models can be used for intent recognition to obtain the first, second, and third intents. The correlation between two intents can be determined by vectorizing each intent separately and calculating their similarity. If the similarity is greater than a correlation threshold (e.g., 90% or 80%), the two intents are considered related; otherwise, they are considered unrelated. In this embodiment, as... Figure 3 If the first intent and the second intent are unrelated, then the second evaluation label Label2 = r1 is confirmed, indicating a problem with the recall strategy (i.e., the recall strategy urgently needs optimization), resulting in incorrect recall content. The recall strategy can be updated by adjusting relevant parameter settings and optimizing the RAG workflow. For example, an example of incorrect recall could be: in the test sample, the question text is "What documents are needed for a child to apply for an ID card for the first time?", and the recall content could include the four recall texts shown in Table 2 below.

[0075] Table 2 Examples of Recall Content in Error Recalls

[0076]

[0077] In this embodiment, as Figure 3 If the first intention, the second intention, and the third intention Figure 3 If the items are pairwise correlated, it indicates that the recalled content is a correct recall and the response text is a correct response, i.e., Label2 = a2, a1. For example, a correct recall could be: in the test sample, the question text is "Can I reissue an ID card from another city in Ya'an?", and the recalled content could include the four recalled texts shown in Table 3 below:

[0078] Table 3 Examples of correctly recalled recall content

[0079]

[0080] The correct recall text in Table 3 is the question-answer pair consisting of question C and answer C. Tables 1 and 2 are just examples and are not limited here.

[0081] In this embodiment, as Figure 3If the first and second intentions are related and the third intention is not related to either the first or second intention, then Label2=a2, f1 is confirmed. This indicates that the retrieved content is correct, but the response text is inaccurate and is of low quality. This may be because the retrieved text contains other interfering information besides the correct retrieved content, which is given to the large model (for example, in Table 3 above, everything except question C and answer C is interfering information). The large model failed to effectively understand the relationship between the retrieved information and the question, or failed to reasonably integrate high-quality retrieved information into the generated text. In the future, the response quality can be improved by optimizing the information fusion strategy (such as filtering interfering retrieved text during the generation process) or by fine-tuning the large model.

[0082] For example, in the case of correct recall as shown in Table 3 above, regarding the question text "Can I reissue an ID card from another province in Ya'an?", if the response text is "Hello, I am Pan from Digital Panda, specializing in government service consultation. If you have special circumstances in Ya'an that require reissue, you can try contacting the local Public Security Bureau's Exit-Entry Administration Office or Household Registration Section to find out if they provide out-of-town processing services or have other solutions," then the third intent is unrelated to the first or second intent, indicating that the question-and-answer model generated an incorrect answer or a low-quality response with low relevance based on the recalled content text. If the generated response text is "Resident ID cards can be processed across provinces," then the three intents are correlated pairwise, indicating correct recall and correct response. This example is merely illustrative and is not intended to limit the scope of the question.

[0083] S1043. If the similarity of all matched knowledge fragments in the matching results does not exceed the first threshold, then the second evaluation label of the test sample is determined based on the recalled content or response text.

[0084] In this embodiment, if the similarity of all matched knowledge fragments in the matching results does not exceed the first threshold, it means that there is no corpus related to the first question text in the knowledge base. Next, it is necessary to check whether there is recall content in the test sample and whether the response text matches the persona to determine the second evaluation label.

[0085] Therefore, the implementation of step S1043, "determining the second evaluation label of the test sample based on the recall content or response text," can include:

[0086] Step (1): If the recall content of the problem text exists (i.e., the recall content in the current test sample is not empty), then confirm that the second evaluation label of the test sample includes the recall optimization label and the corpus lack label.

[0087] Step (2): If the recall content of the problem sample does not exist (i.e. the recall content in the current test sample is empty), then determine whether the response text matches the response persona. If yes, then confirm that the second evaluation label of the test sample is the correct response label and the corpus lack label. If no, then confirm that the second evaluation label of the test sample includes the prompt word optimization label and the corpus lack label.

[0088] In this embodiment, for step (1), as follows Figure 3 If the similarity of all matched knowledge fragments does not exceed the first threshold and the recalled content of the question text exists in the test sample, then the recalled content is completely unrelated to the question. In this case, Label2=r1 and c1 are confirmed. This indicates that in the absence of relevant corpus in the knowledge base, the original recall strategy misunderstood the current question and mistakenly recalled relevant corpus of other questions. The recall strategy needs to be optimized and relevant corpus needs to be supplemented.

[0089] For step (2), such as Figure 3 If the response text matches the respondent's persona, confirming Label2=a1 and c1, it indicates a lack of relevant corpus, but the prompt words are of good quality and can guide the question-answering model to generate the correct catch-all answer according to the respondent's persona (e.g., Digital Panda Pan, specializing in government service consultation). The model does not rely on its own reasoning ability to generate the response text, reducing the chance of generating misleading illusions and avoiding misleading users due to generated incorrect information in specific fields (such as science-related fields, finance-related fields, etc.). In this case, it is necessary to subsequently supplement the corpus content related to the question text.

[0090] For example, suppose in the test sample, the question text is "childcare subsidy," the recall content is empty (i.e., there is no relevant corpus in the knowledge base), and the response text is "Hello, I am Pan from Digital Panda, specializing in government service consultation. Specific policies and application procedures regarding childcare subsidies may involve the responsibilities of multiple departments, and I cannot directly provide detailed information. However, I can help you understand how to find relevant policies, provide general application process guidance, or direct you to contact relevant departments for more accurate information. Do you need information about local policies or national policies?" This response text's persona matches the persona of a government service assistant, indicating that the question-and-answer model can generate a correct fallback response guided by the prompts. The failure to generate the user's expected answer is due to a lack of relevant corpus in the knowledge base. Subsequent addition of childcare subsidy-related corpus to the knowledge base will improve the response quality. It should be noted that this example is merely illustrative and not intended to be limiting.

[0091] For step (2), such as Figure 3If the response text does not match the respondent's persona, and Label2=p1 and c1 are confirmed, it indicates a lack of relevant corpus and issues with the prompt words, such as a lack of example guidance. This prevents the question-answering model from generating a correct, fallback answer, instead resulting in a prompt word "jailbreak"—relying on the model's own reasoning ability to generate an answer. This cannot avoid the "model illusion" problem, and the confidence of the generated text cannot be guaranteed. Therefore, subsequent work not only requires supplementing the corpus content related to this question text but also optimizing the prompt words to address the prompt word "jailbreak" issue.

[0092] For example, suppose in the test sample, the question text is "Infant registration and birth certificate processing in Tianquan County, Ya'an City," the recall content is empty, and the response text is "According to the information you provided, the process for obtaining a property ownership certificate usually involves the owner applying for the certificate after the developer completes the large property ownership certificate process. If you need more detailed information about the process or required materials, we suggest you contact the developer or the local property registration department for consultation." This response text does not use the persona of the digital panda "Pan" and is irrelevant to the question; it is not a catch-all response. This is because the prompt word "jailbreak" caused the large model to hallucinate and generate an incorrect answer. To avoid misleading hallucinations, subsequent responses need to be optimized (e.g., requiring the model to reference fragments from the context, i.e., the prompt word template might be "persona + context fragment + friendly hints," etc.) to reduce the probability of model hallucinations. It should be noted that this example is only illustrative and is not intended to limit the scope of the response.

[0093] As described above regarding the implementation process of the RAG-based question-answering system optimization method, this invention first divides all test samples into irrelevant and relevant domain groups based on the target knowledge domain focused on by the large question-answering model. Then, it uses the method of judging whether the response text conforms to the response persona to confirm the first evaluation label (correct response label or prompt word optimization label) of each test sample in the irrelevant domain group. It then uses a three-way intent cross-comparison method or the method of judging whether the response text conforms to the response persona to confirm the second evaluation label (including at least one of the following: correct response label, correct recall label, prompt word optimization label, recall optimization label, information fusion optimization label, and corpus lack label) of each test sample in the relevant domain group. Thus, it can not only calculate the response accuracy and recall accuracy based on the number of correct response labels and the number of correct recall labels, but also allow developers to clarify the optimization direction and measures based on other labels. This improves the quality of RAG retrieval and generation while ensuring efficient optimization and debugging of RAG.

[0094] An alternative implementation method for optimization measure 1 mentioned above can be as follows:

[0095] S107. Based on the second evaluation label, including the first question text in each first test sample with the missing corpus label, perform corpus augmentation processing on the knowledge base.

[0096] In this embodiment, the second evaluation label includes any first test sample with a lack of corpus label (such as...). Figure 3 For example, the first test sample is either Label2=r1 or Label2=r1, c1. Please refer to [link to relevant documentation]. Figure 4 The implementation method for expanding the original knowledge base based on the first question text of the first test sample may include the following sub-steps S1071~S1076.

[0097] S1071. Obtain the original files related to the text of the first question.

[0098] In this embodiment, the original file is the initial corpus related to the text of the first question.

[0099] S1072. Preprocess the original file to obtain plain text information.

[0100] In this embodiment, the original file may include other content such as charts and images in addition to text. To ensure data quality, preprocessing is required. Optionally, preprocessing involves parsing the original file to obtain text content and format information, and then performing data cleaning to obtain plain text information. Data cleaning can be at least one of the following cleaning methods: deduplication, abnormal data processing, data table transformation, and structured processing.

[0101] S1073. Determine at least one topic covered by the plain text information, divide the plain text information into at least one data field, and determine the text ID corresponding to each document in each data field.

[0102] In this embodiment, a data domain includes text content corresponding to a topic. This grouping related plain texts describing the same topic into the same knowledge scope to form data domains facilitates the extraction of entities and their corresponding relationships from each data domain later, enabling rapid drawing and formation of a knowledge graph. The text ID can be used to uniquely identify the text content of a data domain.

[0103] S1074. Divide each document into multiple sub-segments, convert each sub-segment into a segment vector, establish a mapping relationship based on each segment vector and its corresponding text ID, write each segment vector and its corresponding text ID into a vector database, and write each document and its text ID into a document database.

[0104] In this embodiment, each document in each data domain can be divided into multiple sub-segments based on preset segmentation rules (such as segmentation by fixed length, segmentation by paragraph, or segmentation by title / chapter). This text segmentation can decompose long texts in the data domain into smaller sub-segments, which helps to quickly and accurately obtain knowledge fragments that are relevant to the LLM context from the knowledge base, and establish a mapping relationship between all sub-segments corresponding to each document based on the text ID corresponding to that document.

[0105] In this embodiment, each document and its text ID in each data field are written to the document database, while the fragment vector and corresponding text ID of each sub-fragment are written to the vector database. In this way, if needed during the subsequent retrieval process, the parent text corresponding to the sub-fragment can be found from the document database based on the correspondence between the text IDs.

[0106] S1075. Based on the text content of each data domain, perform entity recognition, relation extraction and attribute extraction to obtain at least one triplet corresponding to each data domain, and construct a knowledge graph based on at least one triplet corresponding to each data domain.

[0107] In this embodiment, a triple includes three parts: entity, relation, and entity. It should be noted that the steps of constructing the knowledge graph and writing data to the vector database and document database are not set in any particular order; the order described here is merely an example and is not specifically limited.

[0108] S1076. Update the knowledge base based on the written vector database, document database, and knowledge graph.

[0109] For all test samples, whenever a missing label appears in the corpus (i.e., for each first test sample including c1 in Label2), the above steps S1071~S1076 are performed to complete the corpus supplementation.

[0110] In other embodiments, based on the aforementioned text ID, RAG capabilities can be evaluated from several dimensions, including knowledge retrieval accuracy, knowledge coverage breadth, and reasoning fusion. The steps are as follows:

[0111] (1) Obtain each test sample including the correct response label and the correct recall label, and obtain multiple text IDs corresponding to the recall text in the test sample from the vector knowledge base. Determine whether the multiple text IDs are the same. If they are, confirm that the test sample belongs to the single document response sample. Otherwise, determine that the test sample is the multi-document response sample.

[0112] (2) Based on the number of test samples s including correct response labels and correct recall labels, the number of single-document response samples g and the number of multi-document response samples f, calculate the single-point response accuracy (i.e. g / s) and the multi-point response accuracy (i.e. f / s).

[0113] The high accuracy of single-point responses indicates that the recall strategy has high knowledge retrieval precision and that the large model can make correct responses based on multiple knowledge fragments within the same document. The high accuracy of multi-point responses indicates that the recall strategy has high knowledge retrieval precision, the recalled knowledge coverage is broad, and the large model can merge multiple knowledge fragments from different documents to extract effective information for correct responses.

[0114] In one optional implementation, after optimizing the question-answering system by taking at least one of the above optimization measures 1, 2, 3, and 4, the method may further include the following steps S201-S205 to verify the optimization effect:

[0115] S201. Upon receiving the evaluation instruction, obtain the evaluation results corresponding to the test set.

[0116] In this embodiment, the evaluation results include response accuracy, recall accuracy, and a first evaluation label or a second evaluation label for each test sample. The evaluation instruction can be generated by the electronic device in response to the evaluation operation of the R&D personnel. Once the electronic device receives the evaluation instruction, it will execute the above steps S101 to S105 based on the acquired test set to obtain the evaluation results.

[0117] S202. Upon receiving an update instruction, the test set is updated based on the updated knowledge base, updated prompts, updated recall strategy, and updated question-answering model to obtain the updated test set.

[0118] In this embodiment, the update instruction can also be generated by the electronic device in response to the update operation of the R&D personnel. Once the electronic device receives the update instruction, it needs to update the test set. The reason for the update is that some question texts need to be regenerated with new response texts, and some question texts need to be reduced to obtain new recall content and regenerated with new response texts, so as to re-evaluate the subsequent quantitative optimization effect.

[0119] S203. Use the updated test set as the new test set.

[0120] In this embodiment, after updating the test set, if an evaluation instruction is received again, the evaluation result corresponding to the test set can be obtained by executing the above steps S101~S105, which is the new evaluation result.

[0121] S205. When a quantization instruction is received, an optimized quantization result is generated based on all the obtained evaluation results.

[0122] In this embodiment, the quantization command can also be generated by the electronic device in response to the quantization operation performed by the R&D personnel. After each evaluation, the R&D personnel may take optimization measures such as prompt word optimization, corpus supplementation, and recall strategy update. Then, based on the updated knowledge base, updated prompt words, updated recall strategy, and updated question-answering model, the test set can be updated for further evaluation. Thus, after obtaining the Nth evaluation result (N can be a positive integer greater than 2), if a quantization command is received, the N evaluation results can be compared and integrated to obtain an optimized quantization result, which can reflect the effectiveness of the optimization measures.

[0123] For example, the optimization results can be presented in the form of charts or intuitive data to show the optimization effect, such as: the improvement of correct response rate, the improvement of correct recall rate, the reduction of the number of samples with optimized prompt words, the reduction of the number of samples with optimized recall labels, and the reduction of the number of samples with missing labels in the corpus.

[0124] In one optional implementation, the step S202 above, "updating the test set based on the updated knowledge base, updated prompt words, updated recall strategy, and updated question-answering model", may include at least one of steps S2021 to S2024.

[0125] S2021. For the first question text in each first test sample where the second evaluation label includes the missing label of the corpus, determine the new recall content of the first question text from the updated knowledge base, and based on the first question text and its new recall content, call the question-answering big model to generate a new response text for the first question text.

[0126] In this embodiment, to verify the optimization effect of optimization measure 1 (i.e., supplementary corpus), for each first test sample with Label2=r1,c1, a1,c1, or p1,c1, it is necessary to redetermine the new recall content and regenerate the new response text. The new response text for the first question text can be generated based on the original prompt words or the updated prompt words; this is not limited here.

[0127] S2022. For the second question text in each second test sample where the first evaluation label is the prompt word optimization label or the second evaluation label includes the prompt word optimization label, based on the updated prompt words and the second question text and its recall content, call the question-answering big model to generate a new response text for the second question text.

[0128] In this embodiment, when the second test sample (let's call it sample A) is a test sample with Label1=p1, combined with Figure 2 Since the evaluation process of sample A only needs to determine whether the response text responds to the person's profile, at this time, it is only necessary to call the question-answering model to generate a new response text based on the updated prompt words, the second question text of sample A and its recall content.

[0129] In the case that the second test sample (let's call it sample B) is a test sample with Label2=p1,c1, combined with Figure 3 Since the optimization strategy corresponding to p1 is prompt word optimization, in order to judge the effect of prompt word optimization, it is necessary to call the question answering model to generate a new response text based on the updated prompt words, the question text of sample B and its recall content.

[0130] In order to verify the optimization effect of optimization measure 2 (i.e. optimization of prompt words), new response texts are regenerated for the two test samples, Label1=p1 and Label2 including p1, so that the effect of the updated prompt words can be effectively verified during subsequent evaluation and quantification.

[0131] S2023. For the third question text in each third test sample, including the recall optimization label, the updated recall strategy is used to determine the new recall content of the third question text. Based on the third question text and its new recall content, the question-answering big model is called to generate a new response text for the third question text.

[0132] In this embodiment, to verify the optimization effect of optimization measure 3 (i.e., optimizing the recall strategy), for each third test sample where Label2=r1 or Label2=r1,c1, it is necessary to redetermine the new recall content and regenerate the new response text. The new recall content for the third question text can be recalled from the knowledge base before the update or from the updated knowledge base, while the new response text for the third question text can be generated based on the original prompt words or based on the updated prompt words; no limitation is imposed here.

[0133] S2024. For cases where the second evaluation label includes the information fusion optimization label, the question-and-answer big model is fine-tuned to obtain the updated question-and-answer big model. The updated question-and-answer big model is then used to generate new response texts for the fourth question text in each fourth test sample corresponding to the information fusion optimization label.

[0134] In this embodiment, to verify the optimization effect of optimization measure 4 (i.e., large model fine-tuning), the question-answering large model is first fine-tuned to obtain an updated question-answering large model. Then, for the fourth question text in each fourth test sample with Label2=a2 and f1, the updated question-answering large model is invoked to generate a new response text based on the fourth question text and the fourth recall content. The new response text for the fourth question text can be generated based on the original prompt words or on the updated prompt words; this is not limited here.

[0135] By using the steps S2021~S2023 above, some content in the original test set can be replaced with the newly generated recall content, new response text, etc., to obtain the updated test set.

[0136] In one alternative implementation, for any third question text, combined with Figure 5 The implementation of step S2022, "using the updated recall strategy to determine the new recall content of the third question text", may include S20221~S20225.

[0137] S20221. Perform similarity matching between the text of the third question and several knowledge fragments in the knowledge base to obtain the recall results.

[0138] In this embodiment, the knowledge base used for similarity matching can be either the previous or updated version. The recall results can include multiple recalled segments whose similarity exceeds a first threshold. Optionally, the recall results can be obtained in the following ways:

[0139] (1) Vectorize the text of the third question to obtain the vector of the third question;

[0140] (2) Calculate the similarity between each fragment vector in the vector database and the third question vector;

[0141] (3) Sort all fragment vectors in descending order of similarity, and then select the knowledge fragments corresponding to the M fragment vectors with similarity greater than the first threshold (e.g., 85%) as recall fragments.

[0142] For example, assuming the preset number of selected segments is 10, if there are 15 segment vectors with a similarity greater than the first threshold (i.e., the actual number of selected segments is 15), then M is 10, and only the first 10 are selected; if there are 8 segment vectors with a similarity greater than the first threshold (i.e., the actual number of selected segments is 8), then M is 8, and only the first 8 are selected. M is the minimum of the preset number of selected segments and the actual number of selected segments. This example is merely illustrative and is not intended to limit the selection.

[0143] S20222. Perform topic consistency detection on the third question text and each recalled fragment to filter out each recalled fragment with inconsistent topics.

[0144] In this embodiment, for each recalled fragment, the topic similarity between the third question text and the recalled fragment can be calculated first. If the topic similarity is greater than a certain threshold (e.g., 85%), the fragment is considered to be topic-consistent; otherwise, it is considered topic-inconsistent and needs to be filtered. This can reduce the occurrence of irrelevant answers in subsequently generated new response texts, thereby improving recall accuracy and response accuracy.

[0145] Optionally, the topic similarity can be semantic similarity. For example, a powerful pre-trained language model (such as BERT) can be used to convert the third question text and the recalled fragment into semantic vectors and then calculate the semantic similarity.

[0146] S20223. If the remaining recall fragments are zero, then confirm that the new recall content is empty.

[0147] In this embodiment, if the remaining recalled fragments are zero after filtering, it means that the topics discussed by each recalled fragment in the recall results and the third question text are not consistent. In this case, the response text generated directly based on the recall results is not a response that meets the user's expectations. In order to reduce the illusion of a large model, it is necessary to rely on the updated prompt words to generate a correct fallback response that matches the respondent persona, so as to guide the user to carry out the dialogue task in the target knowledge domain.

[0148] S20224. If the remaining recalled fragments are not zero and the third question text belongs to a special knowledge domain, then search the document database for the parent text that matches all the remaining recalled fragments and use the parent text as the new recalled content.

[0149] In this embodiment, the specialized knowledge domain can be highly specialized fields such as law or medicine. These domains are susceptible to incomplete knowledge in the recalled text or misinterpretation due to fine text segmentation. Therefore, when the third question text belongs to a specialized knowledge domain, it is necessary to retrieve the parent text of the recalled fragment from the document database as the new recalled content. This new recalled text provides more comprehensive background knowledge, which helps the question-answering model understand the context based on the new recalled content and thus improve the quality of the generated new response text.

[0150] Optionally, the document database can also include text vectors corresponding to each data field. This allows for the merging / averaging / concatenation of the remaining recalled fragment vectors to obtain a composite vector. Then, the composite vector is used to match the most similar data field from the document database, which is considered to be the parent text.

[0151] S20225. If the remaining recalled fragments are not zero and the third question text does not belong to a special knowledge domain, then the new recalled content is determined based on the total number of words of all remaining recalled fragments.

[0152] In this embodiment, if the remaining recalled fragments are not zero and the third question text does not belong to a special knowledge domain, it can be determined whether the total number of words in all remaining recalled fragments is too small and needs to be expanded, or too large and needs to be summarized or rewritten.

[0153] Therefore, as Figure 5 The implementation of "determining new recall content based on the total number of words of all remaining recalled fragments" can include S1~S3.

[0154] S1. If the total number of words of all remaining recalled fragments is lower than the preset word limit, then obtain the knowledge graph associated with each recalled fragment from the knowledge base, and determine the supplementary content based on the obtained knowledge graph. Then, use all remaining recalled fragments and the supplementary content as the new recalled content.

[0155] In this embodiment, if the total number of words is lower than the preset minimum number of words (e.g., 50 words), the reason for the low number of words may be that the third question text itself is a very simple question (e.g., "How is the weather in Chengdu?" The corresponding reply is usually a simple reply such as "rainy or sunny").

[0156] To enrich the response text and thus stimulate greater user interest, this invention requires obtaining supplementary content based on a related knowledge graph. This supplementary content must be relevant to the third question text. For example, the preset knowledge graph for the aforementioned weather is: weather phenomena - travel advice - clothing guidelines. If the remaining recall fragment is "rainy day," then to enrich the response text, it is necessary to find relevant travel advice and clothing guidelines for rainy days from the preset weather knowledge graph as supplementary content. Finally, all remaining recall fragments and supplementary content are used as the final new recall content.

[0157] S2. If the total number of words of all remaining recalled fragments is between the preset lower limit and the preset upper limit, then all remaining recalled fragments will be used as new recalled content.

[0158] In this embodiment, as Figure 6 If the total number of words in all remaining recalled fragments is not less than the preset lower limit and not more than the preset upper limit (i.e., the total number of words is between the preset lower limit and the preset upper limit), it means that the number of words is moderate and there is no missing key information or redundant content. All remaining recalled fragments can be used as new recalled content.

[0159] S3. If the total number of characters of all remaining recalled fragments exceeds the preset character limit, then based on the preset rewriting prompts, the question-and-answer big model is called to summarize or rewrite all remaining recalled fragments to obtain new recalled content.

[0160] In this embodiment, as Figure 6 If the total number of words in all remaining recalled fragments exceeds the preset word limit (e.g., 700 words), it indicates that the excessive number of words may contain redundant information. Therefore, based on the preset rewriting prompts, it is necessary to summarize the key information of all remaining recalled fragments and simplify them into a short rewritten text (the number of words in the rewritten text does not exceed the preset word limit) to serve as the new recalled content. This can improve the completeness of the key information in the generated new response text.

[0161] After optimization based on the above method, the improvement in the response accuracy of the question-answering model is limited. In some application scenarios with low fault tolerance, it may not be able to achieve the accuracy expected by the user. Therefore, for the above optimization measure 4, the model fine-tuning method can be used to improve the response accuracy. That is, by collecting test samples of correct responses as fine-tuning samples, the question-answering model can be fine-tuned. This process may include steps S4-S5.

[0162] S4. Determine whether the number of information fusion optimization tags or the timestamp of the first information fusion optimization tag meets the triggering condition for fine-tuning the question-answering model. The triggering condition is that the ratio of the number of information fusion optimization tags to the number of samples in the test set exceeds a first preset threshold, or the timestamp of the first information fusion optimization tag is more than a second preset threshold.

[0163] In this application, the proportion of information fusion optimized labels among all test set samples or a preset time period is used as the trigger condition for fine-tuning the large model. The first preset threshold is the maximum value of the proportion of information fusion optimized labels, and the second preset threshold is the maximum time period between the timestamp corresponding to the first information fusion optimized label and the current time. For example, the first preset threshold can be 5% or 10%, and the second preset threshold can be half a month, one month, or three months, etc.

[0164] S5. When the triggering condition is met, obtain the preprocessed high-quality corpus to be fine-tuned. The corpus to be fine-tuned includes a general domain corpus set and a professional domain corpus set that reach a preset ratio. Each training sample in the corpus to be fine-tuned contains a fifth question text, a fifth recall text, a fifth response text, and a fifth prompt word.

[0165] In this embodiment, preprocessing includes text cleaning, standardization, vectorization, text enhancement, and filtering. The specialized domain corpus is extracted from each test sample corresponding to the correct answer label in the second evaluation. The general domain corpus is extracted from each test sample where the correct answer label in the first evaluation is applied. The ratio of the number of samples in the general domain corpus to the specialized domain corpus can be 7:3 to prevent the large model from forgetting the question-answering capabilities of the general domain.

[0166] S5. Iterative steps: For each training sample, the corresponding fifth question text, fifth recall text, and fifth prompt word are used as input texts for the question-answering model to obtain the predicted text; the loss value is calculated based on the predicted text and the fifth response text, and the parameters of the question-answering model are updated based on the loss value until the convergence condition is met to obtain the updated question-answering model.

[0167] The fifth question text (the user's original question), the fifth recall text (related context or knowledge base fragments), and the fifth prompt word (task instructions or format guidance) are concatenated into a complete input text according to a preset template. This input text is then vectorized and fed into the large question-answering model to obtain the predicted text. A loss function is used to calculate the loss between the predicted text and the fifth response text. Based on this loss value, the gradient of the loss with respect to all model parameters is calculated. An optimizer (such as AdamW) is then used to update the parameters of the large question-answering model based on the gradient.

[0168] The convergence criterion of this application is based on whether the reduction value between two adjacent loss values ​​reaches a preset value. If not, the loop steps are repeated until the preset value is reached, resulting in a fine-tuned large model. RAG and fine-tuning work together; the fine-tuning process finely optimizes the model parameters, thereby enhancing the model's depth and accuracy in handling specialized problems and building a professional capability support base to solve the problem of insufficient professional depth in RAG. By using RAG to retrieve external knowledge sources in real time, the boundaries of knowledge are dynamically expanded, compensating for the knowledge lag problem caused by fixed training data in the fine-tuned model. This ensures both real-time knowledge and a more accurate understanding of professional query intent, thus outputting response text that meets user expectations.

[0169] It should be noted that the execution order of each step in the above method embodiments is not limited to that shown in the attached figures, and the execution order of each step shall be subject to the actual application situation.

[0170] Based on the same inventive concept as the aforementioned RAG-based question-answering system optimization method, please refer to... Figure 6 , Figure 6A schematic diagram of the structure of a RAG-based question-answering system optimization device provided in an embodiment of the present invention is shown. The RAG-based question-answering system optimization device 200 includes: an acquisition module 210, a grouping module 220, a first evaluation module 230, a second evaluation module 240, a calculation module 250, and an optimization module 260. Wherein:

[0171] The acquisition module 210 is used to acquire a test set including several test samples, the test samples including question text, recall content and response text;

[0172] Grouping module 220 is used to divide all test samples into relevant domain group and irrelevant domain group based on whether each question text belongs to the target knowledge domain;

[0173] The first evaluation module 230 is used to determine the first evaluation label of each test sample in the unrelated domain group by judging whether the response text of the test sample conforms to the preset response persona; the first evaluation label is either a correct response label or a prompt word optimization label.

[0174] The second evaluation module 240 is used to determine the second evaluation label of each test sample in the relevant domain group by performing intent cross-comparison on the question text, recall content and response text in the test sample or by judging whether the response text of the test sample conforms to the response persona; wherein the second evaluation label includes at least one of the following: correct response label, correct recall label, prompt word optimization label, recall optimization label, information fusion optimization label and corpus lack label;

[0175] The calculation module 250 is used to calculate the response accuracy and recall accuracy based on the number of test samples, the number of correct response tags, and the number of correct recall tags, respectively.

[0176] The optimization module 260 is used to perform corresponding optimization processing based on the first evaluation label or the second evaluation label of each test sample to obtain the optimized question-answering system.

[0177] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the RAG-based question-answering system optimization device 200 described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0178] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 300 includes a processor 310, a memory 320, and a bus 330, with the processor 310 connected to the memory 320 via the bus 330.

[0179] The memory 320 can be used to store software programs or firmware, for example, the software programs or firmware corresponding to the RAG-based question-answering system optimization device 200 or the RAG optimization device 400 described above. The processor 310 executes various functional applications and data processing by running the software programs stored in the memory 320 to implement the RAG-based question-answering system optimization method or RAG optimization method provided in the embodiments of the present invention.

[0180] The memory 320 may be, but is not limited to, RAM (Random Access Memory), ROM (Read Only Memory), FLASH (Flash Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electric Erasable Programmable Read-Only Memory), etc.

[0181] The processor 310 can be an integrated circuit chip with signal processing capabilities, capable of executing software programs, such as the software programs corresponding to the RAG-based question-and-answer system optimization device 200 or the RAG optimization device 400 described above. The processor 310 can be a general-purpose processor, including: CPU (Central Processing Unit), NP (Network Processor), SoC (System on Chip), etc.; it can also be: DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0182] Understandable. Figure 7 The structure shown is for illustrative purposes only; the electronic device 300 may also include components that are more advanced than those shown. Figure 7 The more or fewer components shown, or having the same Figure 7 The different configurations shown. Figure 7 The components shown can be implemented using hardware, software, or a combination thereof.

[0183] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for optimizing a question-answering system based on RAG, characterized in that, include: Obtain a test set comprising several test samples, wherein the test samples include question text, recall content, and response text; Based on whether the question text belongs to the target knowledge domain, all test samples are divided into a relevant domain group and an irrelevant domain group; For each test sample in the unrelated domain group, a first evaluation tag for the test sample is determined by judging whether the response text of the test sample conforms to a preset response persona; the first evaluation tag is either a correct response tag or a prompt word optimization tag. For each test sample in the relevant domain group, an intent cross-comparison is performed on the question text, recall content, and response text in the test sample, or the second evaluation label of the test sample is determined by judging whether the response text of the test sample conforms to the response persona; wherein, the second evaluation label includes at least one of the following: correct response label, correct recall label, prompt word optimization label, recall optimization label, information fusion optimization label, and corpus lack label; Based on the number of test samples, the number of correct response labels, and the number of correct recall labels, the response accuracy and recall accuracy are calculated respectively. Based on the first evaluation label or the second evaluation label of each test sample, corresponding optimization processing is performed to obtain the optimized question-answering system; The step of determining the second evaluation label of the test sample by performing intent cross-comparison on the question text, recall content, and response text in the test sample, or by determining whether the response text of the test sample conforms to the response persona, includes: Based on the question text or the question-answer pair composed of the question text and the answer text, a similarity match is performed with several knowledge fragments in the knowledge base to obtain a matching result; the matching result includes the similarity corresponding to several matched knowledge fragments. If at least one matching knowledge fragment in the matching results has a similarity exceeding a preset first threshold, then the question text, the recall content, and the response text are cross-referenced to determine the second evaluation label of the test sample. If the similarity of all matching knowledge fragments in the matching results does not exceed the first threshold, then the second evaluation label of the test sample is determined based on the recalled content or the response text. The step of determining the second evaluation label of the test sample based on the recall content or the response text includes: If the recalled content of the problem text exists, then the second evaluation label of the test sample is confirmed to include the recall optimization label and the corpus lack label; If the recalled content of the question text does not exist, it is determined whether the response text matches the response persona. If yes, the second evaluation label of the test sample is confirmed to be the correct response label and the corpus lack label. If no, the second evaluation label of the test sample includes the prompt word optimization label and the corpus lack label.

2. The RAG-based question-answering system optimization method according to claim 1, characterized in that, The step of performing intent cross-comparison on the question text, the recall content, and the response text to determine the second evaluation label of the test sample includes: Obtain the first intent, second intent, and third intent corresponding to the question text, the recall content, and the response text, respectively; If the first intent is unrelated to the second intent, then the second evaluation label of the test sample is confirmed as the recall optimization label; If the first intent, the second intent, and the third intent are correlated in pairs, then the second evaluation label of the test sample is confirmed to include a correct recall label and a correct response label. If the first intent and the second intent are related and the third intent is not related to either the first intent or the second intent, then the second evaluation label of the test sample is confirmed to include a correct recall label and an information fusion optimization label.

3. The RAG-based question-answering system optimization method according to claim 1, characterized in that, The question-answering system includes a knowledge base, which includes a vector database and a document database. The step of performing corresponding optimization processing based on the first evaluation label or the second evaluation label of each test sample to obtain the optimized question-answering system includes: performing corpus expansion processing on the knowledge base based on the first question text in each first test sample whose second evaluation label includes the corpus lack label. This step includes: For each of the first question texts, obtain the original files related to the first question text; The original file is preprocessed to obtain plain text information; Determine at least one topic covered by the plain text information, divide the plain text information into at least one data field, and determine the text ID corresponding to each document in each data field. A data field includes the text content corresponding to a topic. Each document is divided into multiple sub-segments, and each sub-segment is converted into a segment vector. A mapping relationship is established based on each segment vector and its corresponding text ID. Each segment vector and its corresponding text ID are written into the vector database, and each document and its text ID are written into the document database. Entity recognition, relation extraction, and attribute extraction are performed based on the text content of each data domain to obtain at least one triplet corresponding to each data domain, and a knowledge graph is constructed based on at least one triplet corresponding to each data domain. The knowledge base is updated based on the vector database, the document database, and the knowledge graph after they have been written.

4. The RAG-based question-answering system optimization method according to claim 1 or 2, characterized in that, The method further includes: Upon receiving an evaluation instruction, the evaluation results corresponding to the test set are obtained; the evaluation results include response accuracy, recall accuracy, and a first evaluation label or a second evaluation label for each test sample; Upon receiving an update instruction, the test set is updated based on the updated knowledge base, updated prompts, updated recall strategy, updated information fusion strategy, and question-answering big model to obtain the updated test set. The updated test set is used as the new test set. The process returns to the step of obtaining the evaluation result corresponding to the test set when the evaluation instruction is received, and a new evaluation result is obtained. This continues until a quantization instruction is received, at which point an optimized quantization result is generated based on all the obtained evaluation results.

5. The RAG-based question-answering system optimization method according to claim 4, characterized in that, The step of updating the test set based on the updated knowledge base, updated prompt words, updated recall strategy, and question-answering big model to obtain an updated test set includes: for the first question text in each first test sample whose second evaluation label includes the missing label of the corpus, determining new recall content for the first question text from the updated knowledge base, and generating a new response text for the first question text based on the first question text and its new recall content by calling the question-answering big model; And / or, For the second question text in each second test sample where the first evaluation label is a prompt word optimization label or the second evaluation label includes a prompt word optimization label, based on the updated prompt words and the second question text and its recall content, the question-answering big model is invoked to generate a new response text for the second question text. And / or, For the third question text in each third test sample including the recall optimization label of the second evaluation label, the updated recall strategy is used to determine the new recall content of the third question text, and based on the third question text and its new recall content, the question-answering big model is called to generate a new response text for the third question text; And / or, For cases where the second evaluation label includes the information fusion optimization label, the question-and-answer big model is fine-tuned to obtain an updated question-and-answer big model. The updated question-and-answer big model is then used to generate a new response text for the fourth question text in each fourth test sample corresponding to the information fusion optimization label.

6. The RAG-based question-answering system optimization method according to claim 5, characterized in that, The step of determining the new recall content of the third question text using the updated recall strategy includes: The text of the third question is matched with several knowledge fragments in the knowledge base to obtain the recall results; the recall results include multiple recalled fragments whose similarity exceeds the first threshold. The third question text is compared with each of the recalled fragments to perform topic consistency detection, so as to filter out each recalled fragment with inconsistent topics; If the remaining recall fragments are zero, then the new recall content is confirmed to be empty; If the remaining recalled fragments are not zero and the third question text belongs to a special knowledge domain, then search the document database for the parent text that matches all the remaining recalled fragments, and use the parent text as the new recalled content; If the remaining recalled fragments are not zero and the third question text does not belong to a special knowledge domain, then the new recalled content is determined based on the total number of words in all remaining recalled fragments.

7. The RAG-based question-answering system optimization method according to claim 5, characterized in that, The steps for fine-tuning the large question-answering model to obtain the updated large question-answering model include: Determine whether the number of information fusion optimization tags or the timestamp of the first information fusion optimization tag meets the triggering condition for fine-tuning the question-answering model. The triggering condition is that the ratio of the number of information fusion optimization tags to the number of samples in the test set exceeds a first preset threshold, or the timestamp of the first information fusion optimization tag is more than a second preset threshold. When the triggering condition is met, a high-quality preprocessed corpus to be fine-tuned is obtained. The corpus to be fine-tuned includes a general domain corpus set and a professional domain corpus set that reach a preset ratio. Each training sample in the corpus to be fine-tuned contains a fifth question text, a fifth recall text, a fifth response text, and a fifth prompt word. The iterative steps are as follows: for each training sample, the corresponding fifth question text, fifth recall text, and fifth prompt word are used as input text for the question-answering model to obtain the predicted text; the loss value is calculated based on the predicted text and the fifth response text, and the parameters of the question-answering model are updated based on the loss value until the convergence condition is met to obtain the updated question-answering model.

8. A question-answering system optimization device based on RAG, characterized in that, include: The acquisition module is used to acquire a test set including several test samples, wherein the test samples include question text, recall content and response text; The grouping module is used to divide all test samples into relevant domain groups and unrelated domain groups based on whether each question text belongs to the target knowledge domain; The first evaluation module is used to determine the first evaluation label of each test sample in the unrelated domain group by judging whether the response text of the test sample conforms to the preset response persona. The first evaluation label is either a correct response label or a prompt word optimization label; The second evaluation module is used to perform intent cross-comparison on the question text, recall content, and response text in each test sample in the relevant domain group, or to determine the second evaluation label of the test sample by judging whether the response text of the test sample conforms to the response persona; wherein the second evaluation label includes at least one of the following: correct response label, correct recall label, prompt word optimization label, recall optimization label, information fusion optimization label, and corpus lack label; The calculation module is used to calculate the response accuracy and recall accuracy based on the number of test samples, the number of correct response tags, and the number of correct recall tags, respectively. The optimization module is used to perform corresponding optimization processing based on the first evaluation label or the second evaluation label of each of the test samples to obtain the optimized question-answering system. Specifically, the second evaluation module, in the process of determining the second evaluation label of the test sample by performing intent cross-comparison on the question text, recall content, and response text in the test sample, or by judging whether the response text of the test sample conforms to the response persona, is used for: Based on the question text or the question-answer pair composed of the question text and the answer text, a similarity match is performed with several knowledge fragments in the knowledge base to obtain a matching result; the matching result includes the similarity corresponding to several matched knowledge fragments. If at least one matching knowledge fragment in the matching results has a similarity exceeding a preset first threshold, then the question text, the recall content, and the response text are cross-referenced to determine the second evaluation label of the test sample. If the similarity of all matching knowledge fragments in the matching results does not exceed the first threshold, then the second evaluation label of the test sample is determined based on the recalled content or the response text. Specifically, the second evaluation module, in the process of determining the second evaluation label of the test sample based on the recall content or the response text, is used for: If the recalled content of the problem text exists, then the second evaluation label of the test sample is confirmed to include the recall optimization label and the corpus lack label; If the recalled content of the question text does not exist, it is determined whether the response text matches the response persona. If yes, the second evaluation label of the test sample is confirmed to be the correct response label and the corpus lack label. If no, the second evaluation label of the test sample includes the prompt word optimization label and the corpus lack label.

Images