A test optimization method and a test optimization system for an intelligent question-answering system

Abstract

The application provides a test optimization method of an intelligent question and answer system, comprising the following steps: constructing an intelligent question and answer system based on business documents in a knowledge base, the intelligent question and answer system being used for receiving user input to generate a target answer; determining a first type of question and answer and a second type of question and answer, wherein the first type of question and answer is a question and answer pair generated based on document information of the business documents, and the second type of question and answer is a question and answer pair generated based on the first type of question and answer or the document information; testing the intelligent question and answer system in a hierarchical progressive test mode based on the first type of question and answer and the second type of question and answer to obtain a verification result; and updating and optimizing the knowledge base and the intelligent question and answer system according to the verification result. The above method can construct a multi-dimensional and all-round test optimization system corresponding to an RAG system, not only pays attention to the accuracy, consistency and other indexes of the generated result, but also includes the retrieval recall rate and other indexes in the evaluation range, so that the comprehensive and objective evaluation of the comprehensive performance of the RAG system is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a testing optimization method and system for an intelligent question-answering system. Background Technology

[0002] With the continuous development of Large Language Models (LLM), Retrieval Augmented Generation (RAG) systems that integrate with external knowledge bases are increasingly being applied in business areas such as intelligent customer service, customer support, and enterprise office work. The performance of a question-answering system (intelligent question-answering system) based on RAG depends on the coverage and accuracy of the knowledge base; therefore, an efficient and reliable testing scheme is needed to verify the actual effectiveness of the RAG system. However, the currently commonly used static test set verification method suffers from problems such as static test data and insufficient scenario coverage. Therefore, a comprehensive and dynamic testing optimization scheme for RAG systems is needed. Summary of the Invention

[0003] This invention provides a testing and optimization method for intelligent question-answering systems to solve the technical problems of static test data and insufficient scenario coverage when using the current static test set verification method to test and optimize intelligent question-answering systems.

[0004] The first aspect of the present invention provides a testing and optimization method for an intelligent question-answering system, comprising: constructing an intelligent question-answering system based on business documents in a knowledge base, wherein the intelligent question-answering system is used to receive user input and generate target answers; determining a first type of question-answer and a second type of question-answer, wherein the first type of question-answer is a question-answer pair generated based on document information of the business documents, and the second type of question-answer is a question-answer pair generated based on the first type of question-answer or document information; testing the intelligent question-answering system using a hierarchical progressive testing method based on the first type of question-answer and the second type of question-answer, and obtaining verification results; and updating and optimizing the knowledge base and the intelligent question-answering system according to the verification results.

[0005] The first type of question and answer and the second type of question and answer here can represent static fixed question and answer and dynamic question and answer, respectively.

[0006] As can be seen, the test optimization method proposed in this application can construct a multi-dimensional and comprehensive test optimization system for the RAG system. It not only focuses on indicators such as the accuracy and consistency of the generated results, but also incorporates indicators such as retrieval recall into the evaluation scope, achieving a comprehensive and objective evaluation of the overall performance of the RAG system. A feedback mechanism is established between the test environment and the production environment. By collecting online user interaction data (such as negative feedback, dislikes, and invalid answer feedback), it identifies knowledge blind spots and low retrieval recall issues in the RAG system in real time, providing real and effective data support for system optimization and facilitating the continuous iterative upgrade of the RAG system.

[0007] In one embodiment of the present invention, the first type of question and answer is used to describe questions and answers about document information, and the second type of question and answer is used to describe questions and answers about semantic extension information or semantic interference information corresponding to the document information.

[0008] In one embodiment of the present invention, the intelligent question-answering system is tested using a hierarchical progressive testing method based on a first type of question-answer and a second type of question-answer to obtain verification results. This includes: calculating the retrieval recall and precision between the target answer generated by the intelligent question-answering system and the answers included in the first type of question-answer and / or the second type of question-answer; and using the comparison results of the retrieval recall and precision with preset test thresholds as verification results. The retrieval recall describes the hit rate of the target answer among the answers included in the first type of question-answer and / or the second type of question-answer, and the precision describes the consistency between the target answer and the answers included in the first type of question-answer and / or the second type of question-answer. The preset test thresholds corresponding to the first type of question-answer and the second type of question-answer are different.

[0009] In one embodiment of the present invention, a hierarchical progressive testing method is used to test the first type of question and answer and the second type of question and answer to obtain the verification result. The method further includes: receiving user questions, calculating the user downvote rate corresponding to the first type of question and answer and the second type of question and answer for the user questions; and using the comparison result of the user downvote rate and the preset downvote threshold as the verification result. The user downvote rate is used to describe the number of negative feedbacks from the user to the final answer obtained from the first type of question and answer and the second type of question and answer after receiving the user questions.

[0010] In one embodiment of the present invention, the second type of question and answer is generated in one or more of the following ways: performing type identification and semantic parsing on the business document, and combining it with a preset question and answer type template to generate a second type of question and answer that covers the rule explanation, scope of application or execution process of the business document; and generating a second type of question and answer based on the first type of question and answer by supplementing constraints, transforming question types or adding interference information.

[0011] In one embodiment of the present invention, the accuracy is calculated as follows: semantic similarity analysis and sentiment consistency analysis are performed on the target answer and the answers included in the first type of question and / or the second type of question and answer, and the accuracy is calculated according to the scoring rules. The semantic similarity analysis is used to describe the semantic consistency between the final answer and the answers included in the first type of question and / or the second type of question and answer, and the sentiment consistency analysis is used to describe the consistency in sentiment tendency between the final answer and the answers included in the first type of question and / or the second type of question and answer.

[0012] In one embodiment of the present invention, the verification result includes at least one of missing information, expression deviation, or logical error.

[0013] In one embodiment of the present invention, updating and optimizing the knowledge base and the intelligent question-answering system includes at least one of supplementing missing semantic units in the knowledge base, correcting or discarding semantic units in the knowledge base, updating business documents, and adjusting the parameters of the intelligent question-answering system.

[0014] A second aspect of the present invention provides a testing and optimization system for an intelligent question-answering system, comprising: a knowledge base module, a question-answer generation module, and a testing module, wherein the knowledge base module is used to store business documents; the question-answer generation module is used to construct first-type and second-type question-answers corresponding to the business documents in the knowledge base; the testing module is used to test the first-type and second-type question-answers using a hierarchical progressive testing method to obtain verification results; and the knowledge base module and the intelligent question-answering system are updated and optimized based on the verification results.

[0015] A third aspect of the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the test optimization method provided in the first aspect.

[0016] The beneficial effects of this invention are as follows: The test optimization method of this application enables the construction of a multi-dimensional and comprehensive test optimization system for the RAG system. It not only focuses on indicators such as the accuracy and consistency of generated results but also incorporates indicators such as retrieval recall into the evaluation scope, achieving a comprehensive and objective evaluation of the overall performance of the RAG system. A feedback mechanism is established between the test environment and the production environment. By collecting online user interaction data (such as negative feedback, dislikes, and invalid answer feedback), it identifies knowledge blind spots and low retrieval recall issues in the RAG system in real time, providing real and effective data support for system optimization and facilitating continuous iterative upgrades of the RAG system. The invention significantly improves the testing efficiency of the RAG system, enabling rapid and large-scale robustness verification after large-scale knowledge base updates, effectively reducing testing costs and shortening the testing cycle, adapting to the application requirements of rapid iteration of the RAG system, and thus ensuring the reliability and stability of the RAG system. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0018] In the attached diagram:

[0019] Figure 1 This is a schematic diagram illustrating a scenario using an intelligent question-answering system according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a testing and optimization method for an intelligent question-answering system provided in one embodiment of the present invention. Figure 3 This is a flowchart illustrating a testing and optimization method for an intelligent question-answering system provided in one embodiment of the present invention. Figure 4 This is a structural diagram of a test optimization system provided in one embodiment of the present invention; Figure 5 This is a structural diagram of an electronic device for a runtime testing and optimization system provided in an embodiment of the present invention. Detailed Implementation

[0020] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0021] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The drawings only show components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the structure, connection relationship, quantity and proportion of each component can be arbitrarily changed, and the component layout may also be more complex.

[0022] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0023] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of a RAG-based enterprise office system 10 provided in an embodiment of the present invention, such as... Figure 1As shown, employees can use a question-and-answer format to generate / provide the financial reimbursement process and precautions for Company A through the question-and-answer module 101 of the enterprise office system 10. These financial reimbursement processes and precautions can be extracted from routine questions and answers in financial-related business documents stored in the knowledge base module 102. For example, if the question-and-answer module 101 can only generate the answer "Accommodation receipts and transportation receipts" for "What receipts are required for employee travel expense reimbursement?", then the module may not be able to correctly answer the employee's question, "Can I get partial reimbursement for travel expenses if I have transportation receipts but no accommodation receipts?" This indicates that the enterprise office system 10 has insufficient scenario coverage. If maintenance personnel can only manually prepare questions and answers to test the question-and-answer module 101 of the enterprise office system 10, or manually optimize the module, conflicts, errors, inefficiencies, and untimely optimizations will occur between the module and the business documents stored in the knowledge base module 102.

[0024] To address the issues of static data, insufficient scenario coverage, and low testing efficiency in existing RAG systems, this invention aims to provide a testing optimization method for an intelligent question-answering system based on a RAG system. The method includes: loading an intelligent question-answering system based on business documents in a knowledge base, which can receive user input to determine the target answer; generating static fixed questions and answers and dynamic questions and answers; constructing static fixed questions and answers based on the core information of the corresponding business documents in the knowledge base; generating dynamic questions and answers based on the business documents in the knowledge base and static fixed questions and answers, used to cover interfering information and complex business scenarios in the knowledge base; and testing the intelligent question-answering system using a layered and progressive testing approach based on the static fixed questions and answers and dynamic questions and answers, including: verifying the retrieval recall rate, accuracy, and user dislike rate of the answers generated by the intelligent question-answering system according to the questions corresponding to the received static fixed questions and dynamic questions and answers; collecting the verification results of each testing stage; and updating and optimizing the intelligent question-answering system based on the verification results.

[0025] As can be seen, the aforementioned testing and optimization of the RAG system enables the construction of a multi-dimensional and comprehensive RAG system testing and optimization system. This system not only focuses on metrics such as the accuracy and consistency of generated results but also incorporates metrics like retrieval recall into the evaluation scope, achieving a comprehensive and objective assessment of the overall performance of the RAG system. Establishing a closed-loop feedback mechanism between the testing and production environments, by collecting online user interaction data (such as negative feedback, dislikes, and invalid responses), allows for real-time identification of knowledge blind spots and low retrieval recall issues within the RAG system. This provides real and effective data support for system optimization, facilitating continuous iterative upgrades of the RAG system. It significantly improves the testing efficiency of the RAG system, enables robust verification of the knowledge base, effectively reduces testing costs and shortens the testing cycle, adapts to the rapid iteration requirements of the RAG system, and ultimately ensures the reliability and stability of the RAG system in various application fields.

[0026] See Figure 2 , Figure 2 This is a flowchart illustrating a testing and optimization method for an intelligent question-answering system corresponding to the RAG system, provided in an embodiment of the present invention. The testing and optimization method includes the following steps.

[0027] Step S201: Obtain the knowledge base of the RAG system.

[0028] In some embodiments, the knowledge base here refers to a structured carrier deployed on a business system that supports retrieval and updating. Taking an enterprise system as an example, the knowledge base can be formed by filtering, annotating, and structuring various business documents within the enterprise. Compared to the knowledge base, an intelligent question-and-answer system (also known as a question-and-answer database) can generate answers to questions asked by enterprise employees about various business documents. These business documents can include rules and regulations, management methods, operating procedures, policy documents, contract terms, business guidelines, exam question banks, training materials, etc. Business documents can undergo unified preprocessing, be broken down into multiple semantic units (knowledge fragments), and undergo standardized structural parsing to identify headings, chapters, clauses, tables, annotations, exceptions, etc., forming a hierarchical and machine-understandable structured knowledge system. Each semantic unit is annotated, including: basic metadata (e.g., document name, number, version, effective status, issuing department, etc.), business tags (e.g., business domain, applicable scenarios, etc.), and semantic information (e.g., constraints, scope of application, prohibited items, etc.). It can be understood that the above semantic units can constitute an association mapping between questions and answers corresponding to business documents, which is stored in the knowledge base.

[0029] Step S202: For business documents in the knowledge base, build a question-answering library for generating static fixed questions and answers and dynamic questions and answers.

[0030] In some embodiments, the static fixed Q&A can be questions and answers provided by the author of the business document. The static fixed Q&A can be common question-and-answer pairs (also called QA pairs) related to the business document, and can involve basic information (core information) corresponding to the business document. Each business document can have 15 to 30 QA pairs, and each QA pair can include a question description and a standard answer, ensuring the consistency between the static fixed Q&A and the core content of the business document, providing a benchmark reference for subsequent testing and optimization.

[0031] Dynamic question answers can be generated in three ways: semantic parsing based on business documents in a knowledge base, extended generation based on static fixed question answers, and generation by extracting a question bank from business documents. Both static fixed question answers and dynamic question answers can appear as question-and-answer pairs.

[0032] Specifically, for semantic parsing and generation based on business documents in a knowledge base, the process can include the following steps: First, identify and classify each business document to be tested in the knowledge base. Second, predefine corresponding question-and-answer type templates for different types of business documents (rules and regulations, management methods, operating procedures, etc.). For example, for rules and regulations, the question-and-answer types should comprehensively cover core dimensions such as rule explanation, scope of application, execution process, specific clause requirements, and typical case scenarios. Third, combine a Large Language Model (LLM) to perform semantic parsing based on the document content and the pre-defined question-and-answer type templates, generating a certain number of dynamic questions and answers. It can be seen that the dynamic questions and answers generated in this way can include the logical relationship questions and answers of the business documents, simulating and reproducing real user interactions in complex business scenarios.

[0033] Specifically, regarding the extended generation based on static fixed question-and-answer (Q&A), this means using manually constructed static fixed Q&A as the basic data source and performing element analysis and mining on the static fixed Q&A. This can include the following processes: Element supplementation and improvement: Supplementing the constraints in the identified or parsed static fixed Q&A, completing and strengthening any missing or ambiguous conditions to improve the logical rigor of the Q&A. Question type transformation: Transforming the question types corresponding to the questions in the static fixed Q&A. For example, transforming a forward operational question like "How to execute the XX process" into a reverse accountability question like "What risks will occur if the XX process is not executed," expanding the depth and breadth of dynamic Q&A. Adding distracting information: Adding ambiguous or distracting information based on the semantics of the questions in the static fixed Q&A to simulate real-world scenarios such as ambiguous user questions and mixed intentions, increasing the difficulty / deceptiveness of the Q&A.

[0034] Specifically, regarding the extraction and generation of a question bank for business documents, if the enterprise system is configured with an internal examination system or compliance training system, it can retrieve question bank data from these systems to identify questions related to the business documents. Through semantic comparison and filtering, it can extract compliance questions and answers that are well-written, accurate, and logically rigorous, and convert them into static fixed questions and answers or dynamic questions and answers. It can be seen that this method fully utilizes existing resources, providing a large number of correct question and answer examples for the knowledge base.

[0035] Step S203: Perform layered testing and optimization based on the static fixed questions and answers and dynamic questions and answers of the business documents.

[0036] In some embodiments, to verify the data capabilities, scenario capabilities, and real-world application capabilities of the RAG-based intelligent question-answering system, a layered testing approach is adopted. The intelligent question-answering system is tested in multiple stages according to the test scenarios, ranging from simple to complex, from static to dynamic, and from the test environment to the production environment. The inputs, verification objectives, and admission conditions for each stage are shown in Table 1 below.

[0037]

[0038] Table 1 As can be seen from Table 1 above, the first and second stages belong to the offline verification stage in the test environment, while the third stage belongs to the online verification stage in the production environment. The first and second stages are used to verify the basic retrieval capabilities, generation accuracy, and robustness to complex scenarios of the RAG system, ensuring the correctness of the corresponding functions. The third stage is used to verify the stability, usability, and real user satisfaction of the RAG system. During the testing and optimization process in the first and second stages, the question-and-answer system can be tested by calculating retrieval recall and precision.

[0039] Recall, as used here, is a quantitative metric measuring the effectiveness of a Retrieval and Answering (RAG) system in the retrieval process. It reflects the RAG system's ability to successfully retrieve and return semantic units containing the standard answer to a user's question (user input, such as questions from static or dynamic question-and-answer systems, or similar questions) after receiving the user's question. This standard answer can be from static or dynamic question-and-answer systems. In other words, recall is the foundation for generating answers: only by successfully retrieving and recalling correct and relevant semantic units can an accurate answer be generated; if recall fails, incorrect or off-topic answers will inevitably occur. Recall can quickly identify the root causes of problems in the RAG system. A low recall indicates problems in the retrieval strategy, the parameters of the intelligent question-and-answer system (which describe the semantic relationships and contextual dependencies involved in the answer), and knowledge coverage. Conversely, if the recall is adequate but accuracy is low, the problem lies in the answer generation and logical reasoning processes. Referring to Table 1, the first stage (fixed question answering) requires a retrieval recall rate of ≥90%, ensuring that the RAG system has stable recall capabilities for basic information. The second stage (dynamic question answering) requires a retrieval recall rate of ≥80%, ensuring that the RAG system still has stable recall capabilities for information in complex scenarios, with multiple intents, and ambiguous questions. It is understandable that for issues with low retrieval recall, subsequent steps can directly address knowledge blind spots in the knowledge base, knowledge gaps, unreasonable indexing, or mismatches between the retrieval / generation rules of the intelligent question answering system. Low recall issues can be redirected back to the construction and optimization phase of the intelligent question answering system, forming a closed-loop iteration of "testing-positioning-optimization-retesting" to continuously improve the overall system performance.

[0040] The accuracy rate here represents the ratio of consistency and accuracy between the final answer output by the RAG system (AI answer) and the standard answer (response in static fixed questions or dynamic questions). It reflects the RAG system's understanding of the question and its ability to categorize, integrate, summarize, and generate answers, and is a quantitative indicator of whether the final answer is usable and accurate.

[0041] In some embodiments, the accuracy rate can be determined using the following scoring criteria: semantic similarity analysis, sentiment consistency judgment, and comprehensive scoring rules. Specifically, for semantic similarity analysis, a semantic similarity comparison method can be used to compare the final answer with the standard answer, outputting a similarity score in the range of 0 to 1. Here, 1 point indicates complete consistency in core meaning and key information; 0.8 points indicate consistency in main information, with only differences in expression and sentence structure; 0.6 points indicate partial information matching, with minor missing or biased information; and 0.4 points and below indicate largely irrelevant content and mismatched core information. Sentiment consistency judgment identifies and compares the sentiment tendencies of the final answer and the standard answer. The judgment dimensions include: sentiment category: positive, negative, neutral; sentiment intensity: strong, medium, weak. The output result is either consistent (true) or inconsistent (false). The comprehensive scoring rule uses semantic similarity and sentiment consistency as the basis for the final score of each question and answer. If semantic similarity ≥ 0.8 and sentiment consistency is achieved, the comprehensive score is 1; if semantic similarity ≥ 0.6 and sentiment consistency is achieved, the comprehensive score is 0.5; otherwise, the comprehensive score is 0. The RAG system can calculate the accuracy rate of each question and answer based on the comprehensive score of all questions and answers in the knowledge base, serving as a quantitative indicator for the first and second stages. As shown in Table 1, the first stage (fixed questions and answers) requires an accuracy rate ≥ 95%; the second stage (dynamic questions and answers) requires an accuracy rate ≥ 90%. The aforementioned 95%, 90%, and 80% can be referred to as test thresholds, and these values ​​are exemplary; any values ​​can be used in this application embodiment, and no restrictions are imposed here.

[0042] It is understandable that the above formula for calculating the retrieval recall rate can be: Retrieval Recall Rate = Number of successfully retrieved answers / Total number of tests × 100%; the formula for calculating the precision rate can be: Precision Rate = (Number of answers with a score of 1 + 0.5 × Number of answers with a score of 0.5) / Total number of tests × 100%.

[0043] The following is a further description of sentiment consistency judgment. Sentiment consistency judgment is used to standardize the analysis and comparison of the sentiment tendency and stance intensity of the standard answer and the final answer, avoiding unqualified answers with similar semantics but opposite stances / attitudes. The specific implementation process is as follows: Using sentiment classification methods, the sentiment tendency and sentiment intensity related to the business document are determined. The standard answer and the final answer are analyzed separately, and the corresponding sentiment features are extracted. Here, sentiment tendency can include neutral (objective statement), positive (permission, recommendation, support), and negative (prohibition, warning, risk, denial), and sentiment intensity can be strong, medium, or weak. Two comparisons are performed for sentiment tendency and sentiment intensity. Sentiment tendency comparison is used to determine whether the sentiment tendency of the standard answer and the final answer is the same. Sentiment intensity comparison, under the premise that the sentiment tendency is consistent, determines whether the sentiment intensity is in the same range. If the sentiment tendency is consistent and the sentiment intensity matches, it is judged as sentiment consistency (true); if the sentiment tendency conflicts (e.g., the standard answer is "prohibition / negative" and the final answer is "permission / positive"), or the intensity is seriously mismatched → it is judged as sentiment inconsistency (false).

[0044] For example, taking the Q&A section of a knowledge base related to corporate financial reimbursement as an example: Example 1: The standard answer is that employees must provide formal invoices for travel expense reimbursement; reimbursement will not be granted without an invoice. Affective tendency: negative (prohibited, restrictive); Affective intensity: strong. Final answer: Employees need formal invoices for travel expense reimbursement; reimbursement is not possible without an invoice. Affective tendency: negative (prohibited, restrictive); Affective intensity: strong. Judgment: Affective tendency and affective intensity match; the result is affective consistency (true). Example 3: Standard answer: Reimbursement for personal expenses unrelated to work is strictly prohibited. Affective category: negative (strictly prohibited, forbidden); Affective intensity: strong. Final answer: Personal expenses unrelated to work may be reimbursed at the discretion of the authorities. Affective category: positive (allowed, permitted); Affective intensity: moderate. Judgment: Affective categories are completely opposite; the result is affective inconsistency (false).

[0045] It can be seen that sentiment consistency judgment can avoid erroneous answers that are "semantically similar but have completely opposite stances / attitudes" (e.g., the original text of a business document is "illegal reimbursement is prohibited," but the final answer is "discretionary reimbursement is permissible"). This solves the compliance problem of traditional assessments that only look at textual similarity and ignore stance / attitude. It enables the RAG system to not only focus on "whether the answer is correct" but also on "whether the stance is accurate and the tone is reasonable," incorporating sentiment consistency into the test and improving the fit between the test and real-world scenarios. At the same time, sentiment consistency judgment and semantic similarity together constitute a dual verification, changing the single-dimensional deficiency of RAG system testing that only focuses on content accuracy, and achieving a more comprehensive and rigorous test of generated quality.

[0046] For the third stage, in addition to calculating retrieval recall and precision, the user dislike rate can also be used to test the corresponding questions and answers in the knowledge base. Here, the user dislike rate refers to the proportion of negative feedback (dislikes, dissatisfaction, invalid feedback, etc.) that users give to the final answer in a production environment, out of the total number of valid answers (e.g., threshold: <5%). The final answer can be generated by the intelligent question-answering system, or it can be a static fixed question-answer or an answer included in dynamic question-answering. The user dislike rate is a quantitative indicator for questions and answers in the third stage, i.e., in the production environment. It can directly reflect the actual satisfaction of real users with the quality of questions and answers, and measure the actual service capability of the RAG system in a production environment (real-world scenario). For example, the formula for calculating the user downvote rate can be: User downvote rate = Number of downvotes ÷ Total number of answers × 100%. Continuing with the example of questions and answers related to corporate financial reimbursement in the knowledge base, regarding the requirement that employees must provide formal invoices for travel expense reimbursement, and that reimbursement will not be granted without invoices, the total number of answers is 2000. If users downvote 100 times because they find the answers incorrect, incomplete, or non-compliant, then the user downvote rate is 100 ÷ 2000 = 5%. At this point, the user downvote rate of 5% does not meet the threshold (<5%), indicating insufficient user satisfaction. The RAG system needs to optimize the above questions and answers.

[0047] It can be seen that further testing the RAG system's question-and-answer system in a production environment using user downvote rates can overcome the limitations of relying solely on models or algorithms in the first and second stages. Using real user downvote behavior more directly reflects the RAG system's actual performance in ambiguous questions, colloquial questions, and business scenarios, solving the problem of the disconnect between testing scenarios and real-world use. It transforms the subjective feelings of real users into statistically comparable and traceable quantitative indicators, making the RAG system's question-and-answer system more objective and persuasive. Questions and answers with unsatisfactory downvote rates can be directly identified as low-quality answers and returned to the question-and-answer generation stage for optimization, further identifying issues such as knowledge gaps, retrieval failures, and generation anomalies. Simultaneously, user downvote rates can also proactively expose and fix various potential problems in the production environment, reducing user complaints, manual review, and emergency repair costs, ensuring long-term stable system operation, and supporting the smooth operation of the RAG system within the enterprise.

[0048] Through the above Figure 2 The described test optimization method for the RAG system can filter out questions and answers that match various business documents and those that require further optimization. For questions and answers requiring optimization, it can be done through methods such as... Figure 3 The described process is optimized.

[0049] Step S301: Real-time monitoring and collection of user behavior.

[0050] In some embodiments, user behavior can be monitored and collected in real time in the production environment. The collection dimensions may include: explicit feedback behaviors such as users liking or disliking the final answer, as well as abnormal interruption nodes, repeated questions, and invalid follow-up questions in the dialogue process with the system. The above-mentioned data indicators for questions and answers are input into a dedicated testing and optimization system to form a sample pool of the correspondence between questions and answers and data indicators, providing real data support for subsequent root cause analysis and optimization.

[0051] Step S302: Analyze the causes and address the problem.

[0052] In some embodiments, the collected data metrics can be analyzed, categorized into two types, and processed separately. For example, data metrics can represent low-quality and high-quality answers. For low-quality answers, i.e., questions and answers corresponding to user downvotes, abnormal interruptions, and invalid follow-up questions, semantic clustering can be performed to identify high-frequency questions and answers and common problems, further determining the root cause of the problem. For example, low retrieval recall may correspond to problems such as missing information in the knowledge base or incomplete knowledge points; normal retrieval recall but low precision may correspond to problems such as expression bias, logical errors, and answer integration. Problem handling methods include: automatically generating knowledge base update suggestions, including missing questions and answers, errors in business documents, conflicts in business content, etc., prompting writers or knowledge base maintenance personnel to perform maintenance. For high-quality answers, these questions and answers can be used as static questions and answers, serving as a high-quality data source for subsequent basic testing, model iteration, and dynamic expansion.

[0053] Step S303: Update and optimize the knowledge base.

[0054] In some embodiments, knowledge base maintenance personnel can also update and optimize the knowledge base based on the various issues raised in step S302. Update and optimization operations may include: supplementing missing semantic units in the knowledge base, correcting or discarding semantic units in the knowledge base; updating business documents; adjusting parameters of the intelligent question-answering system, etc., and then executing the update / optimization operation again. Figure 2 The description describes a test optimization method based on the RAG system.

[0055] against Figure 2 Step S202 in the described test optimization method involves generating static fixed question answers and dynamic question answers. The specific implementation process of generating dynamic question answers based on the extension of static fixed question answers may include: Step S2021: Preprocess static fixed questions and answers.

[0056] In some embodiments, step S2021 preprocesses the input static fixed questions and answers to provide standardized basic data for subsequent expansion, avoiding expansion deviations caused by non-standard static fixed questions and answers. This includes: firstly, filtering the static fixed questions and answers to remove invalid data, including but not limited to: vague question descriptions, incomplete answers, mismatches between questions and answers, typos, or grammatical errors; retaining clearly described and accurately answered static fixed questions and answers as the data source for generating dynamic questions and answers. Secondly, using a semantic similarity algorithm, static fixed questions and answers with semantic repetition or high similarity are removed to avoid generating duplicate dynamic questions and answers in subsequent expansions, thereby improving expansion efficiency and use case diversity.

[0057] Step S2022: Analyze static fixed questions and answers.

[0058] In some embodiments, step S2022 serves to mine the core information of static fixed questions and answers, deconstruct the inherent relationship between questions and answers, provide data support for subsequent expansion of dynamic questions and answers, and ensure that the expansion process does not deviate from the semantics of static fixed questions and answers. This includes: parsing and extracting question elements, such as core demands, constraints, question types, business entities, etc., where core demands represent the core information that users want to obtain through the question (such as "query process", "confirmation requirements", "scope definition", etc.); constraints represent the limited information explicitly mentioned in the question (such as time, subject, scenario, value, scope, etc.); question types represent the types classified based on core demands (such as query type, judgment type, risk type, explanation type, etc.); core entities represent the business entities involved in the question (such as "reimbursement process", "approval authority", "invoice requirements", etc.); extracting elements from the answer and extracting answer elements that match the question, including: the core conclusion corresponding to the core demand, the basis supporting the core demand (such as document clause number, specific rules), and key entities, to ensure that the answers generated by subsequent expansion can accurately match the question elements. Organize the relationships between questions and answers, establish a one-to-one correspondence between question elements and answer elements, generate a relationship table between questions and answers, and clarify the answer elements corresponding to each question element, so as to provide a basis for consistency verification between questions and answers in the subsequent expansion process.

[0059] Step S2023: Expand the generation of dynamic questions and answers.

[0060] In some embodiments, step S2023 is based on the parsed question elements and answer elements, and expands the static fixed questions and answers through three methods: "element supplementation and improvement, question type transformation, semantic expansion, and semantic interference" to generate multiple sets of questions and answers. The specific execution process is as follows: Element supplementation and improvement: Supplementing the missing or ambiguous constraints in the static fixed questions and answers, simulating the multi-constraint scenario when users ask questions, while ensuring that the supplemented elements conform to the business documents, comparing the constraints of the static fixed questions and answers with the complete constraints of the corresponding core information in the documents, and identifying the reasonable constraints missing in the original question (such as the original question "What receipts do employees need to provide for expense reimbursement?" does not mention "reimbursement type", supplementing with reimbursement type constraints such as "travel expenses" and "office expenses" that are clearly stated in the documents). Question type conversion transforms static, fixed question-and-answer formats into other question types with high relevance and different scenarios. This simulates users asking questions from different perspectives while ensuring that the converted questions can still generate accurate answers based on business documents. A pre-defined type conversion mapping table is used: based on business scenarios, standardized question type conversion rules are established in advance, clarifying the conversion direction for different question types. For example: practical questions ("How to process employee travel expense reimbursements?") are converted to risk questions ("What are the consequences of not processing employee travel expense reimbursements according to regulations?"); query questions ("What are the approval levels for employee travel expense reimbursements?") are converted to judgment questions ("If an employee's travel expense reimbursement amount is 5000 yuan, will it take effect after the department manager's approval?"); and explanatory questions ("What is the scope of application for employee welfare reimbursement?") are converted to exception questions ("Which employees are not subject to the relevant regulations for employee welfare reimbursement?"). Answer adaptation and adjustment are performed based on the converted questions, adjusting the wording of the answers to ensure they meet the requirements of the question (e.g., answers to judgment questions must clearly state "yes / no" and provide supporting evidence; answers to risk questions must clearly state the specific consequences and corresponding document clauses). Semantic extension and semantic interference information (simulating complex questions in real business scenarios): Based on static fixed question and answer systems, reasonable semantic extension and semantic interference information are added to simulate the expression habits of real users when asking questions (such as redundant expressions, mixed intentions, and ambiguous information), improving the realism and complexity of dynamic question and answer systems. This includes ensuring that semantic extensions align with the business scenario and do not introduce irrelevant information; semantic extensions should not change the core requirements and constraints of the question. Interference information: Interference information is content related to the core entity but does not affect the core requirements (e.g., in the question "I am a marketing department employee and I need to process travel expense reimbursement for March 2026. I would like to know what receipts I need to provide?", "marketing department employee" and "March 2026" are non-core interference information); interference information must conform to the business scenario of the business document and should not contain false information outside the document.

[0061] See Figure 4 , Figure 4The present invention provides a schematic diagram of the structure of a test optimization system 40 for a RAG system according to an embodiment of the present invention. The test optimization system 40 can be divided into a knowledge base module 401, a question and answer generation module 402, and a test module 403. The descriptions of each functional module are as follows.

[0062] The knowledge base module 401 is used to store various business documents and multiple semantic units (knowledge fragments) corresponding to the business documents that have undergone unified preprocessing.

[0063] The question-and-answer generation module 402 is used to construct static fixed questions and answers and dynamic questions and answers corresponding to business documents. The static fixed questions and answers are constructed based on the core information of the business documents corresponding to the knowledge base. Dynamic questions and answers can be generated based on the business documents in the knowledge base and the static fixed questions and answers, used to cover interfering information and complex business scenarios in the knowledge base. This question-and-answer generation module 402 can be the intelligent question-and-answer system described above.

[0064] Test module 403 is used to test the intelligent question answering system using a hierarchical and progressive testing approach, including: verifying the retrieval recall rate (retrieval capability), accuracy, and user dislike rate of the final answer generated by the intelligent question answering system for user input (the questions corresponding to static fixed questions and / or dynamic questions and answers) based on the answers corresponding to static fixed questions and / or dynamic questions and answers; collecting the verification results of each test stage; and updating and optimizing the intelligent question answering system based on the verification results.

[0065] like Figure 5 As shown, the present invention provides a method for running Figure 4 The electronic device 50 of the test optimization system 40 of the RAG system shown may include a memory 501, a processor 502 and a bus, and may also include computer programs stored in the memory 501 and run on the processor 502, such as the various functional modules of the test optimization system.

[0066] The memory 501 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 501 can be an internal storage unit of the electronic device 50, such as the portable hard drive of the electronic device 50. In other embodiments, the memory 501 can also be an external storage device of the electronic device 50, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 50. Furthermore, the memory 501 can include both internal and external storage units of the electronic device 50. The memory 501 can be used not only to store application software and various types of data installed on the electronic device 50, such as code for task distribution in testing and optimizing the system, but also to temporarily store data that has been output or will be output.

[0067] In some embodiments, the processor 502 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 502 is the control unit of the electronic device 50, connecting various components of the electronic device 50 through various interfaces and lines. It executes programs or modules stored in the memory 501 (such as control programs for testing and optimizing systems) and calls data stored in the memory 501 to perform various functions and process data of the electronic device 50.

[0068] The processor 502 executes the operating system of the electronic device 50 and various installed application programs. The processor 502 executes the application programs to implement the steps in the test optimization method for the electronic device described above.

[0069] For example, a computer program may be divided into one or more modules, one or more of which are stored in memory 501 and executed by processor 502 to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in electronic device 50.

[0070] The integrated units implemented as software functional modules described above can be stored in a computer-readable storage medium, which can be non-volatile or volatile. The software functional modules stored in the storage medium include several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute some functions of the test optimization method of the electronic device in the various embodiments of this application.

[0071] In one embodiment, a storage medium is provided on which a computer program is stored, and when the computer program is executed by a processor, it can also perform the steps described above when the processor executes the computer program.

[0072] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for testing and optimizing an intelligent question-answering system, characterized in that, include: Construct an intelligent question-answering system based on business documents in a knowledge base, wherein the intelligent question-answering system is used to receive user input and generate target answers; A first type of question and answer and a second type of question and answer are determined, wherein the first type of question and answer is a question and answer pair generated based on the document information of the business document, and the second type of question and answer is a question and answer pair generated based on the first type of question and answer or the document information; Based on the first type of question and answer and the second type of question and answer, the intelligent question answering system was tested using a hierarchical progressive testing method to obtain verification results; Based on the verification results, the knowledge base and the intelligent question-answering system are updated and optimized.

2. The method according to claim 1, characterized in that, The first type of question and answer is used to describe the questions and answers related to the document information, while the second type of question and answer is used to describe the questions and answers related to the semantic extension information or semantic interference information corresponding to the document information.

3. The method according to claim 1, characterized in that, The intelligent question-answering system is tested using a hierarchical, progressive testing approach based on the first type of question-answering and the second type of question-answering, and the verification results are obtained, including: Calculate the retrieval recall and precision between the target answer generated by the intelligent question answering system and the answers included in the first type of question answering and / or the second type of question answering, respectively; The comparison results of the retrieval recall rate and the precision rate with the preset test thresholds are used as the verification results, wherein the retrieval recall rate is used to describe the hit rate of the target answer in the answers included in the first type of question answer and / or the second type of question answer, and the precision rate is used to describe the consistency between the target answer and the answers included in the first type of question answer and / or the second type of question answer, and the preset test thresholds corresponding to the first type of question answer and the second type of question answer are different.

4. The method according to claim 1, characterized in that, The step of testing the intelligent question-answering system using a hierarchical, progressive testing approach based on the first type of question-answering and the second type of question-answering to obtain verification results also includes: Calculate the user downvote rate corresponding to the target answer generated by the intelligent question answering system, the first type of question answer, and / or the second type of question answer; The comparison result between the user downvote rate and the preset downvote threshold is used as the verification result, wherein the user downvote rate is used to describe the number of negative feedbacks from the user to the target answer, the first type of question and answer and / or the answers included in the second type of question and answer.

5. The method according to claim 1, characterized in that, The second type of question and answer is generated through one or more of the following methods: The business document is subjected to type identification and semantic parsing. Combined with a preset question and answer type template, a second type of question and answer is generated that covers the rule explanation, scope of application, or execution process of the business document. Based on the first type of question and answer, the second type of question and answer is generated by supplementing constraints, transforming question types, or adding distracting information.

6. The method according to claim 3, characterized in that, The accuracy rate is calculated in the following way: Semantic similarity analysis and sentiment consistency analysis are performed on the target answer and the answers included in the first type of question answer and / or the second type of question answer respectively. The accuracy rate is calculated according to the scoring rules. The semantic similarity analysis is used to describe the semantic consistency between the final answer and the answers included in the first type of question answer and / or the second type of question answer respectively. The sentiment consistency analysis is used to describe the consistency of sentiment between the final answer and the answers included in the first type of question answer and / or the second type of question answer respectively.

7. The method according to claim 1, characterized in that, The verification result includes at least one of the following: missing information, expression deviation, or logical error.

8. The method according to claim 1, characterized in that, The updating and optimization of the knowledge base and the intelligent question-answering system includes at least one of the following: supplementing missing semantic units in the knowledge base, correcting or discarding semantic units in the knowledge base, updating business documents, and adjusting parameters of the intelligent question-answering system.

9. A testing and optimization system for an intelligent question-answering system, characterized in that, include: The module consists of a knowledge base, a question-and-answer generation module, and a testing module. The knowledge base module is used to store business documents; The question-and-answer generation module is used to construct the first type of question and answer and the second type of question and answer corresponding to the business documents in the knowledge base; The testing module is used to test the intelligent question-answering system using a hierarchical and progressive testing method based on the first type of question-answering and the second type of question-answering, and obtain verification results; based on the verification results, the knowledge base module and the intelligent question-answering system are updated and optimized.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the test optimization method for the intelligent question-answering system as described in any one of claims 1 to 8.

Images