Method and system for testing accuracy of intelligent questions and answers in AI assistant knowledge base
By automatically generating test questions and multi-dimensional evaluations using a large language model, this approach solves the problems of reliance on human input and biased evaluations in existing AI assistant knowledge base question-and-answer testing, achieving efficient and comprehensive question-and-answer testing.
Patent Information
- Application Number
- CN202511395297.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-28
- Publication Date
- 2026-01-13
AI Technical Summary
Existing AI assistant knowledge base intelligent question-answering tests rely on manually designed questions, which is inefficient, lacks coverage, has a single evaluation dimension, and is difficult to fully reflect question-answering ability.
By automatically generating diverse test questions and standard answers through a large language model, and combining them with a multi-dimensional evaluation index system, an automated testing process is achieved, covering common, edge and cross-domain scenarios.
It improves testing efficiency and accuracy, reduces reliance on manual intervention, ensures the objectivity and comprehensiveness of test results, and can accurately pinpoint shortcomings in question-and-answer performance.
Smart Images
Figure CN121326751A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of artificial intelligence, in particular to a method and system for testing the accuracy of AI assistant knowledge base intelligent question answering. BACKGROUND
[0002] With the rapid development of artificial intelligence technology, AI assistants are widely used in finance, medicine, e-commerce, government affairs and other fields. One of the core functions is to provide intelligent question answering services for users based on preset knowledge bases. The accuracy of AI assistant knowledge base intelligent question answering directly determines the user experience and service quality, so it is crucial to test it comprehensively and accurately.
[0003] Currently, the existing AI assistant knowledge base intelligent question answering accuracy test mainly relies on manual methods, which has the following significant problems:
[0004] 1. High dependence on manual work: In the testing process, the design of test questions, the formulation of standard answers, and the evaluation of test results all need to be done manually. On the one hand, designing questions manually requires a lot of time and labor costs, especially when the knowledge base is large and covers a wide range of fields, the testing efficiency is very low. On the other hand, manual evaluation is easily influenced by subjective factors, and different testers have different judgment standards, which makes it difficult to guarantee the objectivity and consistency of the test results.
[0005] 2. Fixed test scenarios: Test questions designed manually are often limited to common scenarios or known high-frequency questions, making it difficult to cover edge scenarios, special cases, and cross-domain scenarios. This makes it impossible to discover the "untested" vulnerabilities that may occur in the actual application of AI assistants, resulting in insufficient test coverage and failing to fully reflect the real question answering ability of AI assistants.
[0006] 3. One-dimensional evaluation: Existing tests focus only on the single dimension of "whether the answer is correct", ignoring important indicators such as the relevance of the answer (e.g. whether it deviates from the core of the user's question), completeness (e.g. whether it omits key information), and logic (e.g. whether the reasoning process is reasonable). One-dimensional evaluation cannot fully assess the quality of AI assistant question answering, which may result in "correct answers but poor practicality" of AI assistants passing the test, affecting the actual application effect.
[0007] Therefore, there is an urgent need for an AI assistant knowledge base intelligent question answering accuracy testing scheme that can reduce manual dependence, enrich test scenarios, and implement multi-dimensional evaluation, in order to improve testing efficiency and accuracy and ensure the service quality of AI assistants. SUMMARY
[0008] The technical task of the present application is to provide a method and system for testing the accuracy of AI assistant knowledge base intelligent question answering, which can overcome the problems of high artificial dependence, fixed test scene, and one-sided evaluation in the existing AI assistant knowledge base intelligent question answering accuracy test, and realize the automatic generation of test questions and standard answers, multi-dimensional automatic evaluation, and full-process automatic testing, thereby improving the test efficiency and accuracy, reducing the labor cost, and comprehensively detecting the question and answer performance of AI assistants in various scenes.
[0009] The technical solution adopted by the present application to solve its technical problems is:
[0010] A method for testing the accuracy of AI assistant knowledge base intelligent question answering, the implementation of the method includes the following steps:
[0011] (1) Analyze the knowledge base data of the AI assistant to be tested, and define a test scene set containing common scenes, edge scenes, and cross-domain scenes;
[0012] (2) Call a large language model to automatically generate diversified test questions and corresponding standard answers based on the knowledge base data and the test scene set;
[0013] (3) Input the test question into the AI assistant to be tested, and collect the evaluation results returned by the AI assistant;
[0014] (4) Build a multi-dimensional evaluation index system including accuracy, relevance, completeness, and logic, and automatically score the evaluation results by combining a large language model with a rule engine;
[0015] (5) Statistically analyze the test results, analyze the performance short board of the AI assistant, and generate a test report.
[0016] This method relies on a large language model to generate test questions and standard answers of multiple types and multiple expressions, breaks the scene fixation, realizes comprehensive and diversified testing, and achieves a breakthrough in scene and question generation; Abandoning a single standard, building a multi-dimensional index system, combining rule engine preliminary judgment and large language model scoring, solving the one-sided subjective evaluation problem, making the evaluation more objective, and optimizing the evaluation system; Create a coherent test link of "knowledge base analysis-question generation-result collection-evaluation scoring-report generation", realize the integration of the test process, without the need for repeated human intervention, while accurately positioning the performance short board of the AI assistant in different scenes and question types, providing a clear direction for subsequent optimization, which greatly improves the test efficiency and guarantees the accuracy of the test results.
[0017] Further, the knowledge base data of the AI assistant to be tested includes text knowledge, structured knowledge, association rules, etc. Through natural language processing technology, including entity recognition, relationship extraction, and knowledge graph construction, the knowledge base is parsed, and knowledge domains (such as financial planning and medical consultation), core knowledge points (such as product functions and disease symptoms), and knowledge association relationships are extracted. Based on actual application scenarios, a set of test scenarios is defined, including common scenarios, edge scenarios (such as rare problems and special condition constraint problems), and cross-domain scenarios (such as "finance + law" cross-problems), providing scene basis for subsequent test question generation.
[0018] Further, the step (2) calls a pre-trained large language model (such as GPT series, LLaMA series, etc.), and inputs the parsed knowledge base information in step (1) and the scene set as input, and guides the large language model to generate diversified test questions and corresponding standard answers through prompt word engineering (Prompt Engineering):
[0019] Test question generation: The large language model generates different types of questions (such as factual questions, reasoning questions, multi-condition query questions, and negative questions) based on the knowledge points in the knowledge base and the scenarios, and ensures that the questions cover all scenarios defined in step (1) to avoid scenario fixation. At the same time, by randomly adjusting the question expression (such as synonym replacement, sentence conversion, and redundant information addition), multiple groups of questions with different expressions but consistent core requirements are generated to improve the comprehensiveness of the test.
[0020] Standard answer generation: The large language model generates a unique standard answer for each test question based on the accurate information in the knowledge base. The standard answer should include key information (such as specific numerical values, core conclusions, and step processes) and have a standard format (such as point-by-point explanation and logical hierarchy) to provide clear basis for subsequent evaluation.
[0021] Further, the step (3) records the interaction log of each test question, including response time and called knowledge base module.
[0022] Further, the multi-dimensional evaluation index system,
[0023] Accuracy dimension index: Whether the evaluated result is consistent with the core information of the standard answer and whether there is incorrect information (such as factual errors and numerical deviations);
[0024] Relevance dimension index: Whether the evaluated result is around the core requirement of the test question and whether there is a deviation from the topic or an irrelevant answer;
[0025] Completeness dimension index: Whether the evaluated result contains all the key information in the standard answer and whether there is information omission (such as missing steps and unexplained conditions).
[0026] Logical dimension index: if the test question is a reasoning question, whether the reasoning process of the result to be evaluated conforms to logic, and whether the argument supports the conclusion.
[0027] Further, the automatic scoring:
[0028] First, the explicit errors (such as numerical errors, core fact errors) are preliminarily scored by the rule engine;
[0029] Second, call a large language model, take the test question, the standard answer, the result to be evaluated and the evaluation index definition as input, guide the large language model to score the result to be evaluated in each dimension (such as using 1-10 point system);
[0030] Finally, according to the weight of each dimension (which can be configured according to actual needs, such as accuracy weight 40%, relevance weight 25%, integrity weight 25%, logic weight 10%), calculate the comprehensive score of each test question.
[0031] Further, the statistical test result, the comprehensive score, the dimension score and the pass rate (such as the comprehensive score ≥8 points is considered to pass) of all test questions are calculated, and the performance short board (such as low pass rate in edge scene, low logic score in reasoning question) of the AI assistant to be tested in different scenes (such as common scene, edge scene) and different question types (such as fact, reasoning) is analyzed;
[0032] Generate a test report, the report includes test coverage scene, test question number, dimension score distribution, performance short board analysis and optimization suggestion.
[0033] The present application also claims a kind of for AI assistant knowledge base intelligent question and answer accuracy test system, comprising:
[0034] Data input layer: the AI assistant to be tested provides knowledge base original data to the system, receives the test question sent by the system at the same time, and returns the question and answer result;
[0035] Core processing layer: including knowledge base analysis module, test generation module, interaction collection module, multi-dimensional evaluation module and result analysis module, each module realizes automatic test through data flow:
[0036] The knowledge base analysis module converts the original knowledge base into structured analysis result and scene set, provides basis for test generation;
[0037] The test generation module generates test questions and standard answers by prompting word engineering driven large language model based on analysis result;
[0038] The interaction collection module realizes the interaction between the system and the AI assistant to be tested, completes the problem issuing and result collection, and records the interaction log.
[0039] The multi-dimensional evaluation module combines a rule engine (preliminary error correction) and a large language model (multi-dimensional scoring) to output an objective score.
[0040] The result analysis module summarizes the score data to generate a test report containing short board analysis and optimization suggestions.
[0041] The result output layer: the test report is displayed to the test personnel through a user terminal (such as a computer or a tablet), or is stored in a database for subsequent review.
[0042] The system realizes the AI assistant knowledge base intelligent question and answer accuracy test through the above method.
[0043] The present application also claims a device for testing the accuracy of AI assistant knowledge base intelligent question and answer, comprising at least one memory and at least one processor.
[0044] The at least one memory is used to store machine readable programs.
[0045] The at least one processor is used to call the machine readable programs to realize the above method.
[0046] The present application also claims a computer readable medium, which stores computer instructions, and the computer instructions can realize the above method when executed by a processor.
[0047] Compared with the prior art, the present application has the following beneficial effects:
[0048] 1. The present application can comprehensively detect the question and answer accuracy of AI assistants in various scenarios by constructing a comprehensive and representative test set covering various question types, different knowledge base modules and different difficulty levels, and designing counterexamples, thereby avoiding the result deviation caused by the limitation of the test set in the prior art.
[0049] 2. The multi-dimensional evaluation standard is used to evaluate the answers of AI assistants from the aspects of factual accuracy, completeness, relevance and the like, thereby overcoming the limitation of single evaluation dimension and making the evaluation result objective and comprehensive.
[0050] 3. The test process is standardized, including three stages of preparation, execution and analysis, and each stage has clear operation steps, thereby ensuring the reproducibility and traceability of the test process and improving the reliability of the test result.
[0051] 4、By combining quantitative indicators and qualitative analysis, the test results can be analyzed to accurately locate the weak links of the AI assistant, provide a clear direction for subsequent optimization and improvement, and help improve the reliability of the AI assistant in practical applications. BRIEF DESCRIPTION OF DRAWINGS
[0052] Figure 1 is a flowchart of a test method for AI assistant knowledge base intelligent question and answer accuracy provided by an embodiment of the application;
[0053] Figure 2 is a test system architecture diagram for AI assistant knowledge base intelligent question and answer accuracy provided by an embodiment of the application. DETAILED DESCRIPTION
[0054] The application will be further described below in conjunction with specific embodiments.
[0055] An embodiment of the application provides a test method for AI assistant knowledge base intelligent question and answer accuracy, and the implementation of the method includes the following steps:
[0056] (1) Analyze the knowledge base data of the AI assistant to be tested, and define a test scenario set containing common scenarios, edge scenarios, and cross-domain scenarios;
[0057] (2) Call a large language model, and automatically generate diversified test questions and corresponding standard answers based on the knowledge base data and the test scenario set;
[0058] (3) Input the test question into the AI assistant to be tested, and collect the evaluation results returned by the AI assistant;
[0059] (4) Construct a multi-dimensional evaluation index system including accuracy, relevance, completeness, and logic, and automatically score the evaluation results by combining a large language model with a rule engine;
[0060] (5) Statistically analyze the test results, analyze the performance shortcomings of the AI assistant, and generate a test report.
[0061] In combination with Figure 1 as shown, the specific implementation of the method is as follows:
[0062] 1. Knowledge base analysis and scenario definition:
[0063] Obtain the knowledge base data of the AI assistant to be tested (including text knowledge, structured knowledge, association rules, etc.), analyze the knowledge base through natural language processing techniques (such as entity recognition, relationship extraction, knowledge graph construction), extract knowledge domains (such as financial planning, medical consultation), core knowledge points (such as product functions, disease symptoms), and knowledge association relationships; at the same time, based on actual application scenarios, define a set of test scenarios, including common scenarios, edge scenarios (such as rare problems, special condition constraint problems), and cross-domain scenarios (such as "finance + law" cross-domain problems), to provide scene basis for subsequent test question generation.
[0064] 2. Test question and standard answer generation based on large language model:
[0065] Call a pre-trained large language model (such as GPT series, LLaMA series, etc.), input the knowledge base information parsed in step 1 and the scene set, and guide the large language model to generate diversified test questions and corresponding standard answers through prompt engineering (Prompt Engineering):
[0066] Test question generation: The large language model generates different types of questions (such as factual questions, reasoning questions, multi-condition query questions, and negative questions) based on the knowledge points and scenarios in the knowledge base, ensuring that the questions cover all scenarios defined in step 1 and avoiding scenario fixation; at the same time, by randomly adjusting the question expression (such as synonym replacement, sentence conversion, and redundant information addition), multiple sets of questions with different expressions but consistent core requirements are generated, improving the comprehensiveness of the test.
[0067] Standard answer generation: The large language model generates a unique standard answer for each test question based on the accurate information in the knowledge base, which includes key information (such as specific numerical values, core conclusions, and step-by-step processes) and is in a standard format (such as point-by-point explanation and logical hierarchy), providing clear evidence for subsequent evaluation.
[0068] 3. AI assistant question and answer interaction and result collection:
[0069] Input the test questions generated in step 2 into the AI assistant to be tested one by one, trigger the knowledge base question and answer function of the AI assistant, and obtain the question and answer results returned by the AI assistant (hereinafter referred to as "results to be evaluated"); at the same time, record the interaction log of each test question (such as response time and called knowledge base module) for subsequent auxiliary analysis.
[0070] 4. Multi-dimensional automatic evaluation:
[0071] Based on the standard answers generated in step 2, construct a multi-dimensional evaluation index system, and automatically score the results to be evaluated through the combination of a large language model and a rule engine:
[0072] Evaluation indicator system: including but not limited to the following dimensions:
[0073] Accuracy: Whether the result to be evaluated is consistent with the core information of the standard answer, and whether there are any errors (such as factual errors or numerical deviations);
[0074] Relevance: Whether the results to be evaluated revolve around the core requirements of the test question, and whether there are any deviations from the topic or irrelevant answers;
[0075] Completeness: Does the result to be evaluated contain all the key information in the standard answer, and are there any missing information (such as missing steps or unspecified conditions)?
[0076] Logicality: If the test question is a reasoning question, whether the reasoning process of the result to be evaluated is logical and whether the arguments support the conclusion.
[0077] Automated scoring:
[0078] First, the rule engine initially scores explicit errors (such as numerical errors and core factual errors). Second, it calls the large language model, taking the test questions, standard answers, results to be evaluated, and evaluation metric definitions as input, to guide the large language model to score the results to be evaluated in each dimension (e.g., using a 1-10 scale). Finally, based on the weights of each dimension (which can be configured according to actual needs, such as 40% for accuracy, 25% for relevance, 25% for completeness, and 10% for logicality), it calculates the comprehensive score for each test question.
[0079] 5. Test Result Analysis and Report Generation:
[0080] The system calculates the overall score, scores for each dimension, and pass rate for all test questions (e.g., an overall score of ≥8 is considered passing). It analyzes the performance shortcomings of the AI assistant under test in different scenarios (e.g., common scenarios and edge scenarios) and different question types (e.g., factual and reasoning questions) (e.g., low pass rate in edge scenarios and low logic scores in reasoning questions). A test report is generated, which includes the test coverage scenarios, the number of test questions, the score distribution for each dimension, analysis of performance shortcomings, and optimization suggestions.
[0081] This method leverages a large language model to generate diverse test questions and standard answers with multiple expressions, breaking away from fixed scenarios and achieving comprehensive and diverse testing, thus breaking through the limitations of scenario generation. It abandons a single standard, establishing a multi-dimensional indicator system that combines initial judgment from a rule engine with scoring from the large language model to address the problem of biased and subjective evaluation, making assessments more objective and optimizing the evaluation system. Furthermore, it creates a coherent testing chain from "knowledge base analysis to question generation, result collection, evaluation scoring, and report generation," achieving integrated testing processes without repeated manual intervention. Simultaneously, it accurately identifies the performance shortcomings of the AI assistant in different scenarios and question types, providing clear direction for subsequent optimization, significantly improving testing efficiency while ensuring the accuracy of test results.
[0082] The invention will be further described in detail below with reference to specific examples.
[0083] 1. Test set construction:
[0084] Taking a product knowledge base as an example, the test suite is constructed as follows:
[0085] Fact-based questions: "Where is product A manufactured?" "What is the warranty period for product B?"
[0086] Reasoning questions: "What are the main differences between product A and product B?" "Based on the parameters of product C, can it support function XX?"
[0087] Multi-turn interactive questions: "First tell me the price of product A, then recommend a few products with similar functions." "How do I turn on product D? How do I connect to WiFi after turning it on?"
[0088] Vague questions include: "My red camera device won't charge, what should I do?" and "Where can I find the instruction manual for this device?"
[0089] Difficulty levels: Simple questions such as "What is the price of product A?" (directly match the original text in the knowledge base); Medium questions such as "What are the installation steps for product B?" (requires integration of 2-3 pieces of information from the knowledge base); Complex questions such as "Compare the advantages and disadvantages of products C and D, and recommend one based on my needs (XX needs)" (requires cross-module reasoning).
[0090] Counterexamples: Questions not covered by the knowledge base such as "When was Product E launched?"; and questions that conflict with the knowledge base such as "Is the warranty period for Product A 3 years?" (1 year in the knowledge base).
[0091] 2. Evaluation criteria:
[0092] Regarding the question "What are the price and delivery time of product A?", if the knowledge base states "Product A is priced at 500 yuan, and will be shipped within 48 hours of payment":
[0093] If the AI answers "Product A is priced at 500 yuan and will be shipped within 48 hours after payment", it is judged as "correct" and meets all evaluation dimensions such as factual accuracy and completeness.
[0094] If the AI answers "Product A is priced at 500 yuan", it is judged as "partially correct". The core facts are correct, but the delivery time is missing, and the completeness is insufficient.
[0095] If the AI answers "Product A is priced at 600 yuan and will be shipped within 3 days after payment", it is judged as "incorrect" because there is a factual error; if the AI answers "Product B is priced at 500 yuan", it is irrelevant and has low relevance, and is judged as "incorrect".
[0096] If the AI answers "Product A is priced at 500 yuan, will be shipped within 48 hours after payment, and supports global shipping" (global shipping is not mentioned in the knowledge base), then it is an "illusion" and is judged as "incorrect".
[0097] 3. Test execution process:
[0098] Step 1: Set up the test environment:
[0099] Limit the knowledge scope of the AI assistant (load only the target knowledge base), disable functions such as "online search" and "external knowledge retrieval" to avoid interference from external information; record the AI version during testing.
[0100] Step 2: Batch Questioning and Result Collection:
[0101] Ask the AI questions in batches using the test set, record the answers (text, speech-to-text, etc.), and label the knowledge base source of the questions (such as specific documents or chapters) for easy verification later.
[0102] Step 3: Result Labeling and Scoring:
[0103] Automatic initial screening: Use tools (such as keyword matching and semantic similarity models) to determine the degree of matching between the answer and the knowledge base.
[0104] 4. Results Analysis:
[0105] Quantitative indicators:
[0106] Accuracy: Number of correct answers ÷ Total number of questions (core indicator, reflecting overall reliability);
[0107] Illusion rate: Number of answers containing fabricated information ÷ Total number of questions (measures the probability of "creating something out of nothing");
[0108] Completeness score: (the average of the number of key information items in the answer ÷ the number of key information items required by the question) (0-1 point);
[0109] Accuracy rates for different difficulty levels / types of questions: such as "accuracy rate for complex reasoning questions" and "accuracy rate for fuzzy questions," to pinpoint the weaknesses of AI.
[0110] Qualitative analysis:
[0111] Summarize error types and trace their root causes:
[0112] Knowledge base issues: The knowledge base contains ambiguity (such as two different expressions for the same concept), missing information (related content is not included), and errors (the knowledge base itself is incorrect);
[0113] AI's ability to understand problems: It cannot resolve ambiguous questions (such as "this thing" with unclear reference) and misidentifies keywords (such as "warranty" being misjudged as "maintenance");
[0114] Reasoning problems: logical breaks during multi-step reasoning (e.g., A→B is correct, but B→C is incorrect), and failure to integrate cross-module information (e.g., missing related conditions when integrating product functions and prices).
[0115] This invention also provides a testing system for the accuracy of intelligent question answering in an AI assistant knowledge base. This system implements the accuracy testing of intelligent question answering in an AI assistant knowledge base using the testing method described in the above embodiments.
[0116] The system's architecture is layered, from top to bottom: data input layer, core processing layer, and result output layer, as follows: Figure 2 As shown, it includes:
[0117] Data input layer: The AI assistant under test provides the system with raw data from the knowledge base, receives test questions sent by the system, and returns the question-and-answer results;
[0118] The core processing layer comprises five core functional modules: knowledge base parsing, test generation, interactive data collection, multi-dimensional evaluation, and result analysis. These modules achieve automated testing through data flow.
[0119] 1. The knowledge base parsing module transforms the original knowledge base into structured parsing results and scenario sets, providing a foundation for test generation;
[0120] 2. Test generation module: Based on the parsing results, it generates test questions and standard answers by driving the large language model through prompt word engineering;
[0121] 3. Interactive data collection module: Enables interaction between the system and the AI assistant under test, completes the distribution of questions and collection of results, and records interaction logs;
[0122] 4. Multi-dimensional evaluation module, combining rule engine (preliminary error correction) and large language model (multi-dimensional scoring) to output objective scores;
[0123] 5. Results analysis module: Summarizes score data and generates a test report that includes weakness analysis and optimization suggestions;
[0124] Results output layer: Test reports are displayed to testers via user terminals (such as computers and tablets) or stored in a database for later review.
[0125] This invention also provides a testing device for the accuracy of intelligent question answering in an AI assistant knowledge base, comprising: at least one memory and at least one processor;
[0126] The at least one memory is used to store a machine-readable program;
[0127] The at least one processor is configured to call the machine-readable program to implement the testing method for the accuracy of intelligent question answering in the knowledge base of an AI assistant, as described in the above embodiments.
[0128] This invention also provides a computer-readable medium storing computer instructions. When executed by a processor, these instructions implement the testing method for the accuracy of intelligent question answering in an AI assistant knowledge base, as described in the above embodiments. Specifically, a system or apparatus equipped with a storage medium storing software program code that implements the functions of any of the embodiments described above can be provided, enabling the computer (or CPU or MPU) of the system or apparatus to read and execute the program code stored in the storage medium.
[0129] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.
[0130] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.
[0131] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0132] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0133] The present invention has been shown and described in detail above with reference to the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above embodiments, those skilled in the art will know that more embodiments of the present invention can be obtained by combining the code review methods in the different embodiments. These embodiments are also within the protection scope of the present invention.
Claims
1. A method for testing the accuracy of intelligent question answering in an AI assistant knowledge base, characterized in that, The implementation of this method includes the following steps: (1) Analyze the knowledge base data of the AI assistant to be tested and define a set of test scenarios including common scenarios, edge scenarios, and cross-domain scenarios; (2) Call the large language model to automatically generate diverse test questions and corresponding standard answers based on knowledge base data and test scenario set; (3) Input the test questions into the AI assistant to be tested and collect the evaluation results returned by the AI assistant; (4) Construct a multi-dimensional evaluation index system that includes accuracy, relevance, completeness and logic, and automatically score the evaluation results by combining a large language model with a rule engine. (5) Statistical analysis of test results, analysis of shortcomings in AI assistant performance, and generation of test report.
2. The method for testing the accuracy of intelligent question answering in an AI assistant knowledge base according to claim 1, characterized in that, The knowledge base data of the AI assistant under test includes text knowledge, structured knowledge, and association rules. Natural language processing technology, including entity recognition, relation extraction, and knowledge graph construction, is used to parse the knowledge base and extract knowledge domains, core knowledge points, and knowledge relationships. Based on real-world application scenarios, a set of test scenarios is defined.
3. The method for testing the accuracy of intelligent question answering in an AI assistant knowledge base according to claim 1, characterized in that, Step (2) calls the pre-trained large language model, taking the knowledge base information and scene set parsed in step (1) as input, and guides the large language model to generate diverse test questions and corresponding standard answers through prompt word engineering: Test question generation: The large language model generates different types of questions based on the knowledge points and scenarios in the knowledge base, and ensures that the questions cover all scenarios defined in step (1); at the same time, by randomly adjusting the question descriptions, multiple sets of questions with different descriptions but consistent core requirements are generated. Standard answer generation: The large language model generates a unique standard answer for each test question based on accurate information in the knowledge base. The standard answer must contain key information and be in a standardized format.
4. The method for testing the accuracy of intelligent question answering in an AI assistant knowledge base according to claim 3, characterized in that, In step (3), the interaction log for each test question is recorded, including the response time and the knowledge base modules called.
5. The method for testing the accuracy of intelligent question answering in an AI assistant knowledge base according to claim 1, characterized in that, The aforementioned multi-dimensional evaluation index system Accuracy Dimension Indicators: Whether the core information of the result to be evaluated is consistent with the standard answer, and whether there are any errors; Relevance metrics: Whether the results to be evaluated revolve around the core requirements of the test question, and whether there are any deviations from the topic or irrelevant answers; Completeness dimension indicators: Whether the result to be evaluated contains all the key information in the standard answer, and whether there is any omission of information; Logicality dimension indicators: If the test question is a reasoning question, whether the reasoning process of the result to be evaluated is logical and whether the arguments support the conclusion.
6. A method for testing the accuracy of intelligent question answering in an AI assistant knowledge base according to claim 1 or 5, characterized in that, The automated scoring: First, a preliminary score is given to the specific errors using a rules engine; Secondly, the large language model is invoked, and the test questions, standard answers, results to be evaluated, and evaluation index definitions are used as input to guide the large language model to score the results to be evaluated in each dimension; Finally, based on the weights of each dimension, a comprehensive score is calculated for each test question.
7. A method for testing the accuracy of intelligent question answering in an AI assistant knowledge base according to claim 1, characterized in that, The statistical test results include the overall score, scores of each dimension and pass rate of all test questions, and analysis of the performance shortcomings of the AI assistant under test in different scenarios and different question types. Generate a test report, which includes test coverage scenarios, number of test issues, score distribution for each dimension, analysis of performance shortcomings, and optimization suggestions.
8. A testing system for the accuracy of intelligent question answering in an AI assistant knowledge base, characterized in that, include: Data input layer: The AI assistant under test provides the system with raw data from the knowledge base, receives test questions sent by the system, and returns the question-and-answer results; The core processing layer includes a knowledge base parsing module, a test generation module, an interactive data collection module, a multi-dimensional evaluation module, and a result analysis module. These modules achieve automated testing through data flow. The knowledge base parsing module transforms the original knowledge base into structured parsing results and scenario sets, providing a foundation for test generation; The test generation module, based on the parsing results, generates test questions and standard answers through prompt word engineering driven by a large language model. The interactive data acquisition module enables the interaction between the system and the AI assistant under test, completes the distribution of questions and the collection of results, and records the interaction log; The multi-dimensional evaluation module, combining a rule engine and a large language model, outputs an objective score. The results analysis module summarizes the score data and generates a test report that includes a weakness analysis and optimization suggestions. Results output layer: Test reports are displayed to testers via user terminals or stored in a database for later review; The system achieves accuracy testing of AI assistant knowledge base intelligent question answering through the method described in any one of claims 1 to 7.
9. A testing device for the accuracy of intelligent question answering in an AI assistant knowledge base, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to implement the method according to any one of claims 1 to 7.
10. A computer-readable medium, characterized in that, The computer-readable medium stores computer instructions that, when executed by a processor, enable the implementation of the method described in any one of claims 1 to 7.