Question and answer model verification method and system, electronic equipment and storage medium
The question-answering model verification method, which combines semantic expansion and hybrid similarity calculation, solves the verification challenge in the iterative upgrade of question-answering models, achieves efficient and accurate automated verification and dynamic expansion, and improves the reliability of the question-answering system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-03
- Publication Date
- 2026-04-03
AI Technical Summary
When existing question-answering models are iterated, upgraded, or replaced across architectures, the verification of output correctness suffers from insufficient semantic generalization ability, weak domain adaptability, poor dynamic scalability, and an imbalance between efficiency and cost, making it difficult to achieve efficient and accurate automated verification.
A dynamically enhanced knowledge base is generated through semantic expansion. Combined with a large language model and knowledge graph, hybrid similarity calculation is performed. Multi-dimensional indicators are used to evaluate the model output, the threshold is adaptively adjusted, and a verification report is generated.
It enables automated and accurate verification of question-answering models, covers diverse question types, improves verification efficiency, supports dynamic expansion of the knowledge base, provides visual optimization guidance, and ensures the reliability of model iteration.
Smart Images
Figure CN121786408A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of model validation, and more specifically to a question-answering model validation method, system, electronic device, and storage medium. Technical Background
[0002] With the rapid development of artificial intelligence technology, knowledge base-based automated question answering systems are increasingly widely used in fields such as intelligent customer service, medical consultation, and financial analysis. The core of such systems lies in accurately parsing user intent through pre-trained language models and matching and generating answers from structured knowledge bases. However, in practical applications, when models are iterated and upgraded or replaced across architectures, how to efficiently and accurately verify the correctness of the output of the new model has become a key challenge to ensure product reliability. Existing verification technologies generally have the following problems: (1) Insufficient semantic generalization ability: It is not possible to effectively handle multiple expressions of the same question, resulting in incomplete verification coverage; (2) Weak domain adaptability: The terminology and logical relationships of professional fields (such as medical and legal fields) are difficult to be accurately captured by general models; (3) Poor dynamic scalability: After the knowledge base is updated, the verification samples cannot be quickly expanded synchronously, and manual annotation is required repeatedly; (4) Imbalance between efficiency and cost: Large-scale question answering relies on high computing power or manual intervention for verification, making it difficult to achieve an automated pipeline. Summary of the Invention
[0003] The present invention aims to solve at least one of the above-mentioned technical problems.
[0004] To address the above problems, this invention provides a question-answering model verification method, system, electronic device, and storage medium.
[0005] In a first aspect, the present invention provides a question-answering model verification method, comprising:
[0006] The original knowledge base Q is automatically generated into a dynamically enhanced knowledge base Q+ through semantic expansion;
[0007] The dynamically enhanced knowledge base Q+ is input into the target model and at least one external benchmark model respectively to obtain the answer set A and at least one benchmark answer set A+.
[0008] The matching degree is obtained by performing a mixed similarity calculation between the answer set A and the benchmark answer set A+.
[0009] The overall similarity is calculated according to preset rules and compared with a preset dynamic threshold to generate a corresponding verification report.
[0010] Optionally, the hybrid similarity calculation includes: calculating the vector cosine similarity between the answer set A and the benchmark answer set A+ using a domain-optimized embedding model; and verifying the matching of key entities and relationships based on a knowledge graph.
[0011] Optionally, the hybrid similarity calculation includes: extracting the core information of the paragraph texts from the answer set A and the benchmark answer set A+, including core viewpoints, key arguments, and main conclusions, with each paragraph text extracting no more than 3 sentences;
[0012] The core information is compared across dimensions including: whether the themes are consistent, the proportion of overlapping viewpoints, and whether the semantic logic is similar.
[0013] The function outputs a similarity score between 0 and 100. If one side is empty, the text is considered empty by default, and the function returns a score of 0.
[0014] Optionally, the semantic expansion includes generating semantically equivalent diverse questions for infinitive and fixed-form questions based on a large language model. The expansion methods include: synonym replacement, random swapping, random deletion, random insertion, sentence and semantic changes, misspelling recognition and replacement, keyboard error simulation, OCR error simulation, back translation, and adversarial training and testing.
[0015] Optionally, the preset rule calculation includes: calculating the similarity between the answer set A and the benchmark answer set A+ based on multi-dimensional indicators, wherein the multi-dimensional indicators and their corresponding weight ratios are: 35% matching degree, 5% accuracy, 10% precision, 10% recall, 10% F1 score, and 30% task completion rate; and obtaining the comprehensive similarity by weighting the obtained comparison similarities according to the preset weight allocation.
[0016] Optionally, the dynamic threshold is adaptively adjusted based on historical data distribution and domain risk level; when the overall similarity is greater than the dynamic threshold or / and each of the comparison similarities is greater than the dynamic threshold, the target model passes the verification condition.
[0017] Optionally, the verification report includes failure attribution and optimization recommendations.
[0018] Secondly, the present invention provides a question-answering model verification system, comprising:
[0019] The extension module automatically generates a dynamically enhanced knowledge base Q+ from the original knowledge base Q through semantic expansion;
[0020] The collaborative module inputs the dynamically enhanced knowledge base Q+ into the target model and at least one external benchmark model respectively, to obtain the answer set A and at least one benchmark answer set A+;
[0021] The comparison module performs a mixed similarity calculation on the answer set A and the benchmark answer set A+ to obtain the matching degree;
[0022] The verification module calculates the overall similarity based on preset rules and compares it with a preset dynamic threshold to generate a corresponding verification report.
[0023] Thirdly, the present invention provides an electronic device, including a memory and a processor;
[0024] The memory is used to store computer programs;
[0025] The processor is configured to implement a question-answering model verification method as described in the first aspect when executing the computer program.
[0026] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a question-answering model verification method as described in the first aspect.
[0027] The technical solution of this invention has the following advantages: (1) Based on the semantic expansion generation technology of the large language model, it automatically expands the diverse question library Q+, effectively covering the long-tail scenario of user questions and solving the verification blind spot caused by the single sample of the traditional method; (2) Accuracy upgrade, adopting a hybrid similarity calculation model, combining domain-optimized semantic vector matching and knowledge graph logic verification, simultaneously capturing the surface semantic association and deep factual consistency of the answer, avoiding misjudgment by a single indicator; (3) Efficiency leap, relying on the automated pipeline to achieve parallel processing of thousands of question-answer pairs and minute-level verification, supporting the dynamic expansion of the knowledge base and the real-time adaptation of model iteration, improving efficiency by hundreds of times compared with manual review; (4) Enhanced interpretability, through the semantic deviation location of failed cases, logical contradiction analysis and knowledge loss tracing, a visual attribution report is generated, directly guiding model optimization and knowledge base completion. In high-precision scenarios such as intelligent customer service and medical consultation, this solution takes into account the breadth, depth and efficiency of verification, providing reliable technical support for the large-scale deployment of AI products. Attached Figure Description
[0028] Figure 1 This is a flowchart illustrating the question-answering model verification method according to an embodiment of the present invention;
[0029] Figure 2 This is a business process diagram of the question-answering model verification method according to an embodiment of the present invention;
[0030] Figure 3 This is a schematic diagram of the architecture of the question-answering model verification system according to an embodiment of the present invention. Detailed Implementation
[0031] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.
[0032] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.
[0033] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.
[0034] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0035] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.
[0036] like Figure 1 and Figure 2 As shown, this embodiment of the invention provides a question-answering model verification method, which verifies the correctness of the target AI model through multiple external AI models; in particular, it verifies the accuracy of semantic recognition by combining fixed and indefinite answers, and verifies whether the target model can produce the correct answer for indefinite questions.
[0037] The entire process is as follows:
[0038] Step S1: Automatically generate a dynamically enhanced knowledge base Q+ from the original knowledge base Q. Specifically, for the infinitive questions and definite questions in the original knowledge base Q, diverse questions with semantic equivalence are generated based on large language models and Prompt engineering. The expansion methods include but are not limited to: synonym replacement, random swapping, random deletion, random insertion, sentence and semantic changes, typo recognition and replacement, keyboard error simulation, OCR error simulation, back translation, and adversarial training testing. For example, for the original question: "How to relieve indigestion?" After variation, it can be generated in various forms such as: "What should I do if I have bloating and indigestion?", "What medicine can improve gastric food retention?", "After meals, I always feel that the food is not digested. What are the solutions?", etc.
[0039] Step S2: Input the dynamically enhanced knowledge base Q+ into the target model and at least one external benchmark model respectively to obtain the answer set A and at least one benchmark answer set A+. Specifically, in this embodiment, the HGT model is used as the target model, and the external benchmark models are selected from one or more of Qwen3.0, deepseek, and chatGPT. Traverse the dynamically enhanced knowledge base Q+, the HGT model returns the answer set A, and the external benchmark models respectively return one or more of the benchmark answer sets A+1, A+2, A+3 correspondingly.
[0040] Step S3: Calculate the mixed similarity between the answer set A and each benchmark answer set A+ respectively to obtain one or more matching degrees. Specifically, the mixed similarity calculation includes: semantic similarity, using a domain-optimized embedding model, such as domain-fine-tuned BERT, to calculate the vector cosine similarity between the answer set A and each benchmark answer set A+; logical consistency, based on the knowledge graph, verify the matching of key entities and relationships, such as "Shenling Baizhu San → efficacy → invigorate the spleen", and the Apache Jena rule engine can be selected. Matching degree = α × semantic similarity + β × logical consistency, where α and β are adaptively adjusted according to the domain (e.g., in the medical field β = 0.4, in the general scenario β = 0.2)
[0041] The specific operations include:
[0042] S31: Respectively extract the core information of the paragraph texts from the answer set A and each benchmark answer set A+, including the core viewpoints, key arguments, and main conclusions. The extraction of each paragraph text does not exceed 3 sentences;
[0043] S32: Compare the core information, and the dimensions include: whether the topics are consistent, the proportion of overlapping viewpoints, and whether the semantic logics are similar;
[0044] S33: Give a similarity score from 0 to 100, with 100 being completely consistent and 0 being completely irrelevant; when one side is empty, it is default that the text is empty, and the return value is 0 points.
[0045] Step S4: Calculate the comprehensive similarity according to preset rules and compare it with a preset dynamic threshold to generate a corresponding verification report. The preset rules include:
[0046] S41. Obtain the similarity between answer set A and each benchmark answer set A+. The similarity is the weighted sum of multiple indicators, specifically: 35% matching accuracy, 5% precision, 10% accuracy, 10% recall, 10% F1 score, and 30% task completion rate. Accuracy is (TP+TN) / (TP+TN+FP+FN); precision is TP / (TP+FP); recall is TP / (TP+FN). The accuracy is calculated by asking the same standard positive question 100 times. The F1 score is the harmonic mean of precision and recall, comprehensively measuring model performance; 2×(Precision×Recall) / (Precision+Recall); task completion rate is the percentage of users who ultimately achieve their goals through dialogue, i.e., the number of completed tasks / total number of tasks. Where TP: Samples correctly predicted as positive. TN: Samples correctly predicted as negative. FP: Samples incorrectly predicted as positive. FN: Mistakenly predicts a positive class as a negative class.
[0047] S42. The obtained similarities from each comparison are weighted according to a pre-set weight allocation to obtain a comprehensive similarity. For example, in this embodiment, if the external benchmark models used are Qwen3.0, deepseek, and chatGPT, then the comprehensive similarity F = 40% * f1 + 30% * f2 + 30% * f3, where f1 is the similarity between answer set A and the benchmark answer set A+1 generated by Qwen3.0, f2 is the similarity between answer set A and the benchmark answer set A+2 generated by deepseek, and f3 is the similarity between answer set A and the benchmark answer set A+3 generated by chatGPT. If only deepseek is used as the external benchmark model, then the comprehensive similarity F = 100% * f2.
[0048] The dynamic threshold is adaptively adjusted based on historical data distribution and domain risk level. For example, the dynamic threshold is adjusted according to the product characteristics of the intelligent agent. For instance, for question-and-answer products that involve a lot of preamble, the dynamic threshold is 80%; for product-based products that require high numerical accuracy, the dynamic threshold is 95%; and for medical products that require high accuracy, the dynamic threshold is 85%.
[0049] The target model passes validation when the overall similarity F is greater than the dynamic threshold; alternatively, for high accuracy requirements, the similarity of each comparison must be greater than the dynamic threshold for the target model to pass validation; or, both the overall similarity F and the similarity of each comparison must be greater than the dynamic threshold for the target model to pass validation. If it fails, the validation report outputs failure attribution and optimization suggestions, such as semantic deviation, logical contradiction, or knowledge gaps; it can also trigger manual review or model optimization alerts.
[0050] This invention provides a question-answering model verification system 300, comprising:
[0051] Extension module 310 automatically generates a dynamically enhanced knowledge base Q+ from the original knowledge base Q through semantic extension;
[0052] The collaborative module 320 inputs the dynamically enhanced knowledge base Q+ into the target model and at least one external benchmark model respectively, and obtains the answer set A and at least one benchmark answer set A+.
[0053] The comparison module 330 performs a mixed similarity calculation between the answer set A and the benchmark answer set A+ to obtain the matching degree.
[0054] The verification module 340 calculates the comprehensive similarity according to preset rules and compares it with a preset dynamic threshold to generate a corresponding verification report.
[0055] An electronic device provided by an embodiment of the present invention includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement a question-answering model verification method as described above when the computer program is executed.
[0056] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements a question-answering model verification method as described above.
[0057] The present invention will now describe electronic devices that can serve as servers or clients of the present invention, which are examples of hardware devices that can be applied to various aspects of the present invention. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0058] Electronic devices include a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM can also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.
[0059] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. In this application, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention according to actual needs. Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units can be implemented in hardware or as software functional units.
[0060] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.
Claims
1. A question-answering model verification method, characterized in that... include: The original knowledge base Q is automatically generated into a dynamically enhanced knowledge base Q+ through semantic expansion; The dynamically enhanced knowledge base Q+ is input into the target model and at least one external benchmark model respectively to obtain the answer set A and at least one benchmark answer set A+. The matching degree is obtained by performing a mixed similarity calculation between the answer set A and the benchmark answer set A+. The overall similarity is calculated according to preset rules and compared with a preset dynamic threshold to generate a corresponding verification report.
2. The question-answering model verification method as described in claim 1, characterized in that... The hybrid similarity calculation includes: The domain-optimized embedding model is used to calculate the vector cosine similarity between the answer set A and the benchmark answer set A+. The matching of key entities and relationships is verified based on knowledge graphs.
3. The question-answering model verification method as described in claim 2, characterized in that... The hybrid similarity calculation includes: Extract the core information from the paragraph texts from the answer set A and the benchmark answer set A+, including the core viewpoints, key arguments, and main conclusions. Each paragraph text should be extracted in no more than 3 sentences. The core information is compared across dimensions including: whether the themes are consistent, the proportion of overlapping viewpoints, and whether the semantic logic is similar. The function outputs a similarity score between 0 and 100. If one side is empty, the text is considered empty by default, and the function returns a score of 0.
4. The question-answering model verification method as described in claim 1, characterized in that... The semantic expansion includes generating semantically equivalent diverse questions based on a large language model for infinitive and fixed-form questions. The expansion methods include: synonym replacement, random swapping, random deletion, random insertion, sentence and semantic changes, misspelling recognition and replacement, keyboard error simulation, OCR error simulation, back translation, and adversarial training and testing.
5. The question-answering model verification method as described in claim 1, characterized in that... The preset rule calculation includes: The similarity between the answer set A and the benchmark answer set A+ is calculated based on multi-dimensional indicators. The multi-dimensional indicators and their corresponding weight ratios are: 35% matching degree, 5% accuracy, 10% precision, 10% recall, 10% F1 score, and 30% task completion rate. The obtained comparison similarities are weighted and calculated according to a preset weight allocation to obtain the comprehensive similarity.
6. The question-answering model verification method as described in claim 5, characterized in that... The dynamic threshold is adaptively adjusted based on historical data distribution and domain risk level; The target model passes the verification condition when the overall similarity is greater than the dynamic threshold or / and each of the comparison similarities is greater than the dynamic threshold.
7. The question-answering model verification method as described in claim 1, characterized in that... The verification report includes failure attribution and optimization recommendations.
8. A question-answering model verification system, characterized in that... include: The extension module automatically generates a dynamically enhanced knowledge base Q+ from the original knowledge base Q through semantic expansion; The collaborative module inputs the dynamically enhanced knowledge base Q+ into the target model and at least one external benchmark model respectively, to obtain the answer set A and at least one benchmark answer set A+; The comparison module performs a mixed similarity calculation on the answer set A and the benchmark answer set A+ to obtain the matching degree; The verification module calculates the overall similarity based on preset rules and compares it with a preset dynamic threshold to generate a corresponding verification report.
9. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement a question-answering model verification method as described in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements a question-answering model verification method as described in any one of claims 1 to 7.