Intelligent question and answer error detection method and device, equipment and medium

Abstract

The invention relates to the field of artificial intelligence, and discloses an intelligent question and answer error detection method, device and equipment and a medium, and the method comprises the steps: obtaining an original answer generated by a question and answer robot based on an original user question, and taking the original answer as a to-be-detected fact; generating at least one follow-up question according to the fact to be detected, and receiving an answer generated by the question-answering robot based on the follow-up question; performing consistency verification according to the follow-up question, the corresponding answer and the fact to be detected; and performing error detection on the fact to be detected according to a consistency verification result, and outputting a corresponding error detection result. According to the method, follow-up question answering is carried out on the fact generated by the question answering robot, and consistency verification is carried out on the follow-up question answering interaction result and the fact, so that zero-sample fact error detection is effectively realized, the efficiency and reliability of fact error detection on the question answering robot are improved, and more reliable information service is provided for a user.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to an error detection method, device, equipment and medium for intelligent question and answering. Background Art

[0002] At present, question-answering robots based on large language models are being used more and more widely in the financial field. Question-answering robots are an intelligent dialogue system based on artificial intelligence technology. They identify, understand and analyze questions raised by users, and then use knowledge graphs, semantic computing and other technologies to filter out relevant information from massive data. Finally, they generate concise and clear answers, and provide users with automated financial question-answering services through a human-computer interaction interface.

[0003] However, a common phenomenon in large language models is hallucination. Hallucination refers to the fact that large language models generate content that seems reasonable but is incorrect or irrelevant to the question. In financial scenarios where accuracy requirements are very high, the hallucinations of question-answering robots are unacceptable, so the generated answers need to be checked for factual errors.

[0004] Existing factual error detection methods usually rely on large amounts of annotated data and complex model structures, which have high computational costs when processing large-scale data and have weak adaptability to novel fields, thus affecting the efficiency and reliability of factual error detection for question-answering services in the financial field. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide an error detection method, device, equipment and medium for intelligent question and answering that can be applied to financial technology or other related fields, the main purpose of which is to improve the efficiency and reliability of factual error detection in question and answer services.

[0006] The technical solution of the present invention is as follows:

[0007] A first aspect of the present invention provides an error detection method for intelligent question answering, comprising:

[0008] Obtain the original answer generated by the question-answering robot based on the original user's question as the fact to be detected;

[0009] generating at least one follow-up question according to the fact to be detected, and receiving an answer generated by the question-and-answer robot based on the follow-up question;

[0010] Performing consistency verification based on the follow-up questions, the corresponding answers, and the facts to be tested;

[0011] According to the consistency verification result, error detection is performed on the fact to be detected, and a corresponding error detection result is output.

[0012] A second aspect of the present invention provides an error detection device for intelligent question answering, comprising:

[0013] An acquisition module is used to acquire the original answer generated by the question-answering robot based on the original user's question as a fact to be detected;

[0014] A question follow-up module, configured to generate at least one follow-up question according to the fact to be detected, and receive an answer generated by the question-and-answer robot based on the follow-up question;

[0015] A consistency verification module, used for performing consistency verification based on the follow-up questions, the corresponding answers and the facts to be detected;

[0016] The fact error detection module is used to perform error detection on the fact to be detected according to the consistency verification result, and output the corresponding error detection result.

[0017] A third aspect of the present invention provides a computer device, comprising at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned intelligent question and answer error detection method.

[0020] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned intelligent question and answer error detection method.

[0021] Beneficial effects: The present invention discloses an error detection method, device, equipment and medium for intelligent question and answer. Compared with the prior art, the embodiment of the present invention obtains the original answer generated by the question and answer robot based on the original user's question as the fact to be detected; generates at least one follow-up question based on the fact to be detected, and receives the answer generated by the question and answer robot based on the follow-up question; performs consistency verification based on the follow-up question, the corresponding answer and the fact to be detected; performs error detection on the fact to be detected based on the consistency verification result, and outputs the corresponding error detection result. By performing follow-up questions and answers on the facts generated by the question and answer robot, and performing consistency verification on the follow-up question and answer interaction results and the facts, zero-sample fact error detection is effectively achieved, and the efficiency and reliability of fact error detection for the question and answer robot are improved, thereby providing users with more reliable information services. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the scheme in the present invention, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A schematic diagram of an application environment of the error detection method for intelligent question answering provided by an embodiment of the present invention;

[0024] Figure 2 A flow chart of an error detection method for intelligent question answering provided by an embodiment of the present invention;

[0025] Figure 3 A flow chart of step S202 in the error detection method of intelligent question and answer provided in an embodiment of the present invention;

[0026] Figure 4 A flow chart of step S203 in the error detection method of intelligent question and answer provided in an embodiment of the present invention;

[0027] Figure 5 A flow chart of step S204 in the error detection method of intelligent question and answer provided in an embodiment of the present invention;

[0028] Figure 6 A schematic diagram of the functional modules of the error detection device for intelligent question and answer provided by an embodiment of the present invention;

[0029] Figure 7 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0031] The error detection method for intelligent question answering provided by the embodiment of the present invention can be applied to Figure 1 In the application environment of FIG. 1 , it includes a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0032] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0033] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0034] The server 105 may be a server that provides various services, such as a background server that provides support for the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (for example only). The background server may analyze and process the received data such as the user request, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device. The server 105 may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server 105 may also be a server for a distributed system, or a server combined with a blockchain.

[0035] It should be noted that the error detection method of the intelligent question and answer provided in the embodiment of the present application can generally be performed by the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the error detection device of the intelligent question and answer provided in the embodiment of the present invention can also be set in the first terminal device 101, the second terminal device 102 or the third terminal device 103. Alternatively, the error detection method of the intelligent question and answer provided in the embodiment of the present invention can also generally be performed by the server 105. Accordingly, the error detection device of the intelligent question and answer provided in the embodiment of the present invention can generally be set in the server 105.

[0036] It should be understood that the numbers of the above terminal devices, networks and servers are only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0037] like Figure 2 As shown, the error detection method of intelligent question answering provided by the embodiment of the present invention specifically includes the following steps:

[0038] S201. Obtain the original answer generated by the question-answering robot based on the original user's question as the fact to be detected.

[0039] In this embodiment, the question-answering robot refers to a system based on artificial intelligence, such as an intelligent customer service system, preferably a question-answering robot constructed and trained based on a large language model (LLM). The large language model can handle a variety of natural language tasks, such as text classification, question-answering, dialogue, etc., so that the question-answering robot can interact with users through natural language and provide corresponding answers based on the users' questions.

[0040] When the Q&A robot interacts with the user, the user first asks a question. After receiving the original user's question, the Q&A robot retrieves the content related to the original user's question through the retrieval model. If a matching QA (question-answer pair) is retrieved, the answer is directly output; if a matching QA is not retrieved, other content will continue to be retrieved and a prompt word with content will be output to the large language model, and the model will generate an answer as the answer. In order to ensure that the answer provided by the Q&A robot to the user is accurate and to minimize the hallucination problem of the Q&A robot, it is necessary to perform fact error detection on the answer of the Q&A robot, that is, before feeding back the answer to the user, the answer generated by the Q&A robot based on the original user's question is obtained as the fact to be detected to ensure the reliability of the Q&A service.

[0041] For example, in a specific implementation, in a financial insurance scenario, a user asks the question-and-answer robot through the front-end interactive interface, "What responsibilities does your company's auto insurance include?" The question-and-answer robot generates a corresponding answer based on the original user's question received, such as "Our company's auto insurance includes compulsory traffic insurance, third-party liability insurance, vehicle damage insurance, and driver and passenger insurance." The answer to the original user's question is obtained as the fact to be tested to further verify its accuracy.

[0042] S202: Generate at least one follow-up question according to the fact to be detected, and receive an answer generated by the question-and-answer robot based on the follow-up question.

[0043] In this embodiment, further question-answering interaction is performed based on the facts to be detected generated by the question-answering robot, and at least one related follow-up question is generated based on the facts to be detected and input into the question-answering robot, so that the question-answering robot answers based on the follow-up question. Specifically, a question bank containing common questions and possible answers can be established in advance, and follow-up questions can be selected from the question bank to reduce the time for generating questions on site. Another large language model can also be used to generate corresponding prompt text based on the facts to be detected to prompt another large language model to generate related questions, thereby reducing human intervention and not requiring a large amount of labeled data, which can effectively improve efficiency.

[0044] For example, in a specific implementation, in a financial insurance scenario, a follow-up question is further generated for the fact to be detected generated by the question-and-answer robot, "Our company's auto insurance includes compulsory traffic insurance, third-party liability insurance, vehicle damage insurance, and driver and passenger insurance," and the answer generated by the question-and-answer robot based on the follow-up question, such as "Yes, our company's auto insurance also includes theft insurance?", is received. Through at least one round of follow-up questioning interaction, more auxiliary information about the fact to be detected is obtained to verify the fact to be detected.

[0045] Among them, the number of follow-up questions can be flexibly set based on the situation of the facts to be detected, the requirements of the detection scenario for accuracy and other factors. For example, the more complete the generated facts to be detected, the fewer rounds of follow-up questions are required, and the higher the requirements of the detection scenario for accuracy, the more rounds of follow-up questions are required, so as to balance the accuracy and response time of the question-answering robot. Preferably, a maximum number of question rounds can be set when following up questions, that is, a maximum number of follow-up questions is set for the facts to be detected. Once this number is reached and no obvious errors or inconsistencies are found, the facts to be detected can be considered correct. Limiting the number of rounds helps to avoid excessive resource consumption and ensure that the question-answering robot can respond to user needs in a timely manner.

[0046] S203: Perform consistency verification based on the follow-up questions, the corresponding answers, and the facts to be detected.

[0047] In this embodiment, since the follow-up questions can require the Q&A robot to provide more details or examples related to the facts to be detected or solutions to hypothetical questions, etc., so as to obtain information from multiple angles, the facts to be detected generated by the Q&A robot in response to the original user's questions, and the subsequent follow-up questions and corresponding answers are verified for consistency, such as detecting whether the answer directly responds to the follow-up questions and whether it is consistent with the facts to be detected, evaluating whether the logic in the answer is consistent with the logic of the follow-up questions and whether it can reasonably support the facts to be detected, etc. Specifically, the quantitative score can be calculated by a preset function, or by using a machine learning model, such as a support vector machine (SVM) or a class of support vector machines (OCSVM), etc. to detect abnormal patterns of multiple information, thereby verifying the consistency between the follow-up questions, answers and facts to be detected.

[0048] S204: Perform error detection on the fact to be detected according to the consistency verification result, and output a corresponding error detection result.

[0049] In this embodiment, error detection is performed based on the consistency verification results between the follow-up questions, answers, and facts to be detected. Specifically, if the consistency verification results reveal that there are inconsistencies between the follow-up questions, answers, and facts to be detected, such as the answers to the follow-up questions are inconsistent with the facts to be detected, the logic in the answers to the follow-up questions is different from or even completely opposite to the logic in the facts to be detected, etc., then it is confirmed that there are errors in the facts to be detected, and such facts to be detected can be marked as potential errors, and further manual review or automatic adjustment can be performed to ensure the reliability of the answers that the question-answering robot finally feeds back to the user; if the consistency verification results do not reveal that there are inconsistencies between the follow-up questions, answers, and facts to be detected, then the facts to be detected are confirmed to be correct, and the facts to be detected are returned to the user as the final answer. Through automated cross-inquiry and fact detection, zero-sample fact error detection can be achieved, errors can be discovered and corrected earlier, and the operational efficiency and reliability of the question-answering robot can be improved.

[0050] For example, in a specific implementation, in a financial insurance scenario, a consistency verification is performed for the fact to be tested generated by the question-and-answer robot, "Our company's auto insurance includes compulsory traffic insurance, third-party liability insurance, vehicle damage insurance, and driver and passenger insurance", and a further follow-up question, "Does your company's auto insurance include theft and robbery insurance?" and the answer generated by the question-and-answer robot based on the follow-up question, such as "Yes, our company's auto insurance also includes theft and robbery insurance". In this example, the consistency verification result does not reveal any inconsistency between the follow-up question, the answer, and the fact to be tested (i.e., the insurance liabilities described by the question-and-answer robot are all correct), and therefore it can be concluded that the fact to be tested is correct.

[0051] In the above embodiment, the present invention discloses an error detection method for intelligent question and answer, which obtains the original answer generated by the question and answer robot based on the original user's question as the fact to be detected; generates at least one follow-up question based on the fact to be detected, and receives the answer generated by the question and answer robot based on the follow-up question; performs consistency verification based on the follow-up question, the corresponding answer and the fact to be detected; performs error detection on the fact to be detected based on the consistency verification result, and outputs the corresponding error detection result. By performing follow-up questions and answers on the facts generated by the question and answer robot, and performing consistency verification on the follow-up question and answer interaction results and the facts, zero-sample fact error detection is effectively achieved, and the efficiency and reliability of fact error detection for the question and answer robot are improved, thereby providing users with more reliable information services.

[0052] In one embodiment, Figure 3 As shown, step S202 includes:

[0053] S301, evaluating and analyzing the fact to be detected using preset evaluation indicators to confirm whether the fact to be detected meets various evaluation indicators;

[0054] S302: Generate corresponding follow-up questions according to the evaluation indicators that the to-be-detected facts do not satisfy, and receive answers generated by the question-and-answer robot based on the follow-up questions.

[0055] In this embodiment, when following up on the interaction of the problem based on the fact to be detected, the fact to be detected is evaluated and analyzed by preset evaluation indicators to confirm whether the fact to be detected meets various evaluation indicators. Specific evaluation indicators may include, for example, completeness, consistency, logic, accuracy, and relevance, etc., wherein, when confirming whether the fact to be detected meets various evaluation indicators, the completeness evaluation can be performed by confirming whether the fact to be detected comprehensively covers all aspects of the original user's question without missing any key information; the consistency evaluation can be performed by comparing whether the fact to be detected is consistent with the known facts and data, and there are no contradictions in the various parts of the fact to be detected; the logic evaluation can be performed by confirming whether the fact to be detected is logical and whether the reasoning process is reasonable; the accuracy evaluation can be performed by checking whether the data and facts in the fact to be detected are accurate; the relevance evaluation can be performed by confirming whether the fact to be detected is closely related to the original user's question and does not deviate from the topic, etc.

[0056] Based on the evaluation and analysis results, corresponding follow-up questions are generated for the evaluation indicators that are not met by the facts to be tested, that is, which evaluation indicators are not met are identified, and follow-up questions are generated based on the unmet evaluation indicators. For example, if the consistency indicator is not met, questions can be generated to further explore the source or evidence of the facts, and then input them into the question-answering robot to obtain answers to follow-up questions. This allows the direction of follow-up questions to be confirmed based on a comprehensive evaluation of the facts to be tested, effectively improving the accuracy of follow-up questions during cross-examination, and providing a more reliable basis for factual error detection.

[0057] In one embodiment, generating corresponding follow-up questions according to the evaluation index not satisfied by the to-be-detected fact, and receiving answers generated by the question-and-answer robot based on the follow-up questions, includes:

[0058] Generate a corresponding number of follow-up questions according to the evaluation indicators that the facts to be detected do not meet, and generate a question list;

[0059] The question list is input into the question-answering robot, a corresponding number of answers generated by the question-answering robot based on the question list are received, and an answer list is generated.

[0060] In this embodiment, for the evaluation indicators that are not met in the facts to be detected, follow-up question-answering interactions are performed in a one-time generation manner, that is, all follow-up questions are generated at one time based on the evaluation indicators that are not met and added to the question list, the question list is input into the question-answering robot, the question-answering robot processes each question in the question list and generates a corresponding answer, and each answer generated by the question-answering robot is matched with the corresponding follow-up question, and summarized into an answer list. Efficient follow-up processing is achieved by generating follow-up questions for interaction in a one-time manner, thereby ensuring the response speed of the question-answering robot.

[0061] For example, a user asks: "Does our company's medical insurance cover heart bypass surgery?" The question-and-answer robot generates a fact to be tested: "Yes, our company's medical insurance policy includes heart bypass surgery." The fact to be tested is evaluated for completeness, consistency, logic, accuracy, and relevance. If the evaluation finds that it lacks completeness, consistency, and logic, corresponding follow-up questions can be generated at one time, such as a follow-up question for completeness: "What are the specific restrictions on the coverage of heart bypass surgery? For example, are there restrictions on specific hospitals or doctors?" A follow-up question for consistency: "Does this coverage apply to all insured persons, or only to specific groups?" A follow-up question for logic: "If heart bypass surgery is required, does the insurance coverage include all relevant preliminary examinations and follow-up treatments?" These follow-up questions are generated into a question list and input into the question-and-answer robot, so that the question-and-answer robot processes each question in the question list sequentially or in parallel and generates corresponding answers, thereby gaining a more comprehensive understanding of whether the facts to be tested are comprehensive and accurate.

[0062] In one embodiment, generating corresponding follow-up questions according to the evaluation index not satisfied by the to-be-detected fact, and receiving answers generated by the question-and-answer robot based on the follow-up questions, includes:

[0063] Prioritizing the evaluation indicators that are not satisfied by the facts to be detected;

[0064] Generate a first follow-up question according to the evaluation indicator with the highest priority, and receive an answer generated by the question-and-answer robot based on the first follow-up question;

[0065] A second follow-up question is generated according to the evaluation indicator with the second highest priority, and an answer generated by the question-and-answer robot based on the second follow-up question is received, and so on, until there are no new update questions or the maximum number of question rounds is reached.

[0066] In this embodiment, after identifying the evaluation indicators that are not met by the fact to be detected, such as accuracy, consistency, completeness and reliability, a priority is assigned to each unmet evaluation indicator based on business needs and policies. For example, if accuracy is critical to the current application scenario, it can be given the highest priority. The evaluation indicators are sorted according to the assigned priorities to ensure that the most important indicators are processed first, so that when following up on the issues, the most important or critical questions for the error detection of the fact to be detected can be asked first.

[0067] When generating follow-up questions, select the evaluation indicator with the highest priority to generate the first follow-up question. For example, if accuracy is the highest priority, the follow-up question may be a request to provide the source or evidence of the fact, submit the generated first follow-up question to the question-answering robot and receive the answer generated by the question-answering robot for the first follow-up question. After processing the highest priority indicator, select the next highest priority evaluation indicator, generate the second follow-up question for the newly selected evaluation indicator, and submit it to the question-answering robot for answering, until all follow-up questions related to the unsatisfied evaluation indicators have obtained corresponding answers, or the preset maximum number of question rounds is reached. Different from the follow-up questions that generate all unsatisfied evaluation indicators at one time in the previous embodiment, the method of sorting by priority and processing questions one by one can dynamically adjust the subsequent follow-up questions according to the situation of each round of answers. For example, if the answers of a certain round meet the requirements of two evaluation indicators at the same time, a round of follow-up question-answering interaction can be omitted later, making the entire question-answering process more flexible and adaptable, and can more effectively utilize resources to avoid wasting time on unimportant issues.

[0068] In one embodiment, Figure 4 As shown, step S203 includes:

[0069] S401, constructing a set of objects to be verified by combining the facts to be detected, each follow-up question and the corresponding answers;

[0070] S402, performing inconsistency calculation on each group of objects to be verified to obtain a corresponding inconsistency score;

[0071] S403: Calculate a cumulative inconsistency score according to the inconsistency score of each group of objects to be verified, and generate a consistency verification result.

[0072] In this embodiment, when the follow-up questions, answers and facts to be detected are checked for consistency through quantitative scores, the facts to be detected, each follow-up question and the corresponding answers are specifically constructed to obtain a set of objects to be verified. By pairing the facts, questions and answers, a structured data set is formed to facilitate subsequent consistency calculations, and each object to be verified contains contextual information, which helps to more accurately evaluate inconsistencies.

[0073] Perform inconsistency calculations on each group of objects to be verified, for example, extract features from the objects to be verified, including text similarity, semantic consistency, logical rationality of facts, etc., and calculate the inconsistency score of each group of objects to be verified based on the extracted features through a preset function to quantify the degree of inconsistency of each group of objects to be verified. Calculate the inconsistency scores of all objects to be verified to obtain a total inconsistency score, i.e., a cumulative inconsistency score. The cumulative inconsistency score can be calculated, for example, by calculating the average of all inconsistency scores, or by assigning different weights to the inconsistency scores of each round of dialogue, and calculating the weighted average of all inconsistency scores. Generate consistency verification results based on the quantitative calculation results, for example, a detailed report can be generated, including the inconsistency scores of each group of objects to be verified and the overall cumulative inconsistency scores, etc., so as to comprehensively evaluate the consistency of the entire fact to be detected, and provide a basis for error detection to determine whether to accept the fact to be detected through the consistency verification results, which helps to improve the reliability of fact error detection of the question-answering robot.

[0074] In one embodiment, performing inconsistency calculation on each group of objects to be verified to obtain a corresponding inconsistency score includes:

[0075] Extract features for each group of objects to be verified to obtain fact features, question features, and answer features;

[0076] Calculating a first consistency score between the answer feature and the question feature, and a second consistency score between the answer feature and the fact feature;

[0077] An inconsistency score for each group of objects to be verified is calculated based on the first consistency score, the second consistency score, and the difference between the consistency scores.

[0078] In this embodiment, when specifically calculating the inconsistency score of each group of objects to be verified, the facts, follow-up questions and answers in each group of objects to be verified are feature extracted to obtain fact features, question features and answer features. For example, a user asks the question-answering robot: "What is the non-performing loan rate of this bank?" The question-answering robot generates a fact C to be detected: "According to our data, the non-performing loan rate of this bank is 1.5%." The follow-up question Q generated based on the fact to be detected is: "In the latest financial report, what changes have occurred in the non-performing loan rate of the bank?" The question-answering robot generates an answer A: "The latest financial report shows that the non-performing loan rate of the bank has dropped slightly." The features of fact C are extracted, such as keywords such as "non-performing loan rate" and "1.5%", the features of question Q are such as keywords such as "financial report", "bank", "non-performing loan rate", "change", and the features of answer A are such as keywords such as "financial report", "bank", "non-performing loan rate", "decline".

[0079] Based on the extracted features, the first consistency score between the answer features and the question features is calculated, that is, the consistency score S between the answer A and the question Q AQ , and calculate the second consistency score between the answer feature and the fact feature, that is, the consistency score S between the answer A and the fact C AQ The specific method of calculating the consistency score can be, for example, calculating the overlap of keywords by using the Jaccard index, or calculating the cosine similarity between two vectors after converting the features into a vector space, etc. This embodiment does not limit this. AQ , the second consistency score S AQ and the difference in consistency scores, i.e. S AQ -S AQ , calculate the inconsistency score of each group of objects to be verified, which can be specifically obtained by the formula:

[0080]

[0081] The inconsistency score D of each group of objects to be verified is calculated, so that quantitative evaluation data can be provided during error detection and the interpretability of factual error detection can be improved.

[0082] In one embodiment, Figure 5 As shown, step S204 includes:

[0083] S501, parsing and obtaining the cumulative inconsistency score in the consistency verification result;

[0084] S502, performing probability mapping on the accumulated inconsistency scores to convert them into corresponding correct probability values;

[0085] S503: When the correct probability value is greater than or equal to a preset threshold, confirm that the fact to be detected is correct; otherwise, confirm that the fact to be detected is wrong.

[0086] In this embodiment, a cumulative inconsistency score calculated based on all follow-up question-answer interaction results is parsed from the consistency verification results. The cumulative inconsistency score integrates the inconsistency scores between all follow-up questions, corresponding answers and the facts to be detected, and can form a comprehensive inconsistency overview.

[0087] The probability mapping function is used to perform probability mapping on the cumulative inconsistency score, such as the logistic function, the sigmoid function, etc., that is, the probability mapping function is applied to the cumulative inconsistency score to calculate the corresponding correct probability value. For example, using the logistic function Where D totalis the cumulative inconsistency score, k is an adjustment parameter, and an optimal value can be determined in advance through cross-validation and other methods to ensure the accuracy of probability mapping.

[0088] The cumulative inconsistency score is converted into a quantitative indicator representing correctness through probability mapping. The correct probability value can be used to compare with the preset probability value to determine its relative correctness. For example, according to the application scenario and business needs, a threshold of the correct probability value is set, such as 0.8 or 0.9. This value determines the minimum correct probability that the fact to be detected needs to reach in order to be confirmed as correct. If the correct probability value is greater than or equal to the threshold, the fact to be detected is confirmed to be correct. If the correct probability value is less than the threshold, the fact to be detected is confirmed to be wrong. Through the error detection method based on quantitative indicators, the detection process is more objective and consistent, ensuring that only the facts to be detected with high confidence are confirmed as correct and fed back to the user, while the facts to be detected that are confirmed to be wrong need to be corrected or manually intervened, effectively realizing zero-sample fact error detection, improving the efficiency and reliability of fact error detection for question-answering robots, and providing users with more reliable information services.

[0089] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0090] Further references Figure 6 , as a response to the above Figure 2 The present invention provides an embodiment of an error detection device for intelligent question answering, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0091] like Figure 6 As shown, the error detection device 60 for intelligent question answering described in this embodiment includes:

[0092] An acquisition module 601 is used to acquire an original answer generated by the question-answering robot based on the original user's question as a fact to be detected;

[0093] A question follow-up module 602, configured to generate at least one follow-up question according to the fact to be detected, and receive an answer generated by the question-answering robot based on the follow-up question;

[0094] A consistency verification module 603, configured to perform consistency verification based on the follow-up questions, the corresponding answers, and the facts to be detected;

[0095] The fact error detection module 604 is used to perform error detection on the fact to be detected according to the consistency verification result, and output the corresponding error detection result.

[0096] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which is more suitable for describing the error detection execution process of intelligent question and answer than a program. For the specific implementation method of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0097] In one embodiment, the problem follow-up module 602 includes:

[0098] An evaluation and analysis unit, used to evaluate and analyze the fact to be detected according to preset evaluation indicators, and confirm whether the fact to be detected meets various evaluation indicators;

[0099] The question updating unit is used to generate corresponding follow-up questions according to the evaluation index that the fact to be detected does not meet, and receive the answer generated by the question-and-answer robot based on the follow-up question.

[0100] In one embodiment, the problem following unit includes:

[0101] A question generating unit, used to generate a corresponding number of follow-up questions according to the evaluation indicators not satisfied by the facts to be detected, and generate a question list;

[0102] The answer receiving unit is used to input the question list into the question-answering robot, receive a corresponding number of answers generated by the question-answering robot based on the question list, and generate an answer list.

[0103] In one embodiment, the problem following unit includes:

[0104] A priority unit, used to prioritize the evaluation indicators that are not satisfied by the facts to be detected;

[0105] A circular follow-up unit is used to generate a first follow-up question according to the evaluation indicator with the highest priority, and receive the answer generated by the question-and-answer robot based on the first follow-up question; and generate a second follow-up question according to the evaluation indicator with the second highest priority and receive the answer generated by the question-and-answer robot based on the second follow-up question, and so on, until there are no new update questions or the maximum number of question rounds is reached.

[0106] In one embodiment, the consistency verification module 603 includes:

[0107] A construction unit, used to construct a set of objects to be verified by combining the facts to be detected, each follow-up question and the corresponding answers;

[0108] An inconsistency calculation unit, used to perform inconsistency calculation on each group of objects to be verified to obtain a corresponding inconsistency score;

[0109] The cumulative calculation unit is used to calculate the cumulative inconsistency score according to the inconsistency score of each group of objects to be verified, and generate a consistency verification result.

[0110] In one embodiment, the inconsistency calculation unit includes:

[0111] A feature extraction unit is used to extract features from each group of objects to be verified to obtain fact features, question features and answer features;

[0112] a first calculation unit, configured to calculate a first consistency score between the answer feature and the question feature, and a second consistency score between the answer feature and the fact feature;

[0113] The second calculation unit is used to calculate the inconsistency score of each group of objects to be verified according to the first consistency score, the second consistency score and the difference between the consistency scores.

[0114] In one embodiment, the fact error detection module 604 includes:

[0115] A parsing unit, configured to parse and obtain a cumulative inconsistency score in the consistency verification result;

[0116] A probability mapping unit, used to perform probability mapping on the accumulated inconsistency score to convert it into a corresponding correct probability value;

[0117] The error detection unit is used to confirm that the fact to be detected is correct when the correct probability value is greater than or equal to a preset threshold, otherwise confirm that the fact to be detected is wrong.

[0118] In the above embodiment, the present invention discloses an error detection device for intelligent question and answer, which obtains the original answer generated by the question and answer robot based on the original user's question as the fact to be detected; generates at least one follow-up question based on the fact to be detected, and receives the answer generated by the question and answer robot based on the follow-up question; performs consistency verification based on the follow-up question, the corresponding answer and the fact to be detected; performs error detection on the fact to be detected based on the consistency verification result, and outputs the corresponding error detection result. By performing follow-up questions and answers on the facts generated by the question and answer robot, and performing consistency verification on the follow-up question and answer interaction results and the facts, zero-sample fact error detection is effectively achieved, and the efficiency and reliability of fact error detection for the question and answer robot are improved, thereby providing users with more reliable information services.

[0119] Another embodiment of the present invention provides a computer device, such as Figure 7 As shown, the computer device 70 includes:

[0120] One or more processors 701 and memory 702, Figure 7 A processor 701 is used as an example for description. The processor 701 and the memory 702 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0121] Processor 701 is used to complete various control logics of computer device 70, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component or any combination of these components. In addition, processor 701 can also be any traditional processor, microprocessor or state machine. Processor 701 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0122] The memory 702 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the error detection method for intelligent question and answer in the embodiment of the present invention. The processor 701 executes various functional applications and data processing of the computer device 70 by running the non-volatile software programs, instructions and units stored in the memory 702, that is, the error detection method for intelligent question and answer in the above method embodiment is implemented.

[0123] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device 70, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to the computer device 70 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. One or more units are stored in the memory 702, and when executed by one or more processors 701, the steps of the error detection method of the intelligent question and answer in any of the above-mentioned method embodiments are executed.

[0124] In the above embodiment, the present invention discloses a computer device, which obtains the original answer generated by the question-and-answer robot based on the original user's question as the fact to be detected; generates at least one follow-up question based on the fact to be detected, and receives the answer generated by the question-and-answer robot based on the follow-up question; performs consistency verification based on the follow-up question, the corresponding answer and the fact to be detected; performs error detection on the fact to be detected based on the consistency verification result, and outputs the corresponding error detection result. By performing follow-up questions and answers on the facts generated by the question-and-answer robot, and performing consistency verification on the follow-up question and answer interaction results and the facts, zero-sample fact error detection is effectively achieved, and the efficiency and reliability of fact error detection for the question-and-answer robot are improved, thereby providing users with more reliable information services.

[0125] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the steps of the error detection method of intelligent question and answer in any of the above method embodiments are executed.

[0126] In the above embodiment, the present invention discloses a non-volatile computer-readable storage medium, which obtains the original answer generated by the question-and-answer robot based on the original user's question as the fact to be detected; generates at least one follow-up question based on the fact to be detected, and receives the answer generated by the question-and-answer robot based on the follow-up question; performs consistency verification based on the follow-up question, the corresponding answer and the fact to be detected; performs error detection on the fact to be detected based on the consistency verification result, and outputs the corresponding error detection result. By performing follow-up questions and answers on the facts generated by the question-and-answer robot, and performing consistency verification on the follow-up question and answer interaction results and the facts, zero-sample fact error detection is effectively achieved, and the efficiency and reliability of fact error detection for the question-and-answer robot are improved, thereby providing users with more reliable information services.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0128] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0129] In summary, the present invention discloses an error detection method, device, equipment and medium for intelligent question and answer, the method includes: obtaining the original answer generated by the question and answer robot based on the original user's question as the fact to be detected; generating at least one follow-up question based on the fact to be detected, and receiving the answer generated by the question and answer robot based on the follow-up question; performing consistency verification based on the follow-up question, the corresponding answer and the fact to be detected; performing error detection on the fact to be detected based on the consistency verification result, and outputting the corresponding error detection result. By performing follow-up questions and answers on the facts generated by the question and answer robot, and performing consistency verification on the follow-up question and answer interaction results and the facts, zero-sample fact error detection is effectively achieved, and the efficiency and reliability of fact error detection for the question and answer robot are improved, thereby providing users with more reliable information services.

[0130] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0131] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of the present application, they are merely used for illustration and do not represent actual use. It should be understood that the application of the present invention is not limited to the above examples, and that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes shall fall within the scope of protection of the claims attached to the present invention.

Claims

1. An error detection method for intelligent question answering, characterized in that: include: Obtain the original answer generated by the question-answering robot based on the original user's question as the fact to be detected; generating at least one follow-up question according to the fact to be detected, and receiving an answer generated by the question-and-answer robot based on the follow-up question; Performing consistency verification based on the follow-up questions, the corresponding answers, and the facts to be tested; According to the consistency verification result, error detection is performed on the fact to be detected, and a corresponding error detection result is output.

2. The error detection method for intelligent question answering according to claim 1, characterized in that: Generating at least one follow-up question according to the fact to be detected, and receiving an answer generated by the question-and-answer robot based on the follow-up question, comprises: Evaluate and analyze the facts to be detected according to preset evaluation indicators to confirm whether the facts to be detected meet various evaluation indicators; Generate corresponding follow-up questions according to the evaluation indicators that the facts to be detected do not meet, and receive answers generated by the question-and-answer robot based on the follow-up questions.

3. The error detection method for intelligent question answering according to claim 2, characterized in that: Generating corresponding follow-up questions according to the evaluation index not satisfied by the to-be-detected fact, and receiving answers generated by the question-and-answer robot based on the follow-up questions, includes: Generate a corresponding number of follow-up questions according to the evaluation indicators that the facts to be detected do not meet, and generate a question list; The question list is input into the question-answering robot, a corresponding number of answers generated by the question-answering robot based on the question list are received, and an answer list is generated.

4. The error detection method for intelligent question answering according to claim 2, characterized in that: Generating corresponding follow-up questions according to the evaluation index not satisfied by the to-be-detected fact, and receiving answers generated by the question-and-answer robot based on the follow-up questions, includes: Prioritizing the evaluation indicators that are not satisfied by the facts to be detected; Generate a first follow-up question according to the evaluation indicator with the highest priority, and receive an answer generated by the question-and-answer robot based on the first follow-up question; A second follow-up question is generated according to the evaluation indicator with the second highest priority, and an answer generated by the question-and-answer robot based on the second follow-up question is received, and so on, until there are no new update questions or the maximum number of question rounds is reached.

5. The error detection method for intelligent question answering according to claim 1, characterized in that: The consistency verification according to the follow-up question, the corresponding answer and the fact to be detected includes: Constructing a set of objects to be verified by combining the facts to be detected, each follow-up question and the corresponding answers; Perform inconsistency calculation on each group of objects to be verified and obtain corresponding inconsistency scores; The cumulative inconsistency score is calculated based on the inconsistency score of each group of objects to be verified, and the consistency verification result is generated.

6. The error detection method for intelligent question answering according to claim 5, characterized in that: The performing inconsistency calculation on each group of objects to be verified to obtain a corresponding inconsistency score includes: Extract features for each group of objects to be verified to obtain fact features, question features, and answer features; Calculating a first consistency score between the answer feature and the question feature, and a second consistency score between the answer feature and the fact feature; An inconsistency score for each group of objects to be verified is calculated based on the first consistency score, the second consistency score, and the difference between the consistency scores.

7. The error detection method for intelligent question answering according to claim 1, characterized in that: The performing error detection on the fact to be detected according to the consistency verification result and outputting the corresponding error detection result includes: Analyze and obtain the cumulative inconsistency score in the consistency verification result; Performing probability mapping on the cumulative inconsistency score to convert it into a corresponding correct probability value; When the correct probability value is greater than or equal to a preset threshold, the fact to be detected is confirmed to be correct, otherwise the fact to be detected is confirmed to be wrong.

8. An error detection device for intelligent question answering, characterized in that: include: An acquisition module is used to acquire the original answer generated by the question-answering robot based on the original user's question as a fact to be detected; A question follow-up module, configured to generate at least one follow-up question according to the fact to be detected, and receive an answer generated by the question-and-answer robot based on the follow-up question; A consistency verification module, used for performing consistency verification based on the follow-up questions, the corresponding answers and the facts to be detected; The fact error detection module is used to perform error detection on the fact to be detected according to the consistency verification result, and output the corresponding error detection result.

9. A computer device, characterized in that: comprising at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the error detection method for intelligent questioning and answering described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the error detection method for intelligent questioning and answering as described in any one of claims 1-7.

Images