Question and answer pair extraction method and related device
Through the collaborative work of multiple answer generation models, the semantic bias and insufficient knowledge coverage caused by a single teacher model are solved, and the confidence of Q&A data and the accuracy of Q&A model are improved.
Patent Information
- Application Number
- CN202510627254.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-15
- Publication Date
- 2025-07-29
AI Technical Summary
In the prior art, a single teacher model is affected by the distribution, structural deviation and domain adaptability of pre-training data when extracting Q&A data, resulting in insufficient semantic understanding deviation and knowledge coverage, resulting in poor accuracy and robustness of the Q&A model.
Multiple answer generation models are used to generate answer sets based on specific knowledge fragments, and target answers are selected through correctness verification to form a question-and-answer pair, and the coordinated work of multiple models is used to reduce the risk of deviation.
It improves the confidence of Q&A with data, improves the knowledge distillation effect of student models, and enhances the accuracy and robustness of Q&A model.
Smart Images
Figure CN120386846A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a question-answer pair extraction method and related devices. Background Art
[0002] Question-and-answer data is crucial for training question-and-answer models in question-and-answer scenarios (such as intelligent customer service and knowledge Q&A). With the rapid development of artificial intelligence (AI), knowledge distillation is widely used in model training. Knowledge distillation can also be used to train question-and-answer models in question-and-answer scenarios. Specifically, a teacher model is first used to extract question-and-answer data. This data is then used to train a student model using knowledge distillation to obtain the resulting question-and-answer model.
[0003] In the existing technology, a single teacher model is often used to extract question-answer pair data, and then the question-answer pair extraction results are used to train the student model through knowledge distillation to obtain a question-answering model. However, the single teacher model is affected by its pre-training data distribution, structural bias, and domain adaptability, which may lead to semantic understanding bias or insufficient knowledge coverage, causing the single teacher model to produce wrong or inaccurate answers, resulting in low confidence in the extracted question-answer pair data, which in turn affects the training effect of the question-answering model, resulting in poor accuracy and robustness of the question-answering model when processing user questions.
[0004] Therefore, how to provide a question-answer pair extraction solution that can improve the confidence of the extracted question-answer pair data to improve the knowledge distillation effect of the subsequent student model has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the Invention
[0005] In view of the above problems, this application provides a question-answer pair extraction method and related devices to achieve the purpose of improving the confidence of the question-answer pair data extracted by the teacher model, thereby improving the knowledge distillation effect of the subsequent student model. The specific solution is as follows:
[0006] The first aspect of the present application provides a question-answer pair extraction method, comprising:
[0007] Acquiring specific pieces of knowledge;
[0008] Determine a target question based on the specific knowledge fragment, where the target question is a question that meets preset question quality inspection conditions;
[0009] Using multiple answer generation models to generate answers to the target question based on the specific knowledge fragments, respectively, to obtain an answer set for the target question;
[0010] Determine a target answer to the target question from the answer set to the target question, wherein the target answer is an answer that has passed the correctness verification;
[0011] Determine a question-and-answer pair based on the target question and the target answer to the target question.
[0012] In a possible implementation, the determining the target question based on the specific knowledge fragment includes:
[0013] Invoke a question generation model to generate multiple questions based on the specific knowledge fragment;
[0014] Perform quality inspection on the multiple questions based on preset question quality inspection conditions to obtain the target question.
[0015] In a possible implementation, the performing quality inspection on the multiple questions based on preset question quality inspection conditions to obtain the target question includes:
[0016] For each question, perform quality inspection on the question from three aspects: whether there is a fuzzy intention, whether coreference resolution is possible, and whether it duplicates with other questions;
[0017] Determine that the question with no fuzzy intention, capable of coreference resolution, and not duplicating with other questions is the target question.
[0018] In a possible implementation, the using multiple answer generation models to respectively generate answers to the target question based on the specific knowledge fragment to obtain an answer set for the target question includes:
[0019] Generate an answer generation prompt instruction based on the specific knowledge fragment and the target question, where the answer generation prompt instruction includes the text content of the specific knowledge fragment, the target question, and description information of the answer generation task, and the answer generation task is used to generate an answer to the target question based on the text content of the specific knowledge fragment;
[0020] For each of the answer generation models, input the answer generation prompt instruction into the answer generation model to obtain the answer to the target question generated by the answer generation model;
[0021] Combine the answers to the target question generated by each answer generation model to obtain the answer set for the target question.
[0022] In a possible implementation, the determining the target answer to the target question from the answer set for the target question includes:
[0023] For each answer in the answer set for the target question, use multiple answer scoring models to respectively score the answer to obtain a scoring set for the answer, where the scoring set for the answer contains the scores of the answer output by each of the answer scoring models;
[0024] A target answer is screened out from the answer set of the target question; the target answer is an answer whose scores in the score set all reach a preset score threshold.
[0025] In a possible implementation, the multiple answer scoring models are used to score the answers respectively to obtain a score set for the answers, including:
[0026] Generate an answer scoring prompt instruction based on the target question, the specific knowledge fragment, and the answer, wherein the answer scoring prompt instruction includes description information of the target question, the specific knowledge fragment, the answer, and an answer scoring task, wherein the answer scoring task is used to generate a score for the answer based on the target question and the specific knowledge fragment;
[0027] For each of the answer scoring models, inputting the answer scoring prompt instruction into the answer scoring model to obtain the score of the answer generated by the answer scoring model;
[0028] The scores of the answers generated by the various answer scoring models are combined to obtain a score set for the answers.
[0029] In a possible implementation, determining a question-answer pair based on the target question and the target answer to the target question includes:
[0030] Inputting the target question and all target answers to the target question into the answer fusion model to obtain the final answer to the target question;
[0031] The target question and the final answer to the target question are combined to obtain the question-answer pair.
[0032] A second aspect of the present application provides a question-answer pair extraction device, comprising:
[0033] an acquisition unit, used to acquire specific pieces of knowledge;
[0034] A target question determination unit, configured to determine a target question based on the specific knowledge fragment, wherein the target question is a question that meets a preset question quality inspection condition;
[0035] an answer generation unit, configured to generate answers to the target question based on the specific knowledge fragments using a plurality of answer generation models, thereby obtaining an answer set for the target question;
[0036] a target answer determination unit, configured to determine a target answer to the target question from a set of answers to the target question, wherein the target answer is an answer that has passed correctness verification;
[0037] A question-and-answer pair determination unit, configured to determine a question-and-answer pair based on the target question and the target answer of the target question.
[0038] In a possible implementation, the target question determination unit includes:
[0039] A question generation unit, configured to call a question generation model to generate multiple questions based on the specific knowledge fragment;
[0040] A question quality inspection unit, configured to perform quality inspection on the multiple questions based on preset question quality inspection conditions to obtain the target question.
[0041] In a possible implementation, the question quality inspection unit is specifically configured to:
[0042] For each question, perform quality inspection on the question from three aspects: whether there is a fuzzy intention, whether reference resolution is performed, and whether it is repeated with other questions;
[0043] Determine that the question that does not have a fuzzy intention, can perform reference resolution, and is not repeated with other questions is the target question.
[0044] In a possible implementation, the answer generation unit is specifically configured to:
[0045] Generate an answer generation prompt instruction based on the specific knowledge fragment and the target question, where the answer generation prompt instruction includes the text content of the specific knowledge fragment, the target question, and description information of the answer generation task, and the answer generation task is used to generate the answer of the target question based on the text content of the specific knowledge fragment;
[0046] For each answer generation model, input the answer generation prompt instruction into the answer generation model to obtain the answer of the target question generated by the answer generation model;
[0047] Combine the answers of the target question generated by each answer generation model to obtain an answer set of the target question.
[0048] In a possible implementation, the target answer determination unit includes:
[0049] A scoring unit, configured to, for each answer in the answer set of the target question, use multiple answer scoring models to score the answer respectively to obtain a scoring set of the answer, where the scoring set of the answer includes the scores of the answer output by each answer scoring model;
[0050] A screening unit, configured to screen out the target answer from the answer set of the target question; the target answer is the answer for which all scores in the scoring set reach a preset scoring threshold.
[0051] In a possible implementation, the scoring unit is specifically configured to:
[0052] Generate an answer scoring prompt instruction based on the target question, the specific knowledge fragment, and the answer. The answer scoring prompt instruction includes the target question, the specific knowledge fragment, the answer, and description information of an answer scoring task, where the answer scoring task is used to generate a score for the answer based on the target question and the specific knowledge fragment;
[0053] For each of the answer scoring models, input the answer scoring prompt instruction into the answer scoring model to obtain the score of the answer generated by the answer scoring model;
[0054] Combine the scores of the answers generated by each answer scoring model to obtain a set of scores for the answer.
[0055] In a possible implementation, the Q&A pair determination unit is specifically configured to:
[0056] Input the target question and all target answers of the target question into an answer fusion model to obtain the final answer to the target question;
[0057] Combine the target question and the final answer to the target question to obtain the Q&A pair.
[0058] A third aspect of the present application provides a computer program product, including computer-readable instructions, which when running on an electronic device, enable the electronic device to implement the Q&A pair extraction method in the first aspect or any implementation manner of the first aspect.
[0059] A fourth aspect of the present application provides an electronic device, including at least one processor and a memory connected to the processor, where:
[0060] The memory is used to store a computer program;
[0061] The processor is used to execute the computer program so that the electronic device can implement the Q&A pair extraction method in the first aspect or any implementation manner of the first aspect.
[0062] A fifth aspect of the present application provides a computer-readable storage medium, where the storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the Q&A pair extraction method in the first aspect or any implementation manner of the first aspect.
[0063] With the above technical solutions, the question-answer pair extraction method and related device provided by this application, after obtaining a specific knowledge fragment, first determine a target question that meets the preset question quality inspection conditions based on the specific knowledge fragment, and then use multiple answer generation models to generate answers for the target question based on the specific knowledge fragment respectively to obtain an answer set for the target question; then determine a target answer for the target question that passes the correctness verification from the answer set of the target question; finally, determine the question-answer pair based on the target question and the target answer of the target question. In this solution, answers are generated by multiple answer generation models. Compared with a single model, it can consider all aspects of the question more comprehensively, thus avoiding possible biases or knowledge blind spots of a single model. In addition, verifying the correctness of each answer and selecting the correct answer from them can improve the accuracy of the answer. Therefore, the confidence of the extracted question-answer data can be improved, and using the question-answer pairs extracted by this solution for knowledge distillation of the student model has a better effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] In combination with the accompanying drawings and with reference to the following specific embodiments, the above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and that the elements and components are not necessarily drawn to scale.
[0065] Figure 1 It is a flowchart showing a method for extracting question-answer pairs provided by an embodiment of this application;
[0066] Figure 2 It is a schematic diagram showing the process of extracting a target question from a specific knowledge fragment provided by an embodiment of this application;
[0067] Figure 3 It is a schematic diagram showing the process of generating an answer set for a certain target question provided by an embodiment of this application;
[0068] Figure 4 It is a schematic diagram showing the process of generating a scoring set for a certain answer to a certain target question provided by an embodiment of this application;
[0069] Figure 5 It is a schematic diagram showing the structure of a question-answer pair extraction device provided by an embodiment of this application;
[0070] Figure 6 It is a schematic diagram showing the structure of an electronic device provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0071] The following describes the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. The terms used in the embodiments of this application are only for explaining the specific embodiments of this application and are not intended to limit this application.
[0072] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0073] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0074] Question-and-answer data is crucial for training question-and-answer models in question-and-answer scenarios (such as intelligent customer service and knowledge Q&A). With the rapid development of artificial intelligence (AI), knowledge distillation is widely used in model training. Knowledge distillation can also be used to train question-and-answer models in question-and-answer scenarios. Specifically, a teacher model is first used to extract question-and-answer data. This data is then used to train a student model using knowledge distillation to obtain the resulting question-and-answer model.
[0075] In the existing technology, a single teacher model is often used to extract question-answer pair data, and then the question-answer pair extraction results are used to train the student model through knowledge distillation to obtain a question-answering model. However, the single teacher model is affected by its pre-training data distribution, structural bias, and domain adaptability, which may lead to semantic understanding bias or insufficient knowledge coverage, causing the single teacher model to produce wrong or inaccurate answers, resulting in low confidence in the extracted question-answer pair data, which in turn affects the training effect of the question-answering model, resulting in poor accuracy and robustness of the question-answering model when processing user questions.
[0076] In order to solve the above problems, existing improvement solutions include using multiple teacher models to extract question-answer pair data. However, they mainly use simple weighted or probability fusion strategies to fuse the question-answer pair extraction results of multiple teacher models, which has poor effects.
[0077] In view of this, an embodiment of the present application provides a question-answer pair extraction method. The question-answer pair extraction method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.
[0078] Reference Figure 1 , Figure 1 A flowchart of a question-answer pair extraction method provided in an embodiment of the present application is shown as follows:Figure 1 As shown in Figure 1 , a method for extracting question-answer pairs provided by an embodiment of the present application may include the following steps, and these steps will be described in detail below.
[0079] S101: Obtain specific knowledge fragments;
[0080] In the present application, the specific knowledge fragments may be documents, and the documents can be understood as a kind of document that provides information, knowledge or solves problems. For example, they can be book chapters, or electronic documents existing on the Internet or in a database. The purpose of the method for extracting question-answer pairs provided in the present application is to extract question-answer pairs from the specific knowledge fragments to enrich the question-answer database or use these question-answer pairs as training data to train a question-answer model for answering user questions.
[0081] In the embodiments of the present application, the documents can be selected according to the actual application scenarios. For example, in the field of enterprise knowledge Q&A, the documents can be enterprise-related documents. Another example is that in the medical field, the documents can be medical papers, medical textbooks, doctor-patient conversation records, and so on.
[0082] In the embodiments of the present application, the documents can be in formats such as PDF, word, Excel, etc., and the present application does not limit this. However, when the document is in the form of an image, in order to extract question-answer pairs from the document, before extracting the question-answer pairs, the document also needs to be scanned, such as scanning the original document with the help of OCR (Optical Character Recognition) technology to obtain a document that can be recognized by a computer program.
[0083] S102: Determine a target question based on the specific knowledge fragment, where the target question is a question that meets the preset question quality inspection conditions;
[0084] In the present application, the preset question quality inspection conditions can be set based on the scenario requirements, and the present application does not make any limitations on this. It should be noted that the number of target questions can be multiple to improve the richness of the question-answer pair extraction results.
[0085] S103: Use multiple answer generation models to generate answers to the target question based on the specific knowledge fragment respectively to obtain an answer set for the target question;
[0086] In this application, the set of answers to the target question contains the answers to the target question output by each answer generation model. For each answer generation model, the answer generation model can be a model with the ability to generate answers. The answer generation model can analyze and process the input question according to its own algorithm and pre-trained knowledge, and independently generate an answer, which reflects the model's understanding of the question and the attempt to answer based on its internal knowledge.
[0087] Different models may have different understandings and representations of knowledge. Using multiple answer generation models to generate answers to the target question based on the specific knowledge fragments respectively can avoid the possible biases or knowledge blind spots of a single model.
[0088] S104: Determine the target answer to the target question from the set of answers to the target question, where the target answer is the answer that has passed the correctness verification;
[0089] It should be noted that if there are multiple target questions, for each target question, it is necessary to determine the target answer to the target question from the set of answers to the target question. The preset correctness verification conditions can be set based on the scenario requirements, and this application does not make any limitations on this. In one possible implementation, the preset correctness verification conditions include that the score of the answer reaches a preset score threshold.
[0090] S105: Determine the question-answer pair based on the target question and the target answer to the target question.
[0091] In this application, if there is only one target answer, the combination of the target question and the target answer can be determined as the question-answer pair. If there are multiple target answers, one of the target answers can be selected to form a question-answer pair with the target question, or the multiple target answers can be further processed to generate a new answer to form a question-answer pair with the target question. This application does not make any limitations on this.
[0092] The Q&A pair extraction method provided in this embodiment, after obtaining a specific knowledge segment, first determines a target question that meets the preset question quality inspection conditions based on the specific knowledge segment, and then uses multiple answer generation models to generate answers to the target question respectively based on the specific knowledge segment to obtain an answer set of the target question; then determines a target answer of the target question that passes the correctness verification from the answer set of the target question; finally, determines a Q&A pair based on the target question and the target answer of the target question. In this solution, answers are generated by multiple answer generation models. Compared with a single model, it can consider various aspects of the question more comprehensively, thereby avoiding possible biases or knowledge blind spots of a single model. In addition, verifying the correctness of each answer and selecting the correct answer from them can improve the accuracy of the answer. Therefore, the confidence of the extracted Q&A data can be improved, and the effect of using the Q&A pairs extracted by this solution for student model knowledge distillation is better.
[0093] In another embodiment of the present application, the specific implementation manner of determining the target question based on the specific knowledge segment is described, and this manner may include the following steps:
[0094] S201: Invoke a question generation model to generate multiple questions based on the specific knowledge segment;
[0095] In the present application, the question generation model can be a large language model with question generation capabilities. The question generation model can generate diverse questions according to its own different structural characteristics and the distribution of pre-training data, etc., so as to ensure the richness of the questions. In a possible implementation, a question generation prompt instruction can be generated based on the specific knowledge segment. The question generation prompt instruction includes the text content of the specific knowledge segment and the description information of the question generation task. The question generation task is used to generate multiple questions based on the text content of the specific knowledge segment; input the question generation prompt instruction into the question generation model to obtain multiple questions generated by the question generation model.
[0096] For ease of understanding, an example of the question generation prompt instruction is as follows:
[0097] "You are a knowledge robot. Please simulate a human conversation to extract 10 questions from the following provided book passage.
[0098] Book passage: [××××]."
[0099] Among them, "Book passage: [××××]" is a filling slot for filling the text content of the specific knowledge segment.
[0100] S202: Perform quality inspection on the multiple questions based on the preset question quality inspection conditions to obtain the target question.
[0101] In this application, the target questions are the high-quality questions among the multiple questions that pass the quality inspection based on the preset question quality inspection conditions. In a possible implementation, the preset question quality inspection conditions include, but are not limited to, no ambiguous intention, ability to resolve anaphora, and no duplication with other questions. Ambiguous intention means an unclear intention. In this application, if the question itself has an unclear intention, it will have an adverse effect on the performance of the question-and-answer model. Therefore, it is necessary to conduct quality inspection on the questions from the aspect of whether there is an ambiguous intention. Formally, the process of dividing different referents representing the same entity into an equivalent set is called anaphora resolution. If a question lacks a subject and thus has no entity to be replied to, it will also have an adverse effect on the performance of the question-and-answer model. Therefore, it is necessary to conduct quality inspection on the questions from the aspect of whether anaphora can be resolved. In addition, a high degree of question duplication will also have an adverse effect on the performance of the question-and-answer model. Therefore, it is necessary to conduct quality inspection on the questions from the aspect of whether they are duplicated with other questions.
[0102] Then, in a possible implementation, the quality inspection of the multiple questions based on the preset question quality inspection conditions to obtain the target questions includes:
[0103] For each question, conduct quality inspection on the question from three aspects: whether there is an ambiguous intention, whether anaphora can be resolved, and whether it is duplicated with other questions; determine that the questions with no ambiguous intention, the ability to resolve anaphora, and no duplication with other questions are the target questions.
[0104] In this application, questions with ambiguous intention, and / or inability to resolve anaphora, and / or duplication with other questions can be removed through a filtering device.
[0105] For ease of understanding, refer to Figure 2 , Figure 2 which is a schematic diagram of the process of extracting target questions from specific knowledge fragments provided by an embodiment of this application.
[0106] In a possible implementation, the answer generation model is a large language model with the ability to generate answers based on questions and knowledge fragments. Then, in another embodiment of this application, the specific implementation manner of using multiple answer generation models to generate answers to the target questions based on the specific knowledge fragments to obtain the answer set of the target questions is described. This method may include the following steps:
[0107] S301: Generate an answer generation prompt instruction based on the specific knowledge fragment and the target question. The answer generation prompt instruction includes the text content of the specific knowledge fragment, the target question, and the description information of the answer generation task. The answer generation task is used to generate the answer to the target question based on the text content of the specific knowledge fragment;
[0108] S302: For each of the answer generation models, input the answer generation prompt instruction into the answer generation model to obtain the answer to the target question generated by the answer generation model.
[0109] S303: Combine the answers to the target question generated by each answer generation model to obtain the answer set for the target question.
[0110] For ease of understanding, an example of the answer generation prompt instruction is as follows:
[0111] "You are a knowledge robot. Please simulate a conversation between characters and generate the answer to the following question from the book passage provided below.
[0112] Book passage: [××××];
[0113] Question: [××××]."
[0114] Among them, "Book passage: [××××]" is the filling slot for the text content used to fill the specific knowledge fragment. "Question: [××××]" is the filling slot for the target question.
[0115] It should be noted that if there are multiple target questions, for each of the target questions, the target question and the specific knowledge fragment need to be input into multiple answer generation models to obtain the answer set for the target question.
[0116] For ease of understanding, refer to Figure 3 , Figure 3 which is a schematic diagram of the process for generating the answer set for a certain target question provided in an embodiment of the present application. As Figure 3 shown, based on the target question and the specific knowledge fragment, an answer generation prompt instruction is generated. Input the answer generation prompt instruction into answer generation model A to obtain answer A, input the answer generation prompt instruction into answer generation model B to obtain answer B,..., input the answer generation prompt instruction into answer generation model N to obtain answer N. The combination of answers A, B,..., N is the answer set for the target question.
[0117] In another embodiment of the present application, the specific implementation manner for determining the target answer to the target question from the answer set for the target question is described. This manner may include the following steps:
[0118] S401: For each answer in the answer set for the target question, use multiple answer scoring models to score the answer respectively to obtain the scoring set for the answer. The scoring set for the answer contains the scores of the answer output by each answer scoring model.
[0119] In this application, for each answer scoring model, the answer scoring model can be a model capable of scoring answers from multiple dimensions. The multiple dimensions include but are not limited to semantic rationality, knowledge matching degree, structural integrity, and semantic coherence.
[0120] In a possible implementation, the answer scoring model can be the aforementioned answer generation model. There is a set of scoring systems based on its pre-training objectives and algorithm optimizations inside the answer generation model. For the generated answers, the model will score according to factors such as the matching degree between the answer and its pre-trained knowledge, and semantic rationality. For example, the model will evaluate and score according to whether the vocabulary usage in the answer conforms to the learned language patterns and whether the structure of the answer is clear and reasonable.
[0121] S402: Screen out the target answer from the set of answers to the target question; the target answer is the answer for which all scores in the scoring set reach a preset scoring threshold.
[0122] In this application, the scoring threshold can be adjusted according to specific application scenarios and requirements for answer accuracy. For example, in some knowledge Q&A scenarios with extremely high accuracy requirements, a higher scoring threshold can be set to ensure the high reliability of the final answer. In one implementable manner, the average value of all scores of all answers can be calculated as the scoring threshold.
[0123] When all scores in the scoring set of a certain answer reach the scoring threshold, it is determined that the answer is the target answer. In this application, by presetting the scoring threshold, wrong answers can be filtered out and high-confidence answers can be screened out, which can not only prevent the transfer of wrong knowledge and avoid affecting the training effect of subsequent student models, but also improve the training efficiency of subsequent student models.
[0124] In a possible implementation, the answer scoring model can be a large language model capable of scoring answers. Then, in another embodiment of this application, the specific implementation manner of using multiple answer scoring models to score the answer respectively to obtain the scoring set of the answer is described. This method can include the following steps:
[0125] S501: Generate an answer scoring prompt instruction based on the target question, the specific knowledge fragment, and the answer. The answer scoring prompt instruction includes the target question, the specific knowledge fragment, the answer, and description information of the answer scoring task. The answer scoring task is used to generate the score of the answer based on the target question and the specific knowledge fragment.
[0126] S502: For each of the answer scoring models, input the answer scoring prompt instruction into the answer scoring model to obtain the score of the answer generated by the answer scoring model.
[0127] S503: Combine the scores of the answers generated by each answer scoring model to obtain the score set of the answers.
[0128] For ease of understanding, an example of the answer scoring prompt instruction is as follows:
[0129] "You are a scoring model. Please give the MOS score based on the provided question, book passage, and answer; the total score is 5 points; the MOS score needs to consider: correctness, completeness, and extensibility;
[0130] Book passage: [××××];
[0131] Question: [××××];
[0132] Answer: [××××]."
[0133] Among them, "Book passage: [××××]" is the filling slot for the text content used to fill the specific knowledge fragment. "Question: [××××]" is the filling slot for the target question. "Answer: [××××]" is the filling slot for the answer to the target question.
[0134] For ease of understanding, refer to Figure 4 , Figure 4 which is a schematic diagram of the process for generating the score set of an answer to a certain target question provided by an embodiment of the present application. As Figure 4 shown, based on the target question, specific knowledge fragment, and answer, generate the answer scoring prompt instruction, input the answer scoring prompt instruction into the answer scoring model A to obtain the score A of the answer, input the answer scoring prompt instruction into the answer scoring model B to obtain the score B of the answer,..., input the answer scoring prompt instruction into the answer scoring model N to obtain the score N of the answer. The combination of scores A, B,..., N is the score set of this answer.
[0135] It should be noted that after each answer scoring model completes the scoring of all answers, the answers and their corresponding scores can be summarized to obtain the answers and their score sets.
[0136] In another embodiment of the present application, a specific implementation manner for determining the question-and-answer pair based on the target question and the target answer to the target question is described. This manner may include the following steps:
[0137] S601: Input the target question and all target answers of the target question into an answer fusion model to obtain the final answer of the target question;
[0138] In this application, the answer fusion model can be a large language model with answer fusion capabilities. In one possible implementation, a fourth prompt instruction can be generated based on the target question and all answers of the target question. The fourth prompt instruction includes the target question, all target answers of the target question, and description information of the answer fusion task. The answer fusion task is used to fuse all target answers of the target question to generate a new answer; input the fourth prompt instruction into the answer fusion model to obtain the final answer.
[0139] For ease of understanding, an example of the fourth prompt instruction is as follows:
[0140] "You are a knowledge robot. Please fuse the provided answers according to the question and generate a new answer.
[0141] Question: [××××];
[0142] Answer: [××××]."
[0143] Among them, "Question: [××××]" is a filling slot for filling the target question, and "Answer: [××××]" is a filling slot for filling the target answer.
[0144] S602: Combine the target question and the final answer of the target question to obtain the question-answer pair.
[0145] In summary, the question-answer extraction method provided in the embodiments of this application reduces the deviation risk of a single model through the collaborative work of multiple models. Even if a certain model makes a mistake or is interfered with, as long as other models can correctly identify the answer, the correctness of the final answer can still be guaranteed. Therefore, it can improve the confidence of question-answer pair data, provide a high-quality knowledge source for subsequent optimization of the student model, thereby improving the quality and efficiency of data distillation, and ultimately improving the performance of the model trained based on distilled data, making it more robust.
[0146] The above introduces a question-answer pair extraction method provided in the embodiments of this application. The following will introduce the device for executing the above question-answer pair extraction method.
[0147] Please refer to Figure 5 , Figure 5 which is a schematic structural diagram of a question-answer pair extraction device provided in the embodiments of this application. As Figure 5 shown, the question-answer pair extraction device includes:
[0148] An acquisition unit 11, configured to acquire specific knowledge fragments;
[0149] A target problem determination unit 12, configured to determine a target problem based on the specific knowledge fragment, where the target problem is a problem that meets a preset problem quality inspection condition;
[0150] An answer generation unit 13, configured to use multiple answer generation models to generate answers to the target problem based on the specific knowledge fragment respectively, to obtain an answer set of the target problem;
[0151] A target answer determination unit 14, configured to determine a target answer to the target problem from the answer set of the target problem, where the target answer is an answer that passes the correctness verification;
[0152] A question-and-answer pair determination unit 15, configured to determine a question-and-answer pair based on the target problem and the target answer to the target problem.
[0153] In a possible implementation, the target problem determination unit includes:
[0154] A problem generation unit, configured to call a problem generation model to generate multiple problems based on the specific knowledge fragment;
[0155] A problem quality inspection unit, configured to perform quality inspection on the multiple problems based on a preset problem quality inspection condition, to obtain the target problem.
[0156] In a possible implementation, the problem quality inspection unit is specifically configured to:
[0157] For each problem, perform quality inspection on the problem from three aspects: whether there is a fuzzy intention, whether reference resolution is performed, and whether it is repeated with other problems;
[0158] Determine that a problem that does not have a fuzzy intention, can perform reference resolution, and is not repeated with other problems is the target problem.
[0159] In a possible implementation, the answer generation unit is specifically configured to:
[0160] Generate an answer generation prompt instruction based on the specific knowledge fragment and the target problem, where the answer generation prompt instruction includes the text content of the specific knowledge fragment, the target problem, and description information of an answer generation task, and the answer generation task is used to generate an answer to the target problem based on the text content of the specific knowledge fragment;
[0161] For each of the answer generation models, input the answer generation prompt instruction into the answer generation model to obtain the answer to the target problem generated by the answer generation model;
[0162] Combine the answers to the target question generated by each answer generation model to obtain the set of answers to the target question.
[0163] In a possible implementation, the target answer determination unit includes:
[0164] A scoring unit, configured to score each answer in the set of answers to the target question by using multiple answer scoring models respectively, to obtain a set of scores for the answer, where the set of scores for the answer includes the scores of the answer output by each of the answer scoring models;
[0165] A screening unit, configured to screen out the target answer from the set of answers to the target question; the target answer is an answer for which all scores in the set of scores reach a preset score threshold.
[0166] In a possible implementation, the scoring unit is specifically configured to:
[0167] Generate an answer scoring prompt instruction based on the target question, the specific knowledge fragment, and the answer, where the answer scoring prompt instruction includes the target question, the specific knowledge fragment, the answer, and description information of an answer scoring task, and the answer scoring task is used to generate a score for the answer based on the target question and the specific knowledge fragment;
[0168] For each of the answer scoring models, input the answer scoring prompt instruction into the answer scoring model to obtain the score of the answer generated by the answer scoring model;
[0169] Combine the scores of the answer generated by each answer scoring model to obtain the set of scores for the answer.
[0170] In a possible implementation, the Q&A pair determination unit is specifically configured to:
[0171] Input the target question and all target answers to the target question into an answer fusion model to obtain the final answer to the target question;
[0172] Combine the target question and the final answer to the target question to obtain the Q&A pair.
[0173] An electronic device is further provided in an embodiment of the present application. Refer to Figure 6 As shown, it shows a schematic structural diagram of an electronic device suitable for implementing the electronic device in the embodiment of the present application. The electronic device in the embodiment of the present application may include, but is not limited to, fixed terminals such as mobile phones, laptop computers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), desktop computers, and the like. Figure 6The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.
[0174] As Figure 6 shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage device 608 into the random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The input / output (I / O) interface 605 is also connected to the bus 604.
[0175] Generally, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 6 an electronic device with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices may be implemented or had.
[0176] In the embodiments of the present application, there is also provided a computer program product including computer-readable instructions, which, when running on an electronic device, enable the electronic device to implement any one of the question-and-answer pair extraction methods provided by the embodiments of the present application.
[0177] In the embodiments of the present application, there is also provided a computer-readable storage medium carrying one or more computer programs, which, when executed by an electronic device, can enable the electronic device to implement any one of the question-and-answer pair extraction methods provided by the embodiments of the present application.
[0178] In addition, it should be noted that the device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the attached drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines.
[0179] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware. Of course, it can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures for implementing the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits, etc. However, for the present application, in more cases, software program implementation is a better embodiment. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a readable storage medium, such as a floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disc of a computer, etc., and includes several instructions for causing a computer device (which can be a personal computer, training device, or network device, etc.) to execute the methods described in various embodiments of the present application.
[0180] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.
[0181] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, training device, or data center to another website, computer, training device, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store, or a data storage device such as a training device or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.
Claims
1. A method for extracting question-answer pairs, characterized in that, including: obtaining specific knowledge fragments; determining a target question based on the specific knowledge fragment, where the target question is a question that meets the preset question quality inspection conditions; using multiple answer generation models to generate answers to the target question based on the specific knowledge fragment respectively, to obtain an answer set of the target question; determining a target answer to the target question from the answer set of the target question, where the target answer is an answer that passes the correctness verification; determining a question-answer pair based on the target question and the target answer to the target question.
2. The method according to claim 1, wherein The determining the target question based on the specific knowledge fragment includes: invoking a question generation model to generate multiple questions based on the specific knowledge fragment; performing quality inspection on the multiple questions based on the preset question quality inspection conditions to obtain the target question.
3. The method according to claim 2, wherein The performing quality inspection on the multiple questions based on the preset question quality inspection conditions to obtain the target question includes: for each question, performing quality inspection on the question from three aspects: whether there is a fuzzy intention, whether anaphora resolution is possible, and whether it is repeated with other questions; determining that a question that has no fuzzy intention, can be resolved anaphorically, and is not repeated with other questions is the target question.
4. The method according to claim 1, wherein The using multiple answer generation models to generate answers to the target question based on the specific knowledge fragment respectively, to obtain an answer set of the target question includes: generating an answer generation prompt instruction based on the specific knowledge fragment and the target question, where the answer generation prompt instruction includes the text content of the specific knowledge fragment, the target question, and description information of an answer generation task, and the answer generation task is used to generate an answer to the target question based on the text content of the specific knowledge fragment; for each answer generation model, inputting the answer generation prompt instruction into the answer generation model to obtain the answer to the target question generated by the answer generation model; combining the answers to the target question generated by each answer generation model to obtain the answer set of the target question.
5. The method according to claim 1, wherein The determining the target answer to the target question from the answer set of the target question includes: for each answer in the answer set of the target question, using multiple answer scoring models to score the answer respectively to obtain a scoring set of the answer, where the scoring set of the answer contains the scores of the answer output by each answer scoring model; screening out a target answer from the answer set of the target question; the target answer is an answer for which all scores in the scoring set reach a preset scoring threshold.
6. The method according to claim 5, characterized in that The using multiple answer scoring models to score the answer respectively to obtain a scoring set of the answer includes: generating an answer scoring prompt instruction based on the target question, the specific knowledge fragment, and the answer, where the answer scoring prompt instruction includes the target question, the specific knowledge fragment, the answer, and description information of an answer scoring task, and the answer scoring task is used to generate a score of the answer based on the target question and the specific knowledge fragment; For each of the answer scoring models, input the answer scoring prompt instruction into the answer scoring model to obtain the score of the answer generated by the answer scoring model. Combine the scores of the answers generated by each answer scoring model to obtain the answer score set.
7. The method according to claim 1, wherein The determination of the question-answer pair based on the target question and the target answer of the target question includes: Input the target question and all target answers of the target question into the answer fusion model to obtain the final answer of the target question. Combine the target question and the final answer of the target question to obtain the question-answer pair.
8. A question-and-answer pair extraction device, characterized in that It includes: An acquisition unit for acquiring specific knowledge fragments. A target question determination unit for determining a target question based on the specific knowledge fragment, where the target question is a question that meets the preset question quality inspection conditions. An answer generation unit for using multiple answer generation models to generate answers to the target question based on the specific knowledge fragment respectively to obtain an answer set of the target question. A target answer determination unit for determining the target answer of the target question from the answer set of the target question, where the target answer is an answer verified to be correct. A question-answer pair determination unit for determining a question-answer pair based on the target question and the target answer of the target question.
9. A computer program product, characterized in that, It includes computer-readable instructions that, when running on an electronic device, cause the electronic device to implement the question-answer pair extraction method according to any one of claims 1 to 7.
10. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, where: The memory is used to store a computer program. The processor is used to execute the computer program so that the electronic device can implement the question-answer pair extraction method according to any one of claims 1 to 7.
11. A computer-readable storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, can cause the electronic device to implement the question-answer pair extraction method according to any one of claims 1 to 7.