Question and answer pair generation method and device, storage medium and program product

Abstract

One or more embodiments of the invention provide a question and answer pair generation method and device, a storage medium and a program product. The question-answer pair generation method comprises the steps of obtaining an original question-answer pair of a target domain; the trained reply generation model is utilized to answer original questions contained in the original question and answer pair, and reasoning information and candidate answers in the answering process are obtained; verifying the accuracy of the candidate answers based on the original answers contained in the original question and answer pair; under the condition that the candidate answers pass verification, knowledge points related to the target domain are extracted from the reasoning information, and at least one question and answer pair is generated according to the knowledge points. Accurate professional domain question and answer data can be automatically obtained.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of artificial intelligence, and in particular to a question and answer pair generation method, an electronic device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] With the rapid development of large model technology, it has shown strong ability in complex tasks such as intelligent question and answer, knowledge reasoning and task execution. However, in order to further improve the professional performance of large models in specific fields, it still needs to rely on a large amount of high-quality domain question and answer data for fine-tuning and optimization.

[0003] The currently disclosed question and answer data resources are mostly concentrated in open fields or general tasks, and lack in-depth coverage of professional vertical fields such as medicine, law and finance. In particular, in these professional fields, the knowledge structure is complex and the language expression is professional, which makes it difficult for general question and answer data to meet the high requirements of professional and accuracy in real scenarios.

[0004] Therefore, how to effectively construct high-quality professional domain question and answer pairs has become one of the key problems to promote the application of large models. SUMMARY

[0005] Therefore, one or more embodiments of the present specification provide a question and answer pair generation method, an electronic device, a computer readable storage medium, and a computer program product.

[0006] To achieve the above-mentioned purpose, one or more embodiments of the present specification provide technical solutions as follows:

[0007] According to a first aspect of one or more embodiments of the present specification, a question and answer pair generation method is provided, comprising:

[0008] Obtaining an original question and answer pair of a target domain;

[0009] Using a trained answer generation model to answer the original question contained in the original question and answer pair, obtaining reasoning information and a candidate answer in the answering process;

[0010] Verifying the accuracy of the candidate answer based on the original answer contained in the original question and answer pair;

[0011] In the case where the candidate answer passes the verification, extracting knowledge points related to the target domain from the reasoning information, and generating at least one question and answer pair according to the knowledge points.

[0012] According to a second aspect of the embodiments of the present specification, an electronic device is provided, comprising:

[0013] A processor;

[0014] a memory for storing processor-executable instructions;

[0015] wherein the processor, when executing the executable instructions, is configured to implement the method of the first aspect.

[0016] According to a third aspect of the embodiments of the present specification, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of the method of the first aspect.

[0017] According to a fourth aspect of the embodiments of the present specification, a computer program product is provided, and the computer program product comprises a computer program. The computer program is executed by a processor to implement the steps of the method of the first aspect.

[0018] The technical solutions provided by the embodiments of the present specification can include the following beneficial effects:

[0019] In the embodiments of the present specification, the trained reply generation model can be used to answer the original question contained in the original question and answer pair, obtain the reasoning information and the candidate answer of the original question, and combine the original answer contained in the original question and answer pair to verify the accuracy of the candidate answer, so as to filter out high-quality reasoning information with high credibility. The reasoning information usually contains some professional knowledge used by the model in the thinking process, and then by extracting relevant knowledge points from the reasoning information, the relevance to the target field and the content diversity can be further improved, so as to construct accurate professional field question and answer data based on the extracted knowledge points.

[0020] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present specification. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a flowchart of a question and answer pair generation method provided by an exemplary embodiment.

[0022] Figure 2 is a flowchart of another question and answer pair generation method provided by an exemplary embodiment.

[0023] Figure 3 is a flowchart of a question and answer pair answerability detection method provided by an exemplary embodiment.

[0024] Figure 4 is a schematic diagram of a response process when the candidate answer fails to pass the verification provided by an exemplary embodiment.

[0025] Figure 5 is a structural schematic diagram of an electronic device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0026] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is presented in connection with the drawings and describes embodiments of the exemplary embodiments. However, the exemplary embodiments are not intended to be limited to the embodiments described herein. Rather, the exemplary embodiments are intended to be accorded the widest scope consistent with the principles and spirit of the exemplary embodiments as described herein.

[0027] It should be noted that the steps of the methods in other embodiments are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps of the methods can be more or less than those described in this specification. Furthermore, a single step described in this specification can be split into multiple steps in other embodiments; and multiple steps described in this specification can be combined into a single step in other embodiments.

[0028] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0029] Here, relevant terms are explained:

[0030] 1. Large Language Model: A large language model refers to a machine learning model with a super large number of parameters. It is usually based on a deep learning architecture such as Transformer. Through training on a large amount of data, it can capture complex language patterns and knowledge structures, and has strong generalization ability and multi-functionality. For example, a large language model can complete tasks such as text generation, question answering, translation, reasoning, etc., and is widely used in natural language processing (NLP) and other artificial intelligence fields. Its core features are large scale, strong ability, and wide adaptability.

[0031] 2. Long Reasoning Model: A long reasoning model is a large language model that has been further trained to solve multi-step reasoning tasks. These models outperform traditional autoregressive large language models (LLMs) in logical, mathematical, or programmatic tasks, have backtracking capabilities, and enhance the quality of the final answer through long thought chains when reasoning.

[0032] 3. Chain of Thought (CoT): Chain of Thought is a method that simulates the step-by-step reasoning process of humans to solve complex problems. It decomposes a complex task into multiple logically clear and coherent intermediate steps, gradually deriving the final answer. In the field of artificial intelligence, Chain of Thought is often used to enhance the reasoning ability of large language models, especially when dealing with multi-step logic, mathematical, or common-sense problems.

[0033] 4. Data Synthesis: The process of generating virtual or simulated data using artificial intelligence techniques to supplement, replace, or enhance real-world data. It is typically based on existing data samples, domain knowledge, or specific rules, and automatically generates high-quality data with a similar structure and content to the original data through algorithms and models (such as large language models, generative adversarial networks, etc.).

[0034] 5. Domain datasets: These are datasets that are collected, organized, and labeled specifically for a particular domain or industry. They are typically used to train and evaluate machine learning or artificial intelligence models to solve specific tasks or problems within that domain.

[0035] To truly enable large-scale models to achieve the capabilities of human experts in specific domain tasks / industries, further fine-tuning of these models using relevant domain-specific question-and-answer data is needed to meet diverse user needs. The market urgently requires specialized large-scale models across various domains. However, currently available question-and-answer data resources are mostly concentrated in open domains or general tasks, lacking in-depth coverage of specialized vertical fields such as medicine, law, and finance. Especially in these specialized fields, the complex knowledge structures and specialized language make it difficult for general-purpose question-and-answer data to meet the high requirements of professionalism and accuracy in real-world scenarios.

[0036] One related technology offers a templated question-and-answer data generation method. This method typically uses pre-defined question and answer templates, filling them with domain-related knowledge points or keywords to construct new question-and-answer pairs. This approach is primarily used in structured scenarios, such as FAQ (Frequently Asked Questions) systems. After classification and keyword extraction, the original question-and-answer data is expanded using a set of predefined templates to generate diverse question-and-answer samples. However, this method cannot fully utilize the versatility of large language models in the reasoning process, resulting in relatively homogeneous content and a lack of dynamic adjustment capabilities. Due to the limited number of templates, the generated results for similar questions tend to be consistent, making it difficult to meet diverse expression needs. Furthermore, the templates require manual design and maintenance by domain experts, leading to high development costs, poor response times, and difficulty adapting to rapidly changing domain requirements.

[0037] A data expansion method based on a knowledge graph is also provided in the related art. This kind of method usually constructs a knowledge graph for a specific field, and takes each knowledge node in the graph as the basis for generating a question and answer pair. When generating a question and answer pair, the system can retrieve a target knowledge node from the knowledge graph, and then construct a corresponding question and answer, to realize automatic question and answer data generation for structured knowledge. However, constructing a high-quality field knowledge graph often needs to rely on manual expert annotation or semi-automatic processes, which requires a large upfront investment and a long construction period. Once the knowledge graph is constructed, its knowledge range is relatively fixed, and it is difficult to cover emerging concepts or dynamically expanding knowledge content outside the graph, which limits the coverage and flexibility of question and answer generation.

[0038] Based on this, please refer to Figure 1 The embodiments of the present specification provide a question and answer pair generation method, which can be executed by an electronic device, including but not limited to a physical server, a virtual server, a smart phone / mobile phone, a tablet computer, a personal digital assistant (PDA), a laptop computer, a desktop computer, a wearable device, or any other type of device. The method includes:

[0039] In S100, an original question and answer pair of a target field is obtained.

[0040] In this step, the electronic device obtains the original question and answer pair related to the target field from the pre-set field data set. The field data set can come from an existing field knowledge base, a question and answer system log, a manually sorted data set, or a network public resource, etc. The original question and answer pair contains an original question and an original answer matched therewith, and has a basic semantic correspondence and content accuracy. Through this step, the electronic device obtains a preliminary data basis that can be used for knowledge extraction and generation modeling.

[0041] In S102, the original question contained in the original question and answer pair is answered by using a trained reply generation model, to obtain reasoning information and a candidate answer in the answering process.

[0042] In this step, the electronic device calls a pre-trained reply generation model (for example, constructed based on a large language model and a long reasoning model), inputs the original question, and outputs the corresponding candidate answer by the reply generation model. Meanwhile, the electronic device further obtains the internal reasoning path or reasoning basis information (for example, a reasoning chain, a quoted content, a knowledge point matching result, etc.) in the reply generation process, as the key basis for subsequent knowledge point extraction. The reasoning information can reflect the semantic clues and knowledge structure relied on by the model when generating the candidate answer.

[0043] In S104, the accuracy of the candidate answer is verified based on the original answer contained in the original question and answer pair.

[0044] The step is used for evaluating the correctness of the candidate answer. For example, the verification can be performed by comparing the semantic similarity, keyword coverage, answer logical structure, etc. between the candidate answer and the original answer. The verification can be performed by rule matching, text similarity calculation, manual assistance correction, or a trained answer correctness verification model for automatic judgment. If the verification is passed, it means that the candidate answer can be consistent with the original answer at the semantic level, thereby improving the quality and reliability of the generated data.

[0045] For example, an answer correctness verification model can be pre-trained. The input of the answer correctness verification model is the original answer and the candidate answer, and the output of the answer correctness verification model is a first identifier indicating that the verification is passed or a second identifier indicating that the verification is failed.

[0046] In S106, when the candidate answer passes the verification, the knowledge points related to the target field are extracted from the reasoning information, and at least one question and answer pair is generated according to the knowledge points.

[0047] When the candidate answer passes the verification, the electronic device enters the knowledge extraction and generation phase. First, the domain knowledge points used to answer the question are identified in combination with the reasoning information in the model generation process, such as definitions, principles, causal relationships, and factual statements. Then, the question and answer pair generation model constructs new questions and answers by referring to the above knowledge points, realizes the expansion of structured and high-quality question and answer pairs, and provides rich data support for the training or fine-tuning of large models in specific fields.

[0048] The question and answer pair generation method provided in the embodiment can obtain the reasoning information and the answer of the original question by using the trained answer generation model, and accurately verify the candidate answer generated in combination with the original answer, so as to filter out high-quality and high-reliability reasoning information. The reasoning information usually contains some professional field knowledge used by the model in the thinking process, and the related knowledge points can be further extracted from the reasoning information to further improve the relevance of the target field and the content diversity, which is helpful to construct clear structure and accurate knowledge of professional field question and answer data.

[0049] In some embodiments, after the knowledge points related to the target field are extracted from the reasoning information, the electronic device removes the knowledge points that are repeated with the generated historical knowledge points from all the extracted knowledge points. By judging the repetition of the extracted knowledge points with the historical generation record and removing the repeated knowledge points, semantic redundancy or repeated content samples in the subsequent question and answer pair generation process can be effectively avoided, thereby improving the diversity and coverage of the question and answer data.

[0050] In addition, the deduplication mechanism helps to reduce the generation proportion of invalid question-answer pairs, save computing resources, improve the efficiency and quality control ability of the overall question-answer generation process, and has a significant advantage in large-scale question-answer data construction tasks. Further, the mechanism also embodies dynamic knowledge management capability in the generation task, ensuring that newly generated content is based on "incremental knowledge", thereby supporting the construction of content-rich, more balanced, and high-quality domain knowledge datasets.

[0051] In some embodiments, the following is an exemplary description of the generation process of question-answer pairs:

[0052] In the first possible implementation, the electronic device can directly construct the input content for question-answer generation based on the extracted individual knowledge points. For example, multiple knowledge points can be combined into question-answer generation input content through a preset format such as natural language segment splicing, structured prompt, question-answer template, etc. Then, the question-answer generation input content is input into a pre-trained question-answer pair generation model (e.g., based on a large language model, a long reasoning model), and at least one semantically complete and logically clear question-answer pair is generated by the model based on the question-answer generation input content. This approach can quickly convert extracted knowledge points into training samples, has high execution efficiency and implementation flexibility, is convenient for batch generation of question-answer pairs, and is particularly suitable for quickly constructing a pre-training or fine-tuning dataset with wide coverage of domain knowledge. At the same time, since the model is directly used for generation, the generation result can exhibit a certain language diversity, improving the expressiveness of the data.

[0053] In the second possible implementation, the electronic device can determine the types to which the individual knowledge points related to the target domain belong, including but not limited to background knowledge, definitional description, domain core theory, auxiliary fact, and conclusion statement. Then, the electronic device organizes the knowledge points related to the target domain into a tree structure with hierarchical relationships according to the association relationships between the types to which the individual knowledge points belong (e.g., from background knowledge to reasoning, and then to conclusion). Finally, at least one question-answer pair is generated by traversing the knowledge points at different levels in the tree structure.

[0054] For example, the electronic device can obtain at least one knowledge path from the root node to the leaf node from the tree structure, and the knowledge path includes multiple knowledge points with an association relationship, and each path represents a semantically coherent knowledge chain. Then, based on the multiple knowledge points contained in the knowledge path, the question-answer generation input content is constructed, and then the pre-trained question-answer pair generation model is used to generate at least one question-answer pair based on the question-answer generation input content.

[0055] The implementation expresses the semantic hierarchy and logical association between knowledge points through a tree structure, making the generated questions and answers more logically complete and contextually coherent. This approach is particularly suitable for building "sequential" question and answer, and helps to improve the controllability and professional depth of the generated content, adapting to more complex domain question and answer scenarios. In addition, the introduction of tree structure is also conducive to systematically covering different knowledge levels in the field, achieving more organized knowledge expression.

[0056] In a third possible implementation, the electronic device can analyze the causal relationship between all knowledge points related to the target domain, construct at least one causal chain, and the causal chain represents a sequence of knowledge points arranged according to the causal relationship, used to represent the cause, process and result of a certain event, concept or phenomenon. For example, through semantic dependency analysis or causal relationship extraction technology based on pre-trained models, a knowledge point pair of the "cause→result" type can be identified and linked to form a causal chain with a clear direction. Then the electronic device constructs the question and answer generation input content based on each causal chain; for example, the knowledge points in the causal chain can be converted into natural language question prompts, such as "Why does X cause Y?", "What happens after X occurs?", "How can you infer C from A?" and the like; or the causal chain is spliced as input context to guide the question and answer generation model to generate structured and semantically coherent questions and answers. Finally, the electronic device uses the pre-trained question and answer pair generation model to generate at least one question and answer pair based on the question and answer generation input content.

[0057] In this embodiment, the question and answer pairs generated based on the causal chain have greater logical depth, which helps to construct complex questions with reasoning, explanation and hierarchy, thereby improving the intelligence and value of the generated question and answer. It is suitable for application in professional fields such as science, medicine and engineering. In these fields, there are often clear causal relationships between knowledge points, and the causal chain driven approach can more realistically simulate the thinking process of human experts, improving the understanding and reasoning ability of the domain model.

[0058] In a fourth possible implementation, the electronic device can cluster all knowledge points related to the target field based on semantic similarity between any two knowledge points related to the target field to obtain at least one knowledge point set; and then obtain the target knowledge point from the knowledge point set. For example, for each knowledge point set, the electronic device can obtain the average semantic vector of the set as the center vector of the category based on the semantic vector of each knowledge point in the knowledge point set, and then calculate the distance between the semantic vector of each knowledge point and the center vector, and determine the knowledge point closest to the distance as the target knowledge point in the set; or, a knowledge point in the knowledge point set can be randomly selected as the target knowledge point; or, all knowledge points in the knowledge point set can be summarized to obtain the target knowledge point. Then the electronic device constructs the question and answer generation input content using the target knowledge point; and finally generates at least one question and answer pair by referring to the question and answer generation input content using the pre-trained question and answer pair generation model.

[0059] This way can cover core knowledge points under multiple semantic categories, and realize wide coverage and deep expression of field knowledge points in combination with the question and answer generation model, thereby significantly improving the diversity and quality of generated question and answer data, and providing more stable and rich data support for downstream fine-tuning of large models or industry knowledge transfer.

[0060] In some embodiments, referring to Figure 2 The embodiment of the present specification provides another question and answer pair generation method, which comprises the following steps:

[0061] In S200, the original question and answer pair of the target field is obtained. For details, refer to the description above, which will not be repeated here.

[0062] In S202, the original question contained in the original question and answer pair is detected for answerability to determine whether the original question can be directly answered by the answer generation model.

[0063] The core goal of this step is to perform semantic analysis on the structure and content of the original question to determine whether it has the completeness of being directly answerable by the answer generation model within the current knowledge range. For example, the answerability detection can be based on the following factors for comprehensive judgment: (1) whether the original question contains sufficient semantic information or context description; (2) whether the original question involves external knowledge (such as real-time data, calculation, API call); (3) whether the confidence of the answer generated by the answer generation model for the question is lower than a preset threshold; (4) whether it contains trigger keywords pointing to specific tool calls (such as “please calculate”, “call”, “analyze chart”, etc.).

[0064] In a possible implementation, to implement the answerability judgment of the original question, the electronic device can detect whether the original question contains a keyword related to tool invocation; if yes, output a detection conclusion that the question is not answerable; if no, output a detection conclusion that the question is answerable.

[0065] For example, the electronic device can perform word segmentation processing on the original question to obtain a plurality of candidate keywords; calculate the similarity between each candidate keyword and each reference keyword in a preset keyword set; the preset keyword set is constructed in advance according to historical invocation data related to tool invocation; for example, the preset keyword set can be obtained by mining, cleaning and summarizing user input in historical tool invocation requests, and can include typical imperative phrases or verb sets such as "invoke", "calculate", "query", "analyze chart", "read data", "get interface result", "please search online", and the like. If there is a keyword combination with a similarity higher than a preset similarity threshold, it is determined that the original question contains a keyword related to tool invocation, and a detection conclusion that the question is "unanswerable" is obtained; if there is no keyword combination with a similarity higher than the preset similarity threshold, it is determined that the original question does not contain a keyword related to tool invocation, and a detection conclusion that the question is "answerable" is obtained. Through the keyword similarity-based matching mechanism, the "answerability" preliminary screening judgment of the question can be quickly completed without completely analyzing the semantic structure, the computational overhead is significantly reduced, and the mechanism is suitable for large-scale batch processing scenarios. By judging whether the tool needs to be invoked before the question enters the generation process, the model can effectively avoid the incorrect generation of "lack of key data" questions, and improve the effectiveness and semantic integrity of the generated answers.

[0066] In another possible implementation, the electronic device can use a trained intent recognition model to identify whether the original question can be directly answered by the answer generation model, and obtain a detection conclusion. The intent recognition is obtained by supervised training based on a training data set, and the training data set includes questions as model input and reference conclusions as supervision labels corresponding to the questions. By using the model to perform answerability detection, the need for manual feature design can be reduced, the judgment accuracy can be improved, the misjudgment (such as misjudging a question that needs tool invocation and handing it over to the generation model) can be reduced, and the efficiency and accuracy of the overall question and answer link of the system can be improved.

[0067] Reference conclusions for unanswerable questions include the following cases: (1) The question sample carries keywords related to tool invocation, such as "help me check", "invoke interface", "get the latest data", etc.; (2) The question sample lacks necessary context description, such as pronoun ambiguity ("what's wrong with this?"), omission of core subject or condition ("is this right?"), etc.; (3) The question sample relies on user history information or session state, such as "the last question you said wrong" "the question I asked yesterday"; (4) The question sample involves external documents or image information but is not explicitly given, such as "please analyze the following figure" but the figure is not attached; (5) The question sample has logically incomplete or ambiguous expressions, making it difficult for the answer model to build a stable reasoning path.

[0068] The intent recognition model can be based on a deep neural network architecture and trained on the above supervised labeled training data set to have the ability to identify whether the question can be answered by the answer generation model. After the model training is completed, the electronic device can input the original question into the intent recognition model, output a detection conclusion indicating "answerable" or "unanswerable", and use it as a basis for subsequent process judgment.

[0069] In S204, if the detection conclusion is answerable, the original question is input into the answer generation model to answer the original question by the answer generation model, and the reasoning information and candidate answers in the answering process are obtained.

[0070] This step is suitable for the case where the question semantics is complete, has self-consistent logic, and the model has direct generation capability. The electronic device directly inputs the original question as input to the pre-trained answer generation model. The model can output its internal reasoning path or basis information, such as intermediate conclusions, implicit premises, fact references, etc., which can be used as an important basis for knowledge point extraction and quality verification.

[0071] In S206, if the detection conclusion is unanswerable, the tool indicated by the original question is called to obtain the supplementary content required to answer the original question, the original question is rewritten based on the supplementary content, and the rewritten question is input into the answer generation model to answer the rewritten question by the answer generation model, and the reasoning information and candidate answers in the answering process are obtained.

[0072] When it is judged that the original question cannot be directly answered by the model under the current context, the system can automatically identify the tool requirements implied in the question, such as: calling a calculation module or an external calculation engine when involving a calculation task; calling a data interface when involving real-time data (such as stock prices, weather, current affairs, etc.); calling a retrieval service when involving specific document content. The output of the tool, such as numerical values, descriptions, external references, etc., is embedded or rewritten into the original question to form a more complete rewritten question. Alternatively, the supplementary content and the original question can be input into a trained rewriting model (such as a large language model or a long reasoning model), and the rewriting model can rewrite the original question based on the supplementary content to obtain a rewritten question. Subsequently, the electronic device inputs the rewritten question into the answer generation model to obtain the candidate answer and the reasoning information.

[0073] Optionally, before inputting the rewritten question into the answer generation model, the electronic device can also output the rewritten question to present the rewritten question to the user, so as to obtain feedback information of the user for the rewritten question. For example, the rewritten question can be output to the user end or the calling system through a graphical interface, a voice interface, or an API (Application Programming Interface) interface, and feedback input from the user can be received. The feedback information can include but is not limited to the following forms: (1) explicit acceptance / rejection operation (such as clicking the "confirm / modify" button); (2) evaluative feedback (such as "expression is unclear", "does not conform to the intention", "has been improved"); (3) user's supplementary, modification suggestion or restatement of the natural language text of the rewritten content.

[0074] If the feedback information indicates that the rewritten question is available (for example, the user confirms or does not object), the electronic device can input the rewritten question into the answer generation model. If the feedback information indicates that the rewritten question is not available (for example, the user explicitly indicates that the question is still unclear, does not express the original intention, or suggests modification), the electronic device can re-rewrite the original question based on the feedback information and the supplementary content, for example, inputting the feedback information, the supplementary content, and the original question, the rewritten question, etc. into the rewriting model again, outputting the rewritten question again by the rewriting model, and performing the feedback confirmation process again or entering the next processing process. The introduction of the user feedback mechanism in this embodiment can ensure that the final question expression conforms to the real intention of the user, and improves the interaction friendliness. The user confirmation step can effectively avoid the question that does not answer the question due to the automatic rewriting bias, improve the semantic quality of the generated question from the source, and further enhance the pertinence and logical integrity of the answer generation.

[0075] In S208, the accuracy of the candidate answer is verified based on the original answer contained in the original question and answer pair. For details, refer to the description above, which will not be repeated here.

[0076] In S210, in the case that the candidate answer passes the verification, the knowledge points related to the target domain are extracted from the inference information, and at least one question and answer pair is generated according to the knowledge points. Please refer to the above description, which will not be repeated here.

[0077] In some embodiments, please refer to Figure 3 After generating at least one question and answer pair based on the knowledge points, the electronic device can perform answerability detection on the question and answer pair to determine whether the question contained in the question and answer pair can be directly answered by the answer generation model; if the detection result is that the question is answerable, it means that the question has complete semantics and sufficient context, and does not rely on external supplementary information, and the electronic device adds the question and answer pair to the question and answer pair set; if the detection result is that the question is not answerable, it means that the current question has semantic defects, relies on external data, or the logic is not coherent, and the electronic device can regenerate at least one question and answer pair based on the detection result and the knowledge points, for example, the question and answer generation input content can be reconstructed in combination with the detection result and the knowledge points used to generate the question and answer pair, and the answer generation model can generate a new question and answer pair according to the question and answer generation input content, until the answerability condition is met. This process can form a quality control closed loop mechanism to ensure that the generated question and answer pair has semantic and technical validity.

[0078] By performing answerability detection after question and answer generation, the present embodiment can effectively eliminate questions with unclear semantics, incomplete content, or difficult to answer, and ensure that the data in the final question and answer pair set has solvability and logical integrity, thereby improving the quality of the training data for the model. For the question and answer pair that cannot be answered, the regeneration operation can be performed based on the original knowledge points to realize the fault tolerance and self-recovery mechanism of question and answer construction, and enhance the robustness and intelligence of the overall system. The present scheme not only generates questions and answers, but also builds an evaluation and correction mechanism, and constructs a closed-loop automatic question and answer construction process of generation-judgment-reconstruction, which is helpful to realize large-scale high-quality question and answer generation without manual intervention.

[0079] In some embodiments, please refer to Figure 4 In the case that the candidate answer does not pass the verification, that is, the answer generated by the answer generation model conflicts with, is inconsistent with, or has semantic deviation from the original answer contained in the original question and answer pair, the electronic device can input the difference between the original answer and the candidate answer, and the original question into the answer generation model to guide the answer generation model to re-answer the original question in the manner of "comparative understanding" and "error correction optimization", and output a new candidate answer and its inference information.

[0080] By introducing the difference information between the original answer and the candidate answer, the embodiment can explicitly provide the error prompt and the expected reference to the model, help the model adjust the original answering strategy, and generate an answer that is more consistent with the domain semantics and user expectations. The scheme can be regarded as an answer reconstruction mechanism based on "feedback difference", which avoids discarding the entire question and answer path due to a single generation error, thereby reducing invalid calculation and improving the success rate of question and answer generation. The original answer often comes from domain experts or real user data, and the difference information as input can help the model gradually strengthen the understanding of the standard answer structure and term style in the domain, and help the model iterate and optimize.

[0081] Various technical features in the above embodiments can be combined in any manner, as long as the combination of features does not conflict or contradict, but due to the limited space, not all combinations are described. Therefore, any combination of various technical features in the above embodiments also belongs to the scope disclosed in the specification.

[0082] In some embodiments, the electronic device provided by the embodiments of the present specification also includes a processor, a memory for storing processor executable instructions, wherein the processor implements the method of any one of the above by running the executable instructions.

[0083] Figure 5 is a schematic structural diagram of a device provided by an example embodiment. Please refer to Figure 5 At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and of course, it can also include other hardware required by functions. One or more embodiments of the present specification can be implemented in a software manner, such as reading a corresponding computer program from the non-volatile memory 510 into the memory 508 by the processor 502 and then running. Of course, in addition to the software implementation, one or more embodiments of the present specification do not exclude other implementation manners, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0084] In some embodiments, the question and answer pair generation device can be applied to a device as shown in Figure 5 The question and answer pair generation device can include:

[0085] An original question pair acquisition module is configured to acquire an original question and answer pair in a target domain.

[0086] A question answering module is configured to use a trained answer generation model to answer an original question included in the original question and answer pair, and obtain reasoning information and a candidate answer in the answering process.

[0087] The candidate answer verification module is configured to verify the accuracy of the candidate answer based on the original answer contained in the original question-answer pair.

[0088] The question-answer pair generation module is configured to extract knowledge points related to the target domain from the inference information if the candidate answer passes the verification, and generate at least one question-answer pair based on the knowledge points.

[0089] In an implementation manner, the question-answer pair generation module is specifically configured to determine types to which the knowledge points related to the target domain belong, organize the knowledge points related to the target domain into a tree structure with a hierarchical relationship according to an association relationship between the types to which the knowledge points belong, and generate at least one question-answer pair by traversing the knowledge points at different levels in the tree structure.

[0090] In an implementation manner, the question-answer pair generation module is specifically configured to obtain at least one knowledge path from a root node to a leaf node in the tree structure, the knowledge path including a plurality of knowledge points with an association relationship, construct question-answer generation input content based on the plurality of knowledge points contained in the knowledge path, and generate at least one question-answer pair by referring to the question-answer generation input content by using the pre-trained question-answer pair generation model.

[0091] In an implementation manner, the question-answer pair generation module is specifically configured to analyze a causal relationship between all the knowledge points related to the target domain to construct at least one causal chain, the causal chain representing a sequence of a group of knowledge points arranged according to the causal relationship, construct question-answer generation input content based on each causal chain, and generate at least one question-answer pair by referring to the question-answer generation input content by using the pre-trained question-answer pair generation model.

[0092] In an implementation manner, the question-answer pair generation module is specifically configured to cluster all the knowledge points related to the target domain based on a semantic similarity between any two knowledge points related to the target domain to obtain at least one knowledge point set, obtain a target knowledge point from the knowledge point set, and construct question-answer generation input content by using the target knowledge point, and generate at least one question-answer pair by referring to the question-answer generation input content by using the pre-trained question-answer pair generation model.

[0093] In an implementation manner, the method further includes a reply-ability detection module configured to perform reply-ability detection on the original question to determine whether the original question can be directly answered by the reply generation model, input the original question into the reply generation model if the detection result is that the original question is reply-able, call a tool indicated by the original question to obtain supplementary content required to answer the original question, rewrite the original question based on the supplementary content, and input the rewritten question into the reply generation model to obtain inference information and a candidate answer in a process of answering the rewritten question by the reply generation model.

[0094] In an implementation manner, the answerability detection module is specifically configured to detect whether the original question contains the keyword related to the tool call; if yes, an unanswerable detection conclusion is output; if no, an answerable detection conclusion is output.

[0095] In an implementation manner, the answerability detection module is specifically configured to perform word segmentation processing on the original question to obtain a plurality of candidate keywords; similarity between each candidate keyword and each reference keyword in a preset keyword set is calculated; the preset keyword set is constructed in advance according to historical call data related to the tool call; if there is a keyword combination with a similarity higher than a preset similarity threshold, it is determined that the original question contains the keyword related to the tool call; if there is no keyword combination with a similarity higher than the preset similarity threshold, it is determined that the original question does not contain the keyword related to the tool call.

[0096] In an implementation manner, the answerability detection module is specifically configured to use a trained intent recognition model to identify whether the original question can be directly answered by the answer generation model to obtain a detection conclusion; wherein the intent recognition is obtained by supervised training based on a training data set; the training data set includes: a question sample as a model input, and a reference conclusion as a supervision label corresponding to the question sample; the reference conclusion for the unanswerable question sample includes at least one of the following: the question sample carries the keyword related to the tool call, the question sample lacks context description, the question sample involves an attachment but does not explicitly give, and the question sample has ambiguous expression.

[0097] In an implementation manner, the feedback processing module is further configured to output the rewritten question to obtain feedback information of the user for the rewritten question; if the feedback information indicates that the rewritten question is available, the rewritten question is input into the answer generation model; if the feedback information indicates that the rewritten question is not available, the original question is rewritten based on the feedback information and the supplementary content.

[0098] In an implementation manner, the question answering module is further configured to, in a case where the candidate answer fails to pass the verification, input a difference between the original answer and the candidate answer and the original question into the answer generation model to obtain reasoning information in the answering process and the candidate answer by referring to the difference.

[0099] In an implementation manner, the answerability detection module is further configured to perform answerability detection on the question and answer pair to determine whether a question contained in the question and answer pair can be directly answered by the answer generation model; if the detection conclusion is answerable, the question and answer pair is added to the question and answer pair set; if the detection conclusion is unanswerable, at least one question and answer pair is regenerated based on the detection conclusion and the knowledge point.

[0100] The implementation process of the functions and roles of each module in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0101] Based on the same idea as the above method, the present specification also provides a computer readable storage medium, which stores computer instructions, and the instructions are executed by a processor to realize the steps of the method of any one of the above embodiments.

[0102] The computer readable medium includes permanent and non-permanent, removable and non-removable media, which can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, disk storage, quantum memory, graphene-based storage medium or other magnetic storage device, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition in this paper, computer readable medium does not include transitory computer readable medium, such as modulated data signal and carrier wave.

[0103] Based on the same idea as the above method, the present specification also provides a computer program product, which includes computer program / instructions, and the instructions are executed by a processor to realize the steps of the method described in any one of the above embodiments.

[0104] The above is only the preferred embodiment of one or more embodiments of the present specification, and does not limit one or more embodiments of the present specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of the present specification shall be included in the protection scope of one or more embodiments of the present specification.

Claims

1. A method for generating question and answer pairs, comprising: obtaining original question and answer pairs of a target domain; answering original questions contained in the original question and answer pairs by using a trained reply generation model, to obtain reasoning information and candidate answers in the answering process; verifying the accuracy of the candidate answers based on original answers contained in the original question and answer pairs; in the case where the candidate answers pass the verification, extracting knowledge points related to the target domain from the reasoning information, and generating at least one question and answer pair according to the knowledge points.

2. The method of claim 1, wherein the generating at least one question and answer pair according to the knowledge points comprises: determining the types of each knowledge point related to the target domain; organizing the knowledge points related to the target domain into a tree structure with hierarchical relationships according to the association relationships between the types of each knowledge point; generating at least one question and answer pair by traversing the knowledge points at different levels in the tree structure.

3. The method of claim 2, wherein the generating at least one question and answer pair by traversing the knowledge points at different levels in the tree structure comprises: obtaining at least one knowledge path from a root node to a leaf node in the tree structure, the knowledge path including a plurality of knowledge points with an association relationship; constructing question and answer generation input content based on the plurality of knowledge points contained in the knowledge path; generating at least one question and answer pair by using a pre-trained question and answer pair generation model and referring to the question and answer generation input content.

4. The method of claim 1, wherein the generating at least one question and answer pair according to the knowledge points comprises: analyzing the causal relationships between all knowledge points related to the target domain to construct at least one causal chain, the causal chain representing a sequence of knowledge points arranged according to causal relationships; constructing question and answer generation input content based on each causal chain, and generating at least one question and answer pair by using a pre-trained question and answer pair generation model and referring to the question and answer generation input content; and / or, clustering all knowledge points related to the target domain based on the semantic similarity between any two knowledge points related to the target domain to obtain at least one knowledge point set; obtaining a target knowledge point from the knowledge point set and constructing question and answer generation input content by using the target knowledge point; generating at least one question and answer pair by using a pre-trained question and answer pair generation model and referring to the question and answer generation input content.

5. The method of claim 1, further comprising, before the answering original questions contained in the original question and answer pairs by using a trained reply generation model: detecting the answerability of the original questions to determine whether the original questions can be directly answered by the reply generation model; if the detection result is that the original questions are answerable, inputting the original questions into the reply generation model. ​ If the detection result is that the question cannot be answered, a tool indicated by the original question is invoked to obtain supplementary content required for answering the original question, the original question is rewritten based on the supplementary content, the rewritten question is input into the answer generation model, the answer generation model answers the rewritten question, and reasoning information and a candidate answer in the answering process are obtained.

6. The method of claim 5, wherein the original question is subjected to answerability detection to determine whether the original question can be directly answered by the answer generation model, comprising: performing word segmentation on the original question to obtain a plurality of candidate keywords; calculating similarity between each of the candidate keywords and each reference keyword in a preset keyword set, wherein the preset keyword set is constructed in advance based on historical invocation data related to tool invocation; if there is a keyword combination with a similarity higher than a preset similarity threshold, an unanswerable detection result is output; if there is no keyword combination with a similarity higher than the preset similarity threshold, an answerable detection result is output; and / or using a trained intent recognition model to determine whether the original question can be directly answered by the answer generation model, and obtaining the detection result, wherein the intent recognition is obtained based on supervised training of a training data set, and the training data set includes question samples as model input and reference conclusions as supervision labels corresponding to the question samples; the reference conclusions for unanswerable question samples include at least one of the following: the question samples carry keywords related to tool invocation, the question samples lack context descriptions, the question samples involve attachments but do not explicitly give out, and the question samples have ambiguous expressions.

7. The method of claim 5, wherein before the rewritten question is input into the answer generation model, the method further comprises: outputting the rewritten question to obtain feedback information of a user for the rewritten question; if the feedback information indicates that the rewritten question is usable, inputting the rewritten question into the answer generation model; if the feedback information indicates that the rewritten question is not usable, rewriting the original question based on the feedback information and the supplementary content.

8. The method of claim 1, further comprising: if the candidate answer fails to pass verification, inputting a difference between the original answer and the candidate answer and the original question into the answer generation model to answer the original question again by the answer generation model based on the difference, and obtaining reasoning information and a candidate answer in the answering process.

9. The method of claim 1, further comprising: performing answerability detection on the question and answer pair to determine whether a question included in the question and answer pair can be directly answered by the answer generation model; if the detection result is that the question and answer pair is answerable, adding the question and answer pair to a question and answer pair set; if the detection result is that the question and answer pair is unanswerable, generating at least one question and answer pair again based on the detection result and the knowledge point. a processor; 10. An electronic device comprising: ​ a memory for storing processor executable instructions; wherein the processor implements the steps of the method of any of claims 1-9 by running the executable instructions.

11. A computer readable storage medium having stored thereon computer instructions, the instructions, when executed by a processor, implementing the steps of the method of any of claims 1-9.

12. A computer program product comprising computer program / instructions, the computer program / instructions, when executed by a processor, implementing the steps of the method of any of claims 1-9.

Images