Question-answer pair generation method, device, storage medium and electronic device

Abstract

This specification discloses a method, device, storage medium and electronic device for generating question-answer pairs. In the question-answer pair generation method provided in this specification, a text material for generating a question-answer pair is obtained; the text material is input into a large language model, so that the large language model outputs a pseudo question-answer pair according to the text material; in response to receiving a real question input by a user, a target pseudo question-answer pair matching the real question is determined in each pseudo question-answer pair; the target pseudo question-answer pair is embedded in a prompt template of the large language model, and the real question is input into the large language model to obtain an optimized answer output by the large language model; the real question and the optimized answer are determined as a pending question-answer pair, and the pending question-answer pair is quality evaluated; in response to the pending question-answer pair passing the quality evaluation, the pending question-answer pair is determined as an available question-answer pair.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, storage medium and electronic device for generating question and answer pairs. Background Art

[0002] Open-domain knowledge question answering is a natural language processing task that aims to answer various questions raised by users without special background knowledge, without pre-set domains or restrictions, and covers a wide range of topics. Open-domain knowledge question answering plays a significant role in building domain knowledge bases and expert systems, improving product experience, improving production efficiency, and promoting the development of artificial intelligence.

[0003] At present, the generation of answers to questions mainly relies on two methods: information retrieval and knowledge graph. Among them, information retrieval uses traditional search engine technology to retrieve relevant documents and extract answers from them. It is suitable for processing long texts, but the content relevance is often poor. Knowledge graphs use structured entities and relationships to query answers. Compared with search technology, they can provide higher accuracy and explainability, but knowledge graphs are complex to build and have high data update costs, making it difficult to cover all knowledge in open fields.

[0004] Therefore, how to automatically construct high-quality question-answer data is an urgent problem to be solved. Summary of the invention

[0005] The present specification provides a question-answer pair generation method, device, storage medium and electronic device to at least partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a question-answer pair generation method, comprising:

[0008] Obtain text materials for generating question-answer pairs;

[0009] Inputting the text material into a large language model, so that the large language model outputs a pseudo question-answer pair according to the text material;

[0010] In response to receiving a real question input by a user, determining a target pseudo question-answer pair matching the real question among the pseudo question-answer pairs;

[0011] Embedding the target pseudo question-answer pair into the prompt template of the large language model, and inputting the real question into the large language model to obtain an optimized answer output by the large language model;

[0012] Determine the real question and the optimized answer as a pending question-answer pair, and perform quality assessment on the pending question-answer pair;

[0013] In response to the pending question-answer pair passing the quality evaluation, the pending question-answer pair is determined as an available question-answer pair.

[0014] Optionally, obtaining text materials for generating question-answer pairs specifically includes:

[0015] Determine the professional fields to be covered when generating question-answer pairs;

[0016] For each professional field, use a large language model to obtain keywords in that professional field;

[0017] The keywords are searched to obtain text materials in the professional field that match the keywords.

[0018] Optionally, inputting the text material into a large language model so that the large language model outputs a pseudo question-answer pair according to the text material specifically includes:

[0019] Inputting the text material into a large language model so that the large language model extracts entities contained in the text material;

[0020] Inputting the entity as a preset answer into the large language model, and obtaining a predicted question output by the large language model that matches the preset answer;

[0021] The preset answer and the predicted question are determined as a pseudo question-answer pair.

[0022] Optionally, after determining the preset answer and the predicted question as a pseudo question-answer pair, the method further includes:

[0023] Inputting the text material and the pseudo question-answer pair into the large language model, so that the large language model generates an explanatory text of the pseudo question-answer pair according to the text material;

[0024] The explanatory text is determined as a part of the pseudo question-answer pair.

[0025] Optionally, determining a target pseudo question-answer pair matching the real question in each pseudo question-answer pair specifically includes:

[0026] Encoding the predicted question contained in each pseudo question-answer pair to obtain a first text vector of each predicted question, and encoding the real question to obtain a second text vector of the real question;

[0027] Determining the similarity between the second text vector and each first text vector;

[0028] A target first text vector having the highest similarity to the second text vector is determined, and a pseudo question-answer pair containing a predicted question corresponding to the target first text vector is determined as a pseudo question-answer pair matching the real question.

[0029] Optionally, performing a quality assessment on the pending question-answer pair specifically includes:

[0030] Determine the relevance score and the factual score of the pending question and answer pair, wherein the relevance score is used to characterize the degree of relevance between the optimized answer included in the pending question and answer pair and the real question, and the factual score is used to characterize the accuracy of the optimized answer included in the pending question and answer pair as an answer to the real question.

[0031] Optionally, in response to the pending question-answer pair passing the quality evaluation, determining the pending question-answer pair as an available question-answer pair specifically includes:

[0032] In response to the relevance score of the pending question-answer pair being not less than a first threshold, and the factual score of the pending question-answer pair being not less than a second threshold, the pending question-answer pair is determined as an available question-answer pair.

[0033] This specification provides a question-answer pair generation device, the device comprising:

[0034] An acquisition module, used to acquire text materials for generating question-answer pairs;

[0035] An input module, used for inputting the text material into a large language model, so that the large language model outputs a pseudo question-answer pair according to the text material;

[0036] A determination module, configured to determine, in response to receiving a real question input by a user, a target pseudo question-answer pair matching the real question among the pseudo question-answer pairs;

[0037] An embedding module, used to embed the target pseudo question-answer pair into the prompt template of the large language model, and input the real question into the large language model to obtain an optimized answer output by the large language model;

[0038] An evaluation module, used to determine the real question and the optimized answer as a pending question-answer pair, and perform quality evaluation on the pending question-answer pair;

[0039] A generating module is used to determine the pending question-answer pair as an available question-answer pair in response to the pending question-answer pair passing the quality evaluation.

[0040] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned question-answer pair generation method is implemented.

[0041] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned question-answer pair generating method when executing the program.

[0042] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0043] In the question-answer pair generation method provided in the present specification, a text material for generating a question-answer pair is obtained; the text material is input into a large language model, so that the large language model outputs a pseudo question-answer pair according to the text material; in response to receiving a real question input by a user, a target pseudo question-answer pair matching the real question is determined among the pseudo question-answer pairs; the target pseudo question-answer pair is embedded in a prompt template of the large language model, and the real question is input into the large language model to obtain an optimized answer output by the large language model; the real question and the optimized answer are determined as a pending question-answer pair, and a quality assessment is performed on the pending question-answer pair; in response to the pending question-answer pair passing the quality assessment, the pending question-answer pair is determined as an available question-answer pair.

[0044] When using the question-answer pair generation method provided in this specification to generate available question-answer pairs, after obtaining the text material, a pseudo question-answer pair is automatically generated through a large language model, and the pseudo question-answer pair is combined with the real question raised by the user to obtain an optimized answer to construct a pending question-answer pair, and finally the pending question-answer pair that passes the quality assessment is determined as an available question-answer pair. This method is based on the self-mining method of a large language model, and automatically constructs a set of high-standard knowledge sample libraries throughout the entire process, and more accurately generates question-answer data for the required tasks by adding semantically similar small sample examples during the model reasoning process. At the same time, the relevance and factual evaluation criteria of the generated data are defined, and the score calculation and filtering mechanism is used to ensure the quality consistency of open-domain question-answer pairs. This evaluation framework not only shows significant advantages in open-domain question-answer generation, but also provides important reference value for quality assessment of other data synthesis methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0046] Figure 1 A flowchart of a method for generating question-answer pairs in this specification;

[0047] Figure 2 A schematic diagram of a process for generating keywords in various professional fields using a large language model provided in this manual;

[0048] Figure 3 A schematic diagram of searching for keywords provided in this manual;

[0049] Figure 4 A schematic diagram of a process for extracting entities from text materials using a large language model provided in this specification;

[0050] Figure 5 A flowchart of a method for reversely constructing a question using an entity as an answer provided for this specification;

[0051] Figure 6 A schematic diagram of a process for adding explanatory text to a pseudo question-answer pair provided in this manual;

[0052] Figure 7 A pseudo code schematic diagram for encoding text provided in this specification;

[0053] Figure 8 A schematic diagram of a process provided in this specification for using a target pseudo question-answer pair to help a large language model generate an optimized answer;

[0054] Fig. 9 A schematic diagram of a question-answer pair generating device provided in this specification;

[0055] Fig.10 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0056] In recent years, the Large Language Model (LLM) technology based on massive pre-trained data has been rapidly developed and applied in the industry. Its rich knowledge structure and zero-sample learning and reasoning capabilities provide a new perspective for open-domain knowledge acquisition. Existing studies have shown that large language models have good generalization capabilities and can generate multi-domain and multi-topic question-and-answer information. Even when the answers do not have an exact match to the information needs, their language performance is more natural and fluent. By adding small sample learning examples to the large language model reasoning process, the task performance ability can be improved with limited data. In summary, automatically building high-quality question-and-answer data based on large language models has positive practical significance.

[0057] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0059] Figure 1 The following is a flow chart of a method for generating a question-answer pair in this specification, which specifically includes the following steps:

[0060] S100: Acquire text material for generating question-answer pairs.

[0061] All steps in the question-answer pair generation method provided in this specification can be implemented by any electronic device with computing functions, such as a terminal, a server, etc. For ease of description and understanding, this specification mainly uses the execution subject as a server as an example for description.

[0062] This method is mainly used to generate question-answer pairs. Question-answer pairs are data pairs consisting of questions and answers, and are widely used in various natural language processing tasks, such as question-answering tasks, dialogue tasks, knowledge graph construction tasks, and translation tasks. Based on this, in this step, we first need to obtain text materials that can be used to generate question-answer pairs.

[0063] Text materials can be obtained from a variety of channels, such as various databases, article content, encyclopedia entries, etc., and this specification does not make specific restrictions on this. Taking Wikipedia, which is mainly used in the early implementation of this method, as an example, Wikipedia is a global online encyclopedia that covers a wide range of topics and fields. The content has been strictly edited and reviewed, and has diversity and accuracy. As an open and shared knowledge platform, anyone and any organization can access and contribute content, and it has become one of the largest and most popular reference tools. The data organization of Wikipedia adopts an encyclopedic structure, and the entries are classified by subject. The content is clear and the knowledge dimension is rich, covering the definition, history, application, controversy and other parts of each entry, and its data is dynamically updated, edited and reviewed in real time, which can reflect the latest information and content faster than traditional tools. In summary, these characteristics make Wikipedia one of the world's largest online knowledge bases, which is not only widely used for daily queries, but also has important application value in academic research, data mining and natural language processing.

[0064] Additionally, when acquiring text materials, the text materials can be classified according to different professional fields, so that question-answer pairs for different professional fields can be generated according to different categories later. Specifically, the professional fields covered when generating question-answer pairs can be determined; for each professional field, the keywords in the professional field can be acquired using a large language model; the keywords are searched in a text database to obtain text materials in the professional field that match the keywords.

[0065] Still taking Wikipedia as an example, among the 1,448,838 entries currently existing in Wikipedia, a total of 8 major categories are divided, and 364 subcategories are divided under the 8 major categories. Therefore, according to the classification of Wikipedia, each of its subcategories can be regarded as a professional field to complete the division of professional fields. Of course, according to the needs of the final generation of question-answer pairs, several subcategories can be selected as the professional fields that need to be covered by the generated question-answer pairs, and this manual does not make specific restrictions on this.

[0066] At the same time, considering the limitation of storage space and the efficiency of generating and applying question-answer pairs, it is difficult to simply and directly use all the retrieved contents in a professional field as text materials when obtaining text materials. Therefore, in this method, keywords of various professional fields can be generated through a large language model, and then the generated keywords are retrieved, and the retrieved contents are determined as text materials for subsequent generation of question-answer pairs.

[0067] Figure 2 This is a schematic diagram of the process of using a large language model to generate keywords in various professional fields. Figure 2 As shown, assuming that the professional fields that need to be covered include at least science, sports, history and other fields, then in each professional field, several relevant instructions can be set to be input into the large language model, and the responses given by the large language model can be determined as keywords. For example, in the field of science, the instruction "Please list some scientific discoveries" can be input into the large language model, in the field of sports, the instruction "Please list some sports" can be input into the large language model, and in the field of history, the instruction "Please list some historical events" can be input into the large language model. For each instruction, the large language model will give several corresponding replies, and these replies can be used as keywords.

[0068] Still taking Wikipedia as an example, when searching for keywords, the search is performed in Wikipedia. Figure 3 This is a schematic diagram of a keyword search provided in this manual. Figure 3 As shown, assuming that the keyword "black hole" in the scientific field is searched at this time, the content related to the entry of black hole in Wikipedia can be obtained. At this time, the retrieved content can be determined as text material.

[0069] S102: Input the text material into a large language model, so that the large language model outputs a pseudo question-answer pair according to the text material.

[0070] After completing the acquisition of the text material in step S100, in this step, the text material can be input into the large language model to obtain the pseudo question-answer pair output by the large language model. The large language model used in this specification can be any pre-trained large language model, such as ChatGPT, Llama, etc., and this specification does not make specific restrictions on this.

[0071] In this step, the text material is input into the large language model, and relevant instructions are input into the large language model at the same time. The large language model can then construct a question-answer pair based on the input text material and instructions. Since there may be some differences between the question-answer pairs constructed by the large language model and the real questions raised by the user, these question-answer pairs are not suitable to be directly used as question-answer pairs that can be put into practical applications. In this method, they are used as pseudo question-answer pairs and are used to assist in the subsequent generation of higher-quality question-answer pairs.

[0072] There are many methods for generating pseudo question-answer pairs using a large language model, and this specification provides a specific embodiment for reference. Specifically, the text material can be input into a large language model, so that the large language model extracts entities contained in the text material; the entity is input into the large language model as a preset answer, and a predicted question output by the large language model that matches the preset answer is obtained; the preset answer and the predicted question are determined as a pseudo question-answer pair.

[0073] Since the large language model is a generative model, its free output content cannot be guaranteed to be 100% stable and reliable. Therefore, if the text material is directly input and the large language model is allowed to freely construct pseudo question-answer pairs, the pseudo question-answer pairs obtained are likely to be very poor. Therefore, in this method, the form of reversely constructing the prediction question based on the preset answer is adopted to enable the large language model to generate more stable and reliable pseudo question-answer pairs.

[0074] First, the large language model can be used to extract entities contained in the input text material. Figure 4 This is a flow chart of a method for extracting entities from text materials using a large language model. Figure 4 As shown, the text material is still taken as the material related to "black hole". When the text material is input to the large language model, the instruction "Please extract the named entities and output" is simultaneously input to the large language model. Among them, the "xxx..." in the curly brackets "{}" represents the input text material. The large language model will output the entities contained in the text material according to the instruction, such as "black hole", "quasar", "general relativity", etc.

[0075] Additionally, after the entities are extracted, the entities output by the large language model can be cleaned to ensure the quality of subsequent generation. The main methods may include entity format specification, synonym conversion, sensitive word filtering, etc. For example, the book title marks can be removed from entities with book title marks, and named entities with the same meaning can be merged, etc. The main purpose of sensitive word filtering is to delete some unhealthy words.

[0076] The extracted entities can be used to reversely construct questions. In this method, the core idea of ​​reversely generating questions is to generate corresponding questions from a given answer through a large language model. Compared with the traditional question-answering generation process, it changes the order from "question to answer" and starts from the answer to explore and mine multiple possible questions. This method can generate multiple questions from different angles and different semantic levels, thereby enriching the expression and diversity of question-answering data.

[0077] The large language model needs to be able to generate questions with reasonable content and fluent semantics, which relies on the model's ability to understand context and generate natural language. Therefore, in addition to providing the large language model with entities as answers, the content context information to which the entity belongs also needs to be input to help it better understand and construct questions. Figure 5 A flowchart of a method for reversely constructing a question using an entity as an answer is provided for this specification. Figure 5 As shown, following the above example, the entity "mass" extracted by the large language model from the text material related to "black hole" is input into the large language model as the answer, and the relevant instruction "Here is a paragraph of content as follows: {xxx...}, where [xxx] is the answer to the question, please construct the question of the answer and output it" is input into the large language model. The large language model can then construct the corresponding question according to the input text material and answer in accordance with the instruction. Among them, the "xxx..." in the curly brackets "{}" is the text material, and the "xxx" in the square brackets "[ ]" is the entity as the preset answer. In this embodiment, the large language model uses "mass" as the preset answer, and according to the text material related to "black hole", it outputs the prediction question "General relativity predicts what is tight enough to distort space-time to form a black hole".

[0078] At this point, the preset answer input to the large language model and the predicted question output by the large language model can be determined as a pseudo question-answer pair. Preferably, in order to reduce the impact of the large model hallucination and improve the quality of the pseudo question-answer pair, the reverse construction process can be repeatedly experimented in this step. Specifically, the same entity can be input into the large language model twice as a preset answer. Only when the questions constructed in the two experiments are consistent, the obtained pseudo question-answer pair will be included in the pseudo question-answer set.

[0079] Furthermore, in order to better ensure the integrity of information and enhance the credibility of answers, additional explanatory text may be added to each pseudo question-answer pair. Specifically, the text material and the pseudo question-answer pair may be input into the large language model, so that the large language model generates the explanatory text of the pseudo question-answer pair based on the text material; and the explanatory text is determined as a part of the pseudo question-answer pair.

[0080] Figure 6 A schematic diagram of a process for adding explanatory text to a pseudo question-answer pair provided in this specification. Figure 6 As shown, you can follow Figure 6 The template in the example "Here is a paragraph as follows: {xxx…}. Question: [xxx]. Answer [xxx]. Please give a brief explanation of the above question and answer pair based on the paragraph content and output it", input a paragraph of text material and the pseudo question and answer pair generated based on the paragraph of text material, as well as the corresponding instructions to the large language model, and you can get the relevant explanation of the pseudo question and answer pair by the large language model. Among them, the "xxx…" in the curly brackets "{}" represents the text material, the "xxx" in the first bracket "[]" represents the predicted question in the pseudo question and answer pair, and the "xxx" in the second bracket "[]" represents the preset answer in the pseudo question and answer pair.

[0081] The explanatory text of the pseudo question-answer pair may be a part of the pseudo question-answer pair, for example, may be stored together with the pseudo question-answer pair as an explanation of a preset answer.

[0082] S104: In response to receiving a real question input by a user, determining a target pseudo question-answer pair matching the real question among the pseudo question-answer pairs.

[0083] After the pseudo question-answer pairs are generated in step S102, when the user inputs a real question in this step, a pseudo question-answer pair matching the real question can be found among the pseudo question-answer pairs, and based on this, high-quality usable question-answer pairs can be generated subsequently.

[0084] Although large language models have strong zero-shot learning capabilities, adding small sample data, especially examples with strong semantic relevance, can significantly improve the generation quality and task adaptability of the model, making it more accurately meet specific needs. This is specifically reflected in the following aspects:

[0085] First, adding small sample examples with strong semantic relevance can help the large model understand the task requirements and context more accurately, and reduce the generation of answers that are irrelevant to the question or have misunderstandings. For example, by adding relevant task samples, the model can more accurately judge the user's intention in asking questions and generate answers that are more consistent with this intention.

[0086] Second, although large language models have zero-shot learning capabilities, for some refined tasks or professional fields, the zero-shot mode may not be able to meet high-standard task requirements. Through small sample examples, the model can effectively adapt to more complex or more professional task requirements, such as question answering in specific fields, complex reasoning, and generation tasks that are more in line with actual needs.

[0087] Third, small sample examples provide a concrete frame of reference for the model, preventing the model from being biased due to lack of guidance. Small samples with high semantic relevance allow the model to adapt to the meaning of the problem, contextual requirements, and goals, thereby reducing the error of the results.

[0088] Therefore, in this step, the pseudo question-answer pairs generated in step S102 can be used as small sample examples to assist the large language model in answering real questions raised by users, thereby enhancing the answering ability of the large language model and laying the foundation for the subsequent generation of usable question-answer pairs.

[0089] When determining the pseudo question-answer pair that matches the real question, it can be achieved by, for example, semantic similarity matching. Specifically, the predicted question contained in each pseudo question-answer pair can be encoded to obtain the first text vector of each predicted question, and the real question can be encoded to obtain the second text vector of the real question; the similarity between the second text vector and each first text vector can be determined; the target first text vector with the highest similarity to the second text vector can be determined, and the pseudo question-answer pair containing the predicted question corresponding to the target first text vector can be determined as the target pseudo question-answer pair that matches the real question.

[0090] Text encoding and vectorization is a technology that converts text content into digital vectors, which is mainly used for tasks such as text search, clustering, and recommendation. Once the text content is vectorized, the distance between these vectors, that is, the similarity, can be used to measure the relevance of the text semantic space. The smaller the distance between the vectors, the higher the similarity, which means that the texts corresponding to the two vectors are more relevant; conversely, the larger the distance between the vectors, the lower the similarity, which means that the texts corresponding to the two vectors are less relevant.

[0091] In this method, after the pseudo question-answer pair is stored in a fixed format, such as "question-answer-explanatory text", the existing model interface of OpenAI can be directly called to process the text of the predicted question contained in the pseudo question-answer pair, and the generated vector data can be persistently written into the vector database for subsequent retrieval. On the other hand, when receiving the real question input by the user, the real question can also be encoded in the same way to obtain its corresponding vector. Among them, the pseudo code for encoding the predicted question and the real question is as follows: Figure 7 shown.

[0092] In this method, the vector obtained by encoding the predicted question is used as the first text vector, and the vector obtained by encoding the real question is used as the second text vector. When matching, the similarity between the second text vector and each first text vector can be calculated, and the first text vector with the highest similarity can be selected, and the pseudo question-answer pair corresponding to the predicted question is determined as the target pseudo question-answer pair matching the real question.

[0093] Among them, the similarity can be calculated using the cosine distance formula. Compared with other calculation methods such as Euclidean distance, cosine distance can better reflect the differences in the spatial direction and is better adapted to high-dimensional vectorization. It is more reasonable to use the content itself to calculate similarities and differences.

[0094] S106: Embed the target pseudo question-answer pair into the prompt template of the large language model, and input the real question into the large language model to obtain an optimized answer output by the large language model.

[0095] After determining the target pseudo question-answer pair that matches the real question in step S104, the target pseudo question-answer pair can be used in this step to help the large language model answer the real question raised by the user, and obtain the optimized answer output by the large language model.

[0096] Figure 8 This is a schematic diagram of a process provided in this specification that uses a target pseudo question-answer pair to help a large language model generate an optimized answer. Generally, there is a prompt template in a large language model. The prompt template is generally a designed text fragment used to guide the large language model to generate text that meets the expected results. By designing different prompt templates, the behavior of the large language model can be controlled to enable it to complete different generation tasks. Figure 8 As shown, after determining the target pseudo question-answer pair that matches the real question in step S104, the target pseudo question-answer pair can be embedded into the prompt template of the large language model to guide the content and format of the large language model when answering the question. After the embedding is completed, the real question is input to the large language model, and the large language model can generate an optimized high-quality answer under the guidance of the target pseudo question-answer pair as the prompt template, which is called the optimized answer in this method.

[0097] S108: Determine the real question and the optimized answer as a pending question-answer pair, and perform a quality assessment on the pending question-answer pair.

[0098] After obtaining the optimized answer output by the large language model in step S106, the real question input by the user and the optimized answer output by the large language model can be determined as a set of pending question-answer pairs in this step. Compared with pseudo question-answer pairs generated entirely by the large language model itself based on text materials, the pending question-answer pairs contain the real questions raised by the user and the optimized answers given by the large language model under the guidance of small sample examples, which are higher-quality question-answer pair data.

[0099] However, it should be considered that not all optimized answers are accurate enough to meet the needs of users when asking questions. Therefore, in this step, the quality of the pending question and answer pairs can be additionally evaluated to screen out the pending question and answer pairs with excellent quality as available question and answer pairs, and filter out the pending question and answer pairs that do not meet the needs. Specifically, the relevance score and factuality score of the pending question and answer pair can be determined, wherein the relevance score is used to characterize the degree of correlation between the optimized answer contained in the pending question and answer pair and the real question, and the factual score is used to characterize the accuracy of the optimized answer contained in the pending question and answer pair as an answer to the real question.

[0100] In this method, when performing quality assessment on pending question-answer pairs, the relevance and factuality of the pending question-answer pairs can be mainly considered.

[0101] Relevance refers to whether the generated answer is relevant to the question asked by the user, that is, whether it is relevant to the main point of the question without going off topic. The traditional method for evaluating relevance is manual sampling by domain experts, but this method is costly and inefficient. In this method, the Rouge-L (Longest Common Subsequence) algorithm commonly used in machine translation is used for measurement, where L refers to the longest common subsequence, input is the input text, which represents the real question in the pending question-answer pair in this method, and output is the output text, which represents the optimized answer in this method. The specific formula is as follows:

[0102]

[0103] in, represents the recall rate, represents the longest common subsequence between the real question and the optimized answer, Represents the length of the real question. Applied to this method, this formula means the extent to which the optimized answer generated by the output is the "recall" of the real input question. According to the recall rate of each pending question-answer pair, the relevance score of each pending question-answer pair can be determined.

[0104] In the process of data generation, the authenticity of the evaluation content is a key issue. In this method, factuality refers to whether the answers generated by the model are accurate, reliable, and consistent with real-world knowledge. This is crucial in many application scenarios, especially in the fields of medicine, law, and education, where false information can have serious consequences. For open fields, data diversity and a wide range of knowledge systems make it difficult to accurately and objectively evaluate data generated in an intelligent way.

[0105] In order to accurately evaluate the factuality of the pending question and answer pairs, this method introduces an evaluation method based on Wikipedia fact support to calculate the factuality score of the generated question and answer data. This method makes full use of the characteristics of Wikipedia data, such as strong knowledge structure, high accuracy, and timely updates, to ensure the objectivity and fairness of the evaluation. The specific steps are as follows:

[0106] First, the generated optimized answer is decomposed into text and divided into irreducible "atomic fact units", each of which represents a specific factual statement.

[0107] Secondly, the fact knowledge base with the pseudo question and answer set as reference is used to verify the accuracy of the atomic fact unit obtained above. That is, the system determines whether the atomic fact unit is consistent with the reference content through comparison, and the score is 0 or 1, which represents inconsistency or consistency respectively. The verification method uses the large model reasoning logic. In order to ensure objectivity and fairness, the large model for verification needs to be inconsistent with the large model for generating data. For example, the Ghatgpt4 large language model can be used when generating question and answer pairs, and the Llama3 large language model can be used for scoring and verification.

[0108] Finally, the scores of all the atomic fact units after decomposition are comprehensively considered to determine the factual score of the entire pending question and answer pair.

[0109] S110: In response to the pending question-answer pair passing the quality evaluation, determining the pending question-answer pair as an available question-answer pair.

[0110] In this step, based on the result of quality assessment of the pending question and answer pairs obtained in step S108, the pending question and answer pairs that pass the quality assessment can be retained as the finally generated available question and answer pairs, and the pending question and answer pairs that do not pass the quality assessment can be deleted.

[0111] Still taking the example of mainly evaluating the relevance and factuality of the pending question and answer pair during quality evaluation as an example, the pending question and answer pair may be determined as an available question and answer pair in response to the relevance score of the pending question and answer pair being not less than a first threshold value, and the factuality score of the pending question and answer pair being not less than a second threshold value.

[0112] When performing quality assessments on relevance and factuality, a relevance score and a factuality score will be obtained respectively. For these two aspects, thresholds that can meet the requirements can be pre-set. In this method, the threshold for passing the relevance score is set as the first threshold, and the threshold for passing the factuality score is set as the second threshold. Only when the relevance and factuality assessments are met at the same time, that is, the relevance score is not less than the first threshold, and the factuality score is not less than the second threshold, can the pending question and answer pair be considered to have passed the quality assessment.

[0113] Through the automated evaluation of relevance and factuality, the quality of the generated data is guaranteed, and the access standards for subsequent data expansion are further set. It is worth mentioning that since data generation is a continuous process, when new question-answer pairs that pass the evaluation are added to the collection, the relevance and factuality metrics and thresholds in the evaluation system will also change dynamically. This process is a process of continuous optimization until convergence.

[0114] When using the question-answer pair generation method provided in this specification to generate available question-answer pairs, after obtaining the text material, a pseudo question-answer pair is automatically generated through a large language model, and the pseudo question-answer pair is combined with the real question raised by the user to obtain an optimized answer to construct a pending question-answer pair, and finally the pending question-answer pair that passes the quality assessment is determined as an available question-answer pair. This method is based on the self-mining method of a large language model, and automatically constructs a set of high-standard knowledge sample libraries throughout the entire process, and more accurately generates question-answer data for the required tasks by adding semantically similar small sample examples during the model reasoning process. At the same time, the relevance and factual evaluation criteria of the generated data are defined, and the score calculation and filtering mechanism is used to ensure the quality consistency of open-domain question-answer pairs. This evaluation framework not only shows significant advantages in open-domain question-answer generation, but also provides important reference value for quality assessment of other data synthesis methods.

[0115] The above is the question-answer pair generation method provided in this specification. Based on the same idea, this specification also provides a corresponding question-answer pair generation device, such as Fig. 9 shown.

[0116] Fig. 9 A schematic diagram of a question-answer pair generation device provided in this specification specifically includes:

[0117] An acquisition module 200 is used to acquire text materials for generating question-answer pairs;

[0118] An input module 202, configured to input the text material into a large language model, so that the large language model outputs a pseudo question-answer pair according to the text material;

[0119] A determination module 204 is configured to determine, in response to receiving a real question input by a user, a target pseudo question-answer pair matching the real question among the pseudo question-answer pairs;

[0120] An embedding module 206, configured to embed the target pseudo question-answer pair into the prompt template of the large language model, and input the real question into the large language model to obtain an optimized answer output by the large language model;

[0121] An evaluation module 208, configured to determine the real question and the optimized answer as a pending question-answer pair, and perform a quality evaluation on the pending question-answer pair;

[0122] The generating module 210 is configured to determine the pending question-answer pair as an available question-answer pair in response to the pending question-answer pair passing the quality evaluation.

[0123] Optionally, the acquisition module 200 is specifically used to determine the professional fields covered when generating question-answer pairs; for each professional field, use a large language model to obtain keywords in the professional field; search for the keywords to obtain text materials in the professional field that match the keywords.

[0124] Optionally, the input module 202 is specifically used to input the text material into a large language model so that the large language model extracts entities contained in the text material; inputs the entities as preset answers into the large language model to obtain a predicted question output by the large language model that matches the preset answer; and determines the preset answer and the predicted question as a pseudo question-answer pair.

[0125] Optionally, the input module 202 is further used to input the text material and the pseudo question and answer pair into the large language model, so that the large language model generates an explanatory text of the pseudo question and answer pair based on the text material; and determines the explanatory text as a part of the pseudo question and answer pair.

[0126] Optionally, the determination module 204 is specifically used to encode the predicted question contained in each pseudo question and answer pair to obtain a first text vector of each predicted question, and to encode the real question to obtain a second text vector of the real question; determine the similarity between the second text vector and each first text vector; determine the target first text vector with the highest similarity to the second text vector, and determine the pseudo question and answer pair where the predicted question corresponding to the target first text vector is located as the pseudo question and answer pair that matches the real question.

[0127] Optionally, the evaluation module 208 is specifically used to determine the relevance score and the factual score of the pending question and answer pair, wherein the relevance score is used to characterize the degree of correlation between the optimized answer contained in the pending question and answer pair and the real question, and the factual score is used to characterize the accuracy of the optimized answer contained in the pending question and answer pair as an answer to the real question.

[0128] Optionally, the generation module 210 is specifically configured to determine the pending question and answer pair as an available question and answer pair in response to the relevance score of the pending question and answer pair being not less than a first threshold and the factual score of the pending question and answer pair being not less than a second threshold.

[0129] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The provided question-answer pair generation method.

[0130] This manual also provides Fig.10 The schematic structure diagram of the electronic device shown in FIG. Fig.10 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The question-answer pair generation method described above. Of course, in addition to the software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0131] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0132] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0133] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0134] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0135] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0136] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0137] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0139] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0140] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0141] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0142] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0143] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0144] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0145] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0146] The above description is only an embodiment of this specification and is not intended to limit this specification. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this application.

Claims

1. A method for generating question-answer pairs, characterized in that: include: Obtain text materials for generating question-answer pairs; Inputting the text material into a large language model, so that the large language model outputs a pseudo question-answer pair according to the text material; In response to receiving a real question input by a user, determining a target pseudo question-answer pair matching the real question among the pseudo question-answer pairs; Embedding the target pseudo question-answer pair into the prompt template of the large language model, and inputting the real question into the large language model to obtain an optimized answer output by the large language model; Determine the real question and the optimized answer as a pending question-answer pair, and perform quality assessment on the pending question-answer pair; In response to the pending question-answer pair passing the quality evaluation, the pending question-answer pair is determined as an available question-answer pair.

2. The method according to claim 1, characterized in that Get the text material used to generate question-answer pairs, including: Determine the professional fields to be covered when generating question-answer pairs; For each professional field, use a large language model to obtain keywords in that professional field; The keywords are searched to obtain text materials in the professional field that match the keywords.

3. The method according to claim 1, characterized in that Inputting the text material into a large language model, so that the large language model outputs a pseudo question-answer pair according to the text material, specifically includes: Inputting the text material into a large language model so that the large language model extracts entities contained in the text material; Inputting the entity as a preset answer into the large language model, and obtaining a predicted question output by the large language model that matches the preset answer; The preset answer and the predicted question are determined as a pseudo question-answer pair.

4. The method according to claim 3, characterized in that After determining the preset answer and the predicted question as a pseudo question-answer pair, the method further includes: Inputting the text material and the pseudo question-answer pair into the large language model, so that the large language model generates an explanatory text of the pseudo question-answer pair according to the text material; The explanatory text is determined as a part of the pseudo question-answer pair.

5. The method according to claim 3, characterized in that Determining a target pseudo question-answer pair that matches the real question among the pseudo question-answer pairs specifically includes: Encoding the predicted question contained in each pseudo question-answer pair to obtain a first text vector of each predicted question, and encoding the real question to obtain a second text vector of the real question; Determining the similarity between the second text vector and each first text vector; A target first text vector having the highest similarity to the second text vector is determined, and a pseudo question-answer pair containing a predicted question corresponding to the target first text vector is determined as a pseudo question-answer pair matching the real question.

6. The method according to claim 1, characterized in that Performing quality assessment on the pending question-answer pairs, specifically including: Determine the relevance score and the factual score of the pending question and answer pair, wherein the relevance score is used to characterize the degree of relevance between the optimized answer included in the pending question and answer pair and the real question, and the factual score is used to characterize the accuracy of the optimized answer included in the pending question and answer pair as an answer to the real question.

7. The method according to claim 6, characterized in that In response to the pending question-answer pair passing the quality evaluation, determining the pending question-answer pair as an available question-answer pair specifically includes: In response to the relevance score of the pending question-answer pair being not less than a first threshold, and the factual score of the pending question-answer pair being not less than a second threshold, the pending question-answer pair is determined as an available question-answer pair.

8. A question-answer pair generating device, characterized in that: include: An acquisition module, used to acquire text materials for generating question-answer pairs; An input module, used for inputting the text material into a large language model, so that the large language model outputs a pseudo question-answer pair according to the text material; A determination module, configured to determine, in response to receiving a real question input by a user, a target pseudo question-answer pair matching the real question among the pseudo question-answer pairs; An embedding module, used to embed the target pseudo question-answer pair into the prompt template of the large language model, and input the real question into the large language model to obtain an optimized answer output by the large language model; An evaluation module, used to determine the real question and the optimized answer as a pending question-answer pair, and perform quality evaluation on the pending question-answer pair; A generating module is used to determine the pending question-answer pair as an available question-answer pair in response to the pending question-answer pair passing the quality evaluation.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 7 is implemented.

Images