Model sample generation method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to a model sample generation method and device, equipment and a storage medium. The method comprises the steps that based on input keywords, a plurality of question and answer pairs in a natural language are sequentially generated through a first machine learning model, and a second question and answer pair in the multiple question and answer pairs is generated at least based on replies in a first question and answer pair generated before the second question and answer pair; integrating the plurality of question and answer pairs by using a second machine learning model, and generating a multi-hop question aiming at the question and answer model and a reference answer aiming at the multi-hop question; and in response to determining that the multi-hop question and the reference reply meet the first quality requirement, generating a test sample and / or a training sample of the question and answer model.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly, to a method, apparatus, device, and storage medium for generating a model sample. Background Art

[0002] Model data samples (also called model samples) are the fundamental data units used in the training and testing of machine learning models. Generally speaking, model samples can be divided into training samples and test samples. Training samples are the data used for model training, helping machine learning models adjust parameters through algorithms to capture patterns in the data. Test samples are the data used for model testing, allowing us to evaluate the performance of machine learning models on unseen data.

[0003] In recent years, machine learning models have developed rapidly, but the development of model samples has lagged behind, making it difficult to match the actual needs of machine learning models in training and testing. Summary of the Invention

[0004] In a first aspect of the present disclosure, a method for generating a model sample is provided. The method comprises: sequentially generating multiple question-answer pairs in natural language using a first machine learning model based on input keywords, wherein a second question-answer pair in the multiple question-answer pairs is generated based on at least a response in a first question-answer pair generated before the second question-answer pair; generating multi-hop questions for the question-answer model and reference responses for the multi-hop questions by integrating the multiple question-answer pairs using a second machine learning model; and generating test samples and / or training samples for the question-answer model in response to determining that the multi-hop questions and the reference responses meet a first quality requirement.

[0005] In a second aspect of the present disclosure, a device for model sample generation is provided. The device includes: a question-answer pair generation module, configured to sequentially generate multiple question-answer pairs in natural language based on input keywords using a first machine learning model, wherein the second question-answer pair in the multiple question-answer pairs is generated based on at least the answer in the first question-answer pair generated before the second question-answer pair; a reference answer generation module, configured to generate multi-hop questions for the question-answer model and reference answers for the multi-hop questions by integrating the multiple question-answer pairs using a second machine learning model; and a sample generation module, configured to generate test samples and / or training samples for the question-answer model in response to determining that the multi-hop questions and the reference answers meet the first quality requirement.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0007] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored on the computer-readable storage medium, and the computer-executable instructions can be executed by a processor to implement the method of the first aspect.

[0008] In a fifth aspect of the present disclosure, a computer program product is provided, which includes computer-executable instructions, which, when executed by a processor, implement the method according to the first aspect of the present disclosure.

[0009] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram illustrating an example environment in which embodiments of the present disclosure can be implemented;

[0012] Figure 2 A flowchart illustrating an example process of a method for generating a model sample according to some embodiments of the present disclosure;

[0013] Figure 3 A flowchart illustrating the overall process of generating multi-hop question and answer according to some embodiments of the present disclosure is shown;

[0014] Figure 4 A schematic structural block diagram of an apparatus for generating model samples according to some embodiments of the present disclosure is shown; and

[0015] Figure 5 A block diagram of an electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0016] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0017] It should be noted that the titles of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and any type of embodiment may be included under any section / subsection. Furthermore, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or in different sections / subsections.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0019] The embodiments of the present disclosure may involve user data, data acquisition and / or use, etc. These aspects shall comply with the corresponding laws, regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user is aware of and confirms them. Accordingly, when implementing the various embodiments of the present disclosure, the types, scope of use, and usage scenarios of the data or information that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with the relevant laws and regulations. The specific notification and / or authorization method may vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this respect.

[0020] If this specification and the solutions in the examples involve the processing of personal information, such processing will be done only with a legitimate basis (such as with the consent of the subject of personal information or as necessary for the performance of a contract) and only within the prescribed or agreed scope. A user's refusal to process personal information other than that required for basic functions will not affect the user's use of basic functions.

[0021] As briefly described above, current model samples struggle to match the actual requirements of machine learning model training and / or testing. For ease of discussion, the following example uses a question-answering machine learning model and test samples as an example to further illustrate this issue.

[0022] For example, in the Natural Reasoning dataset, test samples are generated by identifying reasoning content from large-scale pre-training data and extracting questions and answers from the reasoning content. However, in the Natural Reasoning dataset, test samples are generally test samples in a specific natural language (such as English) and do not support test tasks based on other natural languages (such as Chinese), thus limiting its application scenarios.

[0023] In some Chinese short question-answering datasets, test samples are generated by screening initial question-answer pairs from a knowledge base using "question-answer" as a condition, and then manually annotating these initial question-answer pairs. However, in Chinese short question-answering datasets, each test sample has only one logical transition (for example, one question and one answer), and such model samples cannot fully test question-answering models that can perform complex reasoning tasks (such as questions with multiple logical transitions).

[0024] In view of this, an embodiment of the present disclosure provides a scheme for generating model samples. According to the scheme, first, based on the input keywords, a first machine learning model is used to sequentially generate multiple question-answer pairs in natural language, wherein the second question-answer pair in the multiple question-answer pairs is generated based on at least the answer in the first question-answer pair generated before the second question-answer pair. Then, by integrating the multiple question-answer pairs using the second machine learning model, multi-hop questions for the question-answer model and reference answers for the multi-hop questions are generated. Subsequently, in response to determining that the multi-hop questions and the reference answers meet the first quality requirements, test samples and / or training samples for the question-answer model are generated.

[0025] According to the solution of the embodiment of the present disclosure, with the help of the first machine learning model, multiple question-answer pairs can be automatically generated when the user only provides keywords. Since these question-answer pairs are multiple question-answer pairs generated sequentially in natural language, they will have logical associations in natural language. On this basis, the solution of the embodiment of the present disclosure further integrates multiple question-answer pairs through the second machine learning model. Since multiple question-answer pairs have logical associations in natural language, they can be constructed as multi-hop questions with multiple logical transitions. Next, after the multi-hop question meets the first quality requirement, this multi-hop question with higher quality will be used to generate test samples and / or training samples of the question-answer model for use in testing and / or training of the question-answer model.

[0026] In this way, the solution of the embodiments of the present disclosure can generate multi-hop questions with multiple logical transitions and generate test samples and / or training samples based on the multi-hop questions. "Multi-hop questions" refer to complex questions that require multiple steps of logical reasoning to answer. Compared to "single-hop questions" with a single question and a single answer, "multi-hop questions" have more logical transitions, and the test samples and / or training samples generated based on multi-hop questions can effectively test and / or train the complex reasoning tasks of the question-answering model.

[0027] In addition, in the solution of the embodiment of the present disclosure, the number of question-answer pairs can be arbitrarily expanded, so that any style of multi-hop questions can be constructed, allowing users to customize the complexity of multi-hop questions. Since the question-answer pairs are generated by the machine learning model, the question-answer pairs can be question-answer pairs in any natural language (including but not limited to Chinese and English, etc.). Through different combinations of multiple question-answer pairs, the solution of the embodiment of the present disclosure can expand a large number of multi-hop questions in any natural language. Therefore, the solution of the embodiment of the present disclosure can enrich multi-hop questions in various natural languages, thereby improving the situation where multi-hop questions in certain natural languages are scarce.

[0028] Various example implementations of this solution will be described in detail below with reference to the accompanying drawings.

[0029] Figure 1 A schematic diagram of an example environment 100 is shown in which embodiments of the present disclosure can be implemented. Figure 1 , the example environment 100 may include an electronic device 110 .

[0030] In example environment 100, user 120 may submit a request to generate a model sample to electronic device 110. Upon receiving the request, electronic device 110 may generate a matching model sample 130 based on the specific content of the request. Model sample 130 may be, for example, a test sample and / or training sample for question-answering model 140.

[0031] In an embodiment of the present disclosure, the model sample generation request submitted by the user 120 may include keywords. After receiving the keywords, the electronic device 110 will use the first machine learning model to sequentially generate multiple question-answer pairs through multiple cycles. Then, the electronic device 110 uses the second machine learning model to integrate multiple question-answer pairs to generate multi-hop questions. Then, the electronic device 110 generates test samples and / or training samples for the question-answer model 140 based on the currently generated multi-hop questions. The test samples can be used to test the question-answer model 140 to evaluate the reasoning ability of the question-answer model 140 for multi-hop questions. The training samples can be used to train the question-answer model 140 to improve the reasoning ability of the question-answer model 140 for multi-hop questions.

[0032] In some embodiments, the electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the terminal device 110 can also support any type of interface for the user (such as "wearable" circuitry, etc.).

[0033] In some embodiments, the electronic device 110 may also be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The electronic device 110 may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0034] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the present disclosure.

[0035] Figure 2 A flow chart of an example process 200 of a method for generating a model sample according to some embodiments of the present disclosure is shown. Figure 1 The process 200 is described in detail. The process 200 may be implemented at the electronic device 110.

[0036] Reference Figure 2 At block 210, the electronic device 110 sequentially generates multiple question-answer pairs in natural language based on the input keyword using a first machine learning model. A second question-answer pair in the multiple question-answer pairs is generated based on at least a response in a first question-answer pair generated before the second question-answer pair.

[0037] As an example, a keyword can be an entity word with a clear referential meaning entered by user 120. For example, a keyword can be "City A" or "Organization B." As an example, a keyword can be entered in any form. For example, a keyword can be manually entered by user 120 using an input device such as a keyboard or a touch screen. In another example, a keyword can be entered by user 120 via voice.

[0038] As an example, a question-answer pair can be composed of a question and its corresponding answer. As an example, the question in the question-answer pair can be a question with simple logic, such as "How big is the land area of city A?" Such a question can also be called a single-hop question. Correspondingly, the answer in the question-answer pair can be, for example, "C square kilometers." In the embodiments of the present disclosure, the question-answer pair will be used to generate multi-hop questions. For example, the question-answer pair will be used as the smallest component unit for generating multi-hop questions, etc. This will be explained in detail below, so it will not be repeated here.

[0039] In some embodiments, the question-answer pairs can be in any natural language (including but not limited to Chinese and English). By combining multiple question-answer pairs in different ways, the solution of the embodiments of the present disclosure can generate multi-hop questions in any natural language. Thus, the solution of the embodiments of the present disclosure can enrich multi-hop questions in various natural languages, thereby improving the problem of the scarcity of multi-hop questions in certain natural languages.

[0040] As an example, the first machine learning model can be any machine learning model that can perform natural language processing tasks to generate question-answer pairs. For example, the first machine learning model can be a large language model (LLM). For another example, the first machine learning model can also be a model based on a recurrent neural network or other neural networks.

[0041] After receiving the keyword input by the user 120, the electronic device 110 provides the keyword to the first machine learning model. The first machine learning model will perform the task of sequentially generating multiple question-answer pairs based on the received keyword. That is, after the user provides the keyword required for the generation of the first question-answer pair, the generation of one or more subsequent question-answer pairs can be automatically repeated. As an example, the first question-answer pair can be any one of the multiple question-answer pairs, and the second question-answer pair can be any one of the multiple question-answer pairs that is located after the first question-answer pair. Additionally, the second question-answer pair can be a question-answer pair that is located after the first question-answer pair and is adjacent to the first question-answer pair among the multiple question-answer pairs.

[0042] In some embodiments, the first machine learning model may randomly generate a second question-answer pair based on the model capability based on the answer in the first question-answer pair. Specifically, the first machine learning model may generate the second question-answer pair based on the question-answer pair generation requirement and the entity words in the answer to the first question-answer pair. Alternatively, in other embodiments, the first machine learning model may generate the second question-answer pair based on the answer to the first question-answer pair under the instruction of the question-answer pair generation requirement. Specifically, the first machine learning model may generate the second question-answer pair based on the entity words in the answer to the first question-answer pair and the question-answer pair generation requirement.

[0043] As an example, the question-answer pair generation requirement can be provided by the electronic device 110 to the first machine learning model, or it can be pre-configured into the first machine learning model. For example, assuming that the first machine learning model is a large language model, the question-answer pair generation requirement can be processed by the electronic device 110 into a first prompt word information (prompt) and then provided to the first machine learning model. The first prompt word information can guide the first machine learning model to generate a question-answer pair that meets the specific question-answer pair generation requirements, and can indicate the output format of the question-answer pair. The first prompt word information can be used as a model input together with the keyword, and processed by the first machine learning model to obtain the question-answer pair output by the first machine learning model.

[0044] As an example, the entity words in the answer to the first question-answer pair may refer to nouns and / or pronouns in the answer. For example, the entity words in the answer to the first question-answer pair may refer to places, organizations, or other things in the answer. As an example, the first machine learning model may randomly select the target entity words from the answer, or may select the target entity words from the answer under the guidance of the question-answer pair generation requirements.

[0045] As an example, the question-answer pair generation requirements may be configurable. The question-answer pair generation requirements may indicate the question-answer type (e.g., knowledge-based question-answer) of the second question-answer pair (also referred to as the question-answer pair currently to be generated), the part of speech of the question-answer content in the second question-answer pair, the structural form of the second question-answer pair (e.g., the presentation style and presentation position of the question and answer, etc.), and any other appropriate requirements. For example, the question-answer pair generation requirements may also instruct the first machine learning model to prohibit exposing the answer to the second question-answer pair in a specific location (e.g., the question of the second question-answer pair) after generating the second question-answer pair, so as to prevent knowledge leakage problems similar to "the answer appears in the question" when new questions are subsequently generated. As an example, the structural form of the second question-answer pair may be represented in the form of an example to guide the first machine learning model to generate the second question-answer pair according to the paradigm of the example.

[0046] As an example, assuming that the keyword is "Event D", the first question-answer pair is the first question-answer pair among the multiple question-answer pairs, and the second question-answer pair is the second question-answer pair among the multiple question-answer pairs. Then, the first machine learning model can generate a first question-answer pair based on the keyword "Event D" or based on the keyword "Event D" and the question-answer pair generation requirement. For example, the question in the first question-answer pair can be "Which organization is the organization where Event D occurred?", and the answer in the first question-answer pair can be "Organization B". Next, the first machine learning model can generate a second question-answer pair based on the answer "Organization B" in the first question-answer pair or based on the answer "Organization B" in the first question-answer pair and the question-answer pair generation requirement. For example, the question in the second question-answer pair can be "Which city was Organization B founded?", and the answer in the second question-answer pair can be "City A".

[0047] In this way, the embodiments of the present disclosure can generate new questions related to the answer content of the first question and answer pair under the instruction of the question and answer pair generation requirements, so that the newly generated second question and answer pair can have a logical association with the first question and answer pair, which is conducive to providing better data support for the subsequent process of constructing multi-hop questions.

[0048] In some embodiments, during the process of sequentially generating multiple question-answer pairs, the electronic device 110 may perform a quality assessment on the question-answer pairs generated in this round. For example, the electronic device 110 generates a candidate question-answer pair based at least on the responses in the first question-answer pair using the first machine learning model. Then, in response to determining that the candidate question-answer pair meets the second quality requirement, the electronic device 110 determines the candidate question-answer pair as the second question-answer pair. Subsequently, in response to determining that the candidate question-answer pair does not meet the second quality requirement, the electronic device 110 regenerates a candidate question-answer pair based at least on the responses in the first question-answer pair using the first machine learning model.

[0049] For example, a candidate question-answer pair may refer to a question-answer pair currently being generated and subject to quality assessment. The second quality requirement may be configurable. The second quality requirement may indicate requirements for question-answer logic, sensitive words, accuracy, part of speech of the question-answer content, and any other appropriate content in the candidate question-answer pair.

[0050] In some embodiments, electronic device 110 may compare the question and answer content in a candidate question and answer pair with the second quality requirement using any appropriate algorithm to determine whether the candidate question and answer pair meets the second quality requirement. Alternatively, in other embodiments, electronic device 110 may use the second quality requirement and the candidate question and answer pair as model inputs and utilize a fourth machine learning model to determine whether the candidate question and answer pair meets the second quality requirement.

[0051] As an example, the fourth machine learning model can be any machine learning model capable of performing natural language processing tasks to compare candidate questions and answers with the second quality requirement. For example, the fourth machine learning model can be a large language model. For another example, the fourth machine learning model can be a model based on a recurrent neural network or other neural networks.

[0052] As an example, the second quality requirement can be provided to the fourth machine learning model by the electronic device 110, or it can be pre-configured into the fourth machine learning model. For example, assuming that the fourth machine learning model is a large language model, the second quality requirement can be processed by the electronic device 110 into fourth prompt word information and then provided to the fourth machine learning model. The fourth prompt word information can guide the fourth machine learning model to evaluate the quality of the question-answer pair. The fourth prompt word information can be used as a model input together with the question-answer pair and processed by the fourth machine learning model to obtain a quality evaluation result of the question-answer pair output by the fourth machine learning model.

[0053] In this way, the embodiments of the present disclosure can ensure that the generated second question-answer pair has high quality, thereby facilitating providing a reliable data source for the subsequent generation of multi-hop questions.

[0054] In some embodiments, the electronic device 110 may compare the question and answer in the candidate question-and-answer pair as a whole with the second quality requirement to determine whether the candidate question-and-answer pair meets the second quality requirement. Alternatively, in other embodiments, the electronic device 110 may compare the question and answer in the candidate question-and-answer pair separately with the second quality requirement to determine whether the candidate question-and-answer pair meets the second quality requirement.

[0055] For example, in some embodiments, the second quality requirement includes a second question quality requirement and a second answer quality requirement. The electronic device 110 may use a fourth machine learning model to perform quality assessment on the question and answer in the candidate question and answer pair based on the second question quality requirement and the second answer quality requirement. For example, the electronic device 110 uses the fourth machine learning model to determine whether the question in the candidate question and answer pair meets the second question quality requirement. Additionally, the electronic device 110 uses the fourth machine learning model to determine whether the answer in the candidate question and answer pair meets the second answer quality requirement. Subsequently, the electronic device 110 determines that the candidate question and answer pair meets the second quality requirement based on determining that the question in the candidate question and answer pair meets the second question quality requirement and the answer in the candidate question and answer pair meets the second answer quality requirement.

[0056] For example, the second question quality requirement may indicate requirements for question logic, sensitive words, and any other appropriate content in a candidate question-answer pair. The second answer quality requirement may indicate requirements for answer logic, answer accuracy, answer part of speech, and any other appropriate content in a candidate question-answer pair.

[0057] In some embodiments, the fourth machine learning model may determine that the question in the candidate question-answer pair meets the second question quality requirement if the question in the candidate question-answer pair partially or fully meets the second question quality requirement. Similarly, the fourth machine learning model may determine that the answer in the candidate question-answer pair meets the second answer quality requirement if the answer in the candidate question-answer pair partially or fully meets the second answer quality requirement.

[0058] In some embodiments, if it is determined that the question in the candidate question-answer pair does not meet the second question quality requirement, or if it is determined that the answer in the candidate question-answer pair does not meet the second answer quality requirement, the fourth machine learning model may determine that the candidate question-answer pair does not meet the second quality requirement. In this case, the electronic device 110 may regenerate the candidate question-answer pair using the first machine learning model.

[0059] In this way, embodiments of the present disclosure can independently evaluate the quality of questions and answers in candidate question-answer pairs, thereby facilitating a more accurate evaluation of the quality of the candidate question-answer pairs.

[0060] In some embodiments, during the generation of multiple question-answer pairs, in response to determining that the number of question-answer pairs currently generated meets a number requirement, sequential generation of the multiple question-answer pairs is terminated. In response to determining that the number of question-answer pairs currently generated does not meet the number requirement, sequential generation of the multiple question-answer pairs is continued.

[0061] As an example, the number requirement may indicate a threshold number of question-answer pairs, a range of numbers, and / or a proportional relationship to a reference number, etc. The number requirement may be configurable. Assuming that the number requirement indicates a threshold number of question-answer pairs, the number threshold may be 3, 5, or other values.

[0062] In this manner, embodiments of the present disclosure allow user 120 to customize the number of question-answer pairs within a plurality of question-answer pairs. Thus, embodiments of the present disclosure can dynamically control the complexity and diversity of subsequently generated multi-hop questions. For example, a greater number of question-answer pairs results in more logical transitions and correspondingly higher complexity for the subsequently generated multi-hop questions.

[0063] After generating multiple question-answer pairs, in box 220, the electronic device 110 integrates the multiple question-answer pairs by utilizing a second machine learning model to generate multi-hop questions for the question-answer model 140 and reference answers for the multi-hop questions.

[0064] As an example, a multi-hop problem can refer to a complex problem that requires multiple steps of logical reasoning to solve. As an example, the second machine learning model can be any machine learning model capable of performing natural language processing tasks to generate multi-hop problems. For example, the second machine learning model can be a large language model. For another example, the second machine learning model can also be a model based on a recurrent neural network or other neural networks.

[0065] In some embodiments, the multi-hop problem is a multi-hop problem in any natural language (including but not limited to Chinese and English, etc.). Therefore, the solution of the embodiment of the present disclosure can enrich the multi-hop problems in various natural languages, thereby improving the problem that multi-hop problems in certain natural languages are rare.

[0066] As an example, a multi-hop question includes multiple sub-questions corresponding to multiple question-answer pairs, each sub-question generated based on the corresponding question-answer pair. In a multi-hop question, a sub-question can be described as a "question" or in other forms besides a "question." For example, a sub-question can be implicitly included in the multi-hop question through the condition to be inferred. As an example, suppose the question in the first question-answer pair is "Which organization experienced event D?" and the answer in the first question-answer pair is "Organization B." The question in the second question-answer pair is "In which city was Organization B founded?" and the answer in the second question-answer pair is "City A." Then, the multi-hop question could be "For the organization that experienced event D, in which city was it founded?" Here, the multi-hop question actually contains two sub-questions. The first sub-question is "Which organization experienced event D?" This sub-question corresponds to the first question-answer pair and is implicitly included in the multi-hop question through the condition to be inferred. The second sub-question is "Which city is the organization founded in?" This sub-question corresponds to the second question-answer pair. In the multi-hop question, this sub-question is described in the form of a "question".

[0067] In some embodiments, the reference answer to the multi-hop question may be determined by the second machine learning model after performing a logical reasoning task on the generated multi-hop question. Alternatively, in other embodiments, the second machine learning model may identify the main sub-problem in the multiple sub-problems of the multi-hop question, and then determine the answer in the question-answer pair corresponding to the main sub-problem as the reference answer to the multi-hop question. As an example, the main sub-problem may be a sub-problem that describes the "real" question of the multi-hop question. Assuming that the multi-hop question is "For the organization where event D occurred, which city was the organization founded?", then the main sub-problem may be "Which city was the organization founded?". It should be noted that, for the convenience of discussion, the multi-hop question and the reference answer to the multi-hop question will be referred to as multi-hop question and answer below.

[0068] In some embodiments, the second machine learning model can randomly generate multi-hop questions and answers based on multiple question-answer pairs. Alternatively, in other embodiments, the second machine learning model can generate multi-hop questions and answers based on multiple question-answer pairs under the instruction of question-answer pair integration requirements.

[0069] As an example, the question-answer pair integration requirement may be provided by the electronic device 110 to the second machine learning model, or it may be pre-configured into the second machine learning model. For example, assuming that the second machine learning model is a large language model, the question-answer pair integration requirement may be processed by the electronic device 110 into second prompt word information and then provided to the second machine learning model. The second prompt word information may guide the second machine learning model to generate multi-hop questions and answers that meet specific multi-hop question and answer generation requirements, and may indicate the output format of the multi-hop questions and answers. The second prompt word information may be used as a model input together with multiple question and answer pairs, and processed by the second machine learning model to obtain the multi-hop questions and answers output by the second machine learning model.

[0070] As an example, the question-answer pair integration requirements can be configurable. The question-answer pair integration requirements can indicate the order of arrangement of multiple question-answer pairs (which can be used to indicate the description order of multiple sub-questions in a multi-hop question), the structural form of the multi-hop question and answer, the question-answer logic of the multi-hop question and answer, the description coherence of the multi-hop question and answer, the presentation form of the multi-hop question and answer, and any other appropriate requirements. For example, the structural form of the multi-hop question and answer can be represented in the form of an example to guide the second machine learning model to generate multi-hop questions and answers according to the paradigm of the example.

[0071] In block 230 , in response to determining that the multi-hop question and the reference answer meet the first quality requirement, the electronic device 110 generates a test sample and / or a training sample for the question-answering model 140 .

[0072] In some embodiments, electronic device 110 may compare the multi-hop question and answer with the first quality requirement using any appropriate algorithm to determine whether the multi-hop question and answer meets the first quality requirement. Alternatively, in other embodiments, electronic device 110 may use the first quality requirement and the multi-hop question and answer as model inputs and utilize a third machine learning model to determine whether the multi-hop question and answer meets the first quality requirement.

[0073] As an example, the third machine learning model can be any machine learning model capable of performing natural language processing tasks to compare multi-hop question answering with the first quality requirement. For example, the third machine learning model can be a large language model. For another example, the third machine learning model can be a model based on a recurrent neural network or other neural networks.

[0074] As an example, the first quality requirement can be provided by the electronic device 110 to the third machine learning model, or it can be pre-configured into the third machine learning model. For example, assuming that the third machine learning model is a large language model, the first quality requirement can be processed by the electronic device 110 into third prompt word information and then provided to the third machine learning model. The third prompt word information can guide the third machine learning model to evaluate the quality of multi-hop question and answer. The prompt word information can be used as a model input together with the multi-hop question and answer, and processed by the third machine learning model to obtain a quality evaluation result of the multi-hop question and answer output by the third machine learning model.

[0075] In this way, embodiments of the present disclosure can ensure that the generated multi-hop question and answer has higher quality.

[0076] In some embodiments, the electronic device 110 may compare the multi-hop questions and reference answers in the multi-hop question and answer process as a whole with the first quality requirement to determine whether the multi-hop question and answer process meets the first quality requirement. Alternatively, in other embodiments, the electronic device 110 may compare the multi-hop questions and reference answers in the multi-hop question and answer process with the first quality requirement separately to determine whether the multi-hop question and answer process meets the first quality requirement.

[0077] In some embodiments, the first quality requirement includes at least a first question quality requirement. Electronic device 110 utilizes a third machine learning model to determine whether the multi-hop question meets the first question quality requirement. Then, based on determining that the multi-hop question meets the first question quality requirement, electronic device 110 determines that both the multi-hop question and the reference answer meet the first quality requirement.

[0078] As an example, the first question quality requirement may indicate requirements for question logic, sensitive words, and any other appropriate content in the candidate question-answer pairs. In some embodiments, in the input of the second machine learning model, the plurality of question-answer pairs have a predetermined order, and the first question quality requirement at least indicates that the question structure of the multi-hop question matches the predetermined order.

[0079] As an example, the predetermined order may be the same as the generation order of multiple question-answer pairs, or may be different from the generation order of multiple question-answer pairs. This may be determined according to actual needs, and the embodiments of the present disclosure do not limit this. As an example, the problem structure of a multi-hop problem may refer to the description order of multiple sub-problems in the multi-hop problem and / or the logical relationship between multiple sub-problems, etc. For example, the problem structure of a multi-hop problem matches the predetermined order, for example, it may mean that: in the multi-hop problem, the description order of multiple sub-problems is consistent with the predetermined order. For example, the problem structure of a multi-hop problem matches the predetermined order may also mean that: the description order of multiple sub-problems is consistent with the predetermined order, and among the multiple sub-problems, the main sub-problem uses the answer to the sub-problem located before the main sub-problem as the condition to be inferred.

[0080] Specifically, suppose the multi-hop question is "For an organization where event D occurred, in which city was it founded?", and the main sub-question is "In which city was this organization founded?" The "organization" in the main sub-question is the condition to be inferred, and the specific organization can be determined based on the answer to the previous sub-question, "Organization where event D occurred."

[0081] In some embodiments, the third machine learning model may determine that the multi-hop problem meets the first problem quality requirement if the multi-hop problem partially meets or fully meets the first problem quality requirement.

[0082] In some embodiments, a reference answer to a multi-hop question can be determined based on the answers in the question-answer pairs corresponding to the main sub-questions in the multi-hop question. Because the quality of the answers in each question-answer pair has been evaluated before generating the multi-hop question and answer (e.g., as described above regarding the second quality requirement), the quality evaluation of the reference answers can be omitted, thereby improving processing speed.

[0083] In some embodiments, the electronic device 110 may determine whether the multi-hop question and the reference answer meet the first quality requirement by respectively evaluating the multi-hop question and the reference answer using a third machine learning model. For example, in addition to the first question quality requirement, the first quality requirement also includes a first answer quality requirement. The electronic device 110 uses the third machine learning model to determine whether the multi-hop question meets the first question quality requirement. The electronic device 110 uses the third machine learning model to determine whether the reference answer meets the first answer quality requirement. Subsequently, based on determining that the multi-hop question meets the first question quality requirement and the reference answer meets the first answer quality requirement, the electronic device 110 determines that both the multi-hop question and the reference answer meet the first quality requirement.

[0084] As an example, the first answer quality requirement may indicate requirements on answer logic, answer accuracy, answer part of speech, answer uniqueness, or any other appropriate content in the reference answer.

[0085] In some embodiments, the third machine learning model may determine that the reference answer meets the first answer quality requirement if the reference answer partially meets or fully meets the first answer quality requirement.

[0086] In some embodiments, if it is determined that the multi-hop question does not meet the first question quality requirement, or if it is determined that the reference answer does not meet the first answer quality requirement, the third machine learning model may determine that the multi-hop question and answer do not meet the first quality requirement. In some embodiments, in response to determining that the multi-hop question and the reference answer do not meet the first quality requirement, the multi-hop question and the reference answer for the multi-hop question are regenerated based on the multiple question-answer pairs and the first quality requirement. This ensures that the ultimately generated multi-hop question and answer are of high quality.

[0087] In this way, embodiments of the present disclosure can independently evaluate multi-hop questions and reference answers, thereby facilitating a more accurate evaluation of the quality of multi-hop question answering.

[0088] It should be noted that the machine learning models involved in this article (such as the first machine learning model, the second machine learning model, the third machine learning model and the fourth machine learning model) can be models of the same type or models of different types. This can be determined according to actual needs, and the embodiments of the present disclosure do not limit this.

[0089] Figure 3 A flowchart of the overall process 300 of generating multi-hop question and answer according to some embodiments of the present disclosure is shown. Figure 3 More details of generating multi-hop question and answer according to the embodiment of the present disclosure are described.

[0090] At block 310 , question-answer pairs are generated using a first machine learning model based on the seed content 301 .

[0091] As an example, seed content 301 may refer to the input of the first machine learning model when generating each question-answer pair. For example, for the first question-answer pair among multiple question-answer pairs, seed content 301 may be one or more keywords provided by user 120. For other question-answer pairs other than the first question-answer pair, seed content 301 may be one or more entity words in the answer to the previously generated question-answer pair.

[0092] At block 320, a fourth machine learning model is used to determine whether the generated question-answer pairs meet the second quality requirement. If the generated question-answer pairs meet the second quality requirement, at block 330, a determination is made as to whether the generated question-answer pairs meet the quantity requirement. If the generated question-answer pairs do not meet the second quality requirement, the process returns to block 310 and regenerates the question-answer pairs.

[0093] If the number of generated question-answer pairs meets the requirement, a second machine learning model is used to generate multi-hop questions and answers based on the generated question-answer pairs in block 340. If the number of generated question-answer pairs does not meet the requirement, the seed content 301 is updated based on the entity words in the currently generated question-answer pair, and the process returns to block 310 to generate the next question-answer pair.

[0094] At block 350, a third machine learning model is used to determine whether the generated multi-hop question and answer meets the first quality requirement. If the generated multi-hop question and answer meets the first quality requirement, at block 360, test samples and / or training samples are generated based on the multi-hop question and answer. If the generated multi-hop question and answer does not meet the first quality requirement, the process returns to block 340 and regenerates the multi-hop question and answer.

[0095] After obtaining the multi-hop question and answer, the electronic device 110 can generate a test sample for the question and answer model 140 based on the multi-hop question and answer. The test sample can be used to test the question and answer model 140 to evaluate the reasoning ability of the question and answer model 140 for multi-hop questions. Alternatively or additionally, the electronic device 110 can generate a training sample for the question and answer model 140 based on the multi-hop question and answer. The training sample can be used to train the question and answer model 140 to improve the reasoning ability of the question and answer model 140 for multi-hop questions. In some embodiments, the electronic device 110 can provide the generated test sample and / or training sample to the user 120 for the user 120 to review or perform operations such as sample annotation.

[0096] In some embodiments, in addition to generating test samples and / or training samples, electronic device 110 may also use the generated samples to test and / or train question-answering model 140. In some embodiments, electronic device 110 tests question-answering model 140 based on the test samples. Alternatively or additionally, electronic device 110 trains question-answering model 140 based on the training samples. The training samples include at least inference information output by the second machine learning model, where the inference information indicates the inference process from the multi-hop question to the reference answer.

[0097] In some embodiments, after generating the test sample, the electronic device 110 may output the test sample so that the test device can test the question-answering model 140 based on the test sample. Alternatively, in some embodiments, after generating the test sample, the electronic device 110 may also directly test the question-answering model 140 based on the test sample.

[0098] In some embodiments, after generating the training samples, the electronic device 110 may output the training samples for the training device to train the question-answering model 140 based on the training samples. Alternatively, in some embodiments, after generating the training samples, the electronic device 110 may also directly train the question-answering model 140 based on the training samples.

[0099] As an example, when training samples need to be generated, requirements related to generating reasoning information can be included in the question-answer pair integration requirements or the first quality requirements described above, so that the second machine learning model can generate reasoning information at the same time as generating multi-hop questions and answers.

[0100] In this way, the embodiments of the present disclosure can construct test samples and / or training samples with multiple logical transitions in any natural language (such as Chinese). Based on such test samples and / or training samples, the strong reasoning ability of the question-answering model 140 (such as the strong reasoning ability of Chinese) can be comprehensively and deeply tested and / or trained, thereby revealing the intrinsic correlation between the memory and reasoning of the question-answering model 140. In addition, the embodiments of the present disclosure can use a large language model to realize the production and quality evaluation of question-answer pairs and multi-hop question-answering, which is beneficial to the expansion of test samples and / or training samples on the one hand, and on the other hand, it is beneficial to improve the quality of test samples and / or training samples, thereby rapidly increasing the scale of effective test samples and / or training samples.

[0101] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 4 : A schematic structural block diagram of an apparatus 400 for generating model samples according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in the electronic device 110. Each module / component in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0102] Reference Figure 4 , the device 400 includes a question-answer pair generation module 410, a reference answer generation module 420 and a sample generation module 430. The question-answer pair generation module 410 is configured to sequentially generate a plurality of question-answer pairs in natural language based on input keywords using a first machine learning model, wherein a second question-answer pair among the plurality of question-answer pairs is generated based at least on the answer in the first question-answer pair generated before the second question-answer pair. The reference answer generation module 420 is configured to integrate a plurality of question-answer pairs by using a second machine learning model to generate multi-hop questions for the question-answer model and reference answers for the multi-hop questions. The sample generation module 430 is configured to generate test samples and / or training samples for the question-answer model in response to determining that the multi-hop questions and the reference answers meet the first quality requirement.

[0103] In some embodiments, the first quality requirement includes at least a first question quality requirement, and the sample generation module 430 is further configured to: utilize a third machine learning model to determine whether the multi-hop question meets the first question quality requirement, and based on determining that the multi-hop question meets the first question quality requirement, determine that both the multi-hop question and the reference answer meet the first quality requirement.

[0104] In some embodiments, the first quality requirement also includes a first answer quality requirement, and the sample generation module 430 is further configured to: utilize a third machine learning model to determine whether the reference answer meets the first answer quality requirement, and based on determining that the multi-hop question meets the first question quality requirement and the reference answer meets the first answer quality requirement, determine that both the multi-hop question and the reference answer meet the first quality requirement.

[0105] In some embodiments, in the input of the second machine learning model, the plurality of question-answer pairs have a predetermined order, and the first question quality requirement at least indicates that the question structure of the multi-hop question matches the predetermined order.

[0106] In some embodiments, the reference answer generation module 420 is further configured to: in response to determining that the multi-hop question and the reference answer do not meet the first quality requirement, regenerate the multi-hop question and the reference answer for the multi-hop question based on multiple question-answer pairs and the first quality requirement.

[0107] In some embodiments, the question-answer pair generation module 410 is further configured to: generate a candidate question-answer pair based at least on the answers in the first question-answer pair using the first machine learning model; determine the candidate question-answer pair as the second question-answer pair in response to determining that the candidate question-answer pair meets the second quality requirement; and regenerate the candidate question-answer pair based at least on the answers in the first question-answer pair using the first machine learning model in response to determining that the candidate question-answer pair does not meet the second quality requirement.

[0108] In some embodiments, the second quality requirement includes a second question quality requirement and a second answer quality requirement, and the question-answer pair generation module 410 is further configured to utilize a fourth machine learning model to perform the following operations: determine whether the question in the candidate question-answer pair meets the second question quality requirement, determine whether the answer in the candidate question-answer pair meets the second answer quality requirement, and determine that the candidate question-answer pair meets the second quality requirement based on determining that the question in the candidate question-answer pair meets the second question quality requirement and the answer in the candidate question-answer pair meets the second answer quality requirement.

[0109] In some embodiments, the question-answer pair generation module 410 is further configured to: generate a second question-answer pair based on the entity words in the answer to the first question-answer pair and the question-answer pair generation requirements using the first machine learning model.

[0110] In some embodiments, the question-answer pair generation module 410 is further configured to: during the generation process of multiple question-answer pairs, in response to determining that the number of question-answer pairs currently generated meets the number requirement, end the sequential generation of multiple question-answer pairs.

[0111] In some embodiments, the apparatus 400 further includes a processing module. The processing module is configured to: test the question-answering model based on the test sample, and / or train the question-answering model based on the training sample, wherein the training sample at least includes reasoning information output by the second machine learning model, and the reasoning information indicates a reasoning process from the multi-hop question to the reference answer.

[0112] Figure 5 1 is a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. The electronic device 500 may be used to implement, for example, Figure 1 The electronic device 110 or Figure 4 The device 400 is shown. It should be understood that Figure 5 The illustrated electronic device 500 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0113] Reference Figure 5 , electronic device 500 is in the form of a general electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processor 510 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 520. In a multi-processor system, multiple processors execute computer-executable instructions in parallel to increase the parallel processing capabilities of electronic device 500.

[0114] The electronic device 500 typically includes a plurality of computer storage media. Such media can be any available media accessible to the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data and can be accessed within the electronic device 500.

[0115] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5As shown in FIG, a magnetic disk drive for reading from or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. Memory 520 may include a computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0116] The communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0117] Input device 550 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 560 may be one or more output devices, such as a display, a speaker, or a printer. Electronic device 500 may also communicate with one or more external devices (not shown) via communication unit 540 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with electronic device 500, or with any device that allows electronic device 500 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0118] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0119] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0120] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0121] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0122] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0123] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is intended to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for generating a model sample, comprising: Based on the input keywords, using a first machine learning model, sequentially generate a plurality of question-answer pairs in natural language, wherein a second question-answer pair in the plurality of question-answer pairs is generated based on at least a response in a first question-answer pair generated before the second question-answer pair; Generating a multi-hop question for a question-answering model and a reference answer for the multi-hop question by integrating the plurality of question-answer pairs using a second machine learning model; as well as In response to determining that the multi-hop question and the reference answer meet a first quality requirement, a test sample and / or a training sample for the question-answering model is generated.

2. The method according to claim 1 , wherein the first quality requirement comprises at least a first question quality requirement, and determining whether the multi-hop question and the reference answer meet the first quality requirement comprises at least: Using the third machine learning model, determining whether the multi-hop problem meets the first problem quality requirement, and Based on determining that the multi-hop question meets the first question quality requirement, determining that both the multi-hop question and the reference answer meet the first quality requirement.

3. The method of claim 2 , wherein the first quality requirement further comprises a first answer quality requirement, and determining whether the multi-hop question and the reference answer meet the first quality requirement further comprises: Using the third machine learning model, determining whether the reference answer meets the first answer quality requirement, and Based on determining that the multi-hop question meets the first question quality requirement and the reference answer meets the first answer quality requirement, it is determined that both the multi-hop question and the reference answer meet the first quality requirement.

4. The method of claim 2, wherein in the input of the second machine learning model, the plurality of question-answer pairs have a predetermined order, and the first question quality requirement at least indicates that the question structure of the multi-hop question matches the predetermined order.

5. The method according to claim 1, further comprising: In response to determining that the multi-hop question and the reference answer do not meet the first quality requirement, the multi-hop question and the reference answer to the multi-hop question are regenerated based on the plurality of question-answer pairs and the first quality requirement.

6. The method of claim 1 , wherein generating the plurality of question-answer pairs using a first machine learning model comprises: generating candidate question-answer pairs using the first machine learning model based at least on the responses in the first question-answer pairs; In response to determining that the candidate question-answer pair meets a second quality requirement, determining the candidate question-answer pair as the second question-answer pair; as well as In response to determining that the candidate question-answer pair does not meet the second quality requirement, the candidate question-answer pair is regenerated using the first machine learning model based at least on the responses in the first question-answer pair.

7. The method according to claim 6, wherein the second quality requirement includes a second question quality requirement and a second answer quality requirement, and determining whether the candidate question-answer pair meets the second quality requirement comprises: Using the fourth machine learning model, Determine whether the question in the candidate question-answer pair meets the second question quality requirement, determining whether the answer in the candidate question-answer pair meets the second answer quality requirement, and Based on determining that the question in the candidate question-answer pair meets the second question quality requirement, and the answer in the candidate question-answer pair meets the second answer quality requirement, it is determined that the candidate question-answer pair meets the second quality requirement.

8. The method of claim 1 , wherein generating the plurality of question-answer pairs using a first machine learning model comprises: The second question-answer pair is generated by utilizing the first machine learning model based on entity words in the response to the first question-answer pair and the question-answer pair generation requirement.

9. The method according to claim 1, wherein sequentially generating a plurality of question-answer pairs in natural language using a first machine learning model comprises: During the generation of the plurality of question-answer pairs, in response to determining that the number of question-answer pairs currently generated meets the number requirement, the sequential generation of the plurality of question-answer pairs is terminated.

10. The method according to claim 1, further comprising: Based on the test sample, test the question-answering model, and / or Based on the training samples, the question-answering model is trained, wherein the training samples at least include reasoning information output by the second machine learning model, and the reasoning information indicates the reasoning process from the multi-hop question to the reference answer.

11. A device for generating a model sample, comprising: a question-answer pair generation module configured to sequentially generate a plurality of question-answer pairs in natural language using a first machine learning model based on input keywords, wherein a second question-answer pair in the plurality of question-answer pairs is generated based on at least a response in a first question-answer pair generated before the second question-answer pair; a reference answer generation module, configured to generate a multi-hop question for a question-answering model and a reference answer for the multi-hop question by integrating the plurality of question-answer pairs using a second machine learning model; as well as The sample generation module is configured to generate a test sample and / or a training sample for the question-answering model in response to determining that the multi-hop question and the reference answer meet a first quality requirement.

12. An electronic device comprising: at least one processor; as well as At least one memory is coupled to the at least one processor and stores instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 10 when executed by the at least one processor.

13. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions can be executed by a processor to implement the method according to any one of claims 1 to 10.

14. A computer program product comprising computer executable instructions, wherein the computer executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 10.

Images