Method, device, equipment, medium and product for generating training data set

Abstract

Embodiments of the present application provide a method and device for generating a training data set, an apparatus, a medium and a product, relating to the technical field of natural language processing. The method comprises: obtaining a domain knowledge set of a domain to which a first language model is applied; generating a plurality of question and answer pairs according to the domain knowledge set and a preset template; for each question and answer pair, performing provenance verification on the first answer text according to the reasoning text and the first answer text to obtain a similarity between the first answer text and first domain knowledge in the reasoning text; if the similarity is greater than a first preset threshold, taking the first answer text as a target answer text; if the similarity is less than or equal to the first preset threshold, modifying the first answer text according to the first domain knowledge to obtain the target answer text; and adding the first question text and the target answer text corresponding to the first question text to the training data set. The embodiments of the present application can improve the data quality of the training data set.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a training data set generation method and device, equipment, medium and product. BACKGROUND

[0002] In recent years, large model technology has shown an exponential growth trend, and pre-training models with more than 10 billion parameters have continuously broken the performance boundaries of natural language processing and multi-modal understanding. However, the current mainstream pre-training-fine-tuning technology still faces core challenges. Although constructing supervised instruction fine-tuning data sets can effectively activate the knowledge obtained by large models in the pre-training stage, the pre-training-fine-tuning technology has not yet formed a standardized training data set construction method. Currently, the existing training data set construction method usually expands and generates data through a seed instruction set, but this method generates training data through a template-based mechanical combination, which can easily lead to semantic repetition or logical bias in the generated training data, and there is a problem of low data quality of the generated training data set. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a training data set generation method, device, equipment, medium and product, which aims to improve the data quality of the training data set.

[0004] To achieve the above-mentioned purpose, the first aspect of the embodiments of the present application provides a training data set generation method, which comprises:

[0005] obtaining a domain knowledge set related to the application field of a first language model;

[0006] According to the domain knowledge set and a preset template, a plurality of question and answer pairs are generated, each question and answer pair comprising a first question text, a first answer text corresponding to each first question text, and a reasoning text, wherein the reasoning text comprises first domain knowledge obtained from the domain knowledge set in the process of reasoning the first question text to obtain the corresponding first answer text;

[0007] For each question and answer pair, according to the reasoning text and the first answer text in the question and answer pair, the first answer text is traced back and verified to obtain the similarity between the first answer text and the first domain knowledge;

[0008] If the similarity is greater than a first preset threshold, the first answer text is taken as the target answer text corresponding to the first question text;

[0009] If the similarity is less than or equal to the first preset threshold, the first answer text is modified according to the first domain knowledge to obtain the target answer text corresponding to the first question text;

[0010] add the first question text and the target answer text corresponding to the first question text to a training data set, the training data set being used to train the first language model.

[0011] In some embodiments, the generating a plurality of question and answer pairs according to the set of domain knowledge and a preset template comprises:

[0012] generating a plurality of initial question and answer pairs according to the set of domain knowledge and the preset template by a pre-trained second language model, each of the initial question and answer pairs comprising the first question text and a second answer text corresponding to the first question text, the second language model being trained by a plurality of historical training data, each of the historical training data comprising a historical question text and a historical answer text corresponding to the historical question text;

[0013] for each of the initial question and answer pairs, modifying target information in the first question text to obtain a second question text;

[0014] generating, by the second language model, a verification answer text corresponding to the second question text according to the set of domain knowledge and the second question text;

[0015] evaluating the second answer text based on the verification answer text to obtain a confidence of the second answer text;

[0016] if the confidence is greater than a second preset threshold, taking the second answer text as the first answer text to obtain the question and answer pair;

[0017] if the confidence is less than or equal to the second preset threshold, adjusting the second language model, and jumping to the step of generating a plurality of initial question and answer pairs according to the set of domain knowledge and a preset template by a pre-trained second language model until the confidence is greater than a second preset threshold.

[0018] In some embodiments, the obtaining a set of domain knowledge related to a domain applied by the first language model comprises:

[0019] obtaining a plurality of the domain knowledge related to the domain applied by the first language model;

[0020] classifying the plurality of the domain knowledge based on a plurality of preset types to obtain a plurality of domain knowledge subsets, the plurality of preset types comprising a judgment standard type, a concept type and a warning information type, each of the domain knowledge subsets corresponding to one of the preset types;

[0021] obtaining the set of domain knowledge according to the plurality of the domain knowledge subsets.

[0022] In some embodiments, the number of the preset templates is a plurality, each of the preset templates corresponds to an answer text generation rule, and the plurality of domain knowledge subsets include a first domain knowledge subset corresponding to the judgment standard type, a second domain knowledge subset corresponding to the concept type, and a third domain knowledge subset corresponding to the warning information type;

[0023] The generating a plurality of question and answer pairs according to the domain knowledge set and the preset templates comprises:

[0024] extracting second domain knowledge from the second domain knowledge subset;

[0025] filling the second domain knowledge into a plurality of the preset templates to obtain a plurality of the first question texts;

[0026] For each of the first question texts, extracting first domain knowledge corresponding to the first question text from at least one of the first domain knowledge subset and the third domain knowledge subset according to the first question text;

[0027] generating the reasoning text and a first answer text corresponding to the first question text according to the first question text, the first domain knowledge, and an answer text generation rule corresponding to a preset template adopted by the first question text.

[0028] In some embodiments, the generating the reasoning text and the first answer text corresponding to the first question text according to the first question text, the first domain knowledge, and the answer text generation rule corresponding to the preset template adopted by the first question text comprises:

[0029] splitting the first question text into a plurality of question subtexts;

[0030] generating reasoning subtexts and answer subtexts corresponding to each of the question subtexts according to each of the question subtexts, the first domain knowledge, and an answer text generation rule corresponding to a preset template adopted by the first question text;

[0031] generating the first answer text according to the answer subtexts corresponding to each of the question subtexts, and generating the reasoning text according to the reasoning subtexts.

[0032] In some embodiments, the plurality of question subtexts have a hierarchical order;

[0033] The generating the reasoning subtexts and the answer subtexts corresponding to each of the question subtexts according to each of the question subtexts, the first domain knowledge, and the answer text generation rule corresponding to the preset template adopted by the first question text comprises:

[0034] According to a hierarchical order among the plurality of question subtexts, the answer subtext corresponding to each question subtext and the reasoning subtext are generated according to each question subtext, the first domain knowledge, and an answer text generation rule corresponding to a preset template adopted by the first question text in sequence.

[0035] To achieve the above object, a second aspect of the embodiment of the present application provides a device for generating a training data set, the device comprising:

[0036] an acquisition module configured to acquire a domain knowledge set related to a domain to which a first language model is applied;

[0037] a generation module configured to generate a plurality of question and answer pairs according to the domain knowledge set and a preset template, each question and answer pair comprising a first question text, a first answer text corresponding to each first question text, and a reasoning text, the reasoning text comprising first domain knowledge obtained from the domain knowledge set in a process of reasoning the first question text to obtain the corresponding first answer text;

[0038] a verification module configured to, for each question and answer pair, perform provenance verification on the first answer text according to the reasoning text and the first answer text in the question and answer pair to obtain a similarity between the first answer text and the first domain knowledge;

[0039] a determination module configured to, if the similarity is greater than a first preset threshold, take the first answer text as a target answer text corresponding to the first question text;

[0040] a correction module configured to, if the similarity is less than or equal to the first preset threshold, correct the first answer text according to the first domain knowledge to obtain a target answer text corresponding to the first question text;

[0041] an addition module configured to add the first question text and the target answer text corresponding to the first question text to a training data set, the training data set being used to train the first language model.

[0042] To achieve the above object, a third aspect of the embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the method for generating a training data set according to the first aspect when executing the computer program.

[0043] To achieve the above object, a fourth aspect of the embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program implementing the method for generating a training data set according to the first aspect when executed by a processor.

[0044] To achieve the above object, a fifth aspect of the embodiments of the present application provides a computer program product, instructions in the computer program product are executed by a processor of an electronic device as described in the third aspect, so that the electronic device executes the generation method of the training data set as described in the first aspect.

[0045] The generation method of the training data set, the device, the equipment, the medium and the product provided by the present application, by obtaining the domain knowledge set related to the application field of the first language model, and then generating a plurality of question and answer pairs according to the domain knowledge set and the preset template, for each question and answer pair, the first answer text can be traced back to check according to the reasoning text and the first answer text in the question and answer pair, and the similarity between the first answer text and the first domain knowledge is obtained, if the similarity is greater than the first preset threshold, the first answer text is taken as the target answer text corresponding to the first question text, if the similarity is less than or equal to the first preset threshold, the first answer text is modified according to the first domain knowledge, and the target answer text corresponding to the first question text is obtained, so that the reliability of the generated first question text and the answer text corresponding to the first question text can be improved. Then, the first question text and the target answer text corresponding to the first question text can be added to the training data set for training the first language model, so that the data quality of the training data set can be improved, and the reasoning generalization ability of the first language model trained based on the training data set in a specific field can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a flowchart of the generation method of the training data set provided by the embodiments of the present application;

[0047] Figure 2 is another flowchart of the generation method of the training data set provided by the embodiments of the present application;

[0048] Figure 3 is a structure schematic diagram of the generation device of the training data set provided by the embodiments of the present application;

[0049] Figure 4 is a hardware structure schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0050] In order to make the objects, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0051] It should be noted that although the functional modules are divided in the device schematic diagram, the logical order is shown in the flowchart, but in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to limit the present application.

[0053] To solve the problems existing in the prior art, the embodiments of the present application provide a training data set generation method, device, equipment, medium and product, aiming to improve the data quality of the training data set.

[0054] The training data set generation method, device, equipment, medium and product provided by the embodiments of the present application are specifically explained by the following embodiments. First, the training data set generation method in the embodiments of the present application is described.

[0055] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system.

[0056] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0057] The method for generating a training data set provided in the embodiments of the present application relates to the technical field of natural language processing. The method for generating a training data set provided in the embodiments of the present application can be applied in a terminal, can also be applied in a server end, and can be software running in the terminal or the server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; the software can be an application for implementing the method for generating a training data set, etc., but is not limited to the above forms.

[0058] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0059] Please refer to Figure 1 , Figure 1 is a flowchart of the method for generating a training data set provided in the embodiments of the present application. The method for generating a training data set provided in the embodiments of the present application can be applied in an electronic device, for example, a server. Figure 1 The method in the embodiments of the present application can include but is not limited to including steps 101 to 106.

[0060] Step 101, obtaining a domain knowledge set related to a domain to which a first language model is applied.

[0061] The first language model can be applied to the medical field, the legal field, the financial field, or other fields. For example, when the first language model is applied to the medical field, a domain knowledge set including domain knowledge such as professional papers, medical diagnosis information, and popular science information related to the medical field can be obtained. Alternatively, when the first language model is applied to the legal field, a domain knowledge set including domain knowledge such as legal provisions, professional papers, and popular science information related to the legal field can be obtained. Alternatively, when the first language model is applied to the financial field, a domain knowledge set including domain knowledge such as financial product information, regulations, and basic concepts related to the financial field can be obtained.

[0062] At step 102, a plurality of question and answer pairs are generated according to the domain knowledge set and a preset template. Each question and answer pair includes a first question text, a first answer text corresponding to each first question text, and reasoning text including first domain knowledge obtained from the domain knowledge set in the process of reasoning the corresponding first answer text from the first question text.

[0063] A preset template can be obtained in advance. The preset template can include at least one of a memory type template, an application type template, and a reasoning type template. In this way, according to the domain knowledge included in the domain knowledge set and the preset template, a plurality of question and answer pairs of different types can be generated. Each question and answer pair includes a first question text, a first answer text corresponding to each first question text, and reasoning text describing the reasoning process for reasoning the corresponding first answer text from the first question text. Moreover, the reasoning text includes domain knowledge, i.e., first domain knowledge, relied on to reason the corresponding first answer text from the first question text. How to generate a plurality of question and answer pairs according to the domain knowledge set and the preset template can be achieved in various ways. For example, when the preset template includes a question template, a reasoning template, and an answer template, the domain knowledge in the domain knowledge set is filled into the preset template in a template filling manner, thereby generating a plurality of question and answer pairs. Alternatively, a knowledge graph can be constructed based on the domain knowledge set, and then entities and relationships are extracted from the knowledge graph through a graph database, and a plurality of question and answer pairs are generated based on the extracted entities, relationships, and preset templates. Alternatively, when the preset template is a question template, a pre-trained large language model can be used to generate a plurality of first question texts according to the domain knowledge set and the preset template, and then logical reasoning is performed on each first question text based on the domain knowledge set to obtain a first answer text corresponding to each first question text and reasoning text.

[0064] At step 103, for each question and answer pair, the first answer text is traced back and verified according to the reasoning text and the first answer text in the question and answer pair, and a similarity between the first answer text and the first domain knowledge is obtained.

[0065] After generating the plurality of question and answer pairs, the accuracy of the first answer text in each question and answer pair can be verified. Specifically, for each question and answer pair, the first answer text can be verified according to the reasoning text and the first answer text in the question and answer pair, so as to determine the similarity between the first answer text and the first domain knowledge. That is, the first domain knowledge in the reasoning text and the first answer text are respectively converted into vectors, and then the similarity between the first answer text and the first domain knowledge is determined. The similarity can be a cosine similarity or a Jaccard similarity.

[0066] In step 104, if the similarity is greater than a first preset threshold, the first answer text is taken as the target answer text corresponding to the first question text.

[0067] The first preset threshold set according to the actual situation can be obtained, and the relative relationship between the similarity and the first preset threshold is determined. If the similarity is greater than the first preset threshold, it can be determined that the first answer text is more consistent with the first domain knowledge, and the reliability of the first answer text is higher, so the first answer text can be taken as the target answer text corresponding to the first question text.

[0068] In step 105, if the similarity is less than or equal to the first preset threshold, the first answer text is modified according to the first domain knowledge to obtain the target answer text corresponding to the first question text.

[0069] If the similarity is less than or equal to the first preset threshold, it can be determined that the first answer text does not conform to the first domain knowledge, and the reliability of the first answer text is lower. At this time, the first answer text needs to be modified according to the first domain knowledge to obtain the target answer text corresponding to the first question text. For example, if the first answer text is "all non-ST segment elevation myocardial infarction patients should immediately undergo percutaneous coronary intervention treatment", and the first domain knowledge is "high-risk patients are recommended to undergo interventional treatment within 24 hours", the similarity between the first answer text and the first domain knowledge is less than or equal to the first preset threshold. Then, the first answer text can be modified according to the first domain knowledge to obtain the target answer text such as "for high-risk non-ST segment elevation myocardial infarction patients, it is recommended to undergo percutaneous coronary intervention treatment within 24 hours".

[0070] In step 106, the first question text and the target answer text corresponding to the first question text are added to a training data set, and the training data set is used to train the first language model.

[0071] After determining the target answer text corresponding to the first question text in each question and answer pair, the first question text and the target answer text corresponding to the first question text can be added to the training data set, thereby obtaining a training data set with high data quality, so as to subsequently train the first language model based on the training data set.

[0072] The steps 101 to 106 shown in the embodiments of the present application obtain a domain knowledge set related to the domain to which the first language model is applied, and then generate a plurality of question and answer pairs according to the domain knowledge set and a preset template. Subsequently, for each question and answer pair, the first answer text in the question and answer pair can be traced back to check the first answer text, to obtain the similarity between the first answer text and the first domain knowledge. If the similarity is greater than a first preset threshold, the first answer text is taken as the target answer text corresponding to the first question text. If the similarity is less than or equal to the first preset threshold, the first answer text is modified according to the first domain knowledge to obtain the target answer text corresponding to the first question text. In this way, the reliability of the generated first question text and the answer text corresponding to the first question text can be improved. Then, the first question text and the target answer text corresponding to the first question text can be added to the training data set for training the first language model. In this way, the data quality of the training data set can be improved, and the reasoning generalization ability of the first language model trained based on the training data set in a specific domain can be improved.

[0073] In some embodiments, the step 101 of obtaining a domain knowledge set related to the domain to which the first language model is applied comprises:

[0074] Obtaining a plurality of domain knowledge related to the domain to which the first language model is applied;

[0075] Classifying the plurality of domain knowledge based on a plurality of preset types to obtain a plurality of domain knowledge subsets, the plurality of preset types including a judgment standard type, a concept type, and a warning information type, and each domain knowledge subset corresponds to a preset type;

[0076] Obtaining the domain knowledge set according to the plurality of domain knowledge subsets.

[0077] The domain knowledge set includes a plurality of domain knowledge subsets. Specifically, a plurality of domain knowledge related to the domain to which the first language model is applied can be obtained, such as professional papers, medical diagnosis information, popular science information related to the medical field, or legal provisions, professional papers, popular science information related to the legal field. Or, the plurality of domain knowledge can also be concept information, theory information and practical application information obtained after information extraction on complex content (for example, professional papers related to the medical field, medical diagnosis information, popular science information, etc.). Subsequently, based on a plurality of preset types, the plurality of domain knowledge can be classified to obtain a plurality of domain knowledge subsets. The plurality of preset types includes a judgment standard type, a concept type and a warning information type, and each domain knowledge subset corresponds to a preset type. For example, continuing with the medical field, if domain knowledge such as "myocardial infarction", "ST segment elevation" and "troponin" is obtained, it can be divided into a domain knowledge subset corresponding to the concept type. Or, if domain knowledge such as "{ST segment elevation myocardial infarction: ["chest pain> 30min", "adjacent lead ST elevation≥2mm]}" is obtained, it can be divided into a domain knowledge subset corresponding to the judgment standard type. Or, if domain knowledge such as "ST elevation ≠ must immediately thrombolysis" and "non-typical symptom identification" is obtained, it can be divided into a domain knowledge subset corresponding to the warning information type. In this way, a plurality of domain knowledge subsets can be obtained. Through the above steps, the domain knowledge set can be obtained, which facilitates the generation of the training data set based on the domain knowledge set.

[0078] In addition, after obtaining the domain knowledge set, based on the domain knowledge in the domain knowledge set and the relationship between the domain knowledge, a knowledge graph can be constructed, and a plurality of question and answer pairs can be generated according to the constructed knowledge graph and the preset template, thereby improving the generation efficiency of the question and answer pairs.

[0079] In some embodiments, the number of preset templates is a plurality, each of the preset templates corresponds to an answer text generation rule, and the plurality of domain knowledge subsets include a first domain knowledge subset corresponding to the judgment standard type, a second domain knowledge subset corresponding to the concept type, and a third domain knowledge subset corresponding to the warning information type.

[0080] The step 102 of generating a plurality of question and answer pairs according to the domain knowledge set and the preset template includes:

[0081] extracting second domain knowledge from the second domain knowledge subset;

[0082] filling the second domain knowledge into a plurality of the preset templates to obtain a plurality of the first question texts;

[0083] For each of the first question texts, according to the first question text, first domain knowledge corresponding to the first question text is extracted from at least one of the first domain knowledge subset and the third domain knowledge subset;

[0084] According to the first question text, the first domain knowledge, and an answer text generation rule corresponding to a preset template adopted by the first question text, the reasoning text and the first answer text corresponding to the first question text are generated.

[0085] According to the domain knowledge set and the preset templates, a plurality of question and answer pairs can be generated. The preset templates include a memory type template, an application type template, and an inference type template, and the plurality of domain knowledge subsets include a first domain knowledge subset corresponding to a judgment standard type, a second domain knowledge subset corresponding to a concept type, and a third domain knowledge subset corresponding to a warning information type. In some embodiments, the preset templates are question templates. Different preset templates adopted by the first question texts result in different answer text generation rules adopted when generating the first answer texts corresponding to the first question texts.

[0086] In this way, second domain knowledge can be extracted from the second domain knowledge subset and filled into the plurality of preset templates to obtain a plurality of first question texts. Taking the medical field as an example, based on the memory type template and the second domain knowledge extracted from the second domain knowledge subset, a first question text such as “What are the two necessary conditions for diagnosing non-ST segment elevation myocardial infarction?” can be generated. Based on the application type template and the second domain knowledge extracted from the second domain knowledge subset, a first question text such as “Can non-ST segment elevation myocardial infarction be diagnosed in a patient with troponin T value 0.15 ng / mL (normal <0.014 ng / mL) but no ischemic changes in electrocardiogram?” can be generated. Based on the inference type template and the second domain knowledge extracted from the second domain knowledge subset, a first question text such as “How should a patient with persistent chest pain and cold sweat for 2 hours be diagnosed, who is 58 years old and has electrocardiogram showing ST segment elevation in leads II, III, and aVF?” can be generated.

[0087] Subsequently, according to the preset template adopted by the first question text, an answer text generation rule corresponding to the preset template adopted by the first question text can be determined. Specifically, for a first question text adopting a memory type template, the generated first answer text should reproduce the first domain knowledge more completely and accurately. For a first question text adopting an application type template, the first answer text should be reasonably generated based on the specific application scenario of the first question text. For a first question text adopting an inference type template, the context dependency relationship of the first question text should be determined, and the first question text should be inferred step by step.

[0088] Subsequently, for each first question text, according to the first question text, first domain knowledge related to the first question text can be extracted from at least one of the first domain knowledge subset and the third domain knowledge subset. In this way, the first question text can be reasoned based on the first domain knowledge and the answer text generation rule corresponding to the preset template adopted by the first question text, so as to generate the reasoning text and the first answer text corresponding to the first question text.

[0089] In one example, if the first question text is "Can non-ST segment elevation myocardial infarction be diagnosed if the patient's troponin T value is 0.15 ng / mL (normal <0.014 ng / mL), but the electrocardiogram shows no ischemic changes?", at this time, first domain knowledge such as "2023 ACC guidelines point out that the diagnosis of non-ST segment elevation myocardial infarction must meet: ① Troponin is elevated more than 99% of the upper limit of normal ② There is evidence of ischemia" can be extracted from the first domain knowledge subset. And it can be determined that the preset template adopted by the first question text is an application type template, at this time, the answer text generation rule corresponding to the application type template should be adopted. Then, according to the first domain knowledge and the answer text generation rule corresponding to the preset template adopted by the first question text, the reasoning text and the first answer text corresponding to the first question text can be generated, and the first answer text is "Must meet the biomarker elevation and ischemic evidence, so it does not meet the diagnostic criteria".

[0090] In another example, if the first question text is "How should the patient be diagnosed if he is a 58-year-old male with persistent chest pain and cold sweat for 2 hours, and the electrocardiogram shows ST segment elevation in leads II, III and aVF?", according to the first question text, first domain knowledge such as "chest pain duration > 20 minutes meets the diagnosis standard of ST segment elevation myocardial infarction" and "ST elevation in inferior leads suggests right coronary artery obstruction" can be extracted from the first domain knowledge subset. And it can be determined that the preset template adopted by the first question text is a reasoning type template, at this time, the answer text generation rule corresponding to the reasoning type template should be adopted. Then, according to the first domain knowledge and the answer text generation rule corresponding to the preset template adopted by the first question text, the reasoning text and the first answer text corresponding to the first question text can be generated.

[0091] In some embodiments, the first answer text corresponding to the first question text is generated according to the first question text, the first domain knowledge, and the answer text generation rule corresponding to the preset template adopted by the first question text, including:

[0092] Split the first question text into multiple question subtexts;

[0093] generate, according to each question subtext, the first domain knowledge, and the answer text generation rule corresponding to the preset template adopted by the first question text, an inference subtext and an answer subtext corresponding to each question subtext;

[0094] generate, according to each question subtext, the first domain knowledge, and the answer text generation rule corresponding to the preset template adopted by the first question text, an inference subtext and an answer subtext corresponding to each question subtext;

[0095] Specifically, for a complex first question text, in order to improve the accuracy of the reasoning process and the accuracy of the generated first answer text, the first question text can be split into multiple question subtexts. For example, for the first question text of “The patient is a 58-year-old male with persistent chest pain and cold sweat for 2 hours. The electrocardiogram shows ST segment elevation in leads II, III, and aVF. How should it be diagnosed?”, the first question text can be split into question subtext 1 and question subtext 2, where question subtext 1 is “Which symptoms suggest acute coronary syndrome?” and question subtext 2 is “Which part of myocardial infarction does ST elevation in leads II, III, and aVF correspond to?”.

[0096] Subsequently, according to each question subtext, the first domain knowledge, and the answer text generation rule corresponding to the preset template adopted by the first question text, an inference subtext and an answer subtext corresponding to each question subtext can be generated. The inference subtext is used to describe the reasoning process of deriving the answer subtext from the question subtext. Then, combining each question subtext corresponding to the answer subtext, the first answer text can be generated, and combining each inference subtext, the reasoning text can be generated. In this way, in the case of generating the first question text, the reasoning text and the first answer text corresponding to the first question text can be further generated according to the first question text, the first domain knowledge, and the answer text generation rule corresponding to the preset template adopted by the first question text.

[0097] Continuing with the example of the foregoing, for question subtext 1, inference subtext 1 can be obtained, i.e., “Chest pain duration > 20 minutes meets the diagnostic criteria for ST segment elevation myocardial infarction...”. For question subtext 2, inference subtext 2 can be obtained, i.e., “ST elevation in the lower wall leads suggests right coronary artery obstruction...”. At this time, the answer subtext corresponding to question subtext 1 is “Chest pain duration > 20 minutes...”, and the answer subtext corresponding to question subtext 2 is “ST segment elevation myocardial infarction in the lower wall...”, then the reasoning text can be obtained as [“Chest pain duration > 20 minutes meets the diagnostic criteria for ST segment elevation myocardial infarction...”, “ST elevation in the lower wall leads suggests right coronary artery obstruction...”], and the first answer text is “Initial diagnosis of acute ST segment elevation myocardial infarction in the lower wall”.

[0098] In some embodiments, the plurality of question subtexts have a hierarchical order;

[0099] The answer text generation rule corresponding to the preset template adopted according to each of the question subtexts, the first domain knowledge, and the first question text generates an inference subtext and an answer subtext corresponding to each of the question subtexts, including:

[0100] According to the hierarchical order between the plurality of question subtexts, the answer text generation rule corresponding to the preset template adopted according to each of the question subtexts, the first domain knowledge, and the first question text generates the inference subtext and the answer subtext corresponding to each of the question subtexts in turn.

[0101] After splitting the first question text, a plurality of question subtexts with a hierarchical order can be obtained. The hierarchical order between the plurality of question subtexts is determined according to the logical relationship of the plurality of question subtexts. Generally, the hierarchical order between the plurality of question subtexts can be determined according to the logical progressive relationship, that is, the question subtext with a later hierarchical order should be more deepened than the question subtext with an earlier hierarchical order. Based on this, according to the hierarchical order between the plurality of question subtexts, the answer text generation rule corresponding to the preset template adopted according to each of the question subtexts, the first domain knowledge, and the first question text can be used to gradually infer the plurality of question subtexts to obtain the inference subtext and the answer subtext corresponding to each of the question subtexts. In this way, the complexity of the first question text can be reduced, and the reliability of the inference text and the first answer text generated subsequently can be improved.

[0102] In some embodiments, the step 102 of generating a plurality of question and answer pairs according to the domain knowledge set and the preset template includes:

[0103] According to the domain knowledge set and the preset template, a plurality of initial question and answer pairs are generated by a pre-trained second language model, each of the initial question and answer pairs including the first question text and a second answer text corresponding to the first question text, the second language model being trained by a plurality of historical training data, each of the historical training data including a historical question text and a historical answer text corresponding to the historical question text;

[0104] For each of the initial question and answer pairs, the target information in the first question text is modified to obtain a second question text;

[0105] According to the domain knowledge set and the second question text, the second language model generates a verification answer text corresponding to the second question text;

[0106] Based on the verification answer text, the second answer text is evaluated to obtain a confidence degree of the second answer text;

[0107] If the confidence is greater than a second preset threshold, the second answer text is taken as the first answer text, and the question and answer pair is obtained;

[0108] If the confidence is less than or equal to the second preset threshold, the second language model is adjusted, and the step of generating a plurality of initial question and answer pairs according to the domain knowledge set and the preset template through the pre-trained second language model is executed until the confidence is greater than the second preset threshold.

[0109] Through the pre-trained second language model, a plurality of initial question and answer pairs can be constructed according to the domain knowledge set and the preset template. The second language model is trained through a plurality of historical training data, and each historical training data includes historical question text and historical answer text corresponding to the historical question text. Each initial question and answer pair includes first question text and second answer text corresponding to the first question text. Subsequently, for each initial question and answer pair, the target information in the first question text can be modified to obtain the second question text. The target information is the key information in the first question text, and the target information in the first question text is modified, which will cause the generated answer text to change. For example, if the first question text is "What does 'E' in the ABCDE principle of secondary prevention of coronary heart disease refer to?", at this time, "coronary heart disease secondary prevention" and "ABCDE principle" can be modified as target information to obtain the second question text "What does 'E' in the medical field refer to?". Or, if the first question text is "Which part of myocardial infarction does ST elevation in lead II, III, and aVF correspond to?", at this time, "ST elevation in lead II, III, and aVF" can be modified as target information to obtain the second question text "Which part of myocardial infarction does ST elevation in lead V1-V4 correspond to?".

[0110] Through the second language model, according to the domain knowledge set and the second question text, a verification answer text corresponding to the second question text can be generated. For example, if the second question text is "What does 'E' in the medical field refer to?", the verification answer text can be generated, such as "Estrogen". The number of verification answer texts is not limited, and when the number of verification answer texts is multiple, verification answer texts such as "Electrolyte" and "Emergency PCI" can also be generated. Alternatively, the second question text is "Which part of myocardial infarction does ST elevation in lead V1-V4 correspond to?", and the verification answer text is "considering anterior myocardial infarction". Then, through the verification answer text, the performance of the second language model in generating the initial question and answer pair can be evaluated, and the confidence of the second answer text is obtained.

[0111] The second preset threshold is set according to actual conditions. The confidence can be compared with the second preset threshold. If the confidence is greater than the second preset threshold, the reliability of the initial question and answer pair generated by the second language model is high, and therefore the second answer text is taken as the first answer text, thereby obtaining the question and answer pair. If the confidence is less than or equal to the second preset threshold, the reliability of the initial question and answer pair generated by the second language model is low, and at this time, the second language model can be adjusted, and the multiple initial question and answer pairs are generated again according to the domain knowledge set and the preset template through the adjusted second language model until the confidence is greater than the second preset threshold. In this way, the robustness and generalization ability of the second language model can be improved, thereby improving the generation efficiency of the multiple question and answer pairs while ensuring the reliability of the generated multiple question and answer pairs.

[0112] Please refer to Figure 2 , Figure 2 is another flowchart of the method for generating a training data set provided by the embodiments of the present application. The method for generating a training data set provided by the embodiments of the present application will be exemplified below. Figure 2

[0113] The original document (i.e., multiple domain knowledge) is divided into a knowledge system by a pre-trained review expert model, that is, the multiple domain knowledge is divided according to core concepts (i.e., concept types), review standards (i.e., judgment standard types) and error warning (i.e., warning information types), thereby obtaining concept map data (i.e., domain knowledge set), and storing the concept map data in a structured knowledge base. The review expert model can be a prompt engineering-based expert model. Through this step, the multiple domain knowledge is preprocessed, which can lay a rigorous data foundation for the subsequent information retrieval stage.

[0114] The multiple question and answer pairs are constructed by a pre-trained planning expert model (i.e., a second language model) according to the constructed concept map data and a preset template (including a memory type template, an application type template and a reasoning type template). In this process, the generated first question text can be refined into different types of tasks based on priori, and different types of tasks are given different requirements of generation standards (i.e., answer text generation rules), so as to guarantee the richness of the generated question and answer pair. For example, for the first question text using the memory type template, the first answer text generated should be a relatively complete and accurate reproduction of the first domain knowledge. For the first question text using the application type template, the first answer text should be reasonably generated based on the specific application scenario of the first question text. For the first question text using the reasoning type template, the context dependency relationship of the first question text should be determined, and the first question text should be reasoned step by step.

[0115] ​Subsequently, by tracing and checking a plurality of question and answer pairs and by adversarial verification, a training data set including the first question text and the target answer text corresponding to the first question text can be obtained. Through the above two-stage processing, the original document can be purified into concentrated corpus with high pertinence to the first language model. Subsequently, through the long thinking chain multi-stage reasoning and reflection type verification process of the planning expert model, the planning expert model can generate reasonable and reliable training data sets.

[0116] The training data set generation method provided by the embodiments of the present application has the following advantages:

[0117] (1) By designing a long chain thinking template, complex problems are decomposed into multi-stage reasoning units, and traditional implicit thinking paths are converted into learnable explicit logic chains. For example, in the generation of mathematical problems, the planning expert model is required to output a complete chain of question stem analysis, formula derivation, step-by-step calculation, counterfactual verification, and conclusion verification. In this way, the planning expert model can directly capture the dependency relationship between reasoning steps, significantly reducing the occurrence rate of logical faults in the reasoning process.

[0118] (2) In the data generation stage, an adversarial perturbation mechanism is implanted to achieve fault-tolerant training by actively constructing semantic interference. Dynamic replacement testing is performed on key reasoning nodes (such as legal provisions), triggering the generation of counterfactual samples, so that the large model used for pre-training can be self-corrected based on the counterfactual samples.

[0119] (3) A meta-instruction pluggable system is constructed to realize decoupled design of domain knowledge and reasoning logic. The reasoning, verification, correction, and other logic operators are modularly encapsulated with the domain ontology to support zero-sample adaptation to new domains.

[0120] Please refer to Figure 3 , Figure 3 is a structural schematic diagram of the training data set generation device provided by the embodiments of the present application. The embodiments of the present application also provide a training data set generation device 300, which can implement the above-mentioned training data set generation method. The device 300 comprises:

[0121] The acquisition module 301 is configured to acquire a set of domain knowledge related to the domain to which the first language model is applied.

[0122] The generation module 302 is configured to generate a plurality of question and answer pairs according to the set of domain knowledge and a preset template. Each question and answer pair comprises a first question text, a first answer text corresponding to each first question text, and a reasoning text. The reasoning text comprises first domain knowledge obtained from the set of domain knowledge in the process of reasoning the first question text to obtain the corresponding first answer text.

[0123] The checking module 303 is configured to, for each of the question-answer pairs, perform provenance checking on the first answer text according to the reasoning text and the first answer text in the question-answer pair, to obtain a similarity between the first answer text and the first domain knowledge.

[0124] The determining module 304 is configured to, if the similarity is greater than a first preset threshold, take the first answer text as a target answer text corresponding to the first question text.

[0125] The correcting module 305 is configured to, if the similarity is less than or equal to the first preset threshold, correct the first answer text according to the first domain knowledge, to obtain a target answer text corresponding to the first question text.

[0126] The adding module 306 is configured to add the first question text and the target answer text corresponding to the first question text to a training data set, the training data set being used to train the first language model.

[0127] In some embodiments, the generating module 302 comprises:

[0128] A first generating sub-module is configured to generate a plurality of initial question-answer pairs by a pre-trained second language model according to the set of domain knowledge and a preset template, each of the initial question-answer pairs comprising the first question text and a second answer text corresponding to the first question text, the second language model being trained by a plurality of historical training data, each of the historical training data comprising a historical question text and a historical answer text corresponding to the historical question text.

[0129] A modifying sub-module is configured to, for each of the initial question-answer pairs, modify target information in the first question text to obtain a second question text.

[0130] A second generating sub-module is configured to generate a verification answer text corresponding to the second question text by the second language model according to the set of domain knowledge and the second question text.

[0131] An evaluating sub-module is configured to evaluate the second answer text based on the verification answer text, to obtain a confidence of the second answer text.

[0132] A first judging sub-module is configured to, if the confidence is greater than a second preset threshold, take the second answer text as the first answer text, to obtain the question-answer pair.

[0133] The second determining submodule is configured to, if the confidence degree is less than or equal to the second preset threshold, adjust the second language model, jump to the step of generating a plurality of initial question and answer pairs according to the domain knowledge set and the preset template through the pre-trained second language model, and execute until the confidence degree is greater than the second preset threshold.

[0134] In some embodiments, the obtaining module 301 comprises:

[0135] The obtaining submodule is configured to obtain a plurality of domain knowledge related to the domain to which the first language model is applied.

[0136] The classification submodule is configured to classify the plurality of domain knowledge based on a plurality of preset types to obtain a plurality of domain knowledge subsets, wherein the plurality of preset types include a judgment standard type, a concept type, and a warning information type, and each domain knowledge subset corresponds to one preset type.

[0137] The set generation submodule is configured to obtain the domain knowledge set according to the plurality of domain knowledge subsets.

[0138] In some embodiments, the generating module 302 comprises:

[0139] The first extraction submodule is configured to extract second domain knowledge from the second domain knowledge subset.

[0140] The filling submodule is configured to fill the second domain knowledge into the plurality of preset templates to obtain a plurality of first question texts.

[0141] The second extraction submodule is configured to, for each first question text, extract first domain knowledge corresponding to the first question text from at least one of the first domain knowledge subset and the third domain knowledge subset according to the first question text.

[0142] The third generation submodule is configured to generate the reasoning text and the first answer text corresponding to the first question text according to the first question text, the first domain knowledge, and an answer text generation rule corresponding to the preset template adopted by the first question text.

[0143] In some embodiments, the third generation submodule comprises:

[0144] The splitting unit is configured to split the first question text into a plurality of question subtexts.

[0145] The first generation unit is configured to generate reasoning subtexts and answer subtexts corresponding to each question subtext according to each question subtext, the first domain knowledge, and an answer text generation rule corresponding to the preset template adopted by the first question text.

[0146] The second generation unit is configured to generate the first answer text according to the answer subtext corresponding to each question subtext, and generate the reasoning text according to the reasoning subtext.

[0147] In some embodiments, the first generation unit comprises:

[0148] The generation subunit is configured to generate the reasoning subtext and the answer subtext corresponding to each question subtext according to the answer text generation rule corresponding to each question subtext, the first domain knowledge and the preset template adopted by the first question text in the hierarchical order among the question subtexts.

[0149] The detailed implementation of the training data set generation apparatus 300 can refer to the above-mentioned specific embodiments of the training data set generation method, and will not be repeated here.

[0150] The embodiments of the present application also provide an electronic device, which comprises a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned training data set generation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0151] Please refer to Figure 4 , Figure 4 is a hardware structure schematic diagram of the electronic device provided by the embodiments of the present application, and the electronic device comprises:

[0152] The processor 401 can be implemented in the form of a general-purpose CPU (Central Processing Unit, central processor), a microprocessor, an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), or one or more integrated circuits, etc., for executing related programs to realize the technical solutions provided by the embodiments of the present application.

[0153] The memory 402 can be implemented in the form of a ROM (Read-Only Memory, read-only memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory, random access memory), etc. The memory 402 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are saved in the memory 402 and called and executed by the processor 401 to realize the training data set generation method of the embodiments of the present application.

[0154] The input / output interface 403 is configured to realize information input and output.

[0155] The communication interface 404 is configured to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (for example, a USB, a network cable, or the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, or the like).

[0156] The bus 405 is configured to transmit information between various components (for example, the processor 401, the memory 402, the input / output interface 403, and the communication interface 404) of the device.

[0157] The processor 401, the memory 402, the input / output interface 403, and the communication interface 404 are communicatively connected to each other within the device through the bus 405.

[0158] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above-mentioned method for generating a training data set.

[0159] The memory is a non-transitory computer readable storage medium, which can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory and can further include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0160] The method, device, equipment, medium and product provided by the embodiment of the present application for generating a training data set, by obtaining a domain knowledge set related to the application domain of the first language model, and then generating a plurality of question and answer pairs according to the domain knowledge set and a preset template, for each question and answer pair, the first answer text can be verified according to the reasoning text and the first answer text in the question and answer pair, to obtain the similarity between the first answer text and the first domain knowledge. If the similarity is greater than a first preset threshold, the first answer text is taken as the target answer text corresponding to the first question text. If the similarity is less than or equal to the first preset threshold, the first answer text is modified according to the first domain knowledge to obtain the target answer text corresponding to the first question text. In this way, the reliability of the generated first question text and the answer text corresponding to the first question text can be improved. Then, the first question text and the target answer text corresponding to the first question text can be added to the training data set used to train the first language model. In this way, the data quality of the training data set can be improved, and the reasoning generalization ability of the first language model trained based on the training data set in a specific domain can be improved.

[0161] The embodiments described in the specification are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0162] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0163] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0164] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0165] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0166] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0167] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed mutual ones can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0168] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0169] In addition, each functional unit in each embodiment of the application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0170] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0171] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for generating a training data set, characterized in that, The method comprises: acquiring a domain knowledge set related to a domain to which a first language model is applied; generating a plurality of question and answer pairs according to the domain knowledge set and a preset template, each of the question and answer pairs comprising a first question text, a first answer text corresponding to each of the first question texts, and reasoning text comprising first domain knowledge obtained from the domain knowledge set in a process of reasoning the first question text to obtain the corresponding first answer text; for each of the question and answer pairs, performing provenance verification on the first answer text according to the reasoning text and the first answer text in the question and answer pair to obtain a similarity between the first answer text and the first domain knowledge; if the similarity is greater than a first preset threshold, taking the first answer text as a target answer text corresponding to the first question text; if the similarity is less than or equal to the first preset threshold, modifying the first answer text according to the first domain knowledge to obtain a target answer text corresponding to the first question text; adding the first question text and the target answer text corresponding to the first question text to a training data set, the training data set being used to train the first language model; wherein the generating of the plurality of question and answer pairs according to the domain knowledge set and the preset template comprises: generating a plurality of initial question and answer pairs according to the domain knowledge set and the preset template by a pre-trained second language model, each of the initial question and answer pairs comprising the first question text and a second answer text corresponding to the first question text, the second language model being trained by a plurality of historical training data, each of the historical training data comprising a historical question text and a historical answer text corresponding to the historical question text; for each of the initial question and answer pairs, modifying target information in the first question text to obtain a second question text; generating a verification answer text corresponding to the second question text according to the domain knowledge set and the second question text by the second language model; based on the verification answer text, evaluating the second answer text to obtain a confidence of the second answer text; if the confidence is greater than a second preset threshold, taking the second answer text as the first answer text to obtain the question and answer pair; if the confidence is less than or equal to the second preset threshold, adjusting the second language model and jumping to the step of generating the plurality of initial question and answer pairs according to the domain knowledge set and the preset template by the pre-trained second language model until the confidence is greater than the second preset threshold.

2. The method of claim 1, wherein, The acquiring of the domain knowledge set related to the domain to which the first language model is applied comprises: acquiring a plurality of the domain knowledge related to the domain to which the first language model is applied; based on a plurality of preset types, classifying the plurality of the domain knowledge to obtain a plurality of domain knowledge subsets, the plurality of preset types comprising a judgment standard type, a concept type, and a warning information type, each of the domain knowledge subsets corresponding to one of the preset types; obtaining the domain knowledge set according to the plurality of the domain knowledge subsets.

3. The method of claim 2, wherein, The number of the preset templates is multiple, each of the preset templates corresponds to an answer text generation rule, and the multiple domain knowledge subsets include a first domain knowledge subset corresponding to the judgment standard type, a second domain knowledge subset corresponding to the concept type, and a third domain knowledge subset corresponding to the warning information type; The generating multiple question and answer pairs according to the domain knowledge set and the preset templates comprises: extracting second domain knowledge from the second domain knowledge subset; filling the second domain knowledge into multiple preset templates to obtain multiple first question texts; for each first question text, extracting first domain knowledge corresponding to the first question text from at least one of the first domain knowledge subset and the third domain knowledge subset according to the first question text; generating the reasoning text and the first answer text corresponding to the first question text according to the first question text, the first domain knowledge, and an answer text generation rule corresponding to the preset template adopted by the first question text.

4. The method of claim 3, wherein, The generating the reasoning text and the first answer text corresponding to the first question text according to the first question text, the first domain knowledge, and an answer text generation rule corresponding to the preset template adopted by the first question text comprises: splitting the first question text into multiple question subtexts; generating reasoning subtexts and answer subtexts corresponding to each question subtext according to each question subtext, the first domain knowledge, and an answer text generation rule corresponding to the preset template adopted by the first question text; generating the first answer text according to the answer subtexts corresponding to each question subtext, and generating the reasoning text according to the reasoning subtexts.

5. The method of claim 4, wherein, The multiple question subtexts have a hierarchical order; The generating the reasoning subtexts and the answer subtexts corresponding to each question subtext according to each question subtext, the first domain knowledge, and an answer text generation rule corresponding to the preset template adopted by the first question text comprises: generating the reasoning subtexts and the answer subtexts corresponding to each question subtext according to each question subtext, the first domain knowledge, and an answer text generation rule corresponding to the preset template adopted by the first question text in turn according to the hierarchical order between the multiple question subtexts.

6. An apparatus for generating a training data set, characterized by The apparatus comprises: an acquisition module configured to acquire a domain knowledge set related to a domain to which a first language model is applied; a generation module configured to generate multiple question and answer pairs according to the domain knowledge set and preset templates, each of the question and answer pairs comprising a first question text, a first answer text corresponding to each first question text, and a reasoning text, the reasoning text comprising first domain knowledge obtained from the domain knowledge set in a process of reasoning the first question text to obtain the corresponding first answer text. The verification module is configured to, for each of the question-answer pairs, perform provenance verification on the first answer text according to the reasoning text and the first answer text in the question-answer pair, to obtain a similarity between the first answer text and the first domain knowledge; The determination module is configured to, if the similarity is greater than a first preset threshold, take the first answer text as a target answer text corresponding to the first question text. The correction module is configured to, if the similarity is less than or equal to the first preset threshold, correct the first answer text according to the first domain knowledge, to obtain a target answer text corresponding to the first question text. The adding module is configured to add the first question text and the target answer text corresponding to the first question text to a training data set, the training data set being used to train the first language model. The generation module includes: The first generation submodule is configured to generate a plurality of initial question-answer pairs by a pre-trained second language model according to the set of domain knowledge and a preset template, each of the initial question-answer pairs including the first question text and a second answer text corresponding to the first question text, the second language model being trained by a plurality of historical training data, each of the historical training data including a historical question text and a historical answer text corresponding to the historical question text. The modification submodule is configured to, for each of the initial question-answer pairs, modify target information in the first question text to obtain a second question text. The second generation submodule is configured to generate a verification answer text corresponding to the second question text by the second language model according to the set of domain knowledge and the second question text. The evaluation submodule is configured to evaluate the second answer text based on the verification answer text, to obtain a confidence of the second answer text. The first judgment submodule is configured to, if the confidence is greater than a second preset threshold, take the second answer text as the first answer text, to obtain the question-answer pair. The second judgment submodule is configured to, if the confidence is less than or equal to the second preset threshold, adjust the second language model, and jump to the step of generating a plurality of initial question-answer pairs by the pre-trained second language model according to the set of domain knowledge and the preset template, until the confidence is greater than the second preset threshold.

7. An electronic device, comprising: The electronic device includes a memory and a processor, the memory storing a computer program, and the processor implementing the method for generating a training data set according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the method for generating a training data set according to any one of claims 1 to 5.

9. A computer program product, characterised in that, The instructions in the computer program product are executed by the processor of the electronic device as claimed in claim 7 to cause the electronic device to perform the method for generating a training data set according to any one of claims 1 to 5.

Images