Method for automatically generating multi-task objective question evaluation set in vertical field of large language model

Abstract

The invention provides a large language model vertical field multi-task objective question evaluation set automatic generation method, and relates to the technical field of artificial intelligence, and the method comprises the steps: for each text in a vertical field plain text data set, according to a plurality of preset task types, calling a problem solver to carry out task suitability discrimination on the text, and obtaining a task suitability discrimination result; when the judgment result is appropriate, generating at least one initial objective question stem of the task type; performing ambiguity detection on the initial objective question stem, and calling a question solver to rewrite the ambiguous initial objective question stem to obtain a rewritten objective question stem; and finally, calling a question solver to generate corresponding objective question options, objective question answers and objective question answer analyses for each rewritten objective question stem, and taking the objective question options, objective question answers and objective question answer analyses as an objective question evaluation set of a corresponding task type. By means of the method and device, the problems that in the prior art, time and labor are consumed in the large model evaluation dataset construction process, and automatic generation of the evaluation dataset in the specific vertical field cannot be achieved are solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for automatically generating a large language model vertical field multi-task objective question evaluation set. Background Art

[0002] With the continuous development and progress of Large Language Model (LLM) technology, evaluation sets have played a core role in measuring model effects and guiding model iteration and optimization. The existing mainstream evaluation sets for evaluating the capabilities of large models mainly include subjective questions (subjective questions) and objective multiple-choice questions (objective questions). They are generally obtained directly from existing examination institutions and schools. Since objective multiple-choice questions have the advantages of high accuracy and few interference factors in evaluation, the current representative evaluation benchmarks, such as CMMLU, C-Eval, GSM8K, SUPERGLUE evaluation sets, etc., all use objective multiple-choice questions. Such evaluation sets can usually reflect the general capabilities of the model to a certain extent, or the strength of a certain aspect of the ability, and become an important reference indicator for evaluating model performance.

[0003] However, there are several problems with existing large model evaluation sets. The first is the problem of evaluation set contamination, that is, the evaluation set may have already appeared in the training data set of some large models. If you need to accurately evaluate the capabilities of the large model, you usually need to use the latest data to specifically build a new evaluation set; second, most of the existing evaluation sets are general data sets or evaluation sets of certain capabilities. If you evaluate the performance of large models in vertical fields or private domain data sets, you will inevitably face the problem of building specific evaluation sets.

[0004] In the prior art, there are two main methods for constructing large-model evaluation datasets: one is to ask human experts in a specific field to compile evaluation sets. This type of evaluation set has high quality, but if a large-scale evaluation set needs to be built, it will take a lot of manpower and time; the other method is automatic generation, extracting evaluation sets from existing domain corpora to test the effectiveness of large models. For example, SUPERCLUE is a comprehensive benchmark for general large models in Chinese, and its objective multiple-choice questions are generated from open source question-answer pairs. However, for specific vertical field data, there is often only a single form of unsupervised text, and it is impossible to directly generate objective multiple-choice evaluation sets. Summary of the invention

[0005] The present invention provides a method for automatically generating a large language model vertical field multi-task objective question evaluation set, which is used to solve the problems in the prior art that the construction process of a large model evaluation data set is time-consuming and labor-intensive and the automatic generation of a specific vertical field evaluation data set cannot be achieved.

[0006] The present invention provides a method for automatically generating a large language model vertical field multi-task objective question evaluation set, the method comprising the following steps: Obtain plain text datasets in vertical fields; For each text in the plain text data set, calling a problem solver to judge the suitability of the text for a task according to a plurality of preset task types, and generating at least one initial objective question stem of the task type according to the text when the judgment result is suitable; Performing ambiguity detection on each of the initial objective question stems, and calling the problem solver to rewrite the ambiguous initial objective question stems to obtain rewritten objective question stems; The problem solver is called to generate corresponding objective question options, objective question answers and objective question answer analysis for each rewritten objective question stem as an objective question evaluation set for the corresponding task type.

[0007] In some embodiments, the task types include: information extraction type, knowledge question answering type, computational reasoning type, and task decomposition type, and the method further includes: According to each task type, the problem solver is called to construct an evaluation set for the text to obtain an objective question evaluation set corresponding to each task type.

[0008] In some embodiments, calling the problem solver to determine the suitability of the text for the task includes: Obtaining a judgment prompt word corresponding to a preset task type, wherein the judgment prompt word includes a system prompt word and a task prompt word corresponding to the task type, wherein the system prompt word is used to instruct the problem solver to perform task suitability judgment on the text; The judgment prompt word is concatenated with the text, and the concatenated text is input into a problem solver to perform task suitability judgment to obtain a corresponding judgment result.

[0009] In some embodiments, calling the problem solver to rewrite the ambiguous initial objective question stem to obtain a rewritten objective question stem includes: Obtaining preset disambiguation prompt words, wherein the disambiguation prompt words include semantic disambiguation prompt words and disambiguation example texts, and the semantic disambiguation prompt words are used to instruct the problem solver to rewrite when it is detected that the initial objective question stem is ambiguous; splicing the ambiguity elimination prompt word, the text and the initial objective question stem to obtain an ambiguous spliced ​​text; The problem solver is called to generate the ambiguous concatenated text to obtain a rewritten objective question stem corresponding to the initial objective question stem.

[0010] In some embodiments, the calling of the question solver to generate corresponding objective question options, objective question answers, and objective question answer analysis for each rewritten objective question stem includes: Obtaining a preset option generation prompt word, wherein the option generation prompt word is used to instruct the problem solver to rewrite the objective question stem according to the text to generate corresponding objective question options, and generate corresponding objective question answers and objective question answer analysis; The option generation prompt word, the text and the rewriting objective question stem are spliced ​​together to generate a spliced ​​text; The problem solver is called to generate the generated concatenated text to obtain corresponding objective question options, objective question answers and objective question answer analysis.

[0011] In some embodiments, the method further comprises: The problem solver is called to filter the objective question options and the objective question answers, and the filtering process of the problem solver includes: Obtaining a preset filtering prompt word, wherein the filtering prompt word includes a filtering condition, and the filtering prompt word is used to prompt the problem solver to filter according to the filtering condition; The filtering prompt word, the text, the rewritten objective question stem, the objective question options and the objective question answer are spliced ​​together to obtain a filtered spliced ​​text; The problem solver is called to filter the filtered spliced ​​text according to the filtering condition to obtain the filtered rewritten objective question stem and the corresponding objective question options and objective question answers.

[0012] In some embodiments, the filtering conditions include filtering conditions common to multiple task types and specific filtering conditions for a single task type; The general filtering condition includes at least one of the following: Rewrite the objective test stem to make it unsuitable for objective test; The answer to the objective test is not the correct answer to the objective test question stem; There are multiple correct answers for the objective questions, and the answer to the objective question is one of the correct answers; In the objective question options, wrong answers other than the objective question answers are taken as correct answers; The answers to objective questions do not come from rewriting the objective question stems.

[0013] The present invention also provides a device for automatically generating a large language model vertical field multi-task objective question evaluation set, the device comprising the following modules: The acquisition module is used to obtain plain text datasets in vertical fields; A discrimination module, for each text in the plain text data set, calling a problem solver to discriminate the suitability of the text for a task according to a plurality of preset task types, and generating at least one initial objective question stem of the task type according to the text when the discrimination result is suitable; A detection module, configured to perform ambiguity detection on each of the initial objective question stems, and call the problem solver to rewrite the ambiguous initial objective question stems to obtain rewritten objective question stems; The generation module calls the problem solver to generate corresponding objective question options, objective question answers and objective question answer analysis for each rewritten objective question stem as an objective question evaluation set for the corresponding task type.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for automatically generating a vertical field multi-task objective question evaluation set for a large language model as described in any one of the above-mentioned methods.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for automatically generating a large language model vertical field multi-task objective question evaluation set as described in any of the above methods.

[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for automatically generating a large language model vertical field multi-task objective question evaluation set.

[0017] The method for automatically generating a vertical field multi-task objective question evaluation set for a large language model provided by the present invention utilizes a pure text data set in a vertical field to construct objective questions. During the construction process, a two-stage process is adopted in which the stem and options are generated separately, thereby solving the problem that when the stem and options are generated simultaneously, the distinction between the stem options is not high and the interference is insufficient. The generation quality of objective questions is further improved by judging the suitability of the text and detecting the ambiguity of the stem. In addition, the present invention realizes the end-to-end automatic generation process of the vertical field objective question evaluation set without any human assistance, saving manpower and time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of the method for automatically generating a large language model vertical field multi-task objective question evaluation set provided by the present invention.

[0020] Figure 2 It is a structural schematic diagram of the device for automatically generating a large language model vertical field multi-task objective question evaluation set provided by the present invention.

[0021] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The method for automatically generating a multi-task objective question evaluation set for a large language model in a vertical field provided by the present invention can be applied to the scenario of downstream large language model training, and can be specifically deployed in a server or terminal used for downstream large language model training, that is, the executor of the method can be a server or terminal for downstream large language model training. When the downstream large language model needs to call a data set for training, calling and running the method of the present invention can automatically generate a corresponding objective question evaluation set for the downstream large language model to complete the training process. The method for automatically generating a multi-task objective question evaluation set for a large language model in a vertical field of the present invention is described below in conjunction with the accompanying drawings.

[0024] Figure 1 : is a flow chart of the method for automatically generating a large language model vertical field multi-task objective question evaluation set provided by the present invention, such as Figure 1 As shown, the method specifically includes the following steps 101 to 104, which are described in detail below.

[0025] Step 101: Obtain a plain text dataset in a vertical field.

[0026] Here, plain text datasets in vertical fields can be collected manually, where vertical fields refer to the application of artificial intelligence (AI) technologies and solutions in specific industries or fields. In vertical fields, AI technologies (such as large language models) are specifically used to solve specific problems in that field. For example, healthcare, finance, retail, transportation, etc. are all vertical fields. Therefore, when collecting plain text datasets, datasets in corresponding fields are collected according to the vertical fields to be processed by the downstream large language model.

[0027] The specific collection method can be obtained by crawling or scanning open source databases or data repositories. The data set should contain plain text in the vertical field, denoted as D. The i-th text in the plain text data set D is denoted as , the number of texts is recorded as N, then i∈{0, 1,…, N-1}.

[0028] After the plain text dataset is manually collected and uploaded, the uploaded plain text dataset in the vertical field is obtained to execute the subsequent objective question evaluation set construction process.

[0029] Step 102: for each text in the plain text data set, call the problem solver to judge the task suitability of the text according to multiple preset task types, and when the judgment result is suitable, generate at least one initial objective question stem of the task type according to the text.

[0030] In the process of constructing the objective question evaluation set, for each text in the plain text data set, the problem solver is called to judge the task suitability of the text according to multiple preset task types. In the embodiment of the present invention, the task type refers to the type of objective question to be generated, and there are multiple preset task types, including information extraction type, knowledge question and answer type, computational reasoning type, and task decomposition type.

[0031] These task types are the more common types in objective questions. Simply put, the information extraction type is to extract certain content from the question stem for understanding and answering, the knowledge answering type is to answer the questions in the question stem based on one's own semantic database, the computational reasoning type requires reasoning and calculation based on the semantics of the question stem to make an answer, and the task decomposition type is to decompose the content of the question stem, or to answer whether the decomposed content is accurate and appropriate.

[0032] After the task type is determined, for each text in the plain text dataset D , all need to be judged in advance to determine the text Whether it is suitable to generate objective questions of the current task type (in the embodiment of the present invention, unless otherwise specified, objective questions refer to multiple-choice questions). Because some texts cannot express complete semantics due to lack of content, or the semantics are not suitable for the corresponding task type. For example, a certain text describes historical and cultural information, and computational reasoning cannot be performed at all, so it is not suitable for generating objective questions of computational reasoning type. Therefore, it is necessary to call a problem solver to judge the suitability of the text for the task. The problem solver can be a large language model, for example, it can be a Qwen1.5 72B base large model, and the selection of the large language model is not limited. The specific process of judging the suitability of the problem solver is introduced below.

[0033] First, obtain the judgment prompt words corresponding to the preset task type. The judgment prompt words are divided into two parts, including system prompt words and task prompt words corresponding to the task type. Among them, the system prompt words are used to instruct the problem solver to judge the task suitability of the text, while the task prompt words explain the concept of the task type to the problem solver, helping the large language model understand the semantics of the task type and make a judgment.

[0034] An example of the format of the preset judgment prompt words may be: "system prompt=You are now a military expert. Based on the input military field text, please analyze and determine whether the text is suitable for generating objective multiple-choice questions to evaluate the information extraction ability of the large model, and output at least 5 diversified evaluation questions of different difficulty, length, and complexity, and ensure that each objective question has only one standard answer. If it is not suitable, please output: This paragraph of text is not suitable for objective questions. taskprompt=Information extraction capabilities include entity recognition (NER), event extraction (EE), relationship extraction (RE), etc. Event extraction schema (event trigger word, event type, (event argument, argument role)), relationship extraction schema (relationship, head entity, tail entity), entity recognition schema (entity, entity type). Information extraction questions should be able to evaluate the large model's ability to extract text entities, entity types, relationships, events and other elements. If the input text does not clearly contain the three elements of entity, event, and relationship, please output that the text is not suitable for generating information extraction questions."

[0035] After obtaining the judgment prompt word, the judgment prompt word and the text Perform text concatenation and input the concatenated text into the problem solver for task suitability judgment to obtain the corresponding judgment result. The large language model will automatically combine the semantic hints of the judgment prompt words, understand the concept of the task type, and make the corresponding judgment result through the semantic understanding of the text.

[0036] When the judgment result is inappropriate, the text If it is not suitable to generate objective questions of the current task type, then other task types are changed to continue to determine the suitability of the task. If all task types are judged to be unsuitable, the text is filtered out and the next text is processed.

[0037] When the judgment result is appropriate, it means It is suitable to generate objective questions of the current task type, so it enters the first stage of constructing objective questions, generating at least one initial objective question stem of the task type based on the text. Generate at least one initial objective question stem of the task type. The objective question stem is usually a question, recorded as , the number of objective question stems is denoted as K (K≥5), k represents the kth initial objective question stem, k∈{0, 1,…, K-1}.

[0038] In the embodiment of the present invention, when constructing an objective question evaluation set, the stem of the objective question is first generated based on the pure text alone, which solves the problem in the prior art that when the stem and options are generated simultaneously, the distinction between the stem options is not high and the interference is insufficient. By judging the task suitability of the text, the generation quality of the objective questions of the corresponding task type is improved.

[0039] Step 103: perform ambiguity detection on each initial objective question stem, and call the problem solver to rewrite the ambiguous initial objective question stem to obtain a rewritten objective question stem.

[0040] After generating a plurality of initial objective question stems according to the text through step 102, these stems cannot be directly used as objective questions because some of them may be ambiguous or have unclear entity references. If the stems are ambiguous or have unclear references, it will inevitably affect the quality of the objective questions.

[0041] Based on this, in the embodiment of the present invention, ambiguity detection is performed on each initial objective question stem, and the question solver is called to rewrite the ambiguous initial objective question stem to obtain the rewritten objective question stem. Whether there is ambiguity, the basis for judgment comes from the text If there is ambiguity, the ambiguous initial objective question stem is rewritten by the problem solver to eliminate the ambiguity and obtain the rewritten objective question stem, which is explained in detail below.

[0042] The first step is still to obtain the preset disambiguation prompt words, which include semantic disambiguation prompt words and disambiguation example texts. The semantic disambiguation prompt words are used to instruct the problem solver to rewrite the initial objective question stem when it detects ambiguity, while the disambiguation example texts provide some examples for the large language model to help it understand how to identify ambiguity and rewrite the stem to eliminate ambiguity.

[0043] An example of the format of the semantic disambiguation prompt word may be: "You are a semantic disambiguation expert. You are given an initial objective question stem. Please judge whether the initial objective question stem (without the given context) is ambiguous. If it is ambiguous, please rewrite the initial objective question stem according to the context. The ambiguity mainly includes: 1. Unclear reference: the referential ambiguity caused by the unclear referent or content of referential words such as personal pronouns and demonstrative pronouns in the sentence; for example: xxx; 2. Unclear entity description: the definition and description of the entity in the sentence are not detailed enough and cannot be uniquely determined, causing declarative ambiguity; for example: xxx; if there is no ambiguity, directly output no ambiguity and give reasons; if there is ambiguity, please first answer that there is ambiguity and give reasons; then try to rewrite it into an unambiguous objective question stem according to the given context; when it cannot be rewritten into an unambiguous objective question stem according to the context, directly answer that it cannot be rewritten and give reasons; in order to help you complete the task better, some input and output examples are provided below for your reference: Example 1: xxx. Example 2: xxx".

[0044] After obtaining the preset ambiguity elimination prompt word, the ambiguity elimination prompt word and text and the initial objective test stems The ambiguous concatenated text is processed and the question solver is called to generate the ambiguous concatenated text to obtain the rewritten objective question stem corresponding to the initial objective question stem. The large language model will use the semantic hints of the ambiguity elimination hint words to convert the text into The semantics of is used as background knowledge to judge the K initial objective questions. Is there any ambiguity? If it is judged that there is ambiguity, the initial objective question stem with the ambiguous word will be corrected according to the semantic hint of the ambiguity elimination hint word. Rewrite to eliminate ambiguity and generate K corresponding rewritten objective question stems , k∈{0, 1,…, K-1}.

[0045] According to an embodiment of the present invention, after generating objective question stems according to texts, ambiguity detection is performed on each objective question stem, and the ambiguity is eliminated by rewriting the stems detected by ambiguity detection, thereby greatly reducing the impact that ambiguity may have on objective questions and improving the generation quality of corresponding objective questions.

[0046] Step 104 , calling the problem solver to generate corresponding objective question options, objective question answers, and objective question answer analysis for each rewritten objective question stem as an objective question evaluation set of the corresponding task type.

[0047] After rewriting the objective question stem in step 103, the second stage of constructing the objective question is entered to generate options corresponding to the stem. Here, the problem solver is called to generate corresponding objective question options, objective question answers, and objective question answer analysis for each rewritten objective question stem as an objective question evaluation set for the corresponding task type.

[0048] In the embodiment of the present invention, there are generally four objective question options, namely "ABCD". Of course, in order to make the constructed evaluation set more suitable for the downstream large language model to perform vertical field training. While generating the objective question options, the corresponding objective question answers and objective question answer analysis are also generated, that is, the problem solver is required to give a complete answer process for the objective question. The following specifically describes the generation process of objective question options, objective question answers, and objective question answer analysis.

[0049] First, the preset option generation prompt words are still obtained. The option generation prompt words are used to instruct the problem solver to rewrite the objective question stem according to the text to generate corresponding objective question options, and generate corresponding objective question answers and objective question answer analysis.

[0050] Specifically, using option generation prompts to put forward some requirements for the problem solver. For example, when generating objective question options, it is necessary to ensure that there is only one correct answer in the objective question options as the objective question answer, and then provide the objective question answer analysis. Of course, the requirements are different depending on the task type. For example, in the information extraction type, it is necessary to ensure that the objective question options must strictly come from rewriting the objective question stem, and promotion or expansion is prohibited. For another example, in the knowledge question answering type, it is necessary to ensure that the semantics of the objective question options cannot be ambiguous, there must be obvious differences between the various objective question options, and the source of the objective question answer must strictly come from the text, etc.

[0051] An example of the format of the preset option generation prompt words may be: "You are an expert in setting questions in the history field, and are good at generating objective question options based on rewriting objective question stems to make them complete objective questions; now, in order to evaluate the {information extraction} ability, please generate 4 objective question options for the given "Rewrite Objective Question Stem", of which there is only 1 correct option and 3 incorrect options, to make it a complete objective question, and give the correct answer and objective question answer analysis; Requirements for objective questions that evaluate the {information extraction} ability of large language models: 1. The content of the correct option must be completely and strictly derived from the question stem and must be completely correct. Ambiguity is prohibited, and the use of one's own knowledge to expand and create is prohibited, and reasoning and expansion are prohibited; 2. xxx; 3. xxx.".

[0052] After obtaining the preset option generation prompt word, the option generation prompt word and text are still and rewrite the objective test stems The concatenation process is performed to obtain the generated concatenated text, and finally the question solver is called to generate the generated concatenated text to obtain the corresponding objective question options, objective question answers, and objective question answer analysis. The semantics of the objective questions are used as background knowledge, and the objective questions are rewritten according to the To generate the corresponding 4 objective question options, then make reasoning questions, give one of the objective question answers, and output the corresponding objective question answer analysis.

[0053] Thus, through the above steps, each rewritten objective question stem can generate the corresponding objective question options, objective question answers and objective question answer analysis, thus forming a complete objective question of the corresponding task type. And by processing each text in the plain text dataset, an objective question of the corresponding task type can be constructed, thereby forming an objective question set, which can be used as an objective question evaluation set of the corresponding task type.

[0054] In the embodiment of the present invention, when using a pure text data set in a vertical field to generate objective questions, a two-stage process is adopted in the construction process to generate the stem and options separately, which solves the problem that when the stem and options are generated at the same time, the distinction between the stem and options is not high and the interference is insufficient. In addition, the construction process is automatically generated based on the preset prompt words as prompts, without any manual assistance, saving manpower and time.

[0055] In some embodiments, after the corresponding objective question options, objective question answers, and objective question answer analysis are generated based on the rewritten objective question stem, the question solver is also called to filter the objective question options and objective question answers. This is because some objective question options do not match the semantics of the rewritten objective question stem, or there is an offset in the answer, and there are inevitably non-unique correct answers, etc., which makes the quality of the objective question still low. Therefore, it is necessary to use the problem solver to perform a filtering operation on the constructed objective questions to filter out these low-quality objective questions. The filtering process is described below.

[0056] First, get the preset filter prompt words. The filter prompt words include filter conditions. The filter prompt words are used to prompt the problem solver to filter according to the filter conditions. Here, the filter prompt words need to provide the problem solver with filter conditions as the filtering criteria. Due to the different task types, the set filtering criteria are also different. Specifically, it still includes two parts: system prompt words and task prompt words. The system prompt words include filter conditions common to all task types, instructing the problem solver to filter objective questions according to the filter conditions and the reasons for filtering. The task prompt words are additional filter conditions that need to be supplemented for specific task types.

[0057] The filtering conditions are essentially preset according to the data requirements of the objective question evaluation set. The filtering conditions in the filtering prompt words include filtering conditions that are common to multiple task types and specific filtering conditions for a single task type. Among them, the common filtering conditions include at least one of the following: rewriting the objective question stem is not suitable as an objective question; the answer to the objective question is not the correct answer to rewrite the objective question stem; there are multiple correct answers in the objective question options, and the objective question answer is one of the correct answers; in the objective question options, wrong answers other than the objective question answers are used as correct answers; the objective question answer does not come from rewriting the objective question stem.

[0058] The specific filtering conditions of a single task type are different. For example, for the information extraction type, it is necessary to ensure that the answer to the objective question does not come from rewriting the objective question stem, and reasoning is required. For the computational reasoning type, it is necessary to ensure that the answer to the objective question cannot appear in the objective question stem, and it needs to be calculated based on the content of the stem.

[0059] An example of the format of the preset filter prompt words can be: "system prompt=Please use the given context as a supplement to your background knowledge, and judge whether there is ambiguity in the objective questions and answers given for evaluating the {information extraction} ability of the large model without this context. Please give a judgment first, and then give an explanation in a new line. The ambiguities mainly include: 1. The given questions are not suitable for objective questions; 2. The given answers are incorrect; 3. xxx; 4. xxx; task prompt=In addition, the ambiguities of multiple-choice questions of information extraction also include: 1. The correct options are not completely strictly derived from the questions, and can only be known through reasoning from other knowledge.".

[0060] After obtaining the preset filter prompt word, filter prompt word, text , rewrite the objective test stem , objective question options and objective question answers are concatenated to obtain the filtered concatenated text, and finally the problem solver is called to filter the filtered concatenated text according to the filtering conditions to obtain the filtered rewritten objective question stem and the corresponding objective question options and objective question answers. The semantics of the task is used as background knowledge, and then the objective question options and answers are judged whether they meet the requirements based on the filtering conditions (including general filtering conditions and specific filtering conditions of the task type).

[0061] When all the filtering conditions are met, it means that the answer matches the question stem, and the objective question stem, objective question options, and objective question answers are retained as the objective question evaluation set. When any one or more filtering conditions are not met, it means that the answer does not match the question stem, the objective question is of low quality, and is not suitable as an evaluation set, so the objective question is filtered out.

[0062] The embodiment of the present invention can filter out low-quality objective questions that do not meet the training requirements of the downstream large language model for each objective question generated by the text through a filtering operation, thereby improving the data quality of the entire evaluation set as a whole and facilitating the downstream large language model to perform vertical field data training.

[0063] In addition, considering that the downstream large language model needs to have diversity in data training for vertical fields, the embodiment of the present invention sets corresponding task types and constructs corresponding objective question evaluation sets according to the task types. , and may also be suitable for generating objective questions of multiple task types. For example, a text can generate both objective questions of information extraction type and objective questions of computational reasoning type.

[0064] In order to make full use of the acquired plain text dataset, in an embodiment of the present invention, after acquiring the plain text dataset in a vertical field, the problem solver is called to construct an evaluation set for the text according to each task type, so as to obtain the objective question evaluation set corresponding to each task type.

[0065] That is to say, for a certain task type, after executing the above steps 102 to 104 for each text in the plain text data set to generate an objective question evaluation set for the corresponding task type, then change to another task type, and continue to repeat the above steps 102 to 104 for each text in the plain text data set to generate an objective question evaluation set for the corresponding task type.

[0066] In this way, under different task types, the text evaluation set is constructed by calling the problem solver to obtain the objective question evaluation set corresponding to each task type. For example, there are four types of tasks: information extraction type, knowledge question answering type, computational reasoning type, and task decomposition type. Then each text in the plain text data set can generate up to four objective questions, and objective question evaluation sets of four different task types can be obtained.

[0067] The embodiment of the present invention, by presetting different task types, constructs a corresponding objective question evaluation set for each task type, and can fully utilize the data of the plain text data set, which not only improves the data utilization rate, but also improves the diversity of the objective question evaluation set.

[0068] Through the method of the embodiment of the present invention, objective question evaluation sets of four different task types are constructed by collecting plain text data sets in vertical fields. The number of objective questions in the evaluation sets of different task types is statistically shown in the following table: Table 1:

[0069] In order to demonstrate the vertical domain data training effect of the objective question evaluation set constructed in the embodiment of the present invention on the downstream large language model, a comparative experiment is conducted here with the evaluation sets CMMLU and C-Eval constructed in the prior art in different downstream large language models. The experimental results are shown in the following table: Table 2:

[0070] It can be seen from Table 2 above that the objective question evaluation datasets of the four task types constructed by the method of the embodiment of the present invention are generally higher in training effect than the evaluation sets CMMLU and C-Eval constructed in the prior art. This shows that in the vertical field, the objective question evaluation dataset constructed by the embodiment of the present invention can effectively improve the training effect of the downstream large language model, and it also has certain reference significance for other similar fields.

[0071] The following is a description of the device for automatically generating a vertical field multi-task objective question evaluation set for a large language model provided by the present invention. The device for automatically generating a vertical field multi-task objective question evaluation set for a large language model described below and the method for automatically generating a vertical field multi-task objective question evaluation set for a large language model described above can be referenced to each other.

[0072] like Figure 2 As shown, the automatic generation device of multi-task objective question evaluation set in vertical field of large language model includes: acquisition module 201, discrimination module 202, detection module 203, and generation module 204. Specifically, acquisition module 201 is used to acquire a plain text data set in a vertical field; discrimination module 202 is used to call the problem solver to judge the task suitability of each text in the plain text data set according to multiple preset task types, and when the judgment result is suitable, generate at least one initial objective question stem of the task type according to the text; detection module 203 is used to perform ambiguity detection on each of the initial objective question stems, and call the problem solver to rewrite the ambiguous initial objective question stems to obtain rewritten objective question stems; generation module 204 calls the problem solver to generate corresponding objective question options, objective question answers and objective question answer analysis for each rewritten objective question stem as the objective question evaluation set of the corresponding task type.

[0073] It should be noted that the beneficial effects of the automatic generation device for multi-task objective question evaluation sets in vertical fields with a large language model and the automatic generation method for multi-task objective question evaluation sets in vertical fields with a large language model mentioned above can correspond to each other, so the beneficial effects of the automatic generation device for multi-task objective question evaluation sets in vertical fields with a large language model will not be repeated here.

[0074] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3As shown, the electronic device may include: a processor 310 , a communications interface 320 , a memory 330 and a communication bus 340 , wherein the processor 310 , the communications interface 320 , and the memory 330 communicate with each other via the communication bus 340 . The processor 310 can call the logic instructions in the memory 330 to execute a method for automatically generating a vertical field multi-task objective question evaluation set for a large language model, the method comprising: obtaining a plain text data set in a vertical field; for each text in the plain text data set, calling a problem solver to judge the task suitability of the text according to a plurality of preset task types, and when the judgment result is suitable, generating at least one initial objective question stem of the task type according to the text; performing ambiguity detection on each of the initial objective question stems, and calling the problem solver to rewrite the ambiguous initial objective question stems to obtain rewritten objective question stems; calling the problem solver to generate corresponding objective question options, objective question answers, and objective question answer analyses for each rewritten objective question stem as an objective question evaluation set of the corresponding task type.

[0075] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0076] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the large language model vertical field multi-task objective question evaluation set automatic generation method provided by the above methods, the method including: obtaining a plain text data set in a vertical field; for each text in the plain text data set, according to a plurality of preset task types, calling a problem solver to perform task suitability judgment on the text, and when the judgment result is suitable, generating at least one initial objective question stem of the task type according to the text; performing ambiguity detection on each of the initial objective question stems, and calling the problem solver to rewrite the ambiguous initial objective question stems to obtain rewritten objective question stems; calling the problem solver to generate corresponding objective question options, objective question answers and objective question answer analyses for each rewritten objective question stem as an objective question evaluation set of the corresponding task type.

[0077] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the method for automatically generating a large language model vertical field multi-task objective question evaluation set provided by the above-mentioned methods, the method comprising: obtaining a plain text data set in a vertical field; for each text in the plain text data set, calling a problem solver to perform task suitability judgment on the text according to a plurality of preset task types, and when the judgment result is suitable, generating at least one initial objective question stem of the task type according to the text; performing ambiguity detection on each of the initial objective question stems, and calling the problem solver to rewrite the ambiguous initial objective question stems to obtain rewritten objective question stems; calling the problem solver to generate corresponding objective question options, objective question answers, and objective question answer analyses for each rewritten objective question stem as an objective question evaluation set of the corresponding task type.

[0078] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0079] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically generating a large language model vertical field multi-task objective question evaluation set, characterized in that: The method comprises: Obtain plain text datasets in vertical fields; For each text in the plain text data set, calling a problem solver to judge the suitability of the text for a task according to a plurality of preset task types, and generating at least one initial objective question stem of the task type according to the text when the judgment result is suitable; Performing ambiguity detection on each of the initial objective question stems, and calling the problem solver to rewrite the ambiguous initial objective question stems to obtain rewritten objective question stems; The problem solver is called to generate corresponding objective question options, objective question answers and objective question answer analysis for each rewritten objective question stem as an objective question evaluation set for the corresponding task type.

2. The method for automatically generating a large language model vertical field multi-task objective question evaluation set according to claim 1 is characterized in that: The task types include: information extraction type, knowledge question and answer type, computational reasoning type, and task decomposition type. The method further includes: According to each task type, the problem solver is called to construct an evaluation set for the text to obtain an objective question evaluation set corresponding to each task type.

3. The method for automatically generating a large language model vertical field multi-task objective question evaluation set according to claim 1, characterized in that: The calling of the problem solver to judge the suitability of the text for the task includes: Obtaining a judgment prompt word corresponding to a preset task type, wherein the judgment prompt word includes a system prompt word and a task prompt word corresponding to the task type, wherein the system prompt word is used to instruct the problem solver to perform task suitability judgment on the text; The judgment prompt word is concatenated with the text, and the concatenated text is input into a problem solver to perform task suitability judgment to obtain a corresponding judgment result.

4. The method for automatically generating a large language model vertical field multi-task objective question evaluation set according to claim 1, characterized in that: The calling of the problem solver to rewrite the ambiguous initial objective question stem to obtain a rewritten objective question stem includes: Obtaining preset disambiguation prompt words, wherein the disambiguation prompt words include semantic disambiguation prompt words and disambiguation example texts, and the semantic disambiguation prompt words are used to instruct the problem solver to rewrite when it is detected that the initial objective question stem is ambiguous; splicing the ambiguity elimination prompt word, the text and the initial objective question stem to obtain an ambiguous spliced ​​text; The problem solver is called to generate the ambiguous concatenated text to obtain a rewritten objective question stem corresponding to the initial objective question stem.

5. The method for automatically generating a large language model vertical field multi-task objective question evaluation set according to claim 1, characterized in that: The calling of the question solver to generate corresponding objective question options, objective question answers and objective question answer analysis for each rewritten objective question stem includes: Obtaining a preset option generation prompt word, wherein the option generation prompt word is used to instruct the problem solver to rewrite the objective question stem according to the text to generate corresponding objective question options, and generate corresponding objective question answers and objective question answer analysis; The option generation prompt word, the text and the rewriting objective question stem are spliced ​​together to generate a spliced ​​text; The problem solver is called to generate the generated concatenated text to obtain corresponding objective question options, objective question answers and objective question answer analysis.

6. The method for automatically generating a large language model vertical field multi-task objective question evaluation set according to claim 5, characterized in that: The method further comprises: The problem solver is called to filter the objective question options and the objective question answers, and the filtering process of the problem solver includes: Obtaining a preset filtering prompt word, wherein the filtering prompt word includes a filtering condition, and the filtering prompt word is used to prompt the problem solver to filter according to the filtering condition; The filtering prompt word, the text, the rewritten objective question stem, the objective question options and the objective question answer are spliced ​​together to obtain a filtered spliced ​​text; The problem solver is called to filter the filtered spliced ​​text according to the filtering condition to obtain the filtered rewritten objective question stem and the corresponding objective question options and objective question answers.

7. The method for automatically generating a large language model vertical field multi-task objective question evaluation set according to claim 6, characterized in that: The filtering conditions include common filtering conditions for multiple task types and specific filtering conditions for a single task type; The general filtering condition includes at least one of the following: Rewrite the objective test stem to make it unsuitable for objective test; The answer to the objective test is not the correct answer to the objective test question stem; There are multiple correct answers for the objective questions, and the answer to the objective question is one of the correct answers; In the objective question options, wrong answers other than the objective question answers are taken as correct answers; The answers to objective questions do not come from rewriting the objective question stems.

8. A device for automatically generating a large language model vertical field multi-task objective question evaluation set, characterized in that: The device comprises: The acquisition module is used to obtain plain text datasets in vertical fields; A discrimination module, for each text in the plain text data set, calling a problem solver to discriminate the suitability of the text for a task according to a plurality of preset task types, and generating at least one initial objective question stem of the task type according to the text when the discrimination result is suitable; A detection module, configured to perform ambiguity detection on each of the initial objective question stems, and call the problem solver to rewrite the ambiguous initial objective question stems to obtain rewritten objective question stems; The generation module calls the problem solver to generate corresponding objective question options, objective question answers and objective question answer analysis for each rewritten objective question stem as an objective question evaluation set for the corresponding task type.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the method for automatically generating a large language model vertical field multi-task objective question evaluation set as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for automatically generating a large language model vertical field multi-task objective question evaluation set as described in any one of claims 1 to 7 is implemented.

Images