Vertical field objective test data set construction method, system, equipment and medium

Abstract

The invention discloses a vertical field objective test data set construction method, system and device and a medium, and relates to the technical field of natural language processing, the construction method comprises the following steps: obtaining text data, and carrying out knowledge extraction and arrangement on the text data to obtain formatted text information; taking the formatted text information as input, and performing question generation, correct answer generation, interference option generation and analysis generation through a pre-established large model to obtain test data; performing quality detection on the obtained test data, and screening out the test data meeting the quality detection requirements; and performing self-inspection on the test data meeting the quality detection requirements, and retaining the test data passing the self-inspection to obtain a vertical field objective test data set. According to the method, the vertical domain test questions can be automatically generated on a large scale, the dependency degree of testers on professional domain knowledge is reduced, meanwhile, the labor cost is saved, the problem difficulty is controllable, and the professionality and accuracy of test question generation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and particularly relates to a method, system, device and medium for constructing an objective test data set in a vertical domain. Background Art

[0002] With the rapid development of artificial intelligence and machine learning technologies, the application requirements for natural language processing in various vertical domains (such as finance, healthcare, law, etc.) are increasing continuously. However, different vertical domains have unique language features and knowledge systems, and general models often fail to achieve ideal performance. Therefore, model training and testing for specific domains have become crucial. Especially for large-scale pre-trained models, their adaptability and performance evaluation in vertical domains rely on high-quality test sets.

[0003] In the prior art, Zhang L et al. proposed the test set FinEval in the financial field (Zhang L, Cai W, Liu Z, et al. Fineval: A chinese financial domain knowledge evaluation benchmark for large language models[J]. arXiv preprint arXiv:2308.09975, 2023.), and Wang X et al. proposed the test set CMB in the medical field (Wang X, Chen G, Dingjie S, et al. CMB: A Comprehensive Medical Benchmark in Chinese[C] / / Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024: 6184-6205.). These test sets in vertical domains directly extract question-answer pairs from publicly available industry exam questions or construct test sets from large-scale raw data manually (Liu X, Lei X, Wang S, et al. Alignbench: Benchmarking chinese alignment of large language models[J]. arXiv preprint arXiv:2311.18743, 2023.). The construction method of this kind of test set highly depends on the quantity and quality of open-source datasets, and the labor cost is relatively high. In some vertical domains with relatively few open-source data, such as the power field, its practicability is not strong. Zhu K et al. proposed RAGEval (Zhu K, Luo Y, Xu D, et al. RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework[J]. CoRR, 2024.), which can construct retrieval-enhanced test datasets through large models in the financial, medical, and legal fields. However, it not only requires constructing schema examples for each field, which requires a high labor cost, but also the quality of the constructed test datasets depends on the model's understanding of the knowledge in that field, lacking an evaluation of the professionalism and accuracy of the test data. Summary of the Invention

[0004] The object of the present invention is to provide a method, system, device and medium for constructing an objective test data set in a vertical field, which can automatically generate a large number of test questions in the vertical field, reduce the dependence of testers on professional field knowledge, save labor costs, and control the difficulty of questions, so as to improve the professionalism and accuracy of the generated test questions in view of the problems in the above-mentioned prior art.

[0005] In order to achieve the above object, the present invention has the following technical solutions: In the first aspect, a method for constructing an objective test data set in a vertical field is provided, including: Obtain text data, perform knowledge extraction on the text data, and organize and obtain formatted text information; Use the formatted text information as input, and generate test data through a pre-established large model for question generation, correct answer generation, distracter generation and parsing generation; Perform quality detection on the obtained test data, and screen out the test data that meets the quality detection requirements; Perform self-check on the test data that meets the quality detection requirements, and retain the test data that passes the self-check to obtain an objective test data set in the vertical field.

[0006] As a preferred solution, in the step of obtaining text data, non-formatted different source data is converted into text information through optical character recognition (OCR) or data extraction tools, and the extracted data is cleaned to convert the extracted data into text data that meets the quality requirements.

[0007] As a preferred solution, the step of performing knowledge extraction on the text data and organizing and obtaining formatted text information includes: Segment the text data according to chapters and paragraphs. For each chapter segment obtained by segmentation, use a pre-established large model to generate a corresponding abstract for each chapter; The input of the large model is the title of the corresponding chapter segment, the upper-level title, all titles between the upper-level title and the data big title, and the entire chapter content. After the large model generates the abstract, key knowledge segments in the chapter content are extracted through the same large model according to the abstract and the chapter content; the chapter content, abstract, and key knowledge segments are organized to obtain processed formatted text information.

[0008] As a preferred solution, in the steps of generating questions, correct answers, distractors, and answer analyses through a pre-established large model, a number of seed documents and corresponding questions are given as question generation examples. The large model learns the question generation logic and generation methods for vertical domain scenarios based on the question generation examples, and continuously iteratively optimizes the seed documents and corresponding questions in subsequent generations; according to the processed formatted text information and question generation examples, the large model is used to generate a number of relevant questions. After obtaining the generated questions, the large model generates corresponding correct answers based on the formatted text information and the generated questions; In the process of distractor generation, the question and the correct answer are concatenated and fed into an encoding model for encoding to obtain a question-answer vector All the obtained key knowledge fragments are encoded to obtain a knowledge vector library , calculate and the similarity of all elements in the knowledge vector library to obtain the key knowledge fragments most similar to the question-answer:

[0009]

[0010]

[0011] In the formula, is the encoding model, represents concatenating the question and the answer, represents the question and the knowledge fragment similarity; The key knowledge fragments most similar to the question-answer, the correct answer knowledge fragment, the question, the correct answer, and the content of the chapter to which the correct answer belongs are fed into the large model together to complete the generation of distractors; In the answer analysis generation stage, based on the key knowledge fragments most similar to the question-answer, the correct answer knowledge fragment, the question, the correct answer, and the content of the chapter to which the correct answer belongs, generate answer analyses for the distractors.

[0012] As a preferred solution, the answer analysis is generated in the way of a chain of thought, including: First, analyze the specific aspects examined by the question, then analyze the differences between the knowledge fragments corresponding to the distractors and the knowledge fragments corresponding to the correct answer one by one, answer the reasons why the distractors are incorrect, and finally analyze the reasons why the correct answer is correct.

[0013] As a preferred solution, in the step of performing quality inspection on the obtained test data and screening out the test data that meets the quality inspection requirements, the question, correct answer, interference options, and relevant knowledge fragments and chapter content are input into the large model, and the evaluation is carried out from the following five criteria: clarity, difficulty, completeness, generalization, and material relevance; The large model classifies each generated test question according to the above criteria and gives the result of whether the overall evaluation is reasonable; if the overall evaluation of the model is reasonable, the corresponding data is retained; if the overall evaluation is unreasonable, the corresponding question and options are discarded.

[0014] As a preferred solution, in the step of performing self-inspection on the test data that meets the quality inspection requirements, retaining the test data that passes the self-inspection, and obtaining the vertical domain objective test data set, the self-inspection of the test data is completed by two methods: retrieval-based answering and non-retrieval-based answering; The non-retrieval-based answering adopts the method of directly answering by the large model. The question and each option are sent into the pre-established large model, and the large model directly gives the answer; during the answering process, the options are randomly shuffled, and the large model is allowed to answer several times. If the number of correct answers of the large model is not less than the set number of times, it is judged that the large model answers correctly, otherwise it is judged as answering wrongly; for the questions with wrong answers, first initialize an error pool, screen out some questions lacking answerability and the corresponding reasons and put them into the error pool, then send the questions with wrong answers into the large model to judge whether the large model can answer, and at the same time randomly select several examples from the error pool to assist the large model in answering. If the large model judges that the question cannot be answered, the corresponding question and reason are put into the error pool, and the corresponding data is discarded; The retrieval-based answering adopts the form of retrieval-augmented generation (RAG), allowing the large model to answer the question in combination with the material. The question, the content of the chapter where the question is located, and the content of the chapter where the key knowledge fragments most similar to the question-answer are sent into the large model, and the large model is allowed to answer, requiring the large model to correctly answer the answer and give the specific location of the material required to answer the answer; if the large model can correctly answer the answer and correctly find the specific location of the material corresponding to the answer, the corresponding data is retained, otherwise it is discarded.

[0015] In the second aspect, a vertical domain objective test data set construction system is provided, including: A knowledge extraction module, used to obtain text data, perform knowledge extraction on the text data, and organize and obtain formatted text information; A test data acquisition module, used to take the formatted text information as input, and generate questions, correct answers, interference options, and parsing through a pre-established large model to obtain test data; The test data quality detection module is used to detect the quality of the obtained test data and screen out the test data that meets the quality detection requirements; The test data self - inspection module is used to self - inspect the test data that meets the quality detection requirements, retain the test data that passes the self - inspection, and obtain the vertical domain objective test data set.

[0016] As a preferred solution, the knowledge extraction module converts unformatted different - source data into text information, cleans the extracted data, and converts the extracted data into text data that meets the quality requirements; The text data is segmented according to chapters and paragraphs. For each chapter segment obtained by segmentation, a pre - established large model is used to generate a corresponding abstract for each chapter; The input of the large model is the title of the corresponding chapter segment, the upper - level title, all the titles between the upper - level title and the data big title, and the entire chapter content; after the large model generates the abstract, key knowledge segments in the chapter content are extracted through the same large model according to the abstract and the chapter content; the chapter content, abstract, and key knowledge segments are sorted out to obtain the processed formatted text information.

[0017] As a preferred solution, the test data acquisition module gives several seed documents and corresponding questions as question - generation examples. The large model learns the question - generation logic and generation method of the vertical - domain scenario based on the question - generation examples and continuously iteratively optimizes the seed documents and corresponding questions in subsequent generations; according to the processed formatted text information and question - generation examples, the large model generates several relevant questions. After obtaining the generated questions, the large model generates corresponding correct answers based on the formatted text information and the generated questions; During the generation of interference options, the question and the correct answer are concatenated and sent to the encoding model for encoding to obtain the question - answer vector , all the obtained key knowledge segments are encoded to obtain the knowledge vector library , calculate and all elements in the knowledge vector library to obtain the key knowledge segments that are most similar to the question - answer:

[0018]

[0019]

[0020] In the formula, is the encoding model, It means to splice the question and the answer. It represents the question. and the knowledge fragments similarity; The key knowledge fragments, correct answer knowledge fragments, question, correct answer, and content of the chapter to which the correct answer belongs that are most similar to the question-answer are sent to the large model to complete the generation of distractors; In the parsing and generation stage, based on the key knowledge fragments, correct answer knowledge fragments, question, correct answer, content of the chapter to which the correct answer belongs, and distractors that are most similar to the question-answer, answer analysis is generated.

[0021] As a preferred solution, the test data quality detection module inputs the question, correct answer, distractors, and related knowledge fragments and chapter content into the large model, and evaluates from the following five criteria: Clarity, difficulty, completeness, generalization, and material relevance; The large model classifies each generated test question according to the above criteria and gives the result of whether the overall evaluation is reasonable; if the overall evaluation of the model is reasonable, the corresponding data is retained; if the overall evaluation is unreasonable, the corresponding question and options are discarded.

[0022] As a preferred solution, the test data self-checking module completes the self-checking of the test data through two methods: retrieval-based answering and non-retrieval-based answering; For non-retrieval-based answering, the large model directly answers. The question and each option are sent into the pre-established large model, and the large model directly gives the answer; during the answering process, the options are randomly shuffled, and the large model is asked to answer several times. If the number of correct answers given by the large model is not less than the set number of times, it is judged that the large model answers correctly, otherwise it answers incorrectly; for the questions with incorrect answers, first initialize an error pool, screen out some questions lacking answerability and the corresponding reasons and put them into the error pool, then send the questions with incorrect answers into the large model to judge whether the large model can answer, and at the same time randomly select several examples from the error pool to assist the large model in answering. If the large model judges that the question cannot be answered, the corresponding question and reason are put into the error pool, and the corresponding data is discarded; For retrieval-based answering, it adopts the form of Retrieval-Augmented Generation (RAG). The large model is asked to answer the question in combination with the material. The question, the content of the chapter where the question is located, and the content of the chapter where the key knowledge fragments that are most similar to the question-answer are sent into the large model, and the large model is required to correctly answer the answer and give the specific location of the material required to answer the answer; if the large model can correctly answer the answer and can correctly find the specific location of the material corresponding to the answer, the corresponding data is retained, otherwise it is discarded.

[0023] In a third aspect, an electronic device is provided, including a processor and a memory. The processor is configured to execute a computer program stored in the memory to implement the method for constructing a vertical domain objective test data set as described above.

[0024] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the method for constructing a vertical domain objective test data set as described above is implemented.

[0025] Compared with the prior art, the first aspect of the present invention has at least the following beneficial effects: The method for constructing a vertical domain objective test data set of the present invention reduces the dependence of testers on professional domain knowledge and greatly saves labor costs. Starting from vertical domain data, the method of the present invention uses a large model to summarize and mine knowledge fragments in the data. By using the knowledge fragments and large model generation technology, the present invention constructs an automatic generation method. Through four steps of question generation, correct answer generation, distracter generation, and explanation generation, a large number of vertical domain test questions are automatically generated in batches, and low-quality test questions are automatically filtered through quality detection and self-checking. The construction of the test set can be automatically completed without human intervention. The method of the present invention divides the test question generation into four steps: question generation, correct answer generation, distracter generation, and explanation generation. It can not only generate high-quality standard questions and correct answers, but also proposes a method of selecting similar knowledge fragments to generate distracters by similarity matching, making the question difficulty more controllable and improving the professionalism of the generated test questions.

[0026] Furthermore, the present invention performs quality detection on the obtained test data, screens out the test data that meets the quality detection requirements, inputs questions, correct answers, distracters, and relevant knowledge fragments and chapter contents into the large model, and evaluates them from five criteria: clarity, difficulty, completeness, generalization, and material relevance, ensuring the reliability of the generated data.

[0027] Furthermore, the present invention performs self-checking on the test data that meets the quality detection requirements to ensure the professionalism and answerability of the generated data. After test question generation and quality detection, a batch of high-quality test data sets can be obtained, and then the self-checking of the test data is completed through two methods: retrieval-based answering and non-retrieval-based answering. For retrieval-based answering, the similarities and differences between the test questions and relevant documents are compared to evaluate the professionalism of the test questions; for non-retrieval-based answering, the answerability of the test questions is evaluated.

[0028] It can be understood that the beneficial effects of the above second to fourth aspects can refer to the relevant descriptions in the above first aspect and will not be elaborated here. Brief Description of the Drawings

[0029] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0030] Figure 1 It is a flowchart of the implementation of the method for constructing an objective test data set in the vertical field of the embodiments of the present invention; Figure 2 It is a flowchart of the implementation of mining knowledge fragments in text data in the embodiments of the present invention; Figure 3 It is a flowchart of the implementation of test data generation in the embodiments of the present invention; Figure 4 It is a block diagram of the structure of the system for constructing an objective test data set in the vertical field of the embodiments of the present invention; Figure 5 It is a schematic diagram of the physical structure of an electronic device in the embodiments of the present invention. Detailed Description of the Embodiments

[0031] In the following description, specific details such as specific system structures and technologies are proposed for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0032] Please refer to Figure 1 , the embodiments of the present invention propose a method for constructing an objective test data set in the vertical field, including the following steps: S1. Obtain text data, perform knowledge extraction on the text data, and organize and obtain formatted text information; S2. Use the formatted text information as input, and generate questions, correct answers, interference options, and parsing through a pre-established large model to obtain test data; S3. Perform quality inspection on the obtained test data, and screen out the test data that meets the quality inspection requirements; S4. Perform self-inspection on the test data that meets the quality inspection requirements, and retain the test data that passes the self-inspection to obtain an objective test data set in the vertical field.

[0033] In a possible implementation, high-quality basic data is obtained through step S1. The high-quality basic data refers to data such as books and standards in the corresponding field. The characteristic of this type of data is that it contains a large amount of domain knowledge. Vertical domain data is usually stored in PDF and WORD documents. Through optical character recognition (OCR) or data extraction tools, unformatted data from different sources is converted into text information. The advertisements, garbled characters, and illegal data in the data are cleaned, and finally the extracted data is converted into high-quality text data.

[0034] Furthermore, in step S1, knowledge extraction is performed on the obtained high-quality text data to remove irrelevant information in the data, increase the proportion of key information in the data, and make the attention weight of the model for useful information higher. As Figure 2 shown, specifically, in the embodiment of the present invention, the text data is segmented according to chapters and paragraphs. For each chapter segment obtained by segmentation, a pre-established large model is used to generate a corresponding abstract for each chapter. Among them, the input of the large model is the title of the corresponding chapter segment, the upper-level title, all the titles between the upper-level title and the data's main title, and the entire chapter content. After the large model generates the abstract, key knowledge segments in the chapter content are extracted through the same large model according to the abstract and the chapter content. Through step S1, redundant and complex text data is converted into highly condensed text abstracts and text knowledge segments, which helps the large model understand the overall chapter structure and content, and enables the large model to generate test questions through the key knowledge segments, improving the professionalism of the generated test data. Finally, the chapter content, abstract, and key knowledge segments are sorted out to obtain processed formatted text information.

[0035] In a possible implementation, the large model in the embodiment of the present invention selects the Qwen2.5-72B-Instruct model. Qwen2.5-72B-Instruct is a large language model developed by the Qwen team and is part of the Qwen2.5 series.

[0036] The Qwen2.5-72B-Instruct model has 72 billion parameters and is one of the flagship models in the Qwen2.5 series. Through instruction tuning, Qwen2.5-72B-Instruct demonstrates excellent performance on multiple core tasks. The Qwen2.5-72B-Instruct model can be used in combination with other models in the Qwen2.5 series (such as multimodal models) to achieve more complex cross-modal tasks. The Qwen2.5-72B-Instruct model is pre-trained on 18T tokens of data, and its overall performance has improved by more than 18% compared to the previous Qwen2 model. Qwen2.5-72B-Instruct performs well in multiple authoritative evaluations. Especially in mathematics (MATH) and coding (MBPP) tasks, the model demonstrates outstanding performance. After instruction tuning, the model has stronger adaptability to the diversity of system prompts, can quickly respond to diverse system prompts, and realize tasks such as role-playing and chatbots. The Qwen2.5-72B-Instruct model has learned a large amount of knowledge during the pre-training stage, enabling it to demonstrate a wide range of knowledge reserves when answering various questions. The model supports more than 29 languages, including Chinese, English, French, Spanish, Russian, Japanese, Vietnamese, Arabic, etc., achieving global language processing capabilities. At the same time, the Qwen2.5-72B-Instruct model supports a context length of up to 128K and can generate up to 8K content, giving it a significant advantage in processing long text tasks. The model can provide effective solutions in different application scenarios, demonstrating excellent performance whether in resource-constrained environments or in tasks requiring high-performance computing.

[0037] In a possible implementation, please refer to Figure 3, in step S2 of the embodiment of the present invention, the processed formatted text information is used as input, and a large model is used to generate a number of relevant questions. During the question generation process, the quality of the model generation is constrained from three dimensions through Prompt Engineering: 1) The questions should be complete, providing complete background information without ambiguity. 2) The questions should have a certain degree of professionalism, ensuring that the questions are asked based on the key knowledge points in the data to avoid overly simple or generalized questions such as "Question: After the contract comes into effect, what must the owner and the contractor perform the contract according to to achieve the legal consequences they pursue? Answer: Regulations". 3) The questions should be rigorous without factual errors. The present invention uses few-shot learning for generation. During the question generation process, several seed documents and corresponding questions are given as generation examples, and the model will familiarize itself with the question generation logic and generation methods of the vertical domain scenarios based on these examples. And in subsequent generations, the seed documents and question examples are continuously iteratively optimized using high-quality generated data to ensure that the questions generated by the model are both comprehensive and have sufficient diversity to support question generation in the vertical domain.

[0038] After obtaining the generated questions, the large model generates corresponding correct answers based on the formatted text information and the generated questions. During the generation process, the model is constrained by Prompt Engineering to generate correct and unique answers.

[0039] During the generation of distraction options, it is necessary to ensure that the generated distraction options are reasonable, relevant to the question, and have a certain degree of difficulty and confusion. The question and the correct answer are concatenated and sent to the encoding model for encoding to obtain the question-answer vector , and all the obtained key knowledge fragments are encoded to obtain the knowledge vector library , calculate and all elements in the knowledge vector library to obtain the key knowledge fragments most similar to the question-answer:

[0040]

[0041]

[0042] In the formula, is the encoding model, represents concatenating the question and the answer, represents the question and the knowledge fragment similarity; The Send the key knowledge fragments, correct answer knowledge fragments, questions, correct answers, and the content of the chapter to which the correct answer belongs into the large model to generate distractors.

[0043] In the analysis generation stage, according to the key knowledge fragments, correct answer knowledge fragments, questions, correct answers, the content of the chapter to which the correct answer belongs, and the distractors, generate answer analysis.

[0044] The answer analysis in the embodiments of the present invention is generated in the way of chain of thought, including: First, analyze the specific aspects examined by the question, then analyze the differences between the knowledge fragments corresponding to the distractors and the knowledge fragments corresponding to the correct answer one by one, answer the reasons why the distractors are incorrect, and finally analyze the reasons why the correct answer is correct.

[0045] Finally, the test data examples generated in the embodiments of the present invention are shown in Table 1.

[0046] Table 1

[0047] In a possible implementation manner, step S3 mainly evaluates aspects such as the rationality and generalization of the generated questions and answers, and judges whether the questions and options are reasonable and suitable for answering from the perspective of the question setter. Specifically, let the Qwen2.5-72B-Instruct model act as an expert in the power field, with the input being questions, correct answers, distractors, and relevant knowledge fragments and chapter content, and evaluate from the following five criteria: 1) Clarity: Evaluate whether the question is clear and unambiguous, without confusing language or vague terms. 2) Difficulty: Evaluate whether the options have a certain degree of difficulty and are somewhat confusing. 3) Completeness: Whether the question is complete and self-sufficient. 4) Generalization: Whether the question and answer are generalizable, without referential questions and answers such as "this article" and "this material". 5) Material relevance: Judge whether the question and answer are derived from the given material and whether the logic is accurate. The model will classify each generated test question according to the above criteria and finally give an overall evaluation of reasonable or unreasonable. If the overall evaluation of the model is reasonable, the data of this item will be retained; if the overall evaluation is unreasonable, the question and options will be directly discarded.

[0048] In a possible implementation, a batch of high-quality test data has been obtained through steps S1 to S3, and good assurance has been achieved in terms of the rationality of questions and answers. However, for some questions, they strongly rely on the given materials and lack answerability when separated from the materials. For example, for the question "What is the direct cause of the AC power failure fault at the Ganqika relay station?", since no specific power failure accident is given, it cannot be directly answered. For another part of the questions, the questions themselves are reasonable, but neither the correct answers nor the interference options can fully answer the questions, lacking professionalism and accuracy. For these two types of questions, step S4 in the embodiments of the present invention proposes two post-checking methods, namely retrieval-based answering and non-retrieval-based answering, to complete the self-checking of the test data.

[0049] Non-retrieval-based answering adopts the method of directly answering by the model. The question and each option are directly sent into the Qwen2.5-72B-Instruct model, and the model is made to directly give answers in the few-shot manner. During the answering process, the options will be randomly shuffled, and the model is made to answer 5 times to avoid the deviation caused by randomness during the test. If the number of times the model answers correctly is greater than or equal to 3 times, it is determined that the model answers correctly; otherwise, it answers wrongly. For the questions answered wrongly, first, an error pool is initialized, and some questions lacking answerability and their reasons are screened out through manual screening and put into the error pool. Then, the wrongly answered questions are sent into the large model to determine whether the model can answer them. At the same time, several examples are randomly selected from the error pool to assist the model in answering. If the model determines that the question cannot be answered, the question and the reason are put into the error pool, and this piece of data is discarded.

[0050] Retrieval-based answering adopts the form of Retrieval-Augmented Generation (RAG), making the model answer the question in combination with the materials. RAG is a natural language processing model that combines information retrieval and generation technologies. This technology introduces an external knowledge base, uses the retrieval module to extract relevant information from a large number of documents, and then passes this information to the generation module to generate more accurate and useful answers or texts. Specifically, the question, the content of the chapter where the question is located, and the content of the chapter where the knowledge fragments similar to it are sent into the large model to make the model answer. To ensure the professionalism and accuracy of the questions and answers, it is necessary to make the model correctly answer the answers and give the specific location of the materials required to answer the answers. If the model can correctly answer the answers and correctly find the specific location of the materials corresponding to the answers, this piece of data is retained; otherwise, it is discarded. In this way, the accuracy of the questions and answers and the uniqueness of the answers are ensured, and the quality of the generated data can be continuously improved.

[0051] The method for constructing an objective test data set in a vertical domain proposed by the present invention can be used for generating multiple-choice questions for tests in various vertical domains. Starting from data such as relevant field books and standards, valid knowledge fragments are extracted from the data, and the first batch of test questions are obtained through question generation, answer generation, distracter generation, and analysis generation. Then, a high-quality test data set is filtered out through quality inspection and post-inspection. The generated test data can be used in multiple test scenarios such as large model testing.

[0052] Please refer to Figure 4 , another embodiment of the present invention further proposes a system for constructing an objective test data set in a vertical domain, including: A knowledge extraction module 101, configured to obtain text data, perform knowledge extraction on the text data, and organize and obtain formatted text information; A test data acquisition module 102, configured to use the formatted text information as input, and generate questions, correct answers, distracters, and analysis through a pre-established large model to obtain test data; A test data quality inspection module 103, configured to perform quality inspection on the obtained test data, and screen out the test data that meets the quality inspection requirements; A test data self-inspection module 104, configured to perform self-inspection on the test data that meets the quality inspection requirements, retain the test data that passes the self-inspection, and obtain an objective test data set in a vertical domain.

[0053] In a possible implementation manner, the knowledge extraction module 101 converts non-formatted data from different sources into text information, performs data cleaning on the extracted data, and converts the extracted data into text data that meets the quality requirements; The text data is segmented according to chapters and paragraphs. For each chapter segment obtained by segmentation, a corresponding abstract is generated for each chapter using a pre-established large model; The input of the large model is the title of the corresponding chapter segment, the upper-level title, all titles between the upper-level title and the data major title, and the entire chapter content. After the large model generates an abstract, key knowledge fragments in the chapter content are extracted through the same large model according to the abstract and the chapter content; the chapter content, abstract, and key knowledge fragments are organized to obtain the processed formatted text information.

[0054] In a possible implementation, the test data acquisition module 102 provides several seed documents and corresponding questions as question generation examples. The large model learns the question generation logic and generation method for the vertical domain scenario based on the question generation examples, and continuously iteratively optimizes the seed documents and corresponding questions in subsequent generations. According to the processed formatted text information and question generation examples, the large model generates several relevant questions. After obtaining the generated questions, the large model generates corresponding correct answers based on the formatted text information and the generated questions. During the generation of distractors, the question and the correct answer are concatenated and fed into an encoding model for encoding to obtain a question-answer vector. All the obtained key knowledge fragments are encoded to obtain a knowledge vector library. Calculate and the similarity of all elements in the knowledge vector library to obtain the key knowledge fragments most similar to the question-answer:

[0055]

[0056]

[0057] In the formula, is the encoding model, represents concatenating the question and the answer, represents the question and the knowledge fragment similarity; The key knowledge fragments most similar to the question-answer, the correct answer knowledge fragment, the question, the correct answer, and the content of the chapter to which the correct answer belongs are fed into the large model together to complete the generation of distractors. In the parsing and generation stage, based on the key knowledge fragments most similar to the question-answer, the correct answer knowledge fragment, the question, the correct answer, and the content of the chapter to which the correct answer belongs, generate answer analysis for the distractors.

[0058] In a possible implementation, the test data quality detection module 103 inputs questions, correct answers, distractors, and relevant knowledge fragments and chapter content into the large model, and evaluates from the following five criteria: Clarity, difficulty, completeness, generalization, and material relevance; The large model classifies each generated test question according to the said criteria and gives the result of whether the overall evaluation is reasonable; if the overall evaluation of the model is reasonable, the corresponding data is retained; if the overall evaluation is unreasonable, the corresponding questions and options are discarded.

[0059] In a possible implementation manner, the test data self-checking module 104 completes the self-checking of the test data through two methods: retrieval-based answering and non-retrieval-based answering; For non-retrieval-based answering, the large model directly answers. The questions and each option are sent into the pre-established large model, and the large model directly gives the answers; during the answering process, the options are randomly shuffled, and the large model is asked to answer several times. If the number of correct answers of the large model is not less than the set number of times, it is judged that the large model answers correctly, otherwise it answers incorrectly; for the questions with incorrect answers, first initialize an error pool, screen out some questions lacking answerability and the corresponding reasons and put them into the error pool, then send the questions with incorrect answers into the large model to judge whether the large model can answer, and at the same time randomly select several examples from the error pool to assist the large model in answering. If the large model judges that the question cannot be answered, the corresponding question and reason are put into the error pool, and the corresponding data is discarded; For retrieval-based answering, it adopts the form of Retrieval-Augmented Generation (RAG), allowing the large model to answer the questions in combination with the materials. The questions, the content of the chapter where the questions are located, and the content of the chapter where the most similar key knowledge fragments to the question-answer are sent into the large model, and the large model is required to correctly answer the answers and give the specific location of the materials required to answer the answers; if the large model can correctly answer the answers and correctly find the specific location of the materials corresponding to the answers, the corresponding data is retained, otherwise it is discarded.

[0060] Figure 5 Illustrates a schematic diagram of the physical structure of an electronic device, such as Figure 5As shown in the figure, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540. Among them, the processor 510, the communications interface 520, and the memory 530 complete communication with each other through the communication bus 540. The processor 510 may call the logical instructions in the memory 530 to execute the method for constructing an objective test data set in a vertical domain. The method includes obtaining text data, performing knowledge extraction on the text data, and sorting to obtain formatted text information; using the formatted text information as input, generating questions, correct answers, interference options, and parsing through a pre-established large model to obtain test data; performing quality detection on the obtained test data, screening out the test data that meets the quality detection requirements; performing self-check on the test data that meets the quality detection requirements, and retaining the test data that passes the self-check to obtain an objective test data set in a vertical domain.

[0061] In addition, when the logical instructions in the above-mentioned memory 530 are implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium.

[0062] Another embodiment of the present invention also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method for constructing an objective test data set in a vertical domain provided in the above embodiment.

[0063] An embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores at least one instruction. When the at least one instruction is executed by a processor, the method for constructing an objective test data set in a vertical domain is implemented.

[0064] The computer program includes computer program code, which may be in the form of source code, object code, executable files, or some intermediate form, etc. The computer-readable storage medium may include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice within the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals. For the sake of convenience of explanation, only the part related to the embodiments of the present invention is shown above. For the specific technical details not disclosed, please refer to the method part of the embodiments of the present invention. This computer-readable storage medium is non-transitory and can be stored in a storage device formed by various electronic devices, and can implement the execution process recorded in the method of the embodiments of the present invention.

[0065] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0067] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0068] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable apparatus to generate a computer-implemented process, thereby providing instructions for implementing the steps of the function specified in one process or multiple processes and / or blocks Figure 1 one process or multiple processes and / or blocks Figure 1 or steps of the function specified in multiple blocks.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: it is still possible to modify the specific implementation manners of the present invention or make equivalent replacements, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. A method for constructing an objective test dataset in a vertical field, characterized in that: include: Obtain text data, extract knowledge from the text data, and organize it to obtain formatted text information; Taking formatted text information as input, the pre-established large model is used to generate questions, correct answers, distractor options, and analysis to obtain test data; Perform quality inspection on the obtained test data and select the test data that meets the quality inspection requirements; Conduct self-inspection on the test data that meets the quality inspection requirements, retain the test data that passes the self-inspection, and obtain an objective test data set in the vertical field.

2. The method for constructing a vertical field objective test dataset according to claim 1, characterized in that: The step of acquiring text data converts the non-formatted different source data into text information through optical character recognition (OCR) or data extraction tools, cleans the extracted data, and converts the extracted data into text data that meets the quality requirements.

3. The method for constructing a vertical field objective test dataset according to claim 1, characterized in that: The step of extracting knowledge from text data and arranging to obtain formatted text information includes: The text data is segmented into chapters and paragraphs. For each segmented chapter, a corresponding summary is generated for each chapter using a pre-established large model. The input of the big model is the corresponding chapter fragment title, the previous level title, all titles between the previous level title and the data big title, and the entire chapter content; after the big model generates the summary, the key knowledge fragments in the chapter content are extracted through the same big model based on the summary and chapter content; the chapter content, summary, and key knowledge fragments are sorted to obtain processed formatted text information.

4. The method for constructing a vertical field objective test dataset according to claim 3, characterized in that: In the steps of generating questions, generating correct answers, generating interference options, and generating analysis through the pre-established big model, a number of seed documents and corresponding questions are given as question generation examples. The big model learns the question generation logic and generation method of vertical field scenarios based on the question generation examples, and continuously iteratively optimizes the seed documents and corresponding questions in subsequent generation; based on the processed formatted text information and question generation examples, the big model is used to generate a number of related questions. After obtaining the generated questions, the big model generates the corresponding correct answers based on the formatted text information and the generated questions; In the process of generating distractor options, the question and the correct answer are concatenated and sent to the encoding model for encoding to obtain the question-answer vector , all the key knowledge fragments obtained Encode and obtain the knowledge vector library ,calculate and knowledge vector library The similarity of all elements in , and get the most similar question-answer Key knowledge snippets: In the formula, For the encoding model, Indicates concatenating the question and the answer. Representation problem and knowledge fragments similarity; The most similar question-answer The key knowledge fragments, the correct answer knowledge fragments, the questions, the correct answers, and the chapter contents to which the correct answers belong are all sent to the big model to complete the generation of interference options; In the analysis generation phase, based on the most similar question-answer The answer analysis is generated based on the key knowledge fragments, correct answer knowledge fragments, questions, correct answers, content of the chapter to which the correct answers belong, and interference options.

5. The method for constructing a vertical field objective test dataset according to claim 4, characterized in that: The answer analysis is generated by using a chain of thoughts, including: First, analyze the specific aspects of the question, then analyze the differences between the knowledge fragments corresponding to the interference options and the knowledge fragments corresponding to the correct answer one by one, answer why the interference options are wrong, and finally analyze why the correct answer is correct.

6. The method for constructing a vertical field objective test dataset according to claim 1, characterized in that: In the step of performing quality inspection on the obtained test data and screening out the test data that meets the quality inspection requirements, the questions, correct answers and interference options as well as relevant knowledge fragments and chapter contents are input into the large model and evaluated based on the following five criteria: clarity, difficulty, completeness, generalizability, and relevance of the material; The large model classifies each generated test question according to the standard and gives a result on whether the overall evaluation is reasonable; if the overall evaluation of the model is reasonable, the corresponding data is retained; if the overall evaluation is unreasonable, the corresponding questions and options are discarded.

7. The method for constructing a vertical field objective test dataset according to claim 1, characterized in that: The step of self-checking the test data that meets the quality inspection requirements, retaining the test data that has passed the self-check, and obtaining an objective test data set in a vertical field, and completing the self-check of the test data through both search-based answering and non-search-based answering methods; Non-retrieval-based answering uses a large model to directly answer questions. Questions and options are sent to a pre-established large model, and the large model directly gives answers. During the answering process, the options are randomly shuffled and the large model is asked to answer several times. If the large model answers correctly no less than the set number of times, the large model is judged to have answered correctly, otherwise the answer is wrong. For questions that are answered incorrectly, an error pool is first initialized, and some questions that lack answerability and the corresponding reasons are screened out and put into the error pool. Then, the questions that are answered incorrectly are sent to the large model to determine whether the large model can answer them. At the same time, several examples are randomly selected from the error pool to assist the large model in answering. If the large model determines that the question is unanswerable, the corresponding question and reason are put into the error pool and the corresponding data is discarded. Retrieval-based answering uses the form of retrieval enhancement generation RAG, allowing the large model to answer questions based on the materials, and summarizing the questions, the content of the chapter where the questions are located, and the most similar questions and answers. The chapter content where the key knowledge fragment is located is sent to the big model, and the big model is asked to answer the question. The big model is required to answer the answer correctly and give the specific location of the materials needed to answer the answer. If the big model can answer the answer correctly and can correctly find the specific location of the materials corresponding to the answer, the corresponding data will be retained, otherwise it will be discarded.

8. A vertical field objective test data set construction system, characterized by: include: The knowledge extraction module is used to obtain text data, extract knowledge from the text data, and organize it to obtain formatted text information; A test data acquisition module is used to take formatted text information as input, generate questions, generate correct answers, generate interference options, and generate analysis through a pre-established large model to obtain test data; The test data quality detection module is used to perform quality detection on the obtained test data and select the test data that meets the quality detection requirements; The test data self-inspection module is used to perform self-inspection on the test data that meets the quality inspection requirements, retain the test data that passes the self-inspection, and obtain an objective test data set in the vertical field.

9. The vertical field objective test data set construction system according to claim 8, characterized in that: The knowledge extraction module converts the unformatted different source data into text information, performs data cleaning on the extracted data, and converts the extracted data into text data that meets the quality requirements; The text data is segmented into chapters and paragraphs. For each segmented chapter, a corresponding summary is generated for each chapter using a pre-established large model. The input of the big model is the corresponding chapter fragment title, the previous level title, all titles between the previous level title and the data big title, and the entire chapter content; after the big model generates the summary, the key knowledge fragments in the chapter content are extracted through the same big model based on the summary and chapter content; the chapter content, summary, and key knowledge fragments are sorted to obtain processed formatted text information.

10. The vertical field objective test data set construction system according to claim 9, characterized in that: The test data acquisition module is given several seed documents and corresponding questions as question generation examples. The big model learns the question generation logic and generation method of the vertical field scenario based on the question generation examples, and continuously iterates and optimizes the seed documents and corresponding questions in subsequent generation; according to the processed formatted text information and question generation examples, the big model is used to generate several related questions. After obtaining the generated questions, the big model generates the corresponding correct answers based on the formatted text information and the generated questions; In the process of generating distractor options, the question and the correct answer are concatenated and sent to the encoding model for encoding to obtain the question-answer vector , all the key knowledge fragments obtained Encode and obtain the knowledge vector library ,calculate and knowledge vector library The similarity of all elements in , and get the most similar question-answer Key knowledge snippets: In the formula, For the encoding model, Indicates concatenating the question and the answer. Representation problem and knowledge fragments similarity; The most similar question-answer The key knowledge fragments, the correct answer knowledge fragments, the questions, the correct answers, and the chapter contents to which the correct answers belong are all sent to the big model to complete the generation of interference options; In the analysis generation phase, based on the most similar question-answer The answer analysis is generated based on the key knowledge fragments, correct answer knowledge fragments, questions, correct answers, content of the chapter to which the correct answers belong, and interference options.

11. The vertical field objective test data set construction system according to claim 8, characterized in that: The test data quality detection module inputs questions, correct answers and distractor options as well as relevant knowledge fragments and chapter content into the large model, and evaluates them based on the following five criteria: clarity, difficulty, completeness, generalization, and material relevance; The large model classifies each generated test question according to the standard and gives a result on whether the overall evaluation is reasonable; if the overall evaluation of the model is reasonable, the corresponding data is retained; if the overall evaluation is unreasonable, the corresponding questions and options are discarded.

12. The vertical field objective test data set construction system according to claim 8, characterized in that: The test data self-checking module completes the self-checking of the test data by two methods: search-based answering and non-search-based answering; Non-retrieval-based answering uses a large model to directly answer questions. Questions and options are sent to a pre-established large model, and the large model directly gives answers. During the answering process, the options are randomly shuffled and the large model is asked to answer several times. If the large model answers correctly no less than the set number of times, the large model is judged to have answered correctly, otherwise the answer is wrong. For questions that are answered incorrectly, an error pool is first initialized, and some questions that lack answerability and the corresponding reasons are screened out and put into the error pool. Then, the questions that are answered incorrectly are sent to the large model to determine whether the large model can answer them. At the same time, several examples are randomly selected from the error pool to assist the large model in answering. If the large model determines that the question is unanswerable, the corresponding question and reason are put into the error pool and the corresponding data is discarded. Retrieval-based answering uses the form of retrieval enhancement generation RAG, allowing the large model to answer questions based on the materials, and summarizing the questions, the content of the chapter where the questions are located, and the most similar questions and answers. The chapter content where the key knowledge fragment is located is sent to the big model, and the big model is asked to answer the question. The big model is required to answer the answer correctly and give the specific location of the materials needed to answer the answer. If the big model can answer the answer correctly and can correctly find the specific location of the materials corresponding to the answer, the corresponding data will be retained, otherwise it will be discarded.

13. An electronic device, characterized in that: It comprises a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the method for constructing a vertical field objective test data set as described in any one of claims 1 to 7.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by the processor, it implements the vertical field objective test data set construction method according to any one of claims 1 to 7.

Images