Open test question evaluation method and related device based on domain-based large model

Abstract

The present invention belongs to the field of educational examination data mining, and provides an open test question evaluation method and related devices based on a domain-based large model. The method includes: obtaining an incremental training data set and an instruction fine-tuning set through the subject corpus data, and obtaining a candidate answer set through text recognition; obtaining a domain model based on the incremental training data set and the instruction fine-tuning set; constructing a prompt template for the candidate viewpoint mining task, performing prompt fine-tuning on the obtained domain model, obtaining a fine-tuned domain model, and performing optimization; extracting the candidate's answer viewpoint, supplementing it with a standard answer viewpoint set, and obtaining a final standard answer viewpoint set; assigning points to the candidate's answer based on the final standard answer viewpoint set. The present invention achieves more accurate scoring while protecting innovation by constructing a reusable domain model, reducing sample requirements with the help of the semantic capabilities of the large model, and combining innovative viewpoint mining with expert feedback.

Description

Technical Field

[0001] The present invention relates to the field of educational examination data mining, and in particular to an open test question evaluation method based on a domain-based large model and related devices. Background Art

[0002] The evaluation of open-ended test answers aims to give a reasonable evaluation by analyzing and assessing students' answers to open-ended questions. In recent years, with the continuous advancement of educational informatization, the way of answering and evaluating test questions has gradually shifted towards automation and intelligence. In terms of the evaluation of open-ended test questions, the traditional manual grading method is inefficient and highly subjective, and is easily affected by the emotional bias and fatigue of the examiners. It is also easy to ignore the innovative answers of some students and even cause misjudgment. The existing open-ended test evaluation technology based on capturing simple text features for pattern matching has problems such as overly simple feature extraction, lack of semantic understanding, and neglect of innovative answers. Therefore, a new open-ended test evaluation technology is urgently needed. In response to the above problems, the present invention proposes an open-ended test answer evaluation method based on a domain-based large model.

[0003] The prior art CN108764074B proposes an intelligent grading method and system for subjective questions to automatically grade the test papers, which mainly includes: first, obtaining the answer sheet image and preprocessing it, dividing the answer areas for objective questions and subjective questions and identifying them; then reviewing them separately according to the question type. For objective questions, the answer content is compared with the standard answer to see if it matches. For subjective questions, text features are extracted to train the convolutional neural network so that it can output the scoring range for the subjective question answers; finally, manual intervention is performed for test papers with abnormal scores during the grading process.

[0004] The above-mentioned intelligent marking method for subjective questions based on deep learning relies on convolutional neural networks to process simple text structural features, ignores the semantic understanding of the text, has a weak understanding of the complex semantics, logical relationships and contextual dependencies in long texts, and inevitably has the risk of overfitting. It is unable to accurately analyze the coverage of viewpoints and capture the innovative views of candidates. Summary of the Invention

[0005] The purpose of the present invention is to provide an open-ended test question evaluation method and related devices based on a domain-based large model to solve the problems of weak understanding of the complex semantics, logical relationships and contextual dependencies in long texts, the inevitable risk of overfitting, and the inability to accurately analyze the coverage of viewpoints and capture the innovative viewpoints of candidates.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides an open-ended test question evaluation method based on a domain-based large model, comprising:

[0008] Obtain incremental training datasets and fine-tuning instruction sets from subject corpus data, as well as candidate answer sets. Based on the incremental training datasets and fine-tuning instruction sets, inject domain knowledge into the pre-set base model and align the intent of the instructions to obtain a domain model.

[0009] Based on the candidate answer set, a prompt template for the candidate opinion mining task is constructed. Based on the prompt template for the candidate opinion mining task, the obtained domain model is fine-tuned to obtain the fine-tuned domain model, and the fine-tuned domain model is tuned.

[0010] Extract the examinees' answer perspectives from the optimized domain model and classify them into a standard answer perspective set and an innovative perspective set. Then, supplement the standard answer perspective set with the innovative perspective set to obtain the final standard answer perspective set.

[0011] Candidates' answers are scored based on the final set of standard answer viewpoints.

[0012] Optionally, obtaining an incremental training dataset and an instruction fine-tuning dataset through subject corpus data, and obtaining a candidate's answer set, includes:

[0013] Textbooks, books, papers, and question banks of the subjects to which the open test questions belong are collected as corpus, low-quality data in the corpus is removed through rule-based data filtering methods, and document deduplication is performed to obtain an incremental training data set; an instruction fine-tuning set is generated through question and answer data in the question bank and a large model dialogue based on subject documents, and the instruction fine-tuning set includes a single-round dialogue set and a multi-round dialogue set; the candidate answer set is obtained by image segmentation and optical character recognition.

[0014] Optionally, the method of injecting domain knowledge into the preset base large model and aligning the instruction intent based on the incremental training data set and the instruction fine-tuning set to obtain the domain model includes:

[0015] Using the incremental pre-training set, the base large model is incrementally pre-trained to inject domain knowledge, and then the instruction fine-tuning dataset is used to fine-tune the incremental pre-trained model to align the intention of human instructions, thereby obtaining a discipline-oriented domain large model.

[0016] Optionally, constructing a prompt template for the examinee opinion mining task based on the examinee answer set, and fine-tuning the obtained domain model based on the prompt template for the examinee opinion mining task to obtain the fine-tuned domain model includes:

[0017] The prompt template format is defined based on the candidate's answer set, the prompt data is annotated according to the prompt template, and the low-rank adaptive efficient parameter fine-tuning domain model in the parameter efficient fine-tuning package is used to train the model on samples to adapt to the opinion mining task. The trained domain model is used to output the opinion information and explanation of the candidate's answer in the format defined by the template.

[0018] Optionally, the tuning operation on the fine-tuned domain model includes:

[0019] Based on the fine-tuned domain model, access the external data storage system to perform retrieval optimization on the model output.

[0020] Optionally, extracting examinees' answer perspectives from the tuned domain model, classifying them into a standard answer perspective set and an innovative perspective set, and supplementing the standard answer perspective set based on the innovative perspective set to obtain a final standard answer perspective set, including:

[0021] First, for each answer in the candidate's answer set, a prompt is filled in according to the opinion mining prompt template to obtain a prompt, and then the prompt is input into the optimized domain model to extract the candidate's answer opinion, and obtain the initial opinion set corresponding to all candidate's answers. Then, all opinions in the initial opinion set are de-duplicated, merged, and inferior to obtain the final opinion set. With reference to the preset recommended standard opinion set, opinions that do not appear in the recommended standard opinion set are constructed into an unverified innovative opinion set. Finally, the unverified innovative opinion set is reviewed and evaluated as correct innovative opinions. The correct innovative opinions are expanded to the standard answer opinion set as part of the subsequent grading criteria. The expanded opinion set is used as the final standard answer opinion set.

[0022] Optionally, the step of assigning scores to the examinee's answers based on the final set of standard answer viewpoints includes:

[0023] Based on the final set of standard answer opinions, after the examinee's answers are mined to obtain the examinee's opinion information, the opinion words are compared with the final set of standard answer opinions. Points are awarded if the comparison is successful, and no points are awarded if the comparison is unsuccessful.

[0024] In a second aspect, the present invention provides an open-ended test question evaluation system based on a domain-based large model, comprising:

[0025] The domain model construction module is used to obtain incremental training datasets and fine-tuning instruction sets from subject corpus data, and obtain candidate answer sets through text recognition. Based on the incremental training datasets and fine-tuning instruction sets, domain knowledge is injected and aligned with the intent of human instructions to obtain a domain model.

[0026] The domain model fine-tuning module is used to construct a prompt template for the candidate opinion mining task based on the candidate's answer set, fine-tune the prompts of the obtained domain model, obtain the fine-tuned domain model, and optimize the fine-tuned domain model;

[0027] The standard answer viewpoint acquisition module is used to extract the examinee's answer viewpoints from the tuned domain model, classify them into a standard answer viewpoint set and an innovative viewpoint set, and supplement the standard answer viewpoint set based on the innovative viewpoint set to obtain the final standard answer viewpoint set;

[0028] The scoring output module is used to assign scores to candidates' answers based on the final set of standard answer viewpoints.

[0029] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the open-ended test question evaluation method based on the domain-based large model when executing the computer program.

[0030] In this aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the open-ended test question evaluation method based on the domain-based large model is implemented.

[0031] Compared with the prior art, the present invention has the following technical effects:

[0032] The present invention realizes the evaluation of answers to open test questions through the steps of constructing a domain model, fine-tuning with a few sample prompts, mining answer viewpoints, expert feedback on innovative viewpoints, and automatic scoring. First, based on the open source basic large model, large-scale incremental pre-training is carried out using subject resources and other field knowledge, a multi-task template set is designed to fine-tune instructions, and a large domain model is constructed; secondly, a viewpoint mining and answer summary prompt learning template is designed to fine-tune the domain model with a few sample prompts; then, the viewpoint information of the answer is analyzed and summarized, and the emerging new viewpoints are submitted to experts for review; finally, based on the viewpoint set composed of new and old viewpoints, automatic scoring is performed with reference to the matching degree between the answer viewpoint and the viewpoint set. The present invention fully mines the semantic information of the answers to open test questions and simplifies it into the form of a viewpoint set, which is convenient for extracting and reviewing innovative viewpoints and provides more fine-grained automatic scoring at the semantic level. It has the advantages of sufficient semantic understanding and accurate automatic scoring, which makes the present invention have obvious advantages over other methods based on simple text feature pattern recognition.

[0033] The present invention learns the semantic features of a subject to construct a domain model, which can be reused in all subjects related to the subject after being constructed once, and is not limited to a specific question.

[0034] With the powerful semantic capabilities of the domain model, the present invention only needs to construct a small number of samples for prompt fine-tuning to meet the model construction conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a framework diagram of the open-ended test question answering evaluation method based on the domain-based large model of the present invention.

[0036] Figure 2 It is a flow chart of data collection and processing.

[0037] Figure 3 It is a flowchart for constructing a large domain model.

[0038] Figure 4 This is a flowchart for tuning large domain models.

[0039] Figure 5 It is a flow chart of opinion mining for answering open questions.

[0040] Figure 6 This is the automatic scoring flow chart.

[0041] Figure 7 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0042] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings and examples. It should be noted that the embodiments described herein are only intended to explain the present invention and are not intended to limit the present invention. In addition, the technical features of the embodiments of the present invention may be combined with each other unless there is a conflict.

[0043] Glossary:

[0044] OpenCV: Open Source Computer Vision Library (Open Source Computer Vision Library);

[0045] OCR: Optical Character Recognition;

[0046] Bert: Pre-trained language model (Bidirectional Encoder Representations from Transformers);

[0047] HTML: HyperText Markup Language;

[0048] SimHash: Locality-Sensitive Hashing;

[0049] Block: code block;

[0050] word2vec: word embedding technology;

[0051] PEFT: Parameter-Efficient Fine-Tuning;

[0052] Lora: A low-rank adaptation technique for efficient parameter tuning.

[0053] NLL: Negative Log-Likelihood loss.

[0054] Example 1, please refer to Figure 7 The present invention provides an open-ended test question evaluation method based on a domain-based large model, comprising:

[0055] The incremental training dataset and instruction fine-tuning set are obtained through subject corpus data, and the candidate answer set is obtained through text recognition. Based on the incremental training dataset and instruction fine-tuning set, domain knowledge is injected and aligned with the intent of human instructions to obtain a domain model.

[0056] Based on the candidate's answer set, a prompt template for the candidate's opinion mining task is constructed, the obtained domain model is fine-tuned with prompts, the fine-tuned domain model is obtained, and the fine-tuned domain model is optimized;

[0057] Extract the examinees' answer perspectives from the optimized domain model and classify them into a standard answer perspective set and an innovative perspective set. Then, supplement the standard answer perspective set with the innovative perspective set to obtain the final standard answer perspective set.

[0058] Candidates' answers are scored based on the final set of standard answer viewpoints.

[0059] The present invention fully exploits the semantic information of answers to open-ended questions and simplifies it into the form of a set of opinions, which not only facilitates the extraction and review of innovative opinions, but also provides more fine-grained automatic scoring at the semantic level. It has the advantages of sufficient semantic understanding and accurate automatic scoring, which makes the present invention have obvious advantages over other methods based on simple text feature pattern recognition.

[0060] Example 2, please refer to Figure 1 The present invention provides an open-ended test question evaluation method based on a domain-based large model, specifically comprising:

[0061] 1. Data collection and processing process:

[0062] The specific process of data collection and processing is as follows:

[0063] (1) To obtain and process the examinee’s answer information, we first use OpenCV’s image segmentation technology to extract the image area containing text information. Then, we use the OCR handwriting recognition model to perform text recognition on the segmented answer image to obtain preliminary recognition results. Finally, we use the Bert-based text error correction model to correct the OCR recognition results and obtain accurate examinee answer information.

[0064] (2) In order to acquire and process corpus data in the subject area, we systematically collect multi-source data such as open source question banks, educational resource libraries, e-book libraries, and academic resource libraries through web crawler technology. First, we conduct preliminary data screening based on the preset subject keyword tag system; second, we effectively identify and remove non-text information such as HTML tags and hyperlinks through the characteristics of key noise data; then, we comprehensively use quantitative indicators such as perplexity, punctuation distribution characteristics, and sentence length to evaluate and filter the text data quality; finally, we use the SimHash algorithm to calculate the semantic similarity between documents to achieve accurate deduplication of highly similar document pairs.

[0065] (3) For document type data expected in the subject area, such as e-books, academic resources, and educational resource libraries, use them as incremental pre-training datasets;

[0066] (4) For the question bank data and educational resource library in the corpus, the questions in the question bank data are used as instructions and the answers as responses to construct a single-round dialogue dataset. Then, a large model instruction template is used to generate multi-round dialogues based on the subject documents in the educational resource library. The large model instruction template is constructed as follows:

[0067] Based on the following document, please generate a multi-turn conversation. Role 1 and Role 2 can be a student and a tutor, a reader and an expert, etc. The goal is to help Role 1 understand the key concepts or viewpoints in the document and answer related questions.

[0068] Document Content: {Briefly summarize the document's topic, key messages, or main points};

[0069] Conversation starts: Role 1 (student): "{Generate a relevant question or inquiry about the content of the document}".

[0070] The above steps are as follows Figure 2 As shown, the incremental training dataset is obtained , instruction fine-tuning dataset ,in represents a single-round dialogue set, Represents a multi-round dialogue set and a candidate answer dataset .

[0071] in, Data on different document types expected in the subject area;

[0072] Questions and answers for a single round of conversation;

[0073] Questions and answers for multi-round conversations.

[0074] 2. The process of building a large domain model:

[0075] The construction of large domain models includes two processes: incremental pre-training and multi-task instruction fine-tuning.

[0076] The incremental pre-training process is as follows:

[0077] (1) All the corpus of the incremental training dataset D is concatenated into a continuous text data stream. The text is divided into multiple fixed-length text blocks according to the preset block size. The data in each block will be used for autoregressive training.

[0078] (2) We selected ChatGLM-6B, an open-source conversational language model from the Knowledge Engineering Laboratory of Tsinghua University, as the base model. We performed unsupervised autoregressive training on each text block, with the goal of predicting the character or word at the next position. The model generates the next character or word by learning information from the current context, and calculates the loss function based on the difference between the generated word and the actual word at the next position. Through multiple rounds of training, the loss function is continuously reduced to update the model parameter weights. The loss function is defined as follows:

[0079]

[0080] Where NLL is the negative log-likelihood loss, Smoothed is the smooth loss part. Let M be the loss normalization factor, N be the length of the input text sequence, and L be the total number of predicted labels. is the loss smoothing coefficient, is the one-hot encoded true label, For the model The predicted probability of the labels, the NLL loss and Smoothed loss are defined as follows:

[0081]

[0082] The multi-tasking instruction fine-tuning process is as follows:

[0083] (1) For a single-round dialogue in the instruction fine-tuning set I, the starting dialogue is taken as input x, and the model directly predicts the response, which is defined as , t represents the tth token of the reply. The difference between the model’s predicted reply and the target reply is measured using the cross-entropy loss function, which is defined as follows:

[0084]

[0085] in Indicates that the current model is predicting the sequence number of the token in the target response. Represents the target response The total number, P represents the model's response to the given input x and the generated response Under the condition of , the probability of the tth token is predicted, and the probability of the model generating the target response is maximized by continuously reducing the loss function.

[0086] (2) For multi-round dialogues in the instruction fine-tuning set I, the model needs to generate responses based on the historical dialogue information h and the current instruction x. Therefore, the loss function needs to take the historical dialogue information into account. Compared with the loss function in the case of a single-round dialogue, the multi-round dialogue instruction fine-tuning loss function is defined as follows:

[0087]

[0088] in Indicates that the current model is predicting the sequence number of the token in the target response. Represents the target response The total number, P represents the model's total number of responses given the historical dialogue h, the current instruction x, and the generated partial response. Under the condition of , the probability of the tth token is predicted, and the probability of the model generating the target response is maximized by continuously reducing the loss function.

[0089] The flow chart of the construction process of the large model in the above fields is as follows Figure 3 shown.

[0090] 3. Few-sample prompt fine-tuning process:

[0091] The prompt template for defining the opinion mining task is as follows:

[0092] You are a professional scoring assistant, responsible for extracting perspective information from the examinee's open-ended answers. Please read the following answers carefully and output the examinee's perspective and explanation in the format of "Perspective 1: Explanation 1; Perspective 2: Explanation 2; Perspective 3: Explanation 3."

[0093] Answer content: {Candidate's answer text}

[0094] Output format: View 1: Explanation 1; View 2: Explanation 2; View 3: Explanation 3. ”

[0095] By manually labeling a small amount (hundreds) of prompt data according to the prompt template, using the LoRa efficient parameter fine-tuning configuration in the PEFT package, and training on a small number of samples to adapt to the opinion mining task, the trained model can output the candidate's answer opinion information and explanation according to the format defined by the template.

[0096] 4. Domain Large Model Tuning Process:

[0097] The tuning of large domain models mainly assists in the generation of domain models by retrieving additional knowledge from external knowledge bases.

[0098] First, build an external knowledge base and vectorize it. Due to the existence of knowledge iteration and hallucination problems after domain model training, the knowledge base obtained in step 1 is updated and the external knowledge base is generated. Then, the knowledge base The text is divided into blocks , through the semantic representation model Bert, each text block is vectorized and stored in the vector database to obtain the external knowledge vector library .

[0099] Then, according to the user's instructions Vector Library After the user enters the instruction, the user instruction is vectorized to obtain , calculate its similarity with each text vector in the vector library , extract the one with the highest similarity score Segment text vector, restore it to text data .

[0100] Finally, the domain model is optimized based on the retrieved data. The retrieved text data is combined with the user instructions and provided to the domain model for subsequent generation.

[0101]

[0102] The flowchart of the large model tuning process in the above fields is as follows Figure 4 shown.

[0103] 5. Answer point mining process:

[0104] Each answer a in the candidate's answer set A is filled into the opinion mining prompt template and then input into the model.

[0105] Get the complete set of opinions corresponding to the answers of all n candidates . Then use word2vec to calculate The feature vectors of all opinion words in the dataset are obtained, and the similarity between the two feature vectors is calculated using the cosine similarity calculation formula. For two opinion words with high similarity, the one with the higher total number of occurrences is retained to achieve deduplication and merging. Then, the incorrect words are removed by comparing with the standard dictionary to obtain the final initial opinion set. .

[0106] Standard opinion sets based on standard opinion answers pre-defined by subject experts , The rest are not in The opinions that appear in the article are regarded as a collection of unverified and possibly reasonable innovative opinions. .

[0107] against Views in , query contains viewpoint of candidates answered , and submit it to subject experts for review. If the experts agree that the view is the correct answer, the view will be added to Otherwise, the viewpoint is considered as an incorrect answer. The final expanded viewpoint set The final standard answer viewpoint set .

[0108] The flowchart of the process of mining opinions for answering the above open questions is as follows: Figure 5 shown.

[0109] 6. Automatic scoring process:

[0110] Standard viewpoints from subject matter experts The answer text a of the candidate to be evaluated is mined, and the obtained opinions are compared with the standard opinion set. For comparison, if the opinion word is directly included in The score of the opinion word is directly obtained in the case of word2vec. Otherwise, the opinion word is represented by features and then The cosine similarity score is calculated for the words in . If the score is higher than the threshold p, it is judged as not scored, otherwise it is not scored.

[0111] The above automatic scoring process is as follows Figure 6 shown.

[0112] In yet another embodiment of the present invention, an open-ended test question evaluation system based on a domain-based large model is provided, which can be used to implement the above-mentioned open-ended test question evaluation method based on a domain-based large model. Specifically, the system includes:

[0113] The domain model construction module is used to obtain incremental training datasets and fine-tuning instruction sets from subject corpus data, and obtain candidate answer sets through text recognition. Based on the incremental training datasets and fine-tuning instruction sets, domain knowledge is injected and aligned with the intent of human instructions to obtain a domain model.

[0114] The domain model fine-tuning module is used to construct a prompt template for the candidate opinion mining task based on the candidate's answer set, fine-tune the prompts of the obtained domain model, obtain the fine-tuned domain model, and optimize the fine-tuned domain model;

[0115] The standard answer viewpoint acquisition module is used to extract the examinee's answer viewpoints from the tuned domain model, classify them into a standard answer viewpoint set and an innovative viewpoint set, and supplement the standard answer viewpoint set based on the innovative viewpoint set to obtain the final standard answer viewpoint set;

[0116] The scoring output module is used to assign scores to candidates' answers based on the final set of standard answer viewpoints.

[0117] The module division in the embodiments of the present invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in various embodiments of the present invention may be integrated into a single processor, exist physically as separate modules, or two or more modules may be integrated into a single module. The integrated modules may be implemented in either hardware or software functional modules.

[0118] In another embodiment of the present invention, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the open test question evaluation method based on the domain-based large model.

[0119] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a computer device, used to store programs and data. It is understood that the computer-readable storage medium herein may include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides storage space, which stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for being loaded and executed by a processor. These instructions may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium herein may be high-speed RAM memory or non-volatile memory, such as at least one disk storage device. The processor may load and execute the one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the open-ended test question evaluation method based on the domain-based large model described in the above-mentioned embodiment.

[0120] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0122] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. The open-ended test question evaluation method based on the domain-based large model is characterized by: include: Obtain incremental training data sets and instruction fine-tuning sets through the subject corpus data, and obtain the candidate answer sets; Based on the incremental training dataset and instruction fine-tuning set, domain knowledge is injected into the preset base model and instruction intent is aligned to obtain a domain model; Based on the candidate answer set, a prompt template for the candidate opinion mining task is constructed. Based on the prompt template for the candidate opinion mining task, the obtained domain model is fine-tuned to obtain the fine-tuned domain model, and the fine-tuned domain model is tuned. Extract the examinees' answer perspectives from the optimized domain model and classify them into a standard answer perspective set and an innovative perspective set. Then, supplement the standard answer perspective set with the innovative perspective set to obtain the final standard answer perspective set. Assign scores to candidates' answers based on the final set of standard answer viewpoints; The method of obtaining the incremental training data set and instruction fine-tuning set through the subject corpus data, and obtaining the candidate answer set, includes: Textbooks, books, papers, and question banks of the subject to which the open test questions belong are collected as corpus. Low-quality data in the corpus is removed through rule-based data filtering methods, and documents are deduplicated to obtain an incremental training dataset. A fine-tuning instruction set is generated through question-and-answer data from the question bank and a large model dialogue based on subject documents. The fine-tuning instruction set includes single-round dialogue sets and multi-round dialogue sets. The candidate answer set is obtained through image segmentation and optical character recognition. Based on the incremental training dataset and instruction fine-tuning set, domain knowledge is injected into the preset base large model and instruction intent is aligned to obtain a domain model, including: Using the incremental pre-training dataset, the base model is incrementally pre-trained to inject domain knowledge. Then, using the instruction fine-tuning dataset, the incremental pre-trained model is fine-tuned to align with the intent of human instructions, resulting in a discipline-specific domain model. The method of constructing a prompt template for the candidate opinion mining task based on the candidate answer set, and fine-tuning the obtained domain model based on the prompt template for the candidate opinion mining task to obtain the fine-tuned domain model includes: The prompt template format is defined based on the candidate's answer set, the prompt data is annotated according to the prompt template, and the low-rank adaptive efficient parameter fine-tuning domain model in the parameter efficient fine-tuning package is used to train the model on samples to adapt to the opinion mining task. The trained domain model is used to output the opinion information and explanation of the candidate's answer in the format defined by the template.

2. The open-ended test question evaluation method based on a domain-based large model according to claim 1 is characterized in that: The tuning operation of the fine-tuned domain model includes: Based on the fine-tuned domain model, access the external data storage system to perform retrieval optimization on the model output.

3. The open-ended test question evaluation method based on a domain-based large model according to claim 1 is characterized in that: The candidate's answer viewpoints are extracted from the tuned domain model and classified into a standard answer viewpoint set and an innovative viewpoint set. The standard answer viewpoint set is supplemented based on the innovative viewpoint set to obtain the final standard answer viewpoint set, including: First, for each answer in the candidate's answer set, a prompt is filled in according to the opinion mining prompt template to obtain a prompt, and then the prompt is input into the optimized domain model to extract the candidate's answer opinion, and obtain the initial opinion set corresponding to all candidate's answers. Then, all opinions in the initial opinion set are de-duplicated, merged, and inferior to obtain the final opinion set. With reference to the preset recommended standard opinion set, opinions that do not appear in the recommended standard opinion set are constructed into an unverified innovative opinion set. Finally, the unverified innovative opinion set is reviewed and evaluated as correct innovative opinions. The correct innovative opinions are expanded to the standard answer opinion set as part of the subsequent grading criteria. The expanded opinion set is used as the final standard answer opinion set.

4. The open-ended test question evaluation method based on a domain-based large model according to claim 3 is characterized in that: The scoring of the examinee's answers based on the final set of standard answer viewpoints includes: Based on the final set of standard answer opinions, after the examinee's answers are mined to obtain the examinee's opinion information, the opinion words are compared with the final set of standard answer opinions. Points are awarded if the comparison is successful, and no points are awarded if the comparison is unsuccessful.

5. The open test question evaluation system based on the domain-based large model is characterized by: include: The domain model construction module is used to obtain incremental training datasets and fine-tuning instruction sets from subject corpus data, and obtain candidate answer sets through text recognition. Based on the incremental training datasets and fine-tuning instruction sets, domain knowledge is injected and aligned with the intent of human instructions to obtain a domain model. The domain model fine-tuning module is used to construct a prompt template for the candidate opinion mining task based on the candidate's answer set, fine-tune the prompts of the obtained domain model, obtain the fine-tuned domain model, and optimize the fine-tuned domain model; The standard answer viewpoint acquisition module is used to extract the examinee's answer viewpoints from the tuned domain model, classify them into a standard answer viewpoint set and an innovative viewpoint set, and supplement the standard answer viewpoint set based on the innovative viewpoint set to obtain the final standard answer viewpoint set; Scoring output module, used to assign scores to candidates' answers based on the final set of standard answer viewpoints; The method of obtaining the incremental training data set and instruction fine-tuning set through the subject corpus data, and obtaining the candidate answer set, includes: Textbooks, books, papers, and question banks of the subject to which the open test questions belong are collected as corpus. Low-quality data in the corpus is removed through rule-based data filtering methods, and documents are deduplicated to obtain an incremental training dataset. A fine-tuning instruction set is generated through question-and-answer data from the question bank and a large model dialogue based on subject documents. The fine-tuning instruction set includes single-round dialogue sets and multi-round dialogue sets. The candidate answer set is obtained through image segmentation and optical character recognition. Based on the incremental training dataset and instruction fine-tuning set, domain knowledge is injected into the preset base large model and instruction intent is aligned to obtain a domain model, including: Using the incremental pre-training dataset, the base model is incrementally pre-trained to inject domain knowledge. Then, using the instruction fine-tuning dataset, the incremental pre-trained model is fine-tuned to align with the intent of human instructions, resulting in a discipline-specific domain model. The method of constructing a prompt template for the candidate opinion mining task based on the candidate answer set, and fine-tuning the obtained domain model based on the prompt template for the candidate opinion mining task to obtain the fine-tuned domain model includes: The prompt template format is defined based on the candidate's answer set, the prompt data is annotated according to the prompt template, and the low-rank adaptive efficient parameter fine-tuning domain model in the parameter efficient fine-tuning package is used to train the model on samples to adapt to the opinion mining task. The trained domain model is used to output the opinion information and explanation of the candidate's answer in the format defined by the template.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the open-ended test question evaluation method based on the domain-based large model as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the open-ended test question evaluation method based on the domain-based large model as described in any one of claims 1 to 4 is implemented.

Images