A method for generating test questions based on large models

Abstract

The present invention provides a method for generating test questions based on a large model, which belongs to the field of talent test assessment. First, relevant question information and material text are collected and preprocessed from the own test database to construct a high-quality data set; then, a large model of proposition based on Transformer autoregression is constructed; secondly, a training data set is produced based on the constructed large model of proposition and the model is fine-tuned and trained; thirdly, the trained large model of proposition is deployed in the background to realize the autonomous generation of test questions suitable for assessing candidates; finally, the model is further optimized based on the manual feedback of the test experts on the output of the large model of proposition and its answers, so as to realize closed-loop training. After the above steps, the model can realize the efficient, accurate and objective generation of test questions according to the requirements of the questions, and improve the level of intelligent talent evaluation.

Description

[0001] The present invention belongs to the field of talent examination and evaluation, and in particular relates to a method for generating examination questions based on a large model. Background Art

[0002] Generating test questions based on the examinee's knowledge, learning progress, ability level, etc. can accurately evaluate the learning status of each examinee and reduce errors caused by the mismatch between the difficulty of test questions and the students' actual abilities. It helps training institutions understand the strengths and weaknesses of each examinee, thereby adjusting teaching strategies and providing more targeted tutoring. It is also of great significance for evaluating examinees' learning outcomes, predicting learning potential and formulating personalized training plans.

[0003] Therefore, the present invention proposes a method for generating test questions based on a large model, which can generate test questions according to the categories of candidates and the candidates' mastery of knowledge. Summary of the invention

[0004] In view of the above problems, the first aspect of the present invention proposes a method for generating test questions based on a large model, comprising the following steps:

[0005] Step 1, data set preparation: collect data from the own test database, collect relevant question information and material text, clean the collected data, process the text segmentation, and set labels to finally obtain a high-quality data set;

[0006] Step 2: Build a large model for question generation based on the Transformer autoregressive architecture, integrating the self-attention mechanism and the scaled dot product attention mechanism to effectively capture the semantic dependency between question information and material text;

[0007] Step 3, creating a training data set based on the constructed propositional large model and fine-tuning the model;

[0008] Step 4, model deployment: deploy the large question model trained in step 3 to the backend server, and when relevant question information and material text are input, the corresponding test questions can be quickly generated;

[0009] Step 5: Based on the manual feedback from test experts on the questions and answers output by the large model, the model is further optimized to achieve closed-loop training.

[0010] Preferably, the step 1 specifically includes the following process:

[0011] S201, data collection: collect data from the own examination database, collect relevant question information and material text, specifically, question information It consists of question type, number of questions and question difficulty. Question types include multiple-choice questions, fill-in-the-blank questions, short-answer questions and essay questions. The number of questions is set by the test experts. The difficulty of the questions is the test experts' assessment of the difficulty of the target questions. The upper limit of the question difficulty score is 10 points. The higher the score, the more difficult the question is. Material text information The data content of the material text information corresponding to the relevant questions comes from the material text on the Internet and the material text in the textbooks;

[0012] S202, data preprocessing: and It may contain invalid words or symbols, such as non-text information such as "#", conjunctions and auxiliary words. These data will increase the data dimension, waste resources, and make the model effect fail to achieve the expected training effect. Therefore, it is necessary to clean the data to improve the validity of the data:

[0013] 1) Regular expression replacement: and Replace the "!", "?", "~", "\", "@", "#", "$", and "%" in the Replace the regular titles such as "Multiple Choice", "Fill in the Blank", "Short Answer" and "Essay" with blanks;

[0014] 2) Deletion of invalid data: and Invalid spaces are deleted;

[0015] 3) Text segmentation: using the NLPIR algorithm launched by the Chinese Academy of Sciences and Perform word segmentation processing;

[0016] After the above steps, we can get the preprocessed title information And the pre-processed material text information ;

[0017] S203, label formulation: Hire professional test experts to write corresponding questions and their corresponding answers based on relevant question information requirements and material text information, recorded as .

[0018] Preferably, the step 2 specifically includes the following process:

[0019] The Transformer autoregressive-based proposition model includes a material text information feature extraction network, a question information feature extraction network, a feature fusion network and a transformer;

[0020] S301, constructing a material text information feature extraction network to extract features from the material text information; considering that the material text information contains a large number of text words describing knowledge points, and the Transformer autoregressive-based propositional model is better in capturing the hierarchical relationship and local semantics of text data, this paper uses the propositional model to extract the semantic features of the material text information; the propositional model designed by the present invention has the following specific steps:

[0021] First, Embedding representation is obtained The word embedding matrix composed of the word vectors of each word in , then After feature extraction, the feature map is obtained The calculation formula is as follows:

[0022] ;

[0023] in, Represents feature extraction operation;

[0024] Will Input the maximum pooling layer to obtain the pooled feature map ; Finally, The feature map with spatial dependency is obtained in the input space feature extraction mechanism ;

[0025] S302, construct a topic information feature extraction network; the network includes a transformer head network, a convolution module and an attention layer; the transformer head network includes a word embedding layer and a position encoding layer. After inputting word embedding, we get the word embedding matrix ; The position coding layer uses sine / cosine function coding to capture The sequence information between each word in Input position encoding layer to obtain position encoding matrix ; embed the word into the matrix and the position encoding matrix Sum to get the topic information word segmentation embedding matrix :

[0026] ;

[0027] Will After feature extraction, the feature map is obtained , the specific calculation formula is as follows:

[0028] ;

[0029] Will Input the attention layer to obtain a feature map with word vector interaction features ;

[0030] S303, constructing a feature fusion network; after obtaining the feature graph of the topic information and feature graph of material text information After that, and Input feature fusion network to obtain interactive features of title information and material text information .

[0031] Preferably, the attention layer of step 2 is characterized by comprising the following processes:

[0032] The attention layer uses scaled dot product attention to capture The dependency relationship between word sequences in :

[0033] ;

[0034] in is the key vector, is a value vector, is the Softmax activation function.

[0035] Preferably, the feature fusion network in step 2 is characterized by comprising the following processes:

[0036] The feature fusion network is a fully connected neural network consisting of two linear layers and a ReLU activation function:

[0037] ;

[0038] in, is the weight matrix, is the bias vector.

[0039] Preferably, the generation output of the model in step 2 specifically includes the following process:

[0040] The last layer of the proposition model is an output layer, which is used to directly generate a sequence of questions and answers. The output layer is a sequence generation layer, which includes a linear layer and a softmax layer. The input is the semantic features of the question information. , semantic features of material text and interactive semantic features , the output is the sequence of the next token, and the entire sequence is generated by sequential sampling. The specific calculation formula is as follows:

[0041] ;

[0042] ;

[0043] ;

[0044] in, is the time step The generated token, For the generated sequence, and are the weight matrix and bias vector of the linear layer, respectively. It represents the AR encoding output with the generated sequence and features as input, and finally generates the entire sequence through sequential sampling to obtain the final output test questions and corresponding answer data .

[0045] Preferably, step 3 specifically includes the following process:

[0046] Based on the propositional model constructed as described above, the labeled training data set is input into the model, and the model is trained end-to-end using an autoregressive approach;

[0047] S701, data input and feature extraction: Input the labeled training data set into the model, perform data cleaning and text segmentation on the input data; then use the Transformer autoregressive proposition model to extract semantic features from the material text information and question information, and further obtain the corresponding fusion features;

[0048] S702, get the recognition result: send the fused features to the output layer based on the proposition model to get the final output test questions and corresponding answer data ;

[0049] S703, define the loss function: design a suitable loss function to convert the results obtained by the model Compared with manually annotated labels Compare and build the loss function :

[0050] ;

[0051] in, is the training batch size, and For the The model prediction results and annotated labels in the training batches;

[0052] S704, fine-tuning training of the propositional large model:

[0053] (1) Initialization: Initialize the parameters of the propositional large model with an existing pre-trained model related to the domain, including the weight matrix of each layer of the Transformer, the parameters of the attention layer, and the embedding layer, rather than random initialization. This can utilize the prior knowledge stored in the pre-trained model, accelerate convergence, and provide a better initial point;

[0054] (2) Loss calculation and gradient update: Adopt Adam optimization algorithm, according to the loss function Perform gradient updates on encoder / decoder parameters:

[0055] ;

[0056] in, Represents the parameters to be updated, including all weight matrices and bias vectors in the proposal model. is the learning rate, which controls the step size of each parameter update. Represents the loss function Parameters The gradient of the loss function, which represents the rate of change of the parameter, is multiplied by the learning rate , the parameters can be updated according to the direction and magnitude of the gradient , so that the loss function gradually decreases, the model can better adapt to the training data, and the updated parameters Will be used in the next iteration of the training process;

[0057] S705, adaptive learning rate: assign different learning rates to different layers, and the learning rate of the lower layer is smaller than that of the higher layer, so as to protect the basic features obtained by the lower layer from being destroyed;

[0058] S706, training termination condition: according to the loss function value of the validation set, when the loss cannot be reduced for N consecutive epochs, the training process is terminated;

[0059] S707, model saving: After the fine-tuning training converges, the optimized proposition model parameters are saved as the final proposition generation model as the final deployment model of the proposal.

[0060] Preferably, step 4 specifically includes the following process:

[0061] S801, deploying the trained proposition model to the backend server cluster;

[0062] S802, when receiving the question information (including question type, number and difficulty) and material text input by the test expert, the background big model triggers the automatic generation process according to the input;

[0063] S803: Extract the semantic features of the question information from the input question information and material text using the Transformer autoregressive mechanism in the question model. , semantic features of material text and interactive semantic features ;

[0064] S804: Semantic features of the title information , semantic features of material text and interactive semantic features Input the model output layer, use the attention mechanism to fuse semantic features, and generate candidate sequences of test questions and answers;

[0065] S805, returning the final test questions and their corresponding answer sequences for review by test experts.

[0066] Preferably, the step 5 specifically includes the following process:

[0067] S901, manual feedback collection:

[0068] Construct an exam expert feedback system to collect experts' evaluation feedback on the test questions generated by the large question model, including:

[0069] a) Knowledge point coverage: Are there any missing knowledge points that need to be supplemented?

[0070] b) Semantic expression: whether the test question is clear and unambiguous;

[0071] c) Question difficulty: whether the difficulty of the generated questions is moderate;

[0072] d) Other feedback: any other questions and suggestions on the generated results;

[0073] S902, feedback data preprocessing:

[0074] Data cleaning: remove invalid information and spelling errors in feedback data;

[0075] Classification and annotation: According to the feedback content, it is annotated into different types such as knowledge points, semantics, difficulty, etc.;

[0076] S903, corpus improvement and fine-tuning training:

[0077] Corpus improvement: Improve existing training corpus based on different types of feedback:

[0078] a) Missing knowledge points: mining and supplementing relevant material texts from the examination database;

[0079] b) Unclear semantics: Invite experts to reconstruct relevant test questions and generate high-quality corpus;

[0080] c) Unreasonable difficulty: adjust the difficulty score;

[0081] Model fine-tuning: Based on the improved corpus, fine-tune the existing large proposition model to improve the generation quality;

[0082] Model update: Periodically deploy the fine-tuned optimized model as a new online service model;

[0083] S904, Online Learning Optimization:

[0084] Test question evaluation: Build an automatic evaluation module to evaluate the quality of test questions generated online;

[0085] Data collection: The test input, generated results, and generated results reconstructed from manual feedback are aggregated as online learning data;

[0086] Model analysis: Regularly analyze online learning data to identify potential deviations in the proposition model;

[0087] Continuous fine-tuning: Using online learning data as the training set, continuously fine-tune the online service model to continuously optimize the generation quality.

[0088] The second aspect of the present invention also provides a test question generation device based on a large model, the device comprising at least one processor and at least one memory, the processor and the memory being coupled; the memory storing a computer execution program of a test question generation model based on a large model constructed by the construction method described in the first aspect; when the processor executes the computer execution program stored in the memory, the processor executes a test question generation method based on a large model.

[0089] Compared with the prior art, the innovations and beneficial effects of the present invention are as follows:

[0090] 1. The innovative points of the present invention are:

[0091] (1) A new automatic test question generation method based on a large language model is proposed, which gets rid of the drawbacks of traditional manual question setting, such as inefficiency and lack of objectivity. This method combines the self-attention mechanism with the scaled dot product attention mechanism, which can efficiently capture the contextual dependencies between question information and material text at different semantic levels, laying the foundation for accurately generating high-quality test questions.

[0092] (2) We designed an innovative large-scale model architecture for question generation, which integrates the Transformer autoregressive mechanism, convolutional module, and multi-head attention layer. The Transformer autoregressive mechanism captures sequence information through position encoding; the convolutional module extracts local features from the input; and the multi-head attention layer uses a scaled dot product attention function to fully exploit the long-range dependencies between different features. This architecture can simultaneously obtain multi-level semantic feature representations of question information and material text.

[0093] (3) A feature fusion network is proposed. The question information features and material text features are input into the network. Through linear transformation and nonlinear activation, the interactive relationship between the two is captured to generate rich fused semantic features, which provides a very complete feature representation for the final proposition.

[0094] (4) We innovatively proposed a closed-loop optimization strategy and established an exam expert feedback system. We manually evaluated the model output and classified and annotated it according to the feedback type (such as missing knowledge points, unclear semantics, inappropriate difficulty, etc.). We adopted corresponding corpus improvement and model fine-tuning measures for different feedback to continuously optimize the quality of questions.

[0095] (5) When initializing model training, we use the optimized parameters of a large-scale pre-trained language model instead of the traditional random initialization. This initialization strategy based on the pre-trained model can effectively inherit the prior semantic knowledge learned in the pre-training stage, greatly accelerate convergence, and provide a better parameter starting point for the model.

[0096] 2. Based on the above innovations, the following technical effects are achieved:

[0097] (1) It achieves efficient and automated large-scale question setting capabilities. With the powerful semantic understanding ability of the big model, it can quickly and efficiently generate batches of suitable test questions based on the question information and material text requirements, greatly improving the question setting efficiency and reducing the workload of manual question setting.

[0098] (2) Ensure the objectivity and fairness of the test questions. The test questions generated by the model are completely based on the input question requirements and material content, and are not affected by personal subjective factors. This fundamentally solves the problem of subjectivity and arbitrariness in manual question setting, and creates an absolutely fair and standardized examination environment for candidates.

[0099] (3) The quality of generated test questions has been greatly improved. Through a multi-layer attention mechanism and feature fusion network, the model can deeply mine the semantic information of question elements and material texts. The generated test questions have accurate knowledge point coverage, standardized semantic expression, and moderate difficulty control, reducing common problems such as knowledge gaps and ambiguous expressions.

[0100] (4) A closed-loop training with self-iterative optimization is achieved. By continuously absorbing the feedback from test experts, analyzing and evaluating the generation quality, improving the corpus quality in a targeted manner, and continuously fine-tuning the model parameters, a positive cycle of quality optimization is formed, which continuously improves the ability to answer questions and continuously optimizes the quality of generated test questions.

[0101] (5) It has hot-swappable and scalable capabilities. The solution can be flexibly deployed in different knowledge fields and question type scenarios. By replacing corpus data, customized question setting systems in different fields can be quickly built. At the same time, the model scale can be expanded according to needs, and the generation capability can be improved, which has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] In order to more clearly illustrate the technical solutions of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, what is described below is only one embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0103] Figure 1 The flowchart is a method for generating test questions based on a large proposition model.

[0104] Figure 2 This is the overall structure diagram of the propositional model based on Transformer autoregression.

[0105] Figure 3 It is a semantic feature extraction network for material text information.

[0106] Figure 4 It is the semantic feature extraction network for title information.

[0107] Figure 5 This is a schematic diagram of the structure of the test question generation device based on the large model in Example 2 of the present invention. DETAILED DESCRIPTION

[0108] The invention will be further described below in conjunction with specific embodiments.

[0109] Embodiment 1:

[0110] The present invention designs a method for generating test questions based on a large model. The overall logic of the present invention is as follows Figure 1 As shown, the overall network structure is as follows Figure 2As shown in the figure, specifically: first, collect and preprocess relevant question information and material text from the company's own test database to build a high-quality data set; then, build a Transformer autoregressive (AR)-based question model; secondly, create a training data set based on the constructed question model and fine-tune the model; thirdly, deploy the trained question model in the background to realize the autonomous generation of test questions suitable for assessing candidates; finally, based on the manual feedback of test experts on the questions and answers output by the question model, further optimize the model and realize closed-loop training.

[0111] 1. Dataset preparation: Collect data from the own test database, collect relevant question information and material text, clean the collected data, process the text segmentation, and set labels to finally obtain a high-quality data set. The specific steps are as follows:

[0112] S101, data collection: collect data from the own examination database, collect relevant question information and material text, specifically, question information It consists of question type, number of questions and question difficulty. Question types include multiple-choice questions, fill-in-the-blank questions, short-answer questions and essay questions. The number of questions is set by the test experts. The difficulty of the questions is the test experts' assessment of the difficulty of the target questions. The upper limit of the question difficulty score is 10 points. The higher the score, the more difficult the question is. Material text information The data content of the material text information corresponding to the relevant questions comes from the material text on the Internet and the material text in the textbooks;

[0113] S102, data preprocessing: and It may contain invalid words or symbols, such as non-text information such as "#", conjunctions and auxiliary words. These data will increase the data dimension, waste resources, and make the model effect fail to achieve the expected training effect. Therefore, it is necessary to clean the data to improve the validity of the data:

[0114] 1) Regular expression replacement: and Replace the "!", "?", "~", "\", "@", "#", "$", and "%" in the Replace the regular titles such as "Multiple Choice", "Fill in the Blank", "Short Answer" and "Essay" with blanks;

[0115] 2) Deletion of invalid data: and Invalid spaces are deleted;

[0116] 3) Text segmentation: using the NLPIR algorithm launched by the Chinese Academy of Sciences and Perform word segmentation processing;

[0117] After the above steps, we can get the preprocessed title information And the pre-processed material text information ;

[0118] S103, label formulation: Hire professional test experts to write corresponding questions and their corresponding answers based on relevant question information requirements and material text information, recorded as .

[0119] 2. Build a large model for question generation based on the Transformer autoregressive architecture, integrating the self-attention mechanism and the scaled dot product attention mechanism to effectively capture the semantic dependency between question information and material text;

[0120] The Transformer autoregressive-based proposition model includes a material text information feature extraction network, a question information feature extraction network, a feature fusion network and a transformer;

[0121] S201, construct a material text information feature extraction network, the network structure is as follows Figure 3 As shown, feature extraction is performed on the material text information; considering that the material text information contains a large number of text words describing knowledge points, and the Transformer autoregressive-based propositional model is constructed to capture the hierarchical relationship and local semantics of text data. Therefore, this paper uses the propositional model to extract the semantic features of the material text information; the specific steps of the propositional model designed by the present invention are:

[0122] First, Embedding representation is obtained The word embedding matrix composed of the word vectors of each word in , then After feature extraction, the feature map is obtained The calculation formula is as follows:

[0123] ;

[0124] in, Represents feature extraction operation;

[0125] Will Input the maximum pooling layer to obtain the pooled feature map ; Finally, The feature map with spatial dependency is obtained in the input space feature extraction mechanism ;

[0126] S202, construct a topic information feature extraction network, the network structure is as follows Figure 4 As shown in Figure 2, the network includes a transformer head network, a convolutional module, and an attention layer. The transformer head network includes a word embedding layer and a position encoding layer. After inputting word embedding, we get the word embedding matrix ; The position coding layer uses sine / cosine function coding to capture The sequence information between each word in Input position encoding layer to obtain position encoding matrix ; embed the word into the matrix and the position encoding matrix Sum to get the topic information word segmentation embedding matrix :

[0127] ;

[0128] Will After feature extraction, the feature map is obtained , the specific calculation formula is as follows:

[0129] ;

[0130] Will Input the attention layer to obtain a feature map with word vector interaction features ;

[0131] The attention layer uses scaled dot product attention to capture The calculation formula for the dependency relationship between word sequences in is as follows:

[0132] ;

[0133] in is the key vector, is a value vector, is the Softmax activation function;

[0134] S203, constructing a feature fusion network; obtaining a feature graph of the topic information and feature graph of material text information After that, and Input feature fusion network to obtain interactive features of title information and material text information .

[0135] S204, model generation output: The last layer of the proposition model is an output layer, which is used to directly generate a sequence of questions and answers. The output layer is a sequence generation layer, which includes a linear layer and a softmax layer; the input is the semantic features of the question information , semantic features of material text and interactive semantic features , the output is the sequence of the next token, and the entire sequence is generated by sequential sampling. The specific calculation formula is as follows:

[0136] ;

[0137] ;

[0138] ;

[0139] in, is the time step The generated token, For the generated sequence, and are the weight matrix and bias vector of the linear layer, respectively. It represents the AR encoding output with the generated sequence and features as input, and finally generates the entire sequence through sequential sampling to obtain the final output test questions and corresponding answer data .

[0140] 3. Create a training data set based on the constructed proposition model and fine-tune the model. The specific steps are as follows:

[0141] Based on the propositional model constructed as described above, the labeled training data set is input into the model, and the model is trained end-to-end using an autoregressive approach;

[0142] S301, data input and feature extraction: input the labeled training data set into the model, perform data cleaning and text segmentation on the input data; then use the Transformer autoregressive proposition model to extract semantic features from the material text information and question information, and further obtain the corresponding fusion features;

[0143] S302, get the recognition result: send the fused features to the output layer based on the proposition model to get the final output test questions and corresponding answer data ;

[0144] S303, define the loss function: design a suitable loss function to convert the results obtained by the model Compared with manually annotated labels Compare and build the loss function :

[0145] ;

[0146] in, is the training batch size, and For the The model prediction results and annotated labels in the training batches;

[0147] S304, fine-tuning training of the propositional large model:

[0148] (1) Initialization: Initialize the parameters of the propositional large model with an existing pre-trained model related to the domain, including the weight matrix of each layer of the Transformer, the parameters of the attention layer, and the embedding layer, rather than random initialization. This can utilize the prior knowledge stored in the pre-trained model, accelerate convergence, and provide a better initial point;

[0149] (2) Loss calculation and gradient update: Adopt Adam optimization algorithm, according to the loss function Perform gradient updates on encoder / decoder parameters:

[0150] ;

[0151] in, Represents the parameters to be updated, including all weight matrices and bias vectors in the proposal model. is the learning rate, which controls the step size of each parameter update. Represents the loss function Parameters The gradient of the loss function, which represents the rate of change of the parameter, is multiplied by the learning rate , the parameters can be updated according to the direction and magnitude of the gradient , so that the loss function gradually decreases, the model can better adapt to the training data, and the updated parameters Will be used in the next iteration of the training process;

[0152] S305, adaptive learning rate: assign different learning rates to different layers, and the learning rate of the lower layer is smaller than that of the higher layer, so as to protect the basic features obtained by the lower layer from being destroyed;

[0153] S306, training termination condition: according to the loss function value of the validation set, when the loss cannot be reduced for N consecutive epochs, the training process is terminated;

[0154] S307, model saving: After the fine-tuning training converges, the optimized proposition model parameters are saved as the final proposition generation model, which serves as the final deployment model of the proposal.

[0155] 4. Deploy the large question model trained in step 3 to the backend server. When the relevant question information and material text are input, the corresponding test questions can be quickly generated. The specific steps are as follows:

[0156] S401, deploy the trained proposition model to the backend server cluster;

[0157] S402, when receiving the question information (including question type, number and difficulty) and material text input by the test expert, the background big model triggers the automatic generation process according to the input;

[0158] S403: Extract the semantic features of the question information from the input question information and material text using the Transformer autoregressive mechanism in the question model , semantic features of material text and interactive semantic features ;

[0159] S404: Semantic features of the title information , semantic features of material text and interactive semantic features Input the model output layer, use the attention mechanism to fuse semantic features, and generate candidate sequences of test questions and answers;

[0160] S405, returning the final test questions and their corresponding answer sequences for review by test experts.

[0161] 5. Based on the manual feedback from the test experts on the questions and answers output by the large test model, the model is further optimized to achieve closed-loop training. The specific steps are as follows:

[0162] S501, manual feedback collection:

[0163] Construct an exam expert feedback system to collect experts' evaluation feedback on the test questions generated by the large question model, including:

[0164] a) Knowledge point coverage: Are there any missing knowledge points that need to be supplemented?

[0165] b) Semantic expression: whether the test question is clear and unambiguous;

[0166] c) Question difficulty: whether the difficulty of the generated questions is moderate;

[0167] d) Other feedback: any other questions and suggestions on the generated results;

[0168] S502, feedback data preprocessing:

[0169] Data cleaning: remove invalid information and spelling errors in feedback data;

[0170] Classification and annotation: According to the feedback content, it is annotated into different types such as knowledge points, semantics, difficulty, etc.;

[0171] S503, corpus improvement and fine-tuning training:

[0172] Corpus improvement: Improve existing training corpus based on different types of feedback:

[0173] a) Missing knowledge points: mining and supplementing relevant material texts from the examination database;

[0174] b) Unclear semantics: Invite experts to reconstruct relevant test questions and generate high-quality corpus;

[0175] c) Unreasonable difficulty: adjust the difficulty score;

[0176] Model fine-tuning: Based on the improved corpus, fine-tune the existing large proposition model to improve the generation quality;

[0177] Model update: Periodically deploy the fine-tuned optimized model as a new online service model;

[0178] S504, Online Learning Optimization:

[0179] Test question evaluation: Build an automatic evaluation module to evaluate the quality of test questions generated online;

[0180] Data collection: Input the test questions with lower scores and generate results, which are summarized as online learning data;

[0181] Model analysis: Regularly analyze online learning data to identify potential deviations in propositional models;

[0182] Continuous fine-tuning: Using online learning data as the training set, continuously fine-tune the online serving model to continuously optimize the generation quality.

[0183] The following example results are presented for this process:

[0184] In order to verify the subjective test paper marking method based on a large model proposed in the present invention, this embodiment also provides a comparison of the results of the method of the present invention with the currently well-known models GPT-3.5, T5 and ELECTRA. The evaluation indicator is mAP, and the results are shown in Table 1.

[0185] Table 1 Quantitative analysis of test data sets in different models

[0186] Model mAP GPT-3.5 82.6 T5 80.0 ELECTRA 85.4 Method of the present invention 90.5

[0187] As shown in Table 1, the method proposed in the present invention achieves the maximum value in mAP, which shows that the model has high text generation accuracy.

[0188] Embodiment 2:

[0189] like Figure 5 As shown, the present invention also provides a test question generation device based on a large model, the device includes at least one processor and at least one memory, and also includes a communication interface and an internal bus; a computer execution program is stored in the memory; a computer execution program of a test question generation model based on a large model constructed by the construction method described in Example 1 is stored in the memory; when the processor executes the computer execution program stored in the memory, the processor can execute a test question generation method based on a large model. The internal bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a U disk, a mobile hard disk, a read-only memory, a disk or an optical disk.

[0190] The device may be provided as a terminal, a server or other forms of devices.

[0191] Figure 5 is a block diagram of a device shown as an example. The device may include one or more of the following components: a processing component, a memory, a power component, a multimedia component, an audio component, an input / output (I / O) interface, a sensor component, and a communication component. The processing component generally controls the overall operation of the electronic device, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component may include one or more processors to execute instructions to complete all or part of the steps of the above method. In addition, the processing component may include one or more modules to facilitate interaction between the processing component and other components. For example, the processing component may include a multimedia module to facilitate interaction between the multimedia component and the processing component.

[0192] The memory is configured to store various types of data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, pictures, videos, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0193] The communication component is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0194] In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above methods.

[0195] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0196] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for generating test questions based on a large model, characterized in that: The process includes: Step 1, data set preparation: collect data from the own test database, collect relevant question information and material text, clean the collected data, process the text segmentation, and set labels to finally obtain a high-quality data set; Step 2: Build a large model for question generation based on the Transformer autoregressive architecture, integrating the self-attention mechanism and the scaled dot product attention mechanism to effectively capture the semantic dependency between question information and material text; The Transformer autoregressive-based proposition model includes a material text information feature extraction network, a question information feature extraction network, a feature fusion network and a transformer, and the specific steps are as follows; S101, construct a material text information feature extraction network to extract features from the material text information; use the proposition model to extract semantic features of the material text information; the specific steps of the proposition model are: First, Embedding representation is obtained The word embedding matrix composed of the word vectors of each word in , then After feature extraction, the feature map is obtained The calculation formula is as follows: ； in, Represents feature extraction operation; Will Input the maximum pooling layer to obtain the pooled feature map ; Finally, The feature map with spatial dependencies is obtained in the input space feature extraction mechanism ; S102, construct a topic information feature extraction network; the network includes a transformer head network, a convolution module and an attention layer; the transformer head network includes a word embedding layer and a position encoding layer. After inputting word embedding, we get the word embedding matrix ; The position coding layer uses sine / cosine function coding to capture The sequence information between each word in Input position encoding layer to obtain position encoding matrix ; embed the word into the matrix and the position encoding matrix Sum to get the topic information word segmentation embedding matrix : ； Will After feature extraction, the feature map is obtained , the specific calculation formula is as follows: ； Will Input the attention layer to obtain a feature map with word vector interaction features ; S103, constructing a feature fusion network; after obtaining the feature graph of the topic information and feature graph of material text information After that, and Input feature fusion network to obtain interactive features of title information and material text information ; S104, the output layer of the large proposition model, is used to directly generate a sequence of questions and answers. The output layer is a sequence generation layer, which includes a linear layer and a softmax layer; the input is the semantic features of the question information , semantic features of material text and interactive semantic features , the output is the sequence of the next token, and the entire sequence is generated by sequential sampling. The specific calculation formula is as follows: ； ； ； in, is the time step The generated token, For the generated sequence, and are the weight matrix and bias vector of the linear layer, respectively. Represents the generated sequence and features as input Encoded output, and finally the entire sequence is generated through sequential sampling to obtain the final output test questions and corresponding answer data ; Step 3, creating a training data set based on the constructed propositional large model and fine-tuning the model; Based on the constructed proposition model, the labeled training data set is input into the model, and the model is trained end-to-end using an autoregressive method. The specific steps are as follows: S201, data input and feature extraction: input the labeled training data set into the model, perform data cleaning and text segmentation on the input data; then use the Transformer autoregressive proposition model to extract semantic features from the material text information and question information, and further obtain the corresponding fusion features; S202, obtain the recognition result: send the fused features to the output layer based on the proposition model to obtain the final output test questions and corresponding answer data ; S203, define loss function: Design the loss function and convert the results obtained by the model into Compared with manually annotated labels Compare and build the loss function : ； in, is the training batch size, and For the The model prediction results and annotated labels in the training batches; S204, fine-tuning training of the large proposition model: (1) Initialization: Initialize the parameters of the propositional large model with an existing pre-trained model related to the domain, including the weight matrix of each layer of the Transformer, the parameters of the attention layer, and the embedding layer, rather than random initialization. This will utilize the prior knowledge stored in the pre-trained model, accelerate convergence, and provide a better starting point. (2) Loss calculation and gradient update: Adopt Adam optimization algorithm, according to the loss function Perform gradient updates on encoder / decoder parameters: ； in, Represents the parameters to be updated, including all weight matrices and bias vectors in the proposition model. is the learning rate, which controls the step size of each parameter update. Represents the loss function Parameters The gradient of the loss function, which represents the rate of change of the parameter, is multiplied by the learning rate , update the parameters according to the direction and magnitude of the gradient , so that the loss function gradually decreases, the model can better adapt to the training data, and the updated parameters Will be used in the next iteration of the training process; S205, adaptive learning rate: different learning rates are assigned to different layers, and the learning rate of the lower layer is smaller than that of the higher layer, so as to protect the basic features obtained by the lower layer from being destroyed; S206, training termination condition: according to the loss function value of the validation set, when the loss cannot be reduced for N consecutive epochs, the training process is terminated; S207, model saving: after fine-tuning training converges, the optimized large proposition model parameters are saved as the final proposition generation model as the final deployment model; Step 4, model deployment: deploy the large question model trained in step 3 to the backend server, and when relevant question information and material text are input, the corresponding test questions can be quickly generated; Step 5: Based on the manual feedback from test experts on the questions and answers output by the large model, the model is further optimized to achieve closed-loop training.

2. A method for generating test questions based on a large model according to claim 1, characterized in that: The step 1 specifically includes the following process: S301, data collection: collect data from the own examination database, collect relevant question information and material text, specifically, question information It consists of question type, number of questions and question difficulty. Question types include multiple-choice questions, fill-in-the-blank questions, short-answer questions and essay questions. The number of questions is set by the test experts. The difficulty of the questions is the test experts' assessment of the difficulty of the target questions. The upper limit of the question difficulty score is 10 points. The higher the score, the more difficult the question is. Material text information The data content of the material text information corresponding to the relevant questions comes from the material text on the Internet and the material text in the textbooks; S302, data preprocessing: and Contains invalid words or symbols, non-text information "#", conjunctions and auxiliary words. These data will increase the data dimension, waste resources, and make the model effect fail to achieve the expected training effect. Therefore, it is necessary to clean the data to improve the effectiveness of the data: 1) Regular expression replacement: and Replace "!", "?", "~", "\", "@", "#", "$", and "%" in the Replace the regular titles of "Multiple Choice", "Fill in the Blank", "Short Answer" and "Essay" with blanks; 2) Deletion of invalid data: and Invalid spaces are deleted; 3) Text segmentation: using the NLPIR algorithm launched by the Chinese Academy of Sciences and Perform word segmentation processing; After the above steps, we can get the preprocessed title information And the pre-processed material text information ; S303, label formulation: Hire professional test experts to write corresponding questions and their corresponding answers based on relevant question information requirements and material text information, recorded as .

3. The method for generating test questions based on a large model according to claim 1, characterized in that: The attention layer is specifically: The attention layer uses scaled dot product attention to capture The dependency relationship between word sequences in : ； in is the key vector, is a value vector, is the Softmax activation function.

4. The method for generating test questions based on a large model according to claim 1, characterized in that: The feature fusion network is specifically: The feature fusion network is a fully connected neural network consisting of two linear layers and a ReLU activation function: ； in, is the weight matrix, is the bias vector.

5. The method for generating test questions based on a large model according to claim 1, characterized in that: The step 4 is specifically The process includes: S401, deploying the trained proposition model to the backend server cluster; S402, when receiving the question information and material text input by the test expert, the backend big model triggers the automatic generation process according to the input, wherein the question information includes question type, number and difficulty; S403: Extract the semantic features of the question information from the input question information and material text using the Transformer autoregressive mechanism in the question model , semantic features of material text and interactive semantic features ; S404: Semantic features of the title information , semantic features of material text and interactive semantic features Input the model output layer, use the attention mechanism to fuse semantic features, and generate candidate sequences of test questions and answers; S405, returning the final test questions and their corresponding answer sequences for review by test experts.

6. The method for generating test questions based on a large model according to claim 1, characterized in that: The step 5 is specifically The process includes: S501, manual feedback collection: Construct an exam expert feedback system to collect experts' evaluation feedback on the test questions generated by the big question model, including: a) Knowledge point coverage: whether there are any missing knowledge points that need to be supplemented; b) Semantic expression: whether the test question is clear and unambiguous; c) Question difficulty: whether the difficulty of the generated questions is moderate; d) Other feedback: any other questions and suggestions on the generated results; S502, feedback data preprocessing: Data cleaning: remove invalid information and spelling errors in feedback data; Classification and annotation: According to the feedback content, it is annotated into three categories: knowledge points, semantics, and difficulty; S503, corpus improvement and fine-tuning training: Corpus improvement: Improve existing training corpus based on different types of feedback: a) Missing knowledge points: mining and supplementing relevant material texts from the examination database; b) Unclear semantics: Invite experts to reconstruct relevant test questions and generate high-quality corpus; c) Unreasonable difficulty: adjust the difficulty score; Model fine-tuning: Based on the improved corpus, fine-tune the existing large proposition model to improve the generation quality; Model update: Periodically deploy the fine-tuned optimized model as a new online service model; S504, Online Learning Optimization: Test question evaluation: Build an automatic evaluation module to evaluate the quality of test questions generated online; Data collection: The test input, generated results, and generated results reconstructed from manual feedback are aggregated as online learning data; Model analysis: Regularly analyze online learning data to identify potential deviations in the proposition model; Continuous fine-tuning: Using online learning data as the training set, continuously fine-tune the online service model to continuously optimize the generation quality.

7. A test question generating device based on a large model, characterized in that: The device includes at least one processor and at least one memory, and the processor and the memory are coupled; the memory stores a computer execution program of a large model-based test question generation model constructed by the large model-based test question generation method as described in any one of claims 1 to 4; when the processor executes the computer execution program stored in the memory, the processor executes test question generation.

Images