A deep learning method for automatic review of subjective questions by combining test question classification and scoring learning
Through the deep learning method of joint test question classification and scoring learning, combined with the CNN and Bi-LSTM models, the poor review effect caused by the difference in the automatic review of subjective questions was solved, and more efficient review of subjective questions was achieved.
Patent Information
- Application Number
- CN202210597773.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-05-30
AI Technical Summary
The existing automatic review technology for subjective questions is difficult to effectively deal with the answer sequence of different types of questions, resulting in unsatisfactory review results.
Using the deep learning method of joint test question classification and scoring learning, the test question type classification and answer score are performed separately through the CNN test question classification sub-model with residual connection and the rating model composed of CNN and Bi-LSTM, and combined with the test question type perception. Review.
It improves the accuracy and adaptability of automatic review of subjective questions, can effectively handle reviews of different types of questions, and significantly improves the scoring accuracy.
Smart Images

Figure CN114969260B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the intersection of artificial intelligence and smart education, and in particular to a deep learning method for automatic marking of subjective questions that combines test question classification with scoring learning, which can be widely used in computer automatic marking systems for subjective questions in various disciplines. Background Art
[0002] The questions in the examination papers are generally divided into two categories, objective questions and subjective questions, based on the form of the answers. Single-choice questions, multiple-choice questions, judgment questions, etc., whose answers are expressed in option numbers, are called objective questions, while questions such as short-answer questions, definition questions, and discussion questions, whose answers are expressed in natural language, are called subjective questions. Since the answers to objective questions such as single-choice questions, multiple-choice questions, and definition questions are all expressed in option numbers, when computers automatically grade such questions, they only need to perform a simple matching operation between the option numbers of the standard answers and the option numbers of the student answers. If the match is successful, the answer is correct. This processing technology has achieved good results. However, for the automatic grading technology of subjective questions whose answers are expressed in natural language, such as automatic grading of short-answer questions, definition questions, and discussion questions, the effect is not very ideal due to the theoretical and technical bottlenecks such as natural language understanding and artificial intelligence.
[0003] With the continuous development of artificial neural network technology, many deep learning models such as LSTM-based models, CNN&LSTM-based models and Transformer-based models have been applied to subjective test scoring. These deep learning models use different neural networks to automatically extract semantic features from the answer text, thus providing an end-to-end method without any artificial feature engineering. However, deep learning methods for automatic scoring of subjective questions still have many challenges. Among them, an important challenge is that the answers to subjective questions of some types of questions are sequential, while the answers to subjective questions of other types of questions are not sequential. For example, for the question "Please write the units of storage capacity in order from small to large", the answer points "B, KB, MB, GB" are sequential; but for the question "Briefly describe the advantages of twisted pair cables", the answer points "high flexibility, oil resistance, waterproof, wear resistance, acid resistance, cold resistance, bending resistance" are not sequential, that is, the student's answer "waterproof, cold resistance, bending resistance, oil resistance, wear resistance, acid resistance, high flexibility" is also completely correct. Therefore, how to automatically learn different marking methods for different question types is one of the difficult problems that the deep learning method for automatic marking of subjective questions needs to solve. Summary of the invention
[0004] The present invention discloses a deep learning method for automatic review of subjective questions that combines test question classification with scoring learning. By combining test question classification with scoring learning, the deep learning method for automatic review of subjective questions can learn to adopt different review methods for different question types, thereby solving the problem of automatic review of subjective questions in a more effective way.
[0005] To achieve the above object, the technical solution of the present invention is:
[0006] A deep learning method for automatic review of subjective questions combined with question classification and scoring learning, characterized by comprising the following steps:
[0007] S1. Classify the subjective questions to be reviewed through a CNN question classification sub-model with residual connection to obtain the question type of the subjective questions to be reviewed;
[0008] S2. Connect the question type embedding of the subjective question to be reviewed with each word embedding of the student answer input sequence and the standard answer input sequence to obtain the question type-aware student answer perception input and the standard answer perception input, and then send them to a subjective question paper scoring sub-model composed of CNN and Bi-LSTM for scoring to obtain the scoring grade of the student answer;
[0009] S3. By jointly calculating the loss function of step S1 and step S2, a neural network model for automatic review of subjective questions for joint question classification and scoring learning is trained;
[0010] S4. Use the subjective question automatic review neural network model trained in step S3 to review any student answer;
[0011] The CNN is the abbreviation of Convolutional Neural Networks, which means convolutional neural network, and is proposed in the document “Lawrence S, Giles CL, Tsoi AC et al. Face recognition: A convolutional neural-network approach. Neural Networks, IEEE Transactions on 1997; 8(1): 98-113”.
[0012] The Bi-LSTM is the abbreviation of Didirectional Long Short-Term Memory, which means a bidirectional long short-term memory neural network, wherein the LSTM neural network is proposed by the document “Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. ¨Neural computation, 9(8): 1735–1780”;
[0013] The types of subjective questions include "definition-type subjective questions", "sequence-type subjective questions" and "general-type subjective questions". The "definition-type subjective questions" refer to subjective questions that require students to explain the definition of a concept, "sequence-type subjective questions" refer to subjective questions whose answers include multiple key points given in sequence, and the "general-type subjective questions" refer to subjective questions other than definition-type and sequence-type.
[0014] Furthermore, the step S1 specifically includes:
[0015] S1.1 The word sequence of the test question to be classified Converted into the word embedding sequence T of the test question to be classified through table lookup operation e , the calculation process is as follows:
[0016]
[0017] Where n represents the sentence length of the test question to be classified, OneHot(T) represents converting the test question word sequence T into a one-hot matrix, T o is the one-hot matrix of the test word sequence T, |V| represents the size of the dictionary V, and the dictionary V is a dictionary containing all words established by the model, is the pre-trained word embedding matrix corresponding to the dictionary V, represents the word embedding vector corresponding to the i-th word in the test word sequence T, and h represents the dimension of the word embedding vector;
[0018] S1.2 Embed the words of the test questions to be classified into sequence T e Input into a convolutional neural network with zero padding to obtain the shallow semantic representation T of the test question to be classified c , the calculation process is as follows:
[0019]
[0020] Among them, PCNN(·) represents a convolutional neural network with zero padding, (T c ) ij Indicates T cThe i-th row and j-th column element of is, k is the convolution kernel size, d c is the number of convolution kernels, Represents the word embedding sequence T of the test question to be classified e The result after zero padding is: express The submatrix consisting of the j-th column to the j+(k-1)-th column, represents the weight of the i-th convolution kernel, represents the bias of the i-th convolution kernel, Represents the convolution operation;
[0021] S1.3 Shallow semantic representation of the test questions to be classified T c It is sent to a feedforward neural network consisting of two fully connected layers for enhancement to obtain the enhanced semantic representation of the test questions to be classified. The calculation process is as follows:
[0022]
[0023] in, are the weight matrices of the two fully connected layers respectively, are the bias vectors of the two fully connected layers respectively. ReLU is the abbreviation of “Rectified Linear Unit” in English. ReLU(·) represents the activation function of the rectified linear unit operation.
[0024] S1.4 The shallow semantic representation T of the test questions to be classified c Instead of enhancing the semantic representation T f Perform residual connection to obtain the deep semantic representation T of the test questions to be classified u , the calculation process is as follows:
[0025]
[0026] Among them, LayerNorm(·) represents layer normalization processing;
[0027] S1.5 Deep semantic representation of classification questions T u Perform maximum pooling to obtain the classification vector Z of the test question to be classified c , the calculation process is as follows:
[0028]
[0029] Among them, maxPooling(·) represents the maximum pooling operation;
[0030] S1.6 Classification vector Z for classification questions cExecute the linear transformation of softmax, calculate the probability of the test question type, and obtain the final test question type. The calculation process is as follows:
[0031] o=M c Z c +b o (8)
[0032]
[0033] in, is the test question type representation matrix, is a bias vector, d y is the number of categories of question types, Represents Z c The confidence score vector on all question types, y is a question type, o y Represents Z c The confidence score for question type y is marked as o i Represents Z c The confidence score on the i-th question type, P(y|Z c ) represents the classification vector Z of the test question to be classified c The predicted probability on question type y, where Y is the set of all question types, It means to return the value that makes P(y|Z c ) is the maximum value of the test type, y * is the final classification type of the test questions to be classified, and exp(·) represents an exponential function with the natural constant e as the base.
[0034] Furthermore, the step S2 specifically includes:
[0035] S2.1 performs table lookup operation on the student answer word sequence and the standard answer word sequence according to step S1.1 to obtain the student answer word embedding sequence Ground truth word embedding sequence Then embed the test word into the sequence T e Sequences of word embeddings with student answers Ground truth word embedding sequence Connect the beginning and the end to get the student's answer input sequence A e And the standard answer input sequence S e , the calculation process is as follows:
[0036]
[0037] Among them, “:” represents the concatenation operation of the head and tail of the sequence, n1 is the sentence length of the student’s answer, n2 is the sentence length of the standard answer, and m1 is the student’s answer input sequence A. eThe length of m1=n+n1, m2 is the standard answer input sequence S e The length of m2 and m2 = n + n2;
[0038] S2.2 The test question type y obtained in step S1 * Convert to the corresponding test question type through table lookup operation Then v t Sequence A with student answers e And the standard answer input sequence S e Each word embedding in is concatenated to obtain the student answer perception input A t and the standard answer perception input S t , the calculation process is as follows:
[0039]
[0040]
[0041] in, They are student answer input sequence A e and student answer perception input A t The word embedding vector of the i-th column in d t is the dimension of question type embedding, They are the standard answer input sequence S e and the standard answer perception input S t The word embedding vector of the jth column in , “;” represents the concatenation operation of the vectors;
[0042] S2.3 Input student answer perception into A t and the standard answer perception input S t The results are sent to two convolutional neural networks with zero padding and shared parameters to obtain the shallow semantic representation A of the student's answer. c And the shallow semantic representation S of the standard answer c , the calculation process is as follows:
[0043]
[0044] Among them, sharedPCNN1(·) and sharedPCNN2 represent two convolutional neural networks with zero padding and shared parameters, d c is the number of convolution kernels;
[0045] S2.4 Represent the student's answer in a shallow semantic form A c And the shallow semantic representation S of the standard answer c Send it to an interactive network composed of Bi-LSTM to get the score vector Z of the student's answer g , as shown below:
[0046]
[0047] Among them, Interactive(·) represents an interactive network composed of Bi-LSTM, d L is the dimension of the hidden state of the Bi-LSTM in the interaction network;
[0048] S2.5 The scoring vector Z for the student's answer g Perform a linear transformation of softmax, calculate the probability of the rating level, and obtain the final rating level. The calculation process is as follows:
[0049] x=M g Z g +b g (18)
[0050]
[0051] in, is the rating level representation matrix, is a bias vector, d g is the number of rating levels, It means Z g The vector of confidence scores for all rating levels, g is a rating level, x g Represents Z g The confidence score of the rating level g is labeled, x i Represents Z g The confidence score at the i-th rating level, P(g|Z g ) represents the score vector Z g The predicted probability at rating level g, G is the set of all rating levels, g * The final rating level is It means to return the value that makes P(g|Z g ) is the maximum rating level.
[0052] Furthermore, the step S2.4 specifically includes:
[0053] S2.4.1 Use two Bi-LSTMs with shared parameters to represent the shallow semantics of the student’s answer A c And the shallow semantic representation S of the standard answer c Encode and get the context representation of the student's answer A r and the standard answer context representation S r , the calculation process is as follows:
[0054]
[0055] Among them, sharedBiLSTM1(·) and sharedBiLSTM2(·) represent two Bi-LSTMs with shared parameters. Respectively represent A c With S c The connection result of the bidirectional hidden state corresponding to the i-th time step in Bi-LSTM, Respectively represent A c With S c From left to right The hidden state of the network at the ith time step, Respectively represent A c With S c From right to left The hidden state of the network at the i-th time step;
[0056] S2.4.2 Representing the context of student answers through attention mechanism A r and the standard answer context representation S r Interact and gain mutual attention and The calculation process is as follows;
[0057]
[0058] in, for The i-th element in for The jth element in Respectively express and Perform a transpose operation, is the universal logical quantifier representing "all", Represents the hidden state tuple and The attention weight of
[0059] S2.4.3 will and A with student answers in context r and the standard answer context representation S r Fusion is performed to obtain the student's answer fusion representation A f The fusion of the standard answer S f , the calculation process is as follows:
[0060]
[0061] in, A f The i-th element in For Sf The i-th element in , Fusion(·) represents the fusion operation, and ⊙ represents the matrix element-by-element multiplication operation;
[0062] S2.4.4 Use two Bi-LSTMs with shared parameters to represent the student answers A f The fusion of the standard answer S f Perform enhancement processing to obtain the deep semantic representation A of the student's answer u and the deep semantic representation of the standard answer S u , the calculation process is as follows:
[0063]
[0064] in, Respectively represent A f With S f The concatenation result of the bidirectional hidden states corresponding to the i-th and j-th time steps in the Bi-LSTM;
[0065] S2.4.5 Deep semantic representation of student answers u and the deep semantic representation of the standard answer S u Perform average pooling and maximum pooling, and connect all the pooled vectors after pooling to get the score vector Z of the student's answer g , the calculation process is as follows:
[0066]
[0067] Among them, maxPooling(·) represents the maximum pooling operation, avePooling(·) represents the average pooling operation, Yes A u The maximum pooling vector of Yes A u The average pooling vector of YesS u The maximum pooling vector of YesS u The average pooling vector of , “;” represents the concatenation operation of the vectors.
[0068] Furthermore, the step S3 specifically includes:
[0069] S3.1 uses the cross entropy loss error to calculate the loss function of the test question classification sub-model and the loss function of the subjective question automatic scoring sub-model. The calculation process is as follows:
[0070]
[0071] Among them, Ω is the training set of the automatic review task of subjective questions for joint question classification and scoring learning, |Ω| represents the size of the training set Ω, is the real test question classification of the i-th training sample in Ω, is the true rating of the i-th training sample in Ω, is the classification vector of the i-th training sample in Ω, is the score vector of the student answer of the i-th training sample in Ω, γ1 is the loss function used when training the test question classification sub-model, and γ2 is the loss function used when training the subjective question automatic scoring sub-model;
[0072] S3.2 Use the following formula to calculate the joint loss function of the subjective question automatic review neural network model for joint question classification and scoring learning:
[0073]
[0074] Among them, λ and β are two weight parameters;
[0075] S3.3 The training objective of the neural network model for automatic review of subjective questions with joint question classification and scoring learning is to minimize the joint loss error of formula (40).
[0076] Furthermore, the table lookup operation in step S2.2 specifically includes:
[0077]
[0078] Among them, OneHot(y * ) means to label the question type y * Convert to a one-hot vector, For question type y * The one-hot vector of is the test question type embedding matrix, d y is the number of question type categories, d t is the dimension of question type embedding.
[0079] In view of the fact that different types of subjective questions have different review methods, the present invention proposes a deep learning method for automatic review of subjective questions that combines test question classification with scoring learning. By combining test question classification with scoring learning, the neural network model for automatic review of subjective questions can master the marking methods of different types of test questions, and solve the problem of automatic review of subjective questions in a more effective way. First, the subjective questions to be reviewed are classified through a CNN test question classification sub-model with residual connection to obtain the test question type of the subjective question to be reviewed. Then, the test question type embedding of the subjective question to be reviewed is connected with each word embedding of the student answer input sequence and the standard answer input sequence to obtain the test question type-aware student answer perception input and the standard answer perception input, and the marking is performed on this basis.
[0080] The present invention has the following advantages:
[0081] (1) A CNN test question classification sub-model with residual connections is used to classify the subjective questions to be reviewed, so that the model can grasp the types of test questions;
[0082] (2) Connect the question type embedding of the subjective question to be reviewed with each word embedding of the student answer input sequence and the standard answer input sequence, so that the scoring sub-module can perceive the question type;
[0083] (3) Joint learning of test question classification and test question scoring can further improve the accuracy of test question classification and the adaptability of test question scoring;
[0084] (4) Scoring can be performed using a hybrid neural network consisting of a convolutional neural network and a bidirectional long short-term memory neural network, which can greatly improve the accuracy of scoring. BRIEF DESCRIPTION OF THE DRAWINGS
[0085] Figure 1 It is a schematic flow chart of the method of the present invention.
[0086] Figure 2 It is a schematic diagram of the Bi-LSTM interaction network of the present invention. DETAILED DESCRIPTION
[0087] The present invention is further described below in conjunction with specific embodiments, but the protection scope of the present invention is not limited to the following embodiments.
[0088] according to Figure 1 The flowchart of the method of the present invention shown in the figure implements a deep learning method for automatic review of subjective questions by combining test question classification and scoring learning through the following steps:
[0089] S1. Classify the subjective questions to be reviewed through a CNN question classification sub-model with residual connection to obtain the question type of the subjective questions to be reviewed;
[0090] S2. Connect the question type embedding of the subjective question to be reviewed with each word embedding of the student answer input sequence and the standard answer input sequence to obtain the question type-aware student answer perception input and the standard answer perception input, and then send them to a subjective question paper scoring sub-model composed of CNN and Bi-LSTM for scoring to obtain the scoring grade of the student answer;
[0091] S3. By jointly calculating the loss function of step S1 and step S2, a neural network model for automatic review of subjective questions for joint question classification and scoring learning is trained;
[0092] S4. Use the subjective question automatic review neural network model trained in step S3 to review any student answer;
[0093] The CNN is the abbreviation of Convolutional Neural Networks, which means convolutional neural network, and is proposed in the document “Lawrence S, Giles CL, Tsoi AC et al. Face recognition: A convolutional neural-network approach. Neural Networks, IEEE Transactions on 1997; 8(1): 98-113”.
[0094] The Bi-LSTM is the abbreviation of Didirectional Long Short-Term Memory, which means a bidirectional long short-term memory neural network, wherein the LSTM neural network is proposed by the document “Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. ¨Neural computation, 9(8): 1735–1780”;
[0095] The types of subjective questions include "definition-type subjective questions", "sequence-type subjective questions" and "general-type subjective questions". The "definition-type subjective questions" refer to subjective questions that require students to explain the definition of a concept, "sequence-type subjective questions" refer to subjective questions whose answers include multiple key points given in sequence, and the "general-type subjective questions" refer to subjective questions other than definition-type and sequence-type.
[0096] Furthermore, the step S1 specifically includes:
[0097] S1.1 The word sequence of the test question to be classified Converted into the word embedding sequence T of the test question to be classified through table lookup operation e , the calculation process is as follows:
[0098]
[0099] Where n represents the sentence length of the test question to be classified, OneHot(T) represents converting the test question word sequence T into a one-hot matrix, T o is the one-hot matrix of the test word sequence T, |V| represents the size of the dictionary V, and the dictionary V is a dictionary containing all words established by the model, is the pre-trained word embedding matrix corresponding to the dictionary V, represents the word embedding vector corresponding to the i-th word in the test word sequence T, and h represents the dimension of the word embedding vector;
[0100] S1.2 Embed the words of the test questions to be classified into sequence T e Input into a convolutional neural network with zero padding to obtain the shallow semantic representation T of the test question to be classified c , the calculation process is as follows:
[0101]
[0102] Among them, PCNN(·) represents a convolutional neural network with zero padding, (T c ) ij Indicates T c The i-th row and j-th column element of is, k is the convolution kernel size, d c is the number of convolution kernels, Represents the word embedding sequence T of the test question to be classified e The result after zero padding is: express The submatrix consisting of the j-th column to the j+(k-1)-th column, represents the weight of the i-th convolution kernel, represents the bias of the i-th convolution kernel, Represents the convolution operation;
[0103] S1.3 Shallow semantic representation of the test questions to be classified T c It is sent to a feedforward neural network consisting of two fully connected layers for enhancement to obtain the enhanced semantic representation of the test questions to be classified. The calculation process is as follows:
[0104]
[0105] in, are the weight matrices of the two fully connected layers respectively, are the bias vectors of the two fully connected layers respectively. ReLU is the abbreviation of “Rectified Linear Unit” in English. ReLU(·) represents the activation function of the rectified linear unit operation.
[0106] S1.4 The shallow semantic representation T of the test questions to be classified c Instead of enhancing the semantic representation T f Perform residual connection to obtain the deep semantic representation T of the test questions to be classified u , the calculation process is as follows:
[0107]
[0108] Among them, LayerNorm(·) represents layer normalization processing;
[0109] S1.5 Deep semantic representation of classification questions T u Perform maximum pooling to obtain the classification vector Z of the test question to be classified c , the calculation process is as follows:
[0110]
[0111] Among them, maxPooling(·) represents the maximum pooling operation;
[0112] S1.6 Classification vector Z for classification questions c Execute the linear transformation of softmax, calculate the probability of the test question type, and obtain the final test question type. The calculation process is as follows:
[0113] o=M c Z c +b o (8)
[0114]
[0115] in, is the test question type representation matrix, is a bias vector, d y is the number of categories of question types, Represents Z c The confidence score vector on all question types, y is a question type, o y Represents Z c The confidence score for question type y is marked as o i Represents Z c The confidence score on the i-th question type, P(y|Z c ) represents the classification vector Z of the test question to be classified cThe predicted probability on question type y, where Y is the set of all question types, It means to return the value that makes P(y|Z c ) is the maximum value of the test type, y * is the final classification type of the test questions to be classified, and exp(·) represents an exponential function with the natural constant e as the base.
[0116] Furthermore, the step S2 specifically includes:
[0117] S2.1 performs table lookup operation on the student answer word sequence and the standard answer word sequence according to step S1.1 to obtain the student answer word embedding sequence Ground truth word embedding sequence Then embed the test word into the sequence T e Sequences of word embeddings with student answers Ground truth word embedding sequence Connect the beginning and the end to get the student's answer input sequence A e And the standard answer input sequence S e , the calculation process is as follows:
[0118]
[0119] Among them, “:” represents the concatenation operation of the head and tail of the sequence, n1 is the sentence length of the student’s answer, n2 is the sentence length of the standard answer, and m1 is the student’s answer input sequence A. e The length of m1=n+n1, m2 is the standard answer input sequence S e The length of m2 and m2 = n + n2;
[0120] S2.2 The test question type y obtained in step S1 * Convert to the corresponding test question type through table lookup operation Then v t Sequence A with student answers e And the standard answer input sequence S e Each word embedding in is concatenated to obtain the student answer perception input A t and the standard answer perception input S t , the calculation process is as follows:
[0121]
[0122] in, They are student answer input sequence A e and student answer perception input A t The word embedding vector of the i-th column in d t is the dimension of question type embedding, They are the standard answer input sequence S e and the standard answer perception input S t The word embedding vector of the jth column in , “;” represents the concatenation operation of the vectors;
[0123] S2.3 Input student answer perception into A t and the standard answer perception input S t The results are sent to two convolutional neural networks with zero padding and shared parameters to obtain the shallow semantic representation A of the student's answer. c And the shallow semantic representation S of the standard answer c , the calculation process is as follows:
[0124]
[0125]
[0126] Among them, sharedPCNN1(·) and sharedPCNN2 represent two convolutional neural networks with zero padding and shared parameters, d c is the number of convolution kernels;
[0127] S2.4 Represent the student's answer in a shallow semantic form A c And the shallow semantic representation S of the standard answer c Send it to an interactive network composed of Bi-LSTM to get the score vector Z of the student's answer g , as shown below:
[0128]
[0129] Among them, Interactive(·) represents an interactive network composed of Bi-LSTM, d L is the dimension of the hidden state of the Bi-LSTM in the interaction network;
[0130] S2.5 The scoring vector Z for the student's answer g Perform a linear transformation of softmax, calculate the probability of the rating level, and obtain the final rating level. The calculation process is as follows:
[0131] x=M g Z g +b g (18)
[0132]
[0133] in, is the rating level representation matrix, is a bias vector, d g is the number of rating levels, It means Z g The vector of confidence scores for all rating levels, g is a rating level, x g Represents Z g The confidence score of the rating level g is labeled, x i Represents Z g The confidence score at the i-th rating level, P(g|Z g ) represents the score vector Z g The predicted probability at rating level g, G is the set of all rating levels, g * The final rating level is It means to return the value that makes P(g|Z g ) is the maximum rating level.
[0134] Furthermore, the step S2.4 specifically includes:
[0135] S2.4.1 Use two Bi-LSTMs with shared parameters to represent the shallow semantics of the student’s answer A c And the shallow semantic representation S of the standard answer c Encode and get the context representation of the student's answer A r and the standard answer context representation S r , the calculation process is as follows:
[0136]
[0137] Among them, sharedBiLSTM1(·) and sharedBiLSTM2(·) represent two Bi-LSTMs with shared parameters. Respectively represent A c With S c The connection result of the bidirectional hidden state corresponding to the i-th time step in Bi-LSTM, Respectively represent A c With S c From left to right The hidden state of the network at the ith time step, Respectively represent A c With S c From right to left The hidden state of the network at the i-th time step;
[0138] S2.4.2 Representing the context of student answers through attention mechanism A r and the standard answer context representation S r Interact and gain mutual attention and The calculation process is as follows;
[0139]
[0140] in, for The i-th element in for The jth element in Respectively express and Perform a transpose operation, is the universal logical quantifier representing "all", Represents the hidden state tuple and The attention weight of
[0141] S2.4.3 will and A with student answers in context r and the standard answer context representation S r Fusion is performed to obtain the student's answer fusion representation A f The fusion of the standard answer S f , the calculation process is as follows:
[0142]
[0143] in, A f The i-th element in For S f The i-th element in , Fusion(·) represents the fusion operation, and ⊙ represents the matrix element-by-element multiplication operation;
[0144] S2.4.4 Use two Bi-LSTMs with shared parameters to represent the student answers A f The fusion of the standard answer S f Perform enhancement processing to obtain the deep semantic representation A of the student's answer u and the deep semantic representation of the standard answer S u , the calculation process is as follows:
[0145]
[0146] in, Respectively represent A f With S f The concatenation result of the bidirectional hidden states corresponding to the i-th and j-th time steps in the Bi-LSTM;
[0147] S2.4.5 Deep semantic representation of student answers u and the deep semantic representation of the standard answer S uPerform average pooling and maximum pooling, and connect all the pooled vectors after pooling to get the score vector Z of the student's answer g , the calculation process is as follows:
[0148]
[0149] Among them, maxPooling(·) represents the maximum pooling operation, avePooling(·) represents the average pooling operation, Yes A u The maximum pooling vector of Yes A u The average pooling vector of YesS u The maximum pooling vector of YesS u The average pooling vector of , “;” represents the concatenation operation of the vectors.
[0150] Furthermore, the step S3 specifically includes:
[0151] S3.1 uses the cross entropy loss error to calculate the loss function of the test question classification sub-model and the loss function of the subjective question automatic scoring sub-model. The calculation process is as follows:
[0152]
[0153] Among them, Ω is the training set of the automatic review task of subjective questions for joint question classification and scoring learning, |Ω| represents the size of the training set Ω, is the real test question classification of the i-th training sample in Ω, is the true rating of the i-th training sample in Ω, is the classification vector of the i-th training sample in Ω, is the score vector of the student answer of the i-th training sample in Ω, Υ1 is the loss function used when training the test question classification sub-model, and Υ2 is the loss function used when training the subjective question automatic scoring sub-model;
[0154] S3.2 Use the following formula to calculate the joint loss function of the subjective question automatic review neural network model for joint question classification and scoring learning:
[0155]
[0156] Among them, λ and β are two weight parameters;
[0157] S3.3 The training objective of the neural network model for automatic review of subjective questions with joint question classification and scoring learning is to minimize the joint loss error of formula (40).
[0158] Furthermore, the table lookup operation in step S2.2 specifically includes:
[0159]
[0160] Among them, OneHot(y * ) means to label the question type y * Convert to a one-hot vector, For question type y * The one-hot vector of is the test question type embedding matrix, d y is the number of question type categories, d t is the dimension of question type embedding.
[0161] Application Examples
[0162] 1. Instance Hyperparameters
[0163] Table 1 Example model hyperparameter settings
[0164]
[0165] 2. Dataset
[0166] The data for this example comes from the final exam papers of Computer Networks for undergraduates majoring in software engineering at a certain university. A total of 3948 Chinese subjective questions were obtained. Each question has 1 standard answer and 4 student answers. Finally, a question classification dataset containing 3948 Chinese subjective questions and a Chinese subjective question automatic scoring dataset with 15792 student answers were obtained, as shown in Table 2 and Table 3:
[0167] Table 2 Question classification dataset
[0168]
[0169] Table 3 Automatic scoring dataset division
[0170]
[0171] 3. Example comparison results
[0172] Table 4 gives the classification and comparison results of the test question types of various models, and Table 5 gives the classification and comparison results of the score ranges of student answers of various models:
[0173] Table 4 Comparison of classification results of test question types of various models
[0174] Model Accuracy Recall Precision F1 value Joint scoring test question classification model 96.03 96.1 95.60 95.85 Independent Question Classification Model 94.56 94.21 94.56 94.38
[0175] Table 5 Classification comparison results of student answer scoring intervals of various models
[0176]
[0177] The experimental results in Tables 4 and 5 show that the deep learning method for automatic review of subjective questions that combines question classification and scoring learning proposed in the present invention is significantly better than the independent review model and the independent question classification model, and is significantly better than the review model without a CNN layer and the review model without a Bi-LSTM interaction layer, which fully demonstrates that the method of the present invention is feasible and excellent.
[0178] 4. Examples
[0179] Table 6 gives examples of student answer scores for various question types:
[0180] Table 6 Examples of scoring of student answers for various question types
[0181]
Claims
1. A deep learning method for automatic review of subjective questions by combining test question classification and scoring learning, characterized in that The following steps are involved: S1. Classify the subjective questions to be reviewed through a CNN question classification sub-model with residual connection to obtain the question type of the subjective questions to be reviewed; S2. Connect the question type embedding of the subjective question to be reviewed with each word embedding of the student answer input sequence and the standard answer input sequence to obtain the question type-aware student answer perception input and the standard answer perception input, and then send them to a subjective question paper scoring sub-model composed of CNN and Bi-LSTM for scoring to obtain the scoring grade of the student answer; S3. By jointly calculating the loss function of step S1 and step S2, a neural network model for automatic review of subjective questions for joint question classification and scoring learning is trained; S4. Use the subjective question automatic review neural network model trained in step S3 to review any student answer; The types of subjective questions include "definition-type subjective questions", "sequence-type subjective questions" and "general-type subjective questions". The "definition-type subjective questions" refer to subjective questions that require students to explain the definition of a certain concept, "sequence-type subjective questions" refer to subjective questions whose answers include multiple key points given in sequence, and "general-type subjective questions" refer to subjective questions other than definition-type and sequence-type questions. The step S1 specifically includes: S1.1 The word sequence of the test question to be classified Converted into the word embedding sequence T of the test question to be classified through table lookup operation e , the calculation process is as follows: Where n represents the sentence length of the test question to be classified, OneHot(T) represents converting the test question word sequence T into a one-hot matrix, T o is the one-hot matrix of the test word sequence T, |V| represents the size of the dictionary V, and the dictionary V is a dictionary containing all words established by the model, is the pre-trained word embedding matrix corresponding to the dictionary V, represents the word embedding vector corresponding to the i-th word in the test word sequence T, and h represents the dimension of the word embedding vector; S1.2 Embed the words of the test questions to be classified into sequence T e Input into a convolutional neural network with zero padding to obtain the shallow semantic representation T of the test question to be classified c , the calculation process is as follows: Among them, PCNN(·) represents a convolutional neural network with zero padding, (T c ) ij Indicates T c The i-th row and j-th column element of is, k is the convolution kernel size, d c is the number of convolution kernels, Represents the word embedding sequence T of the test question to be classified e The result after zero padding, (T e p ) j:j+(k-1) express The submatrix consisting of the j-th column to the j+(k-1)-th column, represents the weight of the i-th convolution kernel, represents the bias of the i-th convolution kernel, Represents the convolution operation; S1.3 Shallow semantic representation of the test questions to be classified T c It is sent to a feedforward neural network consisting of two fully connected layers for enhancement to obtain the enhanced semantic representation of the test questions to be classified. The calculation process is as follows: in, are the weight matrices of the two fully connected layers respectively, They are the bias vectors of the two fully connected layers respectively. ReLU is the abbreviation of "Rectified Linear Unit" in English. ReLU() represents the activation function of the rectified linear unit operation. S1.4 The shallow semantic representation T of the test questions to be classified c Instead of enhancing the semantic representation T f Perform residual connection to obtain the deep semantic representation T of the test questions to be classified u , the calculation process is as follows: Among them, LayerNorm(·) represents layer normalization processing; S1.5 Deep semantic representation of classification questions T u Perform maximum pooling to obtain the classification vector Z of the test question to be classified c , the calculation process is as follows: Among them, maxPooling(·) represents the maximum pooling operation; S1.6 Classification vector Z for classification questions c Execute the linear transformation of softmax, calculate the probability of the test question type, and obtain the final test question type. The calculation process is as follows: o=M c Z c +b o (8) in, is the test question type representation matrix, is a bias vector, d y is the number of categories of question types, Represents Z c The confidence score vector on all question types, y is a question type, o y Represents Z c The confidence score for question type y is marked as o i Represents Z c The confidence score on the i-th question type, Ρ(y|Z c ) represents the classification vector Z of the test question to be classified c The predicted probability on question type y, where Y is the set of all question types, It means to return the value such that Ρ(y|Z c ) is the maximum value of the test type, y * is the final classification type of the test questions to be classified, exp(·) represents the exponential function with the natural constant e as the base; The step S2 specifically includes: S2.1 performs table lookup operation on the student answer word sequence and the standard answer word sequence according to step S1.1 to obtain the student answer word embedding sequence Ground truth word embedding sequence Then embed the test word into the sequence T e Sequences of word embeddings with student answers Ground truth word embedding sequence Connect the beginning and the end to get the student's answer input sequence A e And the standard answer input sequence S e , the calculation process is as follows: Among them, ":" represents the concatenation operation of the head and tail of the sequence, n1 is the sentence length of the student's answer, n2 is the sentence length of the standard answer, and m1 is the student's answer input sequence A e The length of m1=n+n1, m2 is the standard answer input sequence S e The length of m2=n+n2; S2.2 The test question type y obtained in step S1 * Convert to the corresponding test question type through table lookup operation Then v t Sequence A with student answers e And the standard answer input sequence S e Each word embedding in is concatenated to obtain the student answer perception input A t and the standard answer perception input S t , the calculation process is as follows: in, They are student answer input sequence A e and student answer perception input A t The word embedding vector of the i-th column in d t is the dimension of question type embedding, The standard answer input sequence S is e and the standard answer perception input S t The word embedding vector of the jth column in , ";" represents the concatenation operation of the vectors; S2.3 Input student answer perception into A t and the standard answer perception input S t The results are sent to two convolutional neural networks with zero padding and shared parameters to obtain the shallow semantic representation A of the student's answer. c And the shallow semantic representation S of the standard answer c , the calculation process is as follows: Among them, sharedPCNN1(·) and sharedPCNN2 represent two convolutional neural networks with zero padding and shared parameters, d c is the number of convolution kernels; S2.4 Represent the student's answer in a shallow semantic form A c And the shallow semantic representation S of the standard answer c Send it to an interactive network composed of Bi-LSTM to get the score vector Z of the student's answer g , as shown below: Among them, Interactive(·) represents an interactive network composed of Bi-LSTM, d L is the dimension of the hidden state of the Bi-LSTM in the interaction network; S2.5 The scoring vector Z for the student's answer g Perform a linear transformation of softmax, calculate the probability of the rating level, and obtain the final rating level. The calculation process is as follows: x=M g Z g +b g (18) in, is the rating level representation matrix, is a bias vector, d g is the number of rating levels, It means Z g The vector of confidence scores for all rating levels, g is a rating level, x g Represents Z g The confidence score of the rating level g is labeled, x i Represents Z g The confidence score at the i-th rating level, Ρ(g|Z g ) represents the score vector Z g The predicted probability at rating level g, G is the set of all rating levels, g * The final rating level is It means to return the value such that Ρ(g|Z g ) is the maximum rating level; The step S2.4 specifically includes: S2.4.1 Use two Bi-LSTMs with shared parameters to represent the shallow semantics of the student’s answer A c And the shallow semantic representation S of the standard answer c Encode and get the context representation of the student's answer A r and the standard answer context representation S r , the calculation process is as follows: Among them, sharedBiLSTM1(·) and sharedBiLSTM2(·) represent two Bi-LSTMs with shared parameters. Respectively represent A c With S c The connection result of the bidirectional hidden state corresponding to the i-th time step in the Bi-LSTM; S2.4.2 Representing the context of student answers through attention mechanism A r and the standard answer context representation S r Interact and get mutual attention r and S r , the calculation process is as follows; in, A r The i-th element in For S r The jth element in Respectively express and Perform a transpose operation, is a universal logical quantifier representing "all". Represents the hidden state tuples respectively and The attention weight of S2.4.3 A r and S r A with student answers in context r and the standard answer context representation S r Fusion is performed to obtain the student's answer fusion representation A f The fusion of the standard answer S f , the calculation process is as follows: in, A f The i-th element in For S f The i-th element in , Fusion() represents the fusion operation, and ⊙ represents the matrix element-by-element multiplication operation; S2.4.4 Use two Bi-LSTMs with shared parameters to represent the student answers A f The fusion of the standard answer S f Perform enhancement processing to obtain the deep semantic representation A of the student's answer u and the deep semantic representation of the standard answer S u , the calculation process is as follows: in, Respectively represent A f With S f The concatenation result of the bidirectional hidden states corresponding to the i-th and j-th time steps in the Bi-LSTM; S2.4.5 Deep semantic representation of student answers u and the deep semantic representation of the standard answer S u Perform average pooling and maximum pooling, and connect all the pooled vectors after pooling to get the score vector Z of the student's answer g , the calculation process is as follows: Among them, maxPooling(·) represents the maximum pooling operation, avePooling(·) represents the average pooling operation, Yes A u The maximum pooling vector of Yes A u The average pooling vector of YesS u The maximum pooling vector of YesS u The average pooling vector of , ";" represents the concatenation operation of the vector.
2. According to claim 1, a deep learning method for automatic review of subjective questions combined with test question classification and scoring learning is characterized by: The step S3 specifically includes: S3.1 uses the cross entropy loss error to calculate the loss function of the test question classification sub-model and the loss function of the subjective question automatic scoring sub-model. The calculation process is as follows: Among them, Ω is the training set of the automatic review task of subjective questions for joint question classification and scoring learning, |Ω| represents the size of the training set Ω, is the real test question classification of the i-th training sample in Ω, g i is the true rating of the i-th training sample in Ω, is the classification vector of the i-th training sample in Ω, is the score vector of the student answer of the i-th training sample in Ω, Υ1 is the loss function used when training the test question classification sub-model, and Υ2 is the loss function used when training the subjective question automatic scoring sub-model; S3.2 Use the following formula to calculate the joint loss function of the subjective question automatic review neural network model for joint question classification and scoring learning: Among them, λ and β are two weight parameters; S3.3 The training objective of the neural network model for automatic review of subjective questions by joint question classification and scoring learning is to minimize the joint loss error of formula (40).
