Chinese subjective question automatic scoring method and system

Abstract

The invention relates to an automatic scoring method and system for Chinese subjective questions, and the method comprises the steps: 1, inputting a user answer and a reference answer into a model, calculating a similarity matrix of the user answer and the reference answer through a bidirectional attention mechanism, and obtaining the attention vectors of the user answer and the reference answer; 2, splicing the two attention vectors by an output layer, and calculating a score of a user answer according to a similarity matrix of the two attention vectors through a convolutional neural network; 3, generating a disturbance data set and calculating instance similarity; 4, calculating a model prediction value; 6, calculating a predicted value of the simple model; 7, obtaining a final simple model; and step 8, calculating and explaining a correlation value between the user answer word and the model score. According to the method and the device, not only can automatic scoring be realized, but also the score can be explained, so that the user trust model gives a score feedback result, and the use experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic marking, and in particular to a method and system for automatically marking Chinese subjective questions. Background Art

[0002] Automatic scoring of subjective questions refers to the establishment of a scoring model based on certain scoring criteria to achieve automatic scoring of user answers to subjective questions. Automatic scoring models were first born in the 1960s. In the early days, such models were mainly used to assist in the actual scoring of large-scale exams such as TOEFL and GRE. With the development of computer technology, the scoring ability of the model has also increased day by day. Now its scoring objects have been expanded to subjective questions such as short answers and oral exams. From a technical point of view, automatic scoring models can be divided into three categories: rule-based, traditional machine learning-based, and deep learning-based. Rule-based automatic scoring models require manual construction of rules in the form of concept maps and corpora based on certain standards to establish models. The model mainly analyzes the composition and grammar of the user's answers to make scoring predictions. For users, the composition and grammar of the answers analyzed by the model can be used as a basis for scoring to help them understand the decision logic of the model. Therefore, such models have good interpretability. However, since the rules designed by humans are often limited, once users have mastered the model's correction rules, they can deceive the model by using a large number of key phrases and complex sentence patterns to get high scores. The automatic scoring model based on traditional machine learning requires people to establish various features in advance, so as to establish a classification or regression model and realize the review of answers. Compared with the rule-based model, this type of model can analyze the user's answer as a whole, and its scoring accuracy is higher. In addition, system developers can display some weight parameters and logical rules of the model in the system in an appropriate way, give the scoring basis of the model, and help users understand its decision logic. The automatic scoring model based on deep learning is a current research hotspot. This type of model has better performance but more complex structure. It can deeply mine the semantic information in the user's answer and achieve accurate scoring. However, the model decision-making process is not transparent, and its application in the intelligent guidance system will bring risk problems to the system and affect the user's experience. Therefore, how to use existing technical methods to explain the model and make the decision-making process of the service function supported by the model in the system open and transparent is a key problem that needs to be solved in the application of this type of model.

[0003] With the support of deep learning technology, the automatic scoring model can extract implicit features from massive data and achieve more accurate automatic scoring of learners' multi-type answers. However, due to the uncontrollable training process, the model is likely to learn rules that violate educational ethics, and the model's decision-making process is like a black box. While giving the scoring results, it cannot provide specific basis for the score. For users, the basis for scoring is the key explanatory information they need to understand the working logic of the system scoring function and correct problems with their answers. Without this information, users will not be able to understand its decision-making logic and will distrust the feedback information. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art. Taking into account the characteristics of the problem of automatic scoring of Chinese subjective questions, a method and system for automatic scoring of Chinese subjective questions are provided based on explainable artificial intelligence technology, and corresponding explanations are given for the scoring.

[0005] The automatic scoring method for Chinese subjective questions provided by the present invention comprises the following steps:

[0006] Step 1: Input the user answer and the reference answer into the model, use the bidirectional attention mechanism to calculate the similarity matrix between the two and obtain the attention vectors of the two. and

[0007] Step 2: The output layer concatenates the two attention vectors and uses a convolutional neural network to calculate the score of the user's answer based on the similarity matrix between the two.

[0008] Step 3: Generate perturbation data set and instance similarity calculation;

[0009] Step 4: Calculate the model prediction value;

[0010] Step 6: Calculation of predicted values ​​of simple model;

[0011] Step 7: Get the final simple model;

[0012] Step 8: Calculate the correlation value between the user answer words and the model score, and generate an explanation.

[0013] Furthermore, in step 1, after the user answer and the reference answer are input into the scoring model, the encoding layer divides the answer into words and encodes both through a bidirectional long short-term memory network to obtain a vector representation of the user answer and the reference answer containing context information, and the attention layer uses a bidirectional attention mechanism to calculate the similarity matrix of the two and obtain the attention vectors of the two.

[0014] Furthermore, in step 2, the output layer concatenates the two attention vectors and the convolutional neural network calculates the score of the user's answer based on the similarity between the two.

[0015] Furthermore, in step 3, first, the user answer is assumed to be a vector, and the user answer is perturbed by the set perturbation function f to generate a perturbed data set. The cosine similarity calculation formula is used to calculate the similarity between each instance in the perturbed data set and the user answer vector, thereby obtaining a similarity numerical sequence of all instances in the perturbed data set relative to the user answer.

[0016] Furthermore, in step 4, all instances in the perturbed data set are input into the scoring model one by one to obtain a scoring model prediction value sequence, where each prediction value in the sequence corresponds to a specific instance in the perturbed data set.

[0017] Furthermore, in step 6, a decision tree model is trained based on the perturbation data set to obtain a simple linear regression model SM. With the help of the calculation logic inside the simple model, a simple model prediction value sequence corresponding to each instance is obtained, which reflects the prediction output result of the simple model for the instance in the perturbation data set.

[0018] Furthermore, it also includes:

[0019] Step 5: Simple model training: train the simple model SM with the perturbation data set and the score model prediction value sequence. The loss function of the simple model is

[0020] Further, in step 7, based on the similarity value sequence between the perturbation instance and the user answer determined in step 3, the scoring model prediction value sequence obtained in step 4, and the simple model prediction value sequence generated in step 6, the local prediction gap is calculated for each instance according to the pre-set local prediction gap calculation rule, thereby determining the prediction gap sequence between the simple model and the scoring model in the local range;

[0021] Furthermore, steps 3 to 7 are executed in a loop, and the simple model with the smallest local prediction gap with the scoring model is used as the final simple model.

[0022] The present invention also provides an automatic scoring system for Chinese subjective questions, including a scoring model, a similarity calculation module, a model prediction value calculation module, a final simple model generation module, and an explanation module, wherein:

[0023] The scoring model is used to calculate the similarity matrix between the user answer and the reference answer and obtain the attention vector of the two. The two attention vectors are concatenated through the output layer, and then the score of the user answer is calculated based on the similarity matrix of the two through the convolutional neural network.

[0024] Similarity calculation module, used to generate perturbation data sets and instance similarity calculation;

[0025] Model prediction value calculation module; used to calculate the prediction value of the scoring module and the simple module for the instance;

[0026] The final simple model generation module is used to calculate the local prediction gap and use the simple model with the smallest gap as the final simple model.

[0027] The explanation module is used to calculate the association value between the user's answer words and the score, and explain the score.

[0028] The advantages of the automatic scoring method and system for Chinese subjective questions proposed by the present invention are:

[0029] (1) It provides explanation services for ratings, which can promote users’ trust in the rating system and their technical acceptance without increasing the cognitive load of users.

[0030] (2) Help users understand which parts of their answers have a greater impact on the score and which scoring points they have achieved, so that they can gradually understand the scoring logic of the service function.

[0031] (3) With the help of scoring criteria, scoring points and question analysis, users can improve their answers in a targeted manner and master the knowledge points tested in the questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of steps 1-2 of an embodiment of the present invention;

[0033] Figure 2 This is a flow chart of steps 3-8 of an embodiment of the present invention;

[0034] Figure 3 The figure is a schematic diagram of the system structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0036] The present invention proposes a method for automatically grading Chinese subjective questions. Figure 1-2 As shown, the following steps are included:

[0037] Step 1: Input the user answer and the reference answer into the scoring model, use the bidirectional attention mechanism to calculate the similarity matrix between the two and obtain the attention vectors of the two. and

[0038] Step 2: The output layer concatenates the two attention vectors and uses a convolutional neural network to calculate the score of the user's answer based on the similarity matrix between the two.

[0039] Step 3: Generate perturbation data set and instance similarity calculation;

[0040] Step 4: Calculate the model prediction value;

[0041] Step 6: Simple model prediction value calculation step;

[0042] Step 7: Local prediction gap calculation;

[0043] Step 8: Calculate the correlation value between the user answer word and the model score.

[0044] In step 1, the scoring model includes an encoding layer, an attention layer, and an output layer. The encoding layer divides the answer into words and encodes the two through a bidirectional long short-term memory network to obtain a vector representation of the user answer and the reference answer containing context information. Then the attention layer uses the bidirectional attention mechanism to calculate the similarity matrix of the two and obtain the attention vectors of the two and

[0045] In step 2, the output layer concatenates the two attention vectors and inputs them into the convolutional neural network of this layer. The score of the user's answer is calculated based on the similarity between the two. The calculation formula is:

[0046]

[0047] Among them, β is the concatenated vector, K represents the category of subjective question scoring, and P(y=k) represents the probability of different scores of user answers.

[0048] The scoring model may adopt an existing model, or may be designed and trained by itself, but the following steps of the present invention will explain its scoring.

[0049] In the present invention, a simple model with prior interpretability is trained to approximate the local boundary of the scoring model, and the correlation value between the scoring model input features and the scoring model output is obtained based on the components of the simple model, so as to explain the decision logic of the scoring model, such as Figure 2 As shown, it specifically includes steps 3 to 8.

[0050] In step 3, first, let the user's answer be a vector A = (a1, a2, ..., a n ), the user's answer is perturbed by the set perturbation function f to generate a perturbation data set D = {f(A)1, f(A)2, ..., f(A) m}, where f(A) i =(b {i1} ,b {i2} ,…,b {in} ) represents the i-th instance in the perturbed data set. Then, the cosine similarity calculation formula is used to calculate each instance f(A) in the perturbed data set. i , respectively calculate the similarity between them and the user answer vector A, so as to obtain the similarity value sequence S = {cosθ1, cosθ2, …, cosθ m}. The formula for calculating cosine similarity is as follows:

[0051]

[0052] In step 4, all instances f(A) in the perturbed dataset D generated in step 3 are i , are input into the scoring model M one by one, and after the internal calculation and processing mechanism of the scoring model, the corresponding model prediction value sequence P = {p1, p2, ... p m}, each predicted value p in the sequence i Each corresponds to a specific instance f(A) in the perturbed dataset i , model prediction value p i =M(f(A) i ).

[0053] In step 6, based on the perturbed data set, the decision tree model is trained using conventional machine learning training methods to obtain a simple linear regression model SM. With the help of the calculation logic within the simple model, the simple model prediction value sequence Q corresponding to each instance is obtained = {q1, q2, ... q m}, which reflects the predicted output of the simple model for the instances in the perturbation dataset, i.e., q i =SM(f(A) i ).

[0054] Specific training process: In view of the fact that the features of the model input instances are all non-zero features, the perturbation data set D generated in step 3 and the score model prediction value sequence P obtained in step 4 are used as the basic data to directly carry out the training operation on the simple model SM. Assume that the loss function of the simple model is,

[0055]

[0056] During the training process, the internal parameters of the simple model are adjusted and optimized according to a specific training algorithm (such as the gradient descent algorithm) and the above basic data, so that the simple model can gradually fit the relationship between the data features and the prediction target, and the loss function L is minimized by continuously updating the parameter θ. The update formula is: Where α is the learning rate.

[0057] In step 7, based on the similarity value sequence S between the perturbation instance and the user answer determined in step 3, the model prediction value sequence P obtained in step 4, and the simple model prediction value sequence Q generated in step 6, the local prediction gap G is calculated for each instance i according to the pre-set local prediction gap calculation formula. i = cosθ i ×(p i -q i ), thereby determining the prediction gap sequence G = {g1, g2, ... g m}. Steps 3 to 7 are executed repeatedly. In each cycle, the local prediction gap G between the obtained simple model and the scoring model is evaluated and compared. Finally, the simple model SMmin with the smallest local prediction gap with the scoring model is selected and retained.

[0058] In step 8, the weight parameters of the simple model SMmin are set to ω = {ω1, ω2, ... ω m}, then the correlation value between each word in the user's answer and the model score is R = {ω1×a1, ω2×a2, …, ω n ×a n}, thereby achieving effective explanation and analysis of the model's decision-making process and results. Among them, a "positive" correlation value means that the input word will increase the probability of the scoring model predicting a certain score for the user's answer, and a "negative" correlation value means that the input word will reduce the probability of the scoring model predicting a certain score for the user's answer. The larger the absolute value of the correlation value, the greater the impact of the input word on the scoring model.

[0059] The following is an example of a system with a total score of 2 points for each subjective question. After the user enters the answer in the answer box and clicks OK, the scoring system scores the user's answer and gives the probability of getting 0, 1, or 2 points. The score with the highest probability of scoring is used as the final score of the user's answer. Next, the scoring system interprets the association value and outputs the scoring result based on the user's score. Specifically, when the user's answer score is 0, the system does not present the basis for scoring judgment, but directly tells the user that he has not answered any scoring points correctly and tells him the score. When the user's answer score is 1, the scoring system calculates the average value of the absolute value of the association value of all words in the user's answer, and highlights the words with an absolute value of the association value greater than the average value as the basis for scoring judgment. Then the system calculates the cosine similarity between the highlighted words and the scoring points of the question, so as to tell the user which points the service function believes he has answered and which points he has not answered, and then makes a scoring judgment. When the user's answer score is 2, the scoring system will still highlight the words with an absolute value of the association value greater than the average value and present the scoring points answered by the user.

[0060] By viewing the scoring process and score analysis, users can understand which parts of their answers have a greater impact on the score and which scoring points they have answered, and gradually understand the scoring logic. In addition, with the help of the scoring basis, scoring points and question analysis, users can improve their answers in a targeted manner and master the knowledge points tested by the questions.

[0061] According to another aspect of the present invention, a Chinese subjective test automatic scoring system is provided. Figure 3 As shown, it includes a scoring model, a similarity calculation module, a model prediction value calculation module, a final simple model generation module, and an explanation module, where:

[0062] Scoring model, used to calculate the similarity matrix between the user's answer and the reference answer and obtain the attention vector of both and The two attention vectors are concatenated through the output layer, and then the score of the user's answer is calculated based on the similarity matrix of the two through a convolutional neural network; the specific implementation is described in steps 1-2.

[0063] The similarity calculation module is used to generate a perturbation data set and calculate instance similarity; the specific implementation is as described in step 3.

[0064] Model prediction value calculation module; used to calculate the prediction value of the scoring module and the simple module for the instance; the specific implementation is described in steps 4 and 6.

[0065] The final simple model generation module is used to calculate the local prediction gap and use the simple model with the smallest gap as the final simple model. The specific implementation is as described in step 7.

[0066] The explanation module is used to calculate the correlation value between the user's answer words and the score, and explain the score; the specific implementation is as described in step 8.

[0067] Finally, it should be noted that the above embodiments are only used to describe the technical solution of the present invention rather than to limit the technical method. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatic scoring of Chinese subjective questions, characterized in that: The following steps are involved: Step 1: Input the user answer and the reference answer into the model, use the bidirectional attention mechanism to calculate the similarity matrix between the two and obtain the attention vectors of the two; Step 2: The output layer concatenates the two attention vectors and uses a convolutional neural network to calculate the score of the user's answer based on the similarity matrix between the two. Step 3: Generate perturbation data set and instance similarity calculation; Step 4: Calculation of model prediction value; Step 6: Calculation of predicted values ​​of simple model; Step 7: Get the final simple model; Step 8: Calculate the correlation value between the user answer words and the model score, and generate an explanation.

2. The method according to claim 1, characterized in that: In step 1, after the user answer and the reference answer are input into the scoring model, the encoding layer divides the answer into words and encodes both through a bidirectional long short-term memory network to obtain a vector representation of the user answer and the reference answer containing context information, and the attention layer uses a bidirectional attention mechanism to calculate the similarity matrix of the two and obtain the attention vectors of the two.

3. The method according to claim 2, characterized in that: In step 2, the output layer concatenates the two attention vectors and the convolutional neural network calculates the score of the user's answer based on the similarity between the two.

4. The method according to claim 2, characterized in that: In step 3, first, the user answer is assumed to be a vector, and the user answer is perturbed by the set perturbation function f to generate a perturbed data set. The cosine similarity calculation formula is used to calculate the similarity between each instance in the perturbed data set and the user answer vector, thereby obtaining a similarity numerical sequence of all instances in the perturbed data set relative to the user answer.

5. The method according to claim 2, characterized in that: In step 4, all instances in the perturbed data set are input into the scoring model one by one to obtain a scoring model prediction value sequence, where each prediction value in the sequence corresponds to a specific instance in the perturbed data set.

6. The method according to claim 2, characterized in that: In step 6, a decision tree model is trained based on the perturbation data set to obtain a simple linear regression model SM. With the help of the calculation logic inside the simple model, a simple model prediction value sequence corresponding to each instance is obtained, which reflects the prediction output result of the simple model for the instance in the perturbation data set.

7. The method according to claim 6, characterized in that: Also includes: Step 5: Simple model training: train the simple model SM with the perturbation data set and the scoring model prediction value sequence. The loss function of the simple model is 8. The method according to claim 2, characterized in that: In step 7, based on the similarity value sequence between the perturbation instance and the user answer determined in step 3, the scoring model prediction value sequence obtained in step 4, and the simple model prediction value sequence generated in step 6, the local prediction gap is calculated for each instance according to the pre-set local prediction gap calculation rule, thereby determining the prediction gap sequence between the simple model and the scoring model in the local range.

9. The method according to claim 8, characterized in that: Steps 3 to 7 are executed repeatedly, and the simple model with the smallest local prediction gap with the scoring model is taken as the final simple model.

10. An automatic scoring system for Chinese subjective questions, characterized in that: It includes a scoring model, a similarity calculation module, a model prediction value calculation module, a final simple model generation module, and an explanation module, among which: The scoring model is used to calculate the similarity matrix between the user answer and the reference answer and obtain the attention vector of the two. The two attention vectors are concatenated through the output layer, and then the score of the user answer is calculated based on the similarity matrix of the two through the convolutional neural network. Similarity calculation module, used to generate perturbation data sets and instance similarity calculation; Model prediction value calculation module; used to calculate the prediction value of the scoring module and the simple module for the instance; The final simple model generation module is used to calculate the local prediction gap and take the simple model with the smallest gap as the final simple model; The explanation module is used to calculate the correlation value between the user's answer words and the score, and explain the score.

Images