Intelligent scoring method for domain-oriented subjective questions based on machine learning

Abstract

This paper presents a machine learning-based intelligent scoring method for domain-specific subjective questions. It constructs a scoring network framework, including a data preprocessing module, a module for extracting association features between standard answers and student answers, a module for extracting student answer features, and a scoring module. The data preprocessing module performs semantic space transformation to address semantic ambiguity in student answer texts. The module for extracting association features between standard answers and student answers uses a word-level attention mechanism to obtain the association features f between standard answers and student answers. r The candidate answer feature extraction module extracts the contextual features f of the candidate's answer. a The scoring module combines f r and f a The method of this invention can effectively handle semantic ambiguity, enhance semantic recognition capabilities, realize intelligent scoring of domain-specific subjective questions, support scoring tasks for subjective questions with multiple standard answers, and is flexibly applicable to different application scenarios. It has demonstrated high accuracy in scoring domain-specific subjective questions, especially in scoring domain-specific subjective questions with multiple answer formats.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and natural language processing technology, and relates to the understanding and evaluation of subjective question answers. It is a machine learning-based intelligent scoring method for domain-specific subjective questions. Background Technology

[0002] Subjective questions, as one of the most basic question types in modern examinations, aim to understand a student's comprehension and mastery of a specific part of a course, testing their abilities in material extraction, problem understanding, language expression, and grasping the essence of the problem. They examine a student's specific situation or personality and are a general term for all text-based question types in examinations. In research on automatic grading of subjective questions, most algorithms compare the degree of matching between student answers and standard answers. However, due to the complexity of Chinese semantics and the existence of synonyms and near-synonyms, the accuracy of this algorithm is greatly affected. With the development of deep learning, many research methods based on neural networks have been applied to the automatic grading of subjective questions, achieving some good results. However, many problems still exist in the research on automatic grading of subjective questions, such as semantic ambiguity in student answers, answer polymorphism, and domain-specific issues. In addition, subjective questions will have scoring points, which are set by the examiners. Each scoring point corresponds to a standard answer. When grading by humans, points are awarded based on the number of correct answers to each scoring point. Therefore, automatic grading of subjective questions also needs to consider whether to grade based on scoring points or the entire question.

[0003] Riordan B et al. [1] Through comparative analysis of various ablation experiments, a Neural SAG model based on short text processing was introduced, and state-of-the-art (SOTA) results were achieved on the ASAP-SAS dataset. Ramachandran L et al. [2] Designing an automatic pattern recognition method based on regular expressions and using a word order graph for answer matching improves the efficiency of manually designed feature extraction. However, this recognition method does not consider semantic issues, and semantic ambiguity can still lead to mismatches between the recognition pattern and the answer. Tomoya Mizumoto et al. [3] The design of scoring and deduction rules based on attention mechanisms, followed by classification training and summarization of the results, has achieved good results. However, the design of deduction rules has limitations and is not suitable for the design of scoring models for most subjective questions.

[0004] References

[0005] [1]Riordan B,Horbach A,Cahill A,et al.Investigating neuralarchitectures for short answer scoring[C].BEA@EMNLP,2017.

[0006] [2]Ramachandran L,Cheng J,Foltz P W.Identifying Patterns For ShortAnswer Scoring Using Graph-based Lexico-Semantic Text Matching[C].BEA@NAACL-HLT,2015.

[0007] [3]Mizumoto T,Ouchi H,Isobe Y,et al.Analytic Score Prediction and Justification Identification in Automated Short Answer Scoring[C].BEA@ACL,2019. Summary of the Invention

[0008] The problem this invention aims to solve is: the decline in the scoring performance of subjective question models due to semantic ambiguity and loss of semantic information in domain subjective questions, and the limitation of subjective question model scoring performance caused by low accuracy of subjective question feature extraction; and a technical solution for scoring subjective questions with multiple answer formats in the domain.

[0009] The technical solution of this invention is as follows: A machine learning-based intelligent scoring method for domain-specific subjective questions. A scoring network framework is constructed, including a data preprocessing module, a standard answer and candidate answer association feature extraction module, a candidate answer feature extraction module, and a scoring module. The data preprocessing module uses a general domain dictionary and a specific domain dictionary, employing a Latent Semantic Analysis (LSA) topic model to model all candidate answers, obtaining an LSA model. Based on the LSA model, core topic words are obtained from all candidate answers. Then, a semantic space transformation mechanism is obtained by designing conversion rules between core topic words and standard answer words, improving the semantic recognition of core words in candidate answers. The standard answer and candidate answer association feature extraction module uses a word-level attention mechanism to obtain candidate answer text vectors p and q that are highly related to the standard answer. By concatenating p and q, the association feature f between the standard answer and candidate answer is obtained. r The candidate answer feature extraction module extracts the contextual feature vector f from the candidate's answer.a The scoring module combines f r and f a We get f = [f a ;f r The score is obtained through a fully connected layer, and the scoring function is:

[0010] score(a)=etasigmoid(w·f+b),f∈R 2h+2d

[0011] Where η is the scaling variable set according to the subjective question scores, w is the learnable parameter vector, b is the bias term, and 2h+2d represents the dimension of f.

[0012] Furthermore, for subjective questions with multiple standard answer formats, semantic association features are extracted between each standard answer format under the scoring points of the subjective question and the candidate's answer. v is the number of scoring points, and s is the number of multiple standard answers among the scoring points. The fusion is performed using an average pooling layer to obtain semantic features based on the intrinsic correlation between multiple standard answers based on score points. Then through the The splicing process yields the correlation feature f between the standard answer and the test taker's answer. r .

[0013] This invention addresses semantic ambiguity from the perspective of semantic space transformation, effectively handling semantic loss and ambiguity caused by polysemy. Furthermore, considering the different characteristics of subjective questions in various domains and the diverse application scenarios of subjective question scoring, this invention further designs an intelligent scoring algorithm for subjective questions. It fully considers the application scenarios of scoring based on specific points and scoring the entire subjective question, as well as the characteristics of multiple answers and the relationships between scoring points. From the perspective of the relationship between the standard answer and the student's answer, it uses semantic space transformation and attention mechanisms to obtain more accurate core semantic features. For subjective questions with multiple answer formats, from the perspective of complex semantic processing, it captures the relationship features between the student's answer and the standard answer in different semantic forms, fully supporting scoring tasks for subjective questions with multiple standard answers. The pooling layer fusion technology fully explores the inherent relationships between multiple standard answers. This invention's machine learning-based intelligent scoring technology for domain-specific subjective questions effectively handles semantic ambiguity, enhances the semantic recognition capability of the subjective question scoring model, achieves intelligent scoring of domain-specific subjective questions, and is flexibly applicable to different application scenarios. It demonstrates high grading accuracy, especially in scoring domain-specific subjective questions with multiple answer formats. Attached Figure Description

[0014] Figure 1This is a flowchart of the semantic space conversion mechanism of the data preprocessing module of the present invention.

[0015] Figure 2 This is the overall structure of the context feature extraction model of the present invention.

[0016] Figure 3 This is a schematic diagram illustrating the semantic feature extraction of the association between the candidate's answer and the standard answer in the method of the present invention.

[0017] Figure 4 This is the overall structure of the domain-oriented subjective question scoring model of the present invention.

[0018] Figure 5 This is a schematic diagram illustrating the semantic feature extraction of candidate answers and multi-form standard answers in the method of the present invention.

[0019] Figure 6 This is the overall structure of the domain-oriented subjective question multi-answer format scoring model of the present invention. Detailed Implementation

[0020] This invention presents a machine learning-based intelligent scoring method for domain-specific subjective questions. It automatically evaluates student answers against the standard answers based on semantic recognition of both the standard and student answers, achieving intelligent scoring. The invention constructs a scoring network framework, including a data preprocessing module, a standard answer and student answer association feature extraction module, a student answer feature extraction module, and a scoring module. The data preprocessing module is equipped with a semantic space transformation mechanism. During model scoring, student answers undergo semantic transformation through this mechanism to resolve semantic ambiguity. The standard answer and student answer association feature extraction module uses a word-level attention mechanism to obtain the association features f between the standard and student answers. r The candidate answer feature extraction module extracts the context feature vector f from the candidate's answer. a The scoring module combines f r and f a We get f = [f a ;f r The score is obtained through a fully connected layer.

[0021] This invention first designs a unique data preprocessing model to achieve semantic space transformation in order to address semantic ambiguity in the text of subjective questions answered by test takers. Then, referring to the characteristics of human evaluation, it designs a feature extraction technique for test takers' answers and standard answers based on an attention mechanism to fully explore the semantic key points of the standard answers and improve the accuracy of feature extraction for test takers' answers. Furthermore, it integrates the above designs to obtain a multi-matching model tightly coupled unified scoring network framework. This framework can not only fully support the scoring task of subjective questions with multiple standard answers, but also fully explore the inherent relationship between multiple standard answers to improve the overall scoring effect.

[0022] The data preprocessing model of this invention, based on the LSA topic model, designs a semantic space transformation mechanism to address semantic ambiguity and semantic loss in test takers' answers, thereby improving the model's semantic perception capability. First, a general domain dictionary is established using a general knowledge base combined with the TF-IDF algorithm and the Jieba word segmentation tool. Then, a specialized domain dictionary is constructed using specialized domain data and expert knowledge. For example... Figure 1 As shown, based on a general-domain dictionary and a specialized-domain dictionary, the LSA (Latent Semantic Analysis) topic modeling algorithm is used to model all candidate answers, resulting in an LSA model. Based on the LSA model, core topic words are extracted from all candidate answers. Combining this with the standard answers, using the semantics of words in the standard answers as conversion conditions, a semantic space conversion mechanism is designed to convert between core topic words and standard answer words. Furthermore, this semantic space conversion mechanism is embedded into a subjective question scoring model. When candidate answers are input into the subjective question scoring model, automatic semantic space conversion is achieved, enabling automated handling of semantic ambiguity issues in candidate answers.

[0023] The aforementioned automated semantic space transformation mechanism establishes semantic space transformation relationships based on the core topic words in all candidate answers using the LSA (Latent Semantic Analysis) topic model. It then uses Singular Value Decomposition (SVD) to decompose the candidate answers and obtain the word vectors t for all candidate answers. e Given the candidate's answer text vector d, where the subscript e represents the number of words in the candidate's answer, calculate the similarity value between the two:

[0024] S e =similarity(t e ,d)

[0025] The m words with the highest similarity values ​​are selected from the answers of the g-th candidate to obtain the core theme word set, "theme". gThe subscript g represents the g-th candidate's answer. The core keyword set of all candidate answers is obtained in the same way as "theme". Based on "theme" and the standard answer, a core keyword transformation rule is designed with the semantics of the standard answer words as the transformation condition, thus realizing a semantic space transformation mechanism.

[0026] like Figure 3 As shown, the feature extraction module for the association between the standard answer and the candidate's answer is based on the attention mechanism. It uses a self-attention mechanism to enhance the semantic features of the candidate's answer and the standard answer, thereby improving the semantic recognition. Then, it uses a word-level attention mechanism to apply to the candidate's answer and the standard answer respectively, to obtain the semantic association features between each word in the candidate's answer and the standard answer, as well as the semantic association feature weights between each word in the standard answer and the candidate's answer. Finally, it is fused using a splicing technique.

[0027] Word-level feature enhancement based on self-attention processing mechanism is as follows:

[0028]

[0029] att(·) represents the self-attention mechanism, which obtains the enhanced semantic feature representation h through (K,V,Q) processing. n (n<=D), where D is the number of words processed by the self-attention mechanism, and K, V, Q are matrix parameters of learnable enhanced semantic features, K=[k1;k2…k D ], V=[v1;v2…v D ], Q=[q1;q2…q D ], s(·) is the self-attention mechanism's weighting function for each word vector, softmax(·) is the normalization function, exp(·) is the exponential function, and β nw This represents the weight of the output of the nth word vector to the input of the wth word vector in the self-attention mechanism.

[0030] Based on word-level self-attention mechanisms, semantic association features between test takers' answers and standard answers are extracted:

[0031]

[0032]

[0033]

[0034] Take X k With X a These represent the initialization of word vectors for the standard answer and the test taker's answer, respectively. and M and N represent the number of words in the standard answer and the number of words in the candidate's answer, respectively. X k With X aH is obtained after feature enhancement through self-attention mechanism. k With H a ,in z i,j This indicates the correlation between the i-th word of the standard answer and the j-th word of the candidate's answer. and These are the semantic relevance weights between the standard answer and the test taker's answer, obtained through a word-level attention mechanism in different contexts. This indicates the relevance between the i-th word in the standard answer and each word in the test taker's answer. This indicates the relevance between the j-th word in the candidate's answer and each word in the standard answer.

[0035] This yields the semantic association feature vector p between each word in the candidate's answer and the standard answer, i.e., the design of aggregated candidate answer vectors highly correlated with the standard answer, and the semantic association feature vector p between each word in the standard answer and the candidate's answer, i.e., the design of aggregated standard answer vectors highly correlated with the candidate's answer.

[0036]

[0037]

[0038] This indicates the correlation between the i-th word of the standard answer and the j-th word of the test taker's answer. This represents the relevance between the j-th word of the candidate's answer and the i-th word of the standard answer. The feature f representing the association between the standard answer and the candidate's answer is obtained by concatenating p and q. r =[p;q].

[0039] The scoring network framework of this invention is as follows: Figure 4 As shown, this association feature is combined with the contextual features of the test taker's answer to score subjective questions. For example, Figure 2 As shown, the context feature extraction process sequentially goes through: a word embedding layer, a bidirectional long short-term memory neural network layer, and a pooling layer. Based on the fusion result of the pooling layer, the context feature vector of the examinee's answer is obtained into f. a The scoring module combines f r and f a The score is obtained through a fully connected layer.

[0040] Figure 4 The scoring network framework shown is an intelligent scoring framework for single-form standard answers. This invention also proposes a multi-matching model tightly coupled unified scoring network framework, which uses a multi-answer semantic capture mechanism to extract complete semantic features of different forms of standard answers for subjective questions with multiple answer forms.

[0041] The multi-answer semantic capture mechanism decomposes the multi-form standard answer into multiple single-form answers. Based on the standard answer and candidate answer association feature extraction module designed in this invention, the semantic association features of different forms of standard answers and candidate answers are extracted one by one based on the attention mechanism, and then the different semantic features are fused based on the average pooling technique.

[0042] like Figure 5 As shown, in the design for obtaining semantic features of multi-form standard answers and candidate answers, the design for fusing semantic features of multiple forms is as follows:

[0043]

[0044]

[0045] For subjective questions, the semantic association features of each standard answer and the candidate's answer are extracted, and the corresponding association feature vector p is obtained. vs ,q vs The feature vectors of the key elements are obtained by concatenation, where v is the number of scoring points and s is the number of multiple standard answers among the scoring points. It is based on the fusion of average pooling layers to obtain the semantic features of the intrinsic correlation between multiple standard answers, and then through all By splicing, an expression that retains the semantic tight coupling of multiple forms of standard answers is obtained: f r .

[0046] Similar to the case of a single-format standard answer, the semantic text features of multiple-format answers also combine the contextual feature vector f from the test taker's answers. a Subjective questions will be graded. a Combined with f r After fusion, we get f = [f a ;f r ],f∈R 2h+2d 2h+2d represents the dimension of f.

[0047] Finally, the scoring function for subjective questions is:

[0048] score(a)=etasigmoid(w·f+b),f∈R 2h+2d

[0049] Where η is a scaling variable set according to the subjective question scores, w is a learnable parameter vector, and b is a bias term.

[0050] The multi-answer scoring network framework of this invention is as follows: Figure 6 As shown, this method uses semantic association features of multiple answer formats combined with contextual features of test takers' answers to score subjective questions with multiple answer formats.

[0051] Considering the real-world need for point-based scoring in subjective question evaluation, the domain-oriented intelligent scoring method for subjective questions based on machine learning in this invention can effectively cover practical application scenarios of both point-based and whole-question scoring. To effectively evaluate the intelligent scoring method of this invention, tests and analyses were conducted in both point-based and whole-question scoring scenarios for subjective questions.

[0052] Table 1

[0053]

[0054] Table 1 presents the experimental results of the subjective question scoring model in the scoring point scenario. From the above experimental results and analysis, it can be seen that the Semantic Space Transformation (SST) mechanism of this invention performs well in VSM, Neural SAG, and the domain-oriented subjective question scoring model (scoring point scenario) RAA-SAG proposed in this invention. Furthermore, the SST+RAA-SAG model of this invention exhibits highly competitive performance among all models. The comprehensive comparison results in Table 1 for RAA-SAG and SST+RAA-SAG both verify that the word-level attention mechanism incorporating the standard answer demonstrates better overall model performance.

[0055] Table 2

[0056]

[0057] Table 2 presents the experimental results of the subjective question scoring model in the whole-question scoring scenario. The test question types include subjective questions with multiple answer formats. From the experimental results in the table, it can be seen that the model performance is significantly improved after introducing the semantic space transformation mechanism. Simultaneously, the experimental results of the multi-answer semantic capture mechanism verify that the model performs better in multi-answer subjective question scoring. Similarly, the domain-oriented subjective question multi-answer format scoring model (whole-question scoring scenario) MRAA-SAG+SST model of this invention achieves highly competitive performance. Furthermore, the experimental results in Table 2 for both MRAA-SAG+SST and MRAA-SAG verify that the semantic features of multi-format standard answers and candidate answers demonstrate better overall model performance.

[0058] Based on the experimental results above, it is evident that the domain-specific subjective question scoring model introduced in this invention, employing a semantic space transformation mechanism, achieves significantly higher performance. This demonstrates the effectiveness, necessity, and universality of the semantic space transformation mechanism in subjective question scoring tasks. Furthermore, in subjective question scoring tasks based on multiple answer formats, the MRAA-SAGModel model of this invention continues to demonstrate highly convincing results. This verifies that the mechanism incorporating multiple answer formats can capture more specific semantic features and achieve better results in evaluating subjective questions.

Claims

1. A machine learning-based intelligent scoring method for domain-specific subjective questions, characterized by: A scoring network framework is constructed, including a data preprocessing module, a standard answer and candidate answer association feature extraction module, a candidate answer feature extraction module, and a scoring module. The data preprocessing module uses a general-domain dictionary and a specific-domain dictionary, employing Latent Semantic Analysis (LSA) to model all candidate answers, resulting in an LSA model. Based on the LSA model, core topic words are extracted from all candidate answers. Then, a semantic space transformation mechanism is designed, combining core topic words with standard answer words to improve the semantic recognition accuracy of core words in candidate answers. The standard answer and candidate answer association feature extraction module uses a word-level attention mechanism to extract candidate answer text vectors that are highly relevant to the standard answer. And the text vector of the standard answer that is highly correlated with the candidate's answer. By splicing and Correlation characteristics between standard answers and test takers' answers The candidate answer feature extraction module extracts contextual feature vectors from the candidate answers. The scoring module is combined with and ,get The score is obtained through a fully connected layer, and the scoring function is: in, It is a scaling variable set based on the scores of subjective questions. It is a learnable parameter vector. It is a bias term. ; For subjective questions with multiple forms of standard answers, semantic association features between different forms of standard answers and test takers' answers are extracted one by one. Based on average pooling technology, different semantic features are fused. The design for fusion of semantic features across multiple forms is as follows: For subjective questions, the semantic association features of each standard answer and the candidate's answer are extracted, and the corresponding association feature vectors are obtained. , Concatenate the features to obtain the key elements, where v is the number of scoring points and s is the number of multiple standard answers among the scoring points. , Based on average pooling layer The inherent semantic features between the multiple standard answers obtained through fusion are then analyzed through all... By splicing together, an expression that retains the semantic tight coupling of multiple forms of standard answers is obtained: .

2. The intelligent scoring method for domain-oriented subjective questions based on machine learning according to claim 1, characterized in that: For subjective questions with multiple standard answer formats, semantic association features are extracted between each standard answer format under the scoring points of the subjective question and the candidate's answer. v represents the number of scoring points, and s represents the number of multiple standard answers among the scoring points. For all The fusion is performed using an average pooling layer to obtain semantic features based on the intrinsic correlation between multiple standard answers based on score points. Then through the The splicing process yields the correlation characteristics between the standard answer and the test taker's answer. .

3. The intelligent scoring method for domain-oriented subjective questions based on machine learning according to claim 1 or 2, characterized in that: The data preprocessing module consists of the following steps: A general domain dictionary is built using a general knowledge base combined with the TF-IDF algorithm and the Jieba word segmentation tool. A specialized domain dictionary is constructed using specialized domain data and expert knowledge. Based on the general and specialized domain dictionaries, a latent semantic analysis (LSA) topic model is used to model all test-takers' answers. The core topic words in all test-takers' answers are obtained based on the LSA model. Using the semantics of words in the standard answers as the conversion condition, conversion rules between the core topic words and standard answer words are designed to obtain a semantic space conversion mechanism. We obtain the word vector of the g-th candidate's answer by using Singular Value Decomposition (SVD) to decompose the candidate's answer. with candidate answer text vector The subscript 'e' indicates the number of words in the candidate's answer, and the similarity score between the two is calculated: Select the m words with the highest similarity values ​​from the answers of the g-th candidate to obtain the core topic word set. The core keyword set of all candidates' answers was obtained in the same way. ,according to Furthermore, by combining the design of the standard answer with the core topic word association rules based on the semantics of the standard answer words, a semantic space transformation mechanism is achieved.

4. The intelligent scoring method for domain-oriented subjective questions based on machine learning according to claim 1 or 2, characterized in that: The feature extraction module for linking standard answers and test takers' answers is based on a word-level attention mechanism, and the feature enhancement of the word-level self-attention mechanism is as follows: This represents the self-attention mechanism, through ( Enhanced semantic feature representation obtained through processing in this manner n≤D, where D is the number of words processed by the self-attention mechanism. All are matrix parameters for learnable enhanced semantic features. , , , It is the function that weights each word vector in the self-attention mechanism. It is a normalization function. It is an exponential function. This represents the weight of the output of the nth word vector to the input of the wth word vector in the self-attention mechanism; Based on the word-level self-attention mechanism, the semantic association features between the test taker's answer and the standard answer are extracted as follows: Pick and These represent the initialization of word vectors for the standard answer and the test taker's answer, respectively. and M and N represent the number of words in the standard answer and the number of words in the candidate's answer, respectively. and After being enhanced by the self-attention mechanism features, it was obtained and ,in , , , , This indicates the correlation between the i-th word of the standard answer and the j-th word of the candidate's answer. These are the semantic relevance weights between the standard answer and the test taker's answer, obtained through a word-level attention mechanism in different contexts. This indicates the relevance between the i-th word in the standard answer and each word in the test taker's answer. This indicates the relevance between the j-th word in the candidate's answer and each word in the standard answer; Therefore, the candidate's answer text vector, which is highly correlated with the standard answer, And the text vector of the standard answer that is highly correlated with the candidate's answer. : By splicing and Characteristics of the correlation between standard answers and test takers' answers , This indicates the correlation between the i-th word of the standard answer and the j-th word of the test taker's answer. This indicates the correlation between the j-th word of the candidate's answer and the i-th word of the standard answer.

Images