Test question multi-knowledge-point labeling method and system based on sequence generation

Abstract

The invention discloses a test question multi-knowledge-point labeling method and system based on sequence generation, and relates to the field of natural language processing. Firstly, a BERT model is adopted to extract a test question text vector, a global attention feature represented by a test question is extracted through an attention mechanism, the global attention feature is input into an LSTM, and a plurality of knowledge points of the test question are predicted in a sequence generation mode. Meanwhile, when the knowledge point sequence is generated in each step, the extracted knowledge point prior information feature vectors are fused by using a sequence generation-based multi-knowledge point marking method, so that the purpose of correcting the generation of the knowledge point sequence is achieved. And finally, completing a test question knowledge point labeling task in a sequence generation mode. According to the method, the marking accuracy can be comprehensively and scientifically improved on the multi-knowledge-point marking task of the mathematical test questions, and an auxiliary effect is provided for teaching of teachers.

Description

Technical Field

[0001] The present invention relates to natural language processing, pre-trained language models, graph neural networks, and sequence generation technologies, and particularly relates to a method for multi-knowledge-point annotation of test questions based on sequence generation. Background Art

[0002] With the advent of the big data era, the number of educational resources has shown a rapid growth trend. Labeling appropriate tags for learning resources has become a research hotspot in the education field. In the context of a vast amount of resources, tagging relevant knowledge points for learning resources is the most common annotation task. Through knowledge-point annotation, the accuracy and efficiency of learning resource storage and retrieval can be improved. As an indispensable resource in the education scenario, test questions can be used in various education scenarios such as automatic problem-solving, personalized test question recommendation, and intelligent test paper generation. The education field faces a huge demand for resource annotation. However, the traditional expert-based annotation method is time-consuming and laborious, and there are limitations in consistency due to different expert standards, which can no longer meet the resource description requirements in the era of education big data. Therefore, the automatic knowledge-point annotation method for test questions has important research significance and practical value.

[0003] The multi-knowledge-point annotation of test questions aims to extract the mapping relationship between test questions and knowledge points and annotate the knowledge points examined by each test question. Currently, many scholars have conducted research on test question annotation. According to the test question features and classification algorithms used in common annotation models, the knowledge-point annotation models for test questions can be divided into three categories: rule-based annotation, machine learning-based annotation, and deep learning-based annotation models. The rule-based knowledge-point annotation model mainly focuses on lexical features such as keywords and special words in the test question text and only has a good annotation effect on specific learning resources, which has limitations. The machine learning-based knowledge-point annotation model constructs a feature engineering and inputs the extracted multiple test question features into a machine learning algorithm, enabling the model to learn shallow test question text features for annotation. However, the shallow features rely on the manually constructed feature engineering and have insufficient perception of the knowledge-point semantics in the test questions, resulting in a low accuracy of the annotation results. The deep learning-based knowledge-point annotation model is the current mainstream model. Researchers use convolutional neural networks or recurrent neural network models to extract richer and deeper context semantic information, thereby improving the annotation performance of the model. BiLSTM-ATT constructs a text feature extraction model using BiLSTM and an attention mechanism and conducts the corresponding automatic knowledge-point annotation task for junior high school geometry math problems; existing researchers have proposed PBAM and KBM models based on recurrent neural networks and conducted multi-knowledge-point automatic annotation research on English multiple-choice questions. Generally speaking, the knowledge-point annotation model first performs feature extraction on the test question text to capture the semantic information of the test questions, and then completes the knowledge-point annotation task of the test questions by constructing a multi-classifier.

[0004] However, existing research still faces huge challenges in terms of multiple knowledge points in test questions. On the one hand, a test question can contain multiple knowledge points. Past research usually followed the principle of "one test question, one knowledge point", ignoring the possibility of multiple knowledge points. In the real world, test questions often comprehensively examine students' knowledge mastery. Therefore, a test question can be associated with multiple and indefinite numbers of knowledge points. On the other hand, there are also correlation relationships between knowledge points. The multiple knowledge points involved in a test question are often not isolated but have internal connections, and even some knowledge points are usually examined simultaneously. However, most existing methods are difficult to consider the connections between knowledge points while capturing the semantic information of test questions, resulting in insufficient comprehensiveness of the labeled knowledge points.

[0005] Therefore, it is of great research significance and practical value to study a sequence-generation-based test question multi-knowledge point annotation model that can annotate the knowledge points examined by each test question, capture the associations between test questions and multiple knowledge points, as well as the associations between multiple knowledge points, and improve the accuracy of annotation. Summary of the Invention

[0006] Aiming at the deficiencies in the current field of knowledge point annotation, the present invention proposes a sequence-generation-based test question multi-knowledge point annotation. By comprehensively using technical methods such as graph attention mechanism, knowledge point co-occurrence, and correlation matrix, it models the complex correlation relationships between knowledge points and the dynamic mapping of test question semantics. Through the graph attention network, it fuses the knowledge point co-occurrence law and text semantic representation, and combines the sequence generation framework to achieve dynamic prediction and correlation correction of multiple knowledge points, significantly improving the comprehensiveness and accuracy of annotation.

[0007] The object of the present invention is achieved through the following technical measures. A sequence-generation-based test question multi-knowledge point annotation method includes the following steps:

[0008] Step 1, construct a test question text encoder, tokenize the test question and generate a sequence, and use the BERT model for semantic encoding to obtain the test question text feature vector;

[0009] Step 2, extract the knowledge point co-occurrence matrix and knowledge point correlation matrix as the prior knowledge information of the test question, and then input them into the constructed knowledge point prior encoder to obtain the knowledge point feature vector with prior knowledge information of the knowledge point;

[0010] Step 3: Construct a multi-knowledge-point annotation decoder. First, obtain the context semantic representation based on the global attention mechanism, combine it with the knowledge-point feature vector, and obtain the knowledge-point score predicted based on the prior information of the knowledge point at a certain decoding moment. Decode the test question text feature vector circularly through the LSTM decoder to obtain the knowledge-point prediction vector. Predict the knowledge-point probability matrix based on the knowledge-point score and the knowledge-point prediction vector, and select the knowledge point corresponding to the maximum probability value in the probability matrix as the predicted knowledge point.

[0011] Furthermore, use the BertTokenizer function to tokenize the test questions to obtain the corresponding sequence. Each token in the sequence is mapped into a high-dimensional vector, that is, the token embedding of the test question text. This token embedding is initialized with the word embedding pre-trained by BERT. After the test question text passes through the position embedding layer and the segment embedding layer, the corresponding position embedding and segment embedding are obtained. Combine the token embedding, and input the vector representation after adding the three vectors into the BERT model to generate the corresponding test question text encoding vector.

[0012] Furthermore, in Step 2, the method for obtaining the knowledge-point co-occurrence matrix is as follows:

[0013] First, construct a knowledge-point co-occurrence matrix and initialize it as a zero matrix. c represents the total number of knowledge points in the dataset. Secondly, traverse the training set to count the co-occurrence times M i of the i-th knowledge point L j and the j-th knowledge point L ij , that is, the number of times the two knowledge points appear together in the same test question. Finally, update the knowledge-point co-occurrence matrix M based on the co-occurrence times. It is a symmetric matrix.

[0014] The method for obtaining the knowledge-point correlation matrix is as follows:

[0015] Define the frequency vector as F, and normalize the knowledge-point co-occurrence matrix M through the frequency vector F to obtain the final correlation matrix P. The calculation formula: P = M / F; where F is the frequency vector, F i represents the frequency of the i-th knowledge point L i in the entire training set. P ij = P(L j |L i ) represents the probability that the knowledge point L i appears when the knowledge point L j appears. The knowledge-point correlation matrix P is asymmetric.

[0016] Furthermore, the prior knowledge information encoder consists of two cascaded GAT networks. In each GAT network layer, the features of knowledge nodes are aggregated from neighbor nodes to form the features of knowledge nodes in the next layer. The specific calculation is as follows:

[0017] H (l+1) =σ(PH l W l )

[0018] where σ(·) represents the activation function, P is the adjacency matrix, i.e., the knowledge correlation matrix, and W l is the convolution weight of the l-th layer of the GAT network, and H l is the feature of the knowledge node in the l-th layer of the GAT network.

[0019] Furthermore, in the prior knowledge information encoder, the input vector of the first-layer GAT network is the knowledge co-occurrence matrix M and the knowledge correlation matrix P; for node i, after being updated by the first-layer GAT network, the node hidden vector is The specific calculation is as follows:

[0020]

[0021] where K represents the number of attention heads, represents the attention weight of node j to node i in the 0-th layer (i.e., the initial layer) of the k-th attention head, and p ij represents the value corresponding to knowledge points i and j in the adjacency matrix, i.e., the co-occurrence relationship coefficient between knowledge points i and j, and N(i) represents the set of all first-order neighbor nodes adjacent to knowledge point i;

[0022] The output vector of the first-layer GAT network will be further input into the lower-layer GAT network. The weight parameters of the two-layer GAT network are not shared. The output of the last-layer GAT network in the prior knowledge information encoder is

[0023]

[0024] where represents the attention weight of node j to node i in the 1-st layer of the k-th attention head, and W (1) represents the trainable weight matrix of the first-layer GAT, which is used for linear transformation of the initial embedding, is the knowledge feature vector obtained after fusing the two prior knowledge information of knowledge points, i.e., knowledge text information and knowledge co-occurrence information, which is denoted as where c represents the number of labels and d represents the dimension of the knowledge feature vector.

[0025] Further, the calculation formula for the attention coefficient is as follows:

[0026] a ij = ReLU((H i W) || (H j W) T )

[0027] where H i and H j represent the feature vectors of knowledge point nodes i and j; ReLU(·) is the activation function, and W is the weight; finally, all the attention coefficients are uniformly normalized to obtain the final attention coefficient.

[0028] Further, the method for obtaining the context semantic representation is as follows:

[0029] First, different attention coefficients are assigned to different words in the text sequence to obtain keyword information with knowledge point information, so as to converge and generate the hidden representation of the knowledge points involved in the test questions; then, the vector h B = [h1, h2, h3, h4…h m obtained by the test question text encoder and the hidden vector s (t) decoded and output at each moment t are used for attention score calculation. The calculation method of the attention score is:

[0030]

[0031] where is the attention score of the i-th word in the test question text sequence at the time step t, represents the weight parameter at the moment t, and s (t) represents the hidden state output by the decoder at the moment t;

[0032] The attention score and h B are weighted and averaged to finally obtain the context semantic representation input at the decoding moment t

[0033] Further, the knowledge point feature vector H g with knowledge point prior information is used to multiply the context semantic representation vector at the decoding moment t to obtain the knowledge point score

[0034] predicted based on the knowledge point prior information at the decoding moment t. (t) :

[0035]

[0036] Among them, s (0) As the initial hidden vector, in the subsequent loop decoding process, for s (t) the calculation is to input the hidden vector s (t-1) output at the previous moment t-1 and the concatenated vector into the LSTM for decoding, where the concatenated vector is obtained by concatenating the knowledge point prediction score vector generated at the previous moment and the input context semantic representation vector ;

[0037] For each moment t, use the context vector at the current moment and the hidden vector s (t) output by the LSTM to calculate the knowledge point prediction vector

[0038]

[0039] where W o , W d , V d are all weight parameters; is the knowledge point vector predicted completely based on the test question text information, and then added to the knowledge point prediction score predicted based on the knowledge point prior information at time t, with the purpose of using to correct the knowledge point prediction vector The calculation method of the finally predicted knowledge point probability matrix is:

[0040]

[0041] Among them is the mask vector used to prevent the repeated generation of knowledge points. If label i has been predicted in all previous time steps, then (I t ) i will be -∞, otherwise 0; through the argmax(·) function, select the knowledge point corresponding to the maximum probability value in the probability matrix, and use this knowledge point as the predicted knowledge point. The predicted knowledge point of the test question is w i ; After each time step ends, a predicted knowledge point w i will be generated. Finally, after multiple time steps of prediction, multiple predicted knowledge points will be generated. The generated multiple test question knowledge points form a set, denoted as the predicted knowledge point set W = {w1, w2,..., w n}, where n is the number of knowledge points.

[0042] The present invention also provides a multi-knowledge point annotation system for test questions based on sequence generation, including:

[0043] A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a method for multi-knowledge-point annotation of test questions based on sequence generation as described in the above technical solution.

[0044] For the method for multi-knowledge-point annotation of test questions based on sequence generation of the present invention, first, a BERT model is used to extract the test question text vector, the global attention feature of the test question representation is extracted through the attention mechanism, and it is input into the LSTM to predict multiple knowledge points of the test question in a sequence generation manner. At the same time, when implementing the knowledge point sequence generation in each step, a multi-knowledge-point marking method based on sequence generation is used to fuse the extracted knowledge point prior information feature vector to achieve the purpose of correcting the knowledge point sequence generation. Finally, the test question knowledge point annotation task is completed through this sequence generation method. The present invention can comprehensively and scientifically improve the annotation accuracy in the multi-knowledge-point annotation task of mathematics test questions and provide an auxiliary function for teachers' teaching. Description of the Drawings

[0045] Figure 1 It is a structural diagram of a multi-knowledge-point marking method based on sequence generation.

[0046] Figure 2 It is a test question text encoder.

[0047] Figure 3 It is the visualization of the knowledge point co-occurrence matrix value.

[0048] Figure 4 It is a graph of the convergence of the model training Loss. Detailed Embodiments

[0049] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0050] The construction of the knowledge point annotation model needs to be based on extracting and understanding the semantics of mathematics test questions and predicting the maximum probability labels of several test question texts. The specific definition is: assume that there are n mathematics test questions in the dataset, the sample space is T = {t1, t2,..., t n}, and there are a total of c knowledge points, and the corresponding knowledge point set is L = {l1, l2,..., l c}

[0051] As Figure 1 shown, a method for multi-knowledge-point annotation of test questions based on sequence generation includes the following parts:

[0052] (1) Test question text encoding.

[0053] To obtain more accurate and rich test question information, the present invention uses the BERT model as the test question text encoder. Compared with traditional models, the BERT model can better consider the context relationship and semantic meaning of vocabulary. By using the BERT model, the test question text can be encoded more comprehensively, thereby providing a more accurate test question text representation for subsequent knowledge point decoding work.

[0054] The test question text encoding includes BERT-based semantic encoding and word segmentation and sequence generation.

[0055] (1-1) Word Segmentation and Sequence Generation

[0056] The present invention uses the BertTokenizer function to tokenize the test question t i to obtain the sequence x = ([CLS], x1, x2, x3, …, [SEP]), where [CLS] represents the start of the sequence and [SEP] represents the end of the sequence. These two identifiers also participate in the training process, enabling the model to learn when to end the sequence generation process. Each token x i is mapped into a high-dimensional vector representation where H is the dimension size of the vector, thereby forming the embedding matrix U n×H , that is, the token embedding of the test question text. The present invention uses the word embedding of the BERT pre-trained model for initialization, and the BERT structure is as Figure 2 shown.

[0057] A certain test question in the test question sample is represented by t i and the Chinese text x = ([CLS], x1, x2, x3,...,[SEP]) is tokenized using the BertTokenizer function to obtain the text word sequence of the test question t i , where [CLS] represents the start of the sequence and [SEP] represents the end of the sequence. These two identifiers also participate in the training process, enabling the model to learn when to end the sequence generation process. Each token x i is mapped into a vector representation where H is the dimension size of the vector, thereby forming a test question text embedding matrix U n×H , which is the token embedding part of the test question text. This embedding matrix is initialized with the word embedding pre-trained by BERT. After the test question text passes through the position embedding layer and the segment embedding layer, the corresponding position embedding and segment embedding parts will be obtained. Combining the previous token embedding, the vector representation X after adding the three vectors is input into the BERT model to generate the corresponding test question text encoding vector b X .

[0058] (1-2) BERT-based Semantic Encoding

[0059] For the text representation obtained by BERT, the general classification model will use the output vector corresponding to the token [CLS] placed at the beginning of the text to complete the subsequent classification work. The [CLS] vector represents the semantic vector of the entire sentence. Although it can also obtain relatively good classification results when applied to the field of text classification, it will inevitably reduce the semantic role of some keywords. The hidden vectors of the words in the text [h1,h2,h3,h4...h m ] can also provide a lot of useful information for text classification. X =[h [CLS] ,h1,h2,h3,h4...h m All hidden vectors in ] will be used, and for b X Different hidden vectors are used separately, mainly divided into the vector h corresponding to the [CLS] tag [CLS] The semantic vector that can represent the entire sentence, as well as the hidden vector part corresponding to the text word token [h1,h2,h3,h4...h m ], where h i Represents the BERT output vector corresponding to the i-th word in the test text.

[0060] (2) Encoding of prior information of knowledge points.

[0061] In order to integrate the prior information of knowledge points, the present invention designs a knowledge point prior information encoder, which first extracts the test question knowledge point text and the knowledge point co-occurrence matrix, and uses them as the prior information of the test question knowledge points, and then inputs them into the graph attention network to obtain the knowledge point embedding representation, which is used to correct the test question knowledge point annotation and improve the knowledge point annotation effect.

[0062] (2-1) Knowledge point co-occurrence matrix

[0063] The present invention uses the test sample of the training set to generate the knowledge point co-occurrence matrix. First, we construct a knowledge point co-occurrence matrix And initialize it to a zero matrix, c represents the total number of knowledge points in the dataset. Secondly, traverse the training set to count the knowledge points L i and L j The number of co-occurrences M ij , that is, the number of times two knowledge points appear together in the same test question; finally, based on the number of co-occurrences, the knowledge point co-occurrence matrix M is updated, which is a symmetric matrix. Figure 3 This visualization shows the co-occurrence matrix of knowledge points in a dataset of math test annotations. Excluding the diagonal blocks, the darker the blocks, the more co-occurrences there are between the corresponding two knowledge points, indicating a greater degree of correlation between the knowledge points.

[0064] (2-2) Knowledge Point Correlation Matrix

[0065] For the knowledge point co-occurrence matrix, this paper further constructs a corresponding correlation matrix P for the calculation of the subsequent modules. This paper defines the frequency vector as F, and then models the co-occurrence dependence of knowledge points in the form of conditional probability, that is, P ij (L j |L i ) represents the probability that knowledge point L i appears when knowledge point L j appears. Since P ij (L j |L i ) is not equal to P ji (L i |L j ), the correlation matrix P is asymmetric. This paper normalizes the co-occurrence matrix through the frequency vector F to obtain the final correlation matrix P, and the calculation formula is: P = M / F; where F is the frequency vector, and F i represents the frequency of knowledge point L i in the entire training set, and P ij = P(L j |L i ) represents the probability that knowledge point L i appears when knowledge point L j appears.

[0066] (2-3) Knowledge Point Representation Based on Graph Attention

[0067] Knowledge point correlation graph. To model the correlation between knowledge points, we first construct a weighted knowledge point correlation graph G=(V, E). For the node V, we regard the knowledge point set as the node of the correlation graph V={v1, v2,..., v c}), and c represents the total number of knowledge points. We use the pre-trained BERT model to encode the text information of knowledge points to obtain the semantic information of knowledge points, which is used as the initial representation of the node d represents the dimension of the node embedding representation. For the edge E, we set the knowledge point correlation matrix P obtained in the previous step as the adjacency matrix of the correlation graph, which is used as the correlation between knowledge point nodes.

[0068] The knowledge point prior information encoder is composed of two cascaded GAT networks, as shown in Figure 1 the (b) module in. In each GAT network layer, the features of knowledge point nodes are aggregated by neighbor nodes to form the features of knowledge point nodes in the next layer, and the specific calculation is shown in formula (1):

[0069] H (l+1) = σ(PH l Wl ) (1)

[0070] where σ(·) represents the activation function, P is the adjacency matrix, and W l is the convolution weight of the l-th layer of the GAT network.

[0071] Graph Attention Mechanism. To obtain a more rich representation of knowledge point vectors, we use multi-head attention to absorb the importance of different knowledge point nodes to the current node. In this paper, K different heads are used to describe capturing knowledge point relationships, that is, the attention layer will be independently replicated K times, and each replication will use different parameters. The output of the multi-head attention layer is the output after concatenating the features. The attention coefficient of the node in the l-th layer is denoted as which is used to describe the importance of the j-th knowledge point node to the i-th node when updating the l-th hidden layer. The general calculation of the attention coefficient is shown in formula (2):

[0072] a ij = ReLU((H i W)||(H j W) T ) (2)

[0073] where, H i and H j represent the feature vectors of knowledge point nodes i and j. ReLU(·) is the activation function, and W is the weight. In this paper, the attention coefficient is evaluated by concatenating features, and || represents the concatenation operation. To make the attention coefficients between different knowledge points comparable, the Softmax function is used when calculating the attention coefficient of node i to node j, and all attention coefficients are uniformly normalized. The final calculation of the attention coefficient is shown in formula (3):

[0074]

[0075] Graph Network Update. In the knowledge point prior information encoder, the input vector of the first layer of the GAT network is the BERT embedding vector matrix M corresponding to the knowledge point text and the adjacency matrix P. For node i, after being updated by the first layer of the GAT network, the node hidden vector obtained is The specific calculation is shown in formula (4):

[0076]

[0077] where K represents the number of attention heads, represents the attention weight of node j to node i in the 0-th layer (initial layer) of the k-th attention head, and p ijDenote the value corresponding to knowledge points i and j in the adjacency matrix, that is, it represents the co-occurrence relationship coefficient between knowledge points i and j. N(i) represents the set of all first-order neighbor nodes adjacent to knowledge point i, W (0) Denotes the trainable weight matrix of the first layer of GAT, which is used for linear transformation of the initial embedding.

[0078] The output vector of the first layer of GAT network Will be further input into the lower layer GAT network, that is, the second layer GAT network of the knowledge point prior information encoder. However, the weight parameters of these two layers of GAT networks are not shared, as shown in formula (5):

[0079]

[0080] The output of the last layer of GAT network in the knowledge point prior information encoder is That is, the knowledge point feature vector obtained after fusing two kinds of prior information of the knowledge point (knowledge point text information and knowledge point co-occurrence information). For the convenience of recording, it is denoted here as Where c represents the number of labels, d represents the dimension of the knowledge point feature vector, where Represents the attention weight of node j to node i in the first layer of the k-th attention head, W (1) Denotes the trainable weight matrix of the first layer of GAT, which is used for linear transformation of the initial embedding.

[0081] (3) Multi-knowledge point annotation decoding.

[0082] In order to fully capture the useful information of the annotated knowledge points in the test question text, in the knowledge point annotation decoding part, the global attention mechanism and the feature fusion decoding module are used.

[0083] (3-1) Global attention mechanism

[0084] The present invention assigns different attention coefficients to different words in the text sequence, obtains keyword information with knowledge point information, and thus converges to generate a hidden representation of the knowledge points involved in the test question; then the vector h obtained by the text encoding module B =[h1, h2, h3, h4...h m and the hidden vector s decoded and output at each time step t (t) Perform attention score calculation. The calculation method of attention:

[0085]

[0086] Where Is the attention score of the i-th word in the test question text sequence at time step t, represents the weight parameter at time t, s (t) represents the output hidden state of the decoder at time t. The present invention uses a linear mapping method to calculate the global attention score, that is, an additive model. For h B and s (t) After weighted summation, a Softmax operation is used to normalize the weighted average, and a probability distribution will be output after normalization which is the attention weight of word i at decoding time t.

[0087] To absorb the similarity information between s (t-1) and h B as much as possible and reduce the interference of the original information in h B , a weighted average calculation is performed on the attention score and h B . m represents the number of word vectors, and finally the context semantic representation input at decoding time t is obtained The specific calculation is formula (7).

[0088]

[0089] (3 - 2) Feature Fusion Decoding Module

[0090] The feature fusion decoding module mainly includes two parts: knowledge point prior information fusion and LSTM cyclic decoding. For the knowledge point prior information fusion part, mainly by using the knowledge point feature vector H with knowledge point prior information g , multiply it with the context semantic representation vector at decoding time t to obtain the knowledge point score predicted based on the knowledge point prior information at decoding time t

[0091]

[0092] The LSTM cyclic decoding process is mainly for decoding based on the test question text information. Through the cyclic decoding of the test question text feature vector by LSTM, multiple mathematical test question knowledge points are generated. At time t, the hidden vector output by the LSTM decoder is s (t) :

[0093]

[0094] Among them, is the knowledge point prediction score vector formed at the previous moment, and s (0) is used as the initial hidden vector and is initialized using the h X vector in b [CLS] . In the subsequent cyclic decoding process, for the calculation of s (t) it is to use the hidden vector s output at the previous moment t - 1(t-1) are simultaneously input into the LSTM for decoding together with the splicing vector, where the splicing vector is the knowledge point prediction score vector generated at the previous moment and the input context semantic representation vector are spliced to obtain. For the tag <bos>Initialization is carried out, representing the start of this sequence generation. For each time step t, using the context vector at the current time step and the hidden vector s output by the LSTM (t) , the knowledge point prediction vector is calculated

[0095]

[0096] where W o , W d , V d are all weight parameters of the model. is the knowledge point vector predicted completely based on the test question text information, and then added to the knowledge point prediction score predicted based on the knowledge point prior information at time step t. The purpose is to use to correct the knowledge point prediction vector so as to achieve the purpose of integrating the knowledge point prior information when generating knowledge points and improve the annotation effect of test question knowledge points. The finally predicted knowledge point probability matrix is calculated as follows:

[0097]

[0098] where is a mask vector used to prevent the repeated generation of knowledge points. If label i has been predicted in all previous time steps, then (I t ) i will be -∞, otherwise it is 0. The predicted knowledge point of the test question is w i . Through the argmax(·) function, the knowledge point corresponding to the maximum probability value in the probability matrix is selected and used as the predicted knowledge point, as shown in formula (12). After each time step ends, a predicted knowledge point w i is generated. Finally, after multiple time steps of prediction, multiple predicted knowledge points will be generated. When the model generates the knowledge point label as <eos>It represents the end of the current sequence generation process, ending the loop. The multiple generated test question knowledge points finally form a set, denoted as the predicted knowledge point set W = {w1, w2,..., w n}, where n is the number of knowledge points.

[0099] The following are the experimental data and evaluation methods for the method of the present invention:

[0100] (1-1) Experimental data and processing

[0101] TMK-PSS is a standard dataset for annotating knowledge points of primary and secondary school mathematics test questions, which comes from a learning big data platform developed by a certain university. The database of this platform stores detailed information of test questions, including question numbers, knowledge points, question stems, answers, difficulty values, subjects and applicable grades, etc. The knowledge points come from the nodes of the subject knowledge map, and are annotated with multi-level knowledge points through the "annotation and review" mechanism of experts. We extract test questions from the platform database, use the first-level knowledge points involved in the test questions as knowledge point labels, and splice the question stems and analyses to form the text content of the test questions. Then, we perform a series of data preprocessing steps, including deleting HTML tags, processing special symbols, removing duplicates, etc. Finally, we obtain the processed standard dataset TMK-PSS, and the statistical information is shown in Table 1. The ratio of the training set, validation set, and test set is 6:2:2.

[0102] Table 1 Detailed information of the multi-knowledge point annotation dataset of test questions

[0103]

[0104] (1-2) Evaluation indicators

[0105] The evaluation indicators used in the research include four types: Hamming loss, accuracy, precision, and recall rate. These indicators are often used to compare the quality of test question annotation.

[0106] (1-3) Comparison with baseline methods

[0107] To verify the effectiveness of the proposed multi-knowledge point annotation method model for test questions, this paper sets multiple baseline methods, and the specific details are as follows:

[0108] TextCNN: TextCNN is a convolutional neural network that uses convolutional kernels of different sizes to capture context information and performs well in short text classification.

[0109] LSTM-ATT: The LSTM model is one of the most commonly used text sequence generation models. By adding an Attention mechanism, it can extract richer text features to improve the text classification effect.

[0110] BiLSTM-ATT: The Bi-LSTM model introduces the forward and backward information of the input sequence on the basis of the LSTM to capture more text features, and is improved by combining the Attention mechanism to enhance the model performance.

[0111] BERT: The BERT model parameters pre-trained by the official Google model are adopted, and the output [CLS] vector is used as the sentence vector for text multi-label prediction.

[0112] UniLM: UniLM adopts a shared Transformer network and a specific self-attention mask mechanism for controlling the context conditions of prediction for unified language modeling. In this paper, the Seq_to_Seq mask mechanism is used to conduct annotation experiments in the way of sequence generation.

[0113] SOMPT: SOMPT proposes a data augmentation method for math field test questions, and uses a framework based on sequence generation to complete the knowledge point annotation task, achieving multi-knowledge point annotation.

[0114] (1-4) Model Parameter Settings

[0115] In the model of the multi-knowledge point annotation method for test questions, we use the base version of the BERT pre-trained model as the initial parameter of the test question text encoder module and allow dynamic updates during training. At the same time, we use the knowledge point prior information encoder module based on the BERT pre-trained model to obtain the test question knowledge point text embedding. We adopt the Adam optimizer with a learning rate of 2e-5 for gradient descent and set two momentum parameters as: β1 = 0.9, β2 = 0.99. In terms of the hyperparameter Batch_Size, we respectively tried settings of 4, 8, 12, 16, and 32, and found that when set to 16, the model of the multi-knowledge point annotation method for test questions performs best.

[0116] (1-5) Experimental Results and Analysis

[0117] Comparison of multi-knowledge point annotation performance: To evaluate the performance of the multi-knowledge point annotation method for test questions in the knowledge point annotation task, we conducted experiments on the TMK-PSS math test question dataset and compared with six baseline methods. The experimental results are shown in Table 2. Among them, the evaluation indicators with (+) indicate that the higher the score, the better the performance, and those with (-) are the opposite.

[0118] Table 3 Annotation performance of the model in this paper and six baseline methods on four indicators

[0119]

[0120] Observing the experimental results in the table, it can be found that compared with the six baseline models, the model of the multi-knowledge-point annotation method for test questions has obtained obvious advantages in the performance of four evaluation indicators. Specifically, compared with the best-performing SOMPT model among the baseline models, the multi-knowledge-point annotation method for test questions has increased by 1.8%, 2.8%, and 1.9% in precision, recall, and accuracy respectively, and has reduced the loss by 5.55% in HL. This indicates that the multi-knowledge-point annotation method for test questions can effectively classify the knowledge points involved in the test questions and improve the accuracy of multi-knowledge-point annotation. Secondly, compared with the baseline, the scores of the multi-knowledge-point annotation method for test questions in precision and recall are the best and relatively balanced, demonstrating its excellent performance in terms of accuracy and comprehensiveness, and the annotation performance is relatively stable. It is worth noting that the experimental results corresponding to the models that complete the knowledge-point annotation task with the sequence generation framework (i.e., UniLM and SOMPT) are better, which shows that designing the model through the sequence generation method helps to complete the text multi-label classification task of knowledge-point annotation. This is because the knowledge-point information predicted in the previous step during the sequence generation process will be added to the process of generating the next knowledge point. To a certain extent, this can help the model capture the connections between knowledge points and play the role of knowledge-point correlation prediction. Finally, by comparing the experimental results of the multi-knowledge-point annotation method for test questions with the SOMPT model, it can be further found that the prior information of knowledge points is important for the knowledge-point annotation task. Because although both of these models use the sequence generation method to predict the knowledge-point sequence in the annotation method, the knowledge-point graph embedding generated by the multi-knowledge-point annotation method model for test questions by aggregating the knowledge text and knowledge-point co-occurrence information can correct the prediction results of knowledge points to a certain extent, which also shows that rich test question information can assist knowledge-point annotation.

[0121] To further analyze the performance of the model, we draw the training loss convergence graphs of the baseline model and the model of the multi-knowledge-point annotation method for test questions, as Figure 4 shown. By comparing the training loss convergence trend with the baseline model, it can be found that the multi-knowledge-point annotation method for test questions has a faster training convergence speed and can approach the convergence state in 5-10 rounds of training. This shows that the model of the multi-knowledge-point annotation method for test questions has better ability to extract useful information from the data and can adjust the model parameters faster and more effectively during training, that is, its ability to mark the correct knowledge points is stronger. In addition, when the model reaches the convergence state, its Loss convergence curve tends to be horizontal and hardly fluctuates anymore. This shows that the model of the multi-knowledge-point annotation method for test questions has better stability in the knowledge-point annotation task.

[0122] On the other hand, the embodiment of the present invention also provides a multi-knowledge-point annotation system for test questions based on sequence generation, including:

[0123] A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a method for multi-knowledge-point annotation of test questions based on sequence generation as described in the above technical solution.

[0124] The content not described in detail in this specification belongs to the prior art well-known to those skilled in the art.

[0125] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.< / eos> < / bos>

Claims

1. A method for multi-knowledge-point annotation of test questions based on sequence generation, characterized in that, It includes the following steps: Step 1: Construct a test question text encoder, tokenize the test questions and generate sequences, and use the BERT model for semantic encoding to obtain test question text feature vectors; Step 2: Extract the knowledge point co-occurrence matrix and the knowledge point correlation matrix as the prior knowledge information of the test questions, and then input them into the constructed prior knowledge encoder of knowledge points to obtain knowledge point feature vectors with prior knowledge information of knowledge points; Step 3: Construct a multi-knowledge point annotation decoder. First, obtain the context semantic representation based on the global attention mechanism, combine the knowledge point feature vectors, and obtain the knowledge point scores predicted based on the prior knowledge information of knowledge points at a certain decoding moment; perform cyclic decoding on the test question text feature vectors through the LSTM decoder to obtain knowledge point prediction vectors; predict the knowledge point probability matrix based on the knowledge point scores and the knowledge point prediction vectors, and select the knowledge point corresponding to the maximum probability value in the probability matrix as the predicted knowledge point.

2. The method for multi-knowledge-point annotation of test questions based on sequence generation according to claim 1, characterized in that: Use the BertTokenizer function to tokenize the test questions to obtain corresponding sequences. Each token in the sequence is mapped into a high-dimensional vector, that is, the token embedding of the test question text. This token embedding is initialized with the word embedding pre-trained by BERT. After the test question text passes through the position embedding layer and the segment embedding layer, the corresponding position embedding and segment embedding are obtained. Combining the token embedding, the vector representation after adding the three vectors is input into the BERT model to generate the corresponding test question text encoding vector.

3. A method for multi-knowledge-point annotation of test questions based on sequence generation according to claim 1, characterized in that: In Step 2, the method for obtaining the knowledge point co-occurrence matrix is as follows: First, construct a knowledge point co-occurrence matrix and initialize it as a zero matrix, where c represents the total number of knowledge points in the dataset; Secondly, traverse the training set to count the i-th knowledge point L i and the jth knowledge point L j The number of co-occurrences M ij , that is, the number of times two knowledge points appear together in the same test question; finally, the knowledge point co-occurrence matrix M is updated based on the co-occurrence number, which is a symmetric matrix; The method for obtaining the knowledge point correlation matrix is as follows: Define the frequency vector as F, and normalize the knowledge point co-occurrence matrix M through the frequency vector F to obtain the final correlation matrix P. The calculation formula is: P = M / F; where F is the frequency vector, and F i represents the frequency of the i-th knowledge point L i in the entire training set, and P ij = P(L j |L i ) represents the probability that the knowledge point L i appears when the knowledge point L j appears. The knowledge point correlation matrix P is asymmetric.

4. The method for multi-knowledge-point annotation of test questions based on sequence generation according to claim 1, characterized in that: The prior knowledge information encoder of knowledge points consists of two cascaded GAT networks. In each GAT network layer, the features of the knowledge point nodes are aggregated by neighbor nodes to form the knowledge point node features of the next layer. The specific calculation is as follows: H (l+1) = σ(PH l W l ) where σ(·) represents the activation function, P is the adjacency matrix, i.e., the knowledge point correlation matrix, and W l is the convolution weight of the l-th layer of the GAT network, and H l is the knowledge point node feature of the l-th layer of the GAT network.

5. The method for multi-knowledge-point annotation of test questions based on sequence generation according to claim 4, wherein: In the prior knowledge information encoder of knowledge points, the input vectors of the first layer of the GAT network are the knowledge point co-occurrence matrix M and the knowledge point correlation matrix P; For node i, after being updated by the first layer of the GAT network, the node hidden vector is The specific calculation is as follows: where K represents the number of attention heads, represents the attention weight of the k-th attention head for node j to node i in the 0-th layer, i.e., the initial layer, and p ij represents the value corresponding to knowledge points i and j in the adjacency matrix, that is, the co-occurrence relationship coefficient between knowledge points i and j, and N(i) represents the set of all first-order neighbor nodes adjacent to knowledge point i; The output vector of the first-layer GAT network will be further input into the lower-layer GAT network. The weight parameters of the two-layer GAT network are not shared. The output of the last-layer GAT network in the prior knowledge encoder is Among them represents the attention weight of the k-th attention head from node j to node i in the first layer, W (1) represents the trainable weight matrix of the first-layer GAT, which is used for linear transformation of the initial embedding is the knowledge point feature vector obtained after fusing the two prior information of the knowledge point, namely the knowledge point text information and the knowledge point co-occurrence information, which is denoted as where c represents the number of labels and d represents the dimension of the knowledge point feature vector 6. The method for multi-knowledge-point annotation of test questions based on sequence generation according to claim 5, wherein: The calculation formula for the attention coefficient is as follows: a ij = ReLU((H i W) || (H j W) T ) Among them, H i and H j represent the feature vectors of knowledge point nodes i and j; ReLU(·) is the activation function, and W is the weight; finally, all attention coefficients are uniformly normalized to obtain the final attention coefficients.

7. The multi-knowledge-point annotation method for test questions based on sequence generation according to claim 1, wherein: The method for obtaining the context semantic representation is as follows: First, different attention coefficients are assigned to different words in the text sequence to obtain keyword information with knowledge point information, so as to converge and generate the hidden representation of the knowledge points involved in the test questions; then the vector h B = [h1, h2, h3, h4... h m and the hidden vector s (t) decoded and output at each moment t are used to calculate the attention score. The calculation method of the attention score is as follows: where is the attention score of the i-th word in the test question text sequence at time step t, represents the weight parameter at time t, s (t) represents the decoder output hidden state at time t; Perform weighted average calculation on the attention score and h B to finally obtain the context semantic representation input at decoding time t 8. A method for multi-knowledge-point annotation of test questions based on sequence generation according to claim 1, characterized in that: Use the knowledge point feature vector H with prior information of knowledge points g , multiply it with the context semantic representation vector at the decoding time t , to obtain the knowledge point score predicted based on the prior information of knowledge points at the decoding time t 9. The method for multi-knowledge-point annotation of test questions based on sequence generation according to claim 1, characterized in that: At time t, the hidden vector output by the LSTM decoder is s (t) : Among them, s (0) As the initial hidden vector, during the subsequent loop decoding process, for s (t) the calculation is to input the hidden vector s (t-1) output at the previous moment t - 1 and the concatenated vector into the LSTM for decoding, where the concatenated vector is obtained by concatenating the knowledge point prediction score vector generated at the previous moment and the input context semantic representation vector ; For each moment t, use the context vector at the current moment and the hidden vector s output by the LSTM (t) , and calculate the knowledge point prediction vector Among which W o , W d , V d are all weight parameters; is the knowledge point vector predicted entirely based on the test question text information, and then added to the knowledge point prediction score predicted at time t based on the prior information of the knowledge point. The purpose is to use to correct the knowledge point prediction vector The calculation method of the finally predicted knowledge point probability matrix is as follows: Among them is a mask vector used to prevent the repeated generation of knowledge points. If label i has been predicted at all previous time steps, then (I t ) i is -∞, otherwise it is 0; through the argmax(·) function, the knowledge point corresponding to the maximum probability value in the probability matrix is selected, and this knowledge point is used as the predicted knowledge point. The knowledge point predicted for the question is w i ; a predicted knowledge point w i is generated after each time step. Finally, after multiple time steps of prediction, multiple predicted knowledge points will be generated. The generated multiple question knowledge points form a set, denoted as the predicted knowledge point set W = {w1, w2,..., w n}, where n is the number of knowledge points.

10. A multi-knowledge-point annotation system for test questions based on sequence generation, characterized in that, It includes: A processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a method for multi-knowledge point annotation of test questions based on sequence generation as described in any one of claims 1-9.

Images