Transform-based student side information fusion knowledge tracking method

Abstract

The invention discloses a Transform-based student side information fusion knowledge tracking method, which comprises the following steps of: constructing a sequence-to-sequence model, and separating a learning record sequence of a student into an exercise sequence and an answer sequence; embedding the obtained sequence into side information to obtain an exercise and answer embedding sequence; a Transformer model based on an attention masking mechanism is constructed, an exercise embedding sequence is input into an encoder, the encoder captures the relation between exercises through a multi-head attention mechanism, then nonlinear features are added to obtain output # imgabs0 #, a decoder combines an answer embedding sequence with the output # imgabs1 # through the multi-head attention mechanism and captures the relation between exercises and answers, and the answer embedding sequence and the output # imgabs1 # are combined through the multi-head attention mechanism. Adding nonlinear features to obtain decoder output; and generating a prediction probability of the student to do exercises. According to the invention, the knowledge level of the student is constructed more comprehensively, and the accuracy of predicting the knowledge state of the student is improved.

Description

Technical Field

[0001] The present invention belongs to the field of smart education technology, and specifically relates to a student-side information fusion knowledge tracking method based on Transformer. Background Art

[0002] Knowledge Tracking (KT) is the task of modeling each learner's mastery of knowledge concepts while participating in a series of learning activities. Specifically, the goal of knowledge tracking is to predict whether students can correctly answer the next question based on their learning history.

[0003] Traditional knowledge tracing methods include Bayesian knowledge tracing (BKT) and collaborative filtering (CF), among which BKT is the first model to solve the knowledge tracing problem. BKT adopts the framework of Hidden Markov Model (HMM), where the hidden state of HMM represents the learner's knowledge state, which is described by a set of binary variables: mastered or not mastered. Specifically, BKT independently models each knowledge concept state, but traditional methods have some obvious shortcomings, such as the binary representation of learners' understanding may be unrealistic.

[0004] In recent years, with the successful application of deep learning in various fields, many knowledge tracing models based on deep learning have been proposed and achieved good performance. Neural network structures such as recurrent neural networks (RNNs) and transformers have become common building blocks of knowledge tracing models. Over the years, people have made many improvements and extensions to traditional methods, such as Deep Knowledge Tracing (DKT) and its variants. DKT is the first to use recurrent neural networks to model students' dynamic knowledge status in a total hidden vector. Dynamic Key-Value Memory Networks (DKVMN) uses memory-enhanced neural networks for KT development, which includes static matrices and dynamic matrices. The static matrix is ​​the key matrix, which is used to store potential knowledge concepts (KCs), while the dynamic matrix is ​​the value matrix, which is used to store the main level of the corresponding KCs. They respectively learn the correlation between exercises, basic knowledge points and students' knowledge status. A Self-Attentive model for Knowledge Tracing (SAKT) is a Transformer-based knowledge tracing model that uses a pure attention mechanism to model the learner's learning process. The DKT model faces the problem that its parameters are uninterpretable. DKVMN is more interpretable than DKT because it explicitly maintains a knowledge point representation matrix (key) and a knowledge state representation matrix (value). However, since all these deep learning models are based on RNN, they face the problem of non-generalization when dealing with sparse data. At the same time, a lot of information is involved in the learner's exercises and behaviors. The most important thing in the exercises is the knowledge points, and there are many other important auxiliary information such as disciplines, question types, question difficulty, etc., and the auxiliary information in the behavior such as the number of interactions, speed of doing questions, accuracy, etc. If these side information can be fully utilized, it will greatly improve the accuracy of tracking and evaluating the learner's learning status in learning activities, and solve the problem of non-generalization when dealing with sparse data. Summary of the invention

[0005] The purpose of the present invention is to provide a student-side information fusion knowledge tracking method based on Transformer, which solves the problem of low prediction accuracy of students' knowledge status in existing methods.

[0006] The technical solution adopted by the present invention is: a student-side information fusion knowledge tracking method based on Transformer, comprising the following steps: Step 1: Build a sequence-to-sequence model to separate the student's learning record sequence into an exercise sequence and an answer sequence; Step 2: embed the sequence obtained in step 1 into the side information through the embedding layer of the sequence-to-sequence model to obtain the question embedding sequence and the answer embedding sequence; Step 3: Build a Transformer model based on the masked attention mechanism. The exercises obtained in step 2 are embedded in the sequence and input into the encoder of the Transformer model. The encoder captures the relationship between exercises through the multi-head attention mechanism, and then adds nonlinear features through the feedforward network to obtain the output. The decoder embeds the answer obtained in step 2 into the sequence and outputs through a multi-head attention mechanism Combine and capture the relationship between the exercise and the answer, and then add nonlinear features through the feedforward network to obtain the decoder output ; Step 4: Decoder output according to step 3 Generates the predicted probability that the student will answer the question correctly.

[0007] The present invention is also characterized in that: In step 1, students’ learning records are recorded as interaction sequences , where the sequence tuple , Indicates the student's answer i Exercises and related information, response Indicates that students Answer and answer relevant information, , i If the answer is correct, ,otherwise .

[0008] Step 2 specifically includes the following steps: Step 2.1: Substitute the sequence tuple Encoding as index , the vector representation of the side information is obtained by calculating the mapping layer of the sequence-to-sequence model. The calculation formula is as follows:

[0009] In the formula, Represents a sequence tuple The side information vector representation of ; Represents the weights of all potential side information; the matrix Covers all relevant potential side information, each column is a vector representation of potential side information; Step 2.2: Use the embedding layer of the sequence-to-sequence model to embed the exercise sequence and answer sequence Embed the side information vector to get the problem embedding sequence Embedded sequence with answer , specifically expressed as follows:

[0010]

[0011] In the formula, Represents the embedding layer, which is used to embed the exercise sequence and answer sequence And the side information contained is encoded as a vector; the embedding matrix is ​​expressed as , Indicates the total number of various side information corresponding to different projects, Represents the embedding dimension of the item.

[0012] Step 3 specifically includes the following steps: Step 3.1: Embed the exercises obtained in step 2.2 into the sequence In the encoder of the input Transformer model, the encoder uses the masked attention layer to extract Extract query from ,key Sum , and then through the matrix , and Each , and Projection to latent space:

[0013] In the formula, ; Step 3.2: According to the query and key The consistency between and The similarity between the query and key is the past interaction Distribute your attention , the calculation formula is as follows:

[0014] In the formula, the matrix is ​​transformed into The upper triangular part of ; Time interval deviation , according to the interaction and The time interval between adjusts the attention weight, where Indicates a timestamp; is the scaling factor, i.e. and Dimensions; Step 3.3: Multiply the weighted sum of past mask attention weights by the value Get the attention head :

[0015] In the formula, yes Function, through After the operation, the attention network resets the attention weights of subsequent positions to zero; Step 3.4: The concatenation result of the attention heads is multiplied by the matrix To summarize the output of different attention heads, the calculation formula is as follows:

[0016] Step 3.5: Add nonlinear features to the attention head output obtained in step 3.4 by the following formula:

[0017]

[0018] In the formula, represents the output of the attention head, , , as well as Respectively represent different The weight matrix and bias vector shared between them; Step 3.6: Output the attention head with nonlinear features added in step 3.5 The output sequence of the encoder is obtained through residual connection and layer normalization in Represents the characteristic representation of each exercise, and the calculation formula is as follows:

[0019] Step 3.7: Embed the answer obtained in step 2.2 into the sequence The decoder is input into the Transformer model, and the decoder embeds the start token through the same multi-head attention mechanism as the encoder. The answer embedding sequence Generates the intermediate representation of the current layer :

[0020] In the formula, The decoder uses the masked attention layer to obtain the The query extracted from ,key Sum ; Step 3.8: The output sequence of the encoder Combined, use a multi-head attention mechanism to capture the embedding start token The answer embedding sequence Embedding sequence with exercises The relationship between :

[0021] Step 3.9: Add nonlinear features to get the final output of the decoder : .

[0022] Output the decoder obtained in step 3 Defined as , indicating that the decoder i The feature representation of the time step, then the student correctly answers the The probability of an exercise is expressed as:

[0023] In the formula, and is the interaction sequence tuple The vector representation is applied to and .

[0024] It includes the contextual information in the answer sequence, the interaction between questions and answers in the encoder output, and feature enhancement after nonlinear processing.

[0025] It also includes the students' correct answers to the The probability setting loss function for each exercise , expressed as: .

[0026] The predicted probability of a student getting the question right in step 4 is expressed as: .

[0027] The beneficial effects of the present invention are: the student-side information fusion knowledge tracking method based on Transformer of the present invention separates the exercise sequence and the answer sequence and inputs them into the improved Transformer model respectively, so that the model can calculate and capture the complex relationship between exercises and answers through the deep attention mechanism. The importance of side information in knowledge tracking is considered during the input process, so that the model has higher flexibility and attention representation ability, makes full use of the students' learning behavior trajectory, and constructs the students' knowledge level more comprehensively, thereby improving the accuracy of predicting the students' knowledge status. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the student-side information fusion knowledge tracking method based on Transformer of the present invention. DETAILED DESCRIPTION

[0029] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Example 1 The present invention provides a student-side information fusion knowledge tracing method based on Transformer, which solves two defects in the prior art: on the one hand, the attention layers of these models are relatively shallow, so they cannot fully capture the complexity of learners' learning activities after a certain period of time. These learning activities are usually presented in the form of high-dimensional and serialized data, and current models often have difficulty processing these data. For example, the network structure mentioned in SAKT does not apply the self-attention mechanism to exercises and answers, but directly uses the latent features of the embedding layer as input to the attention layer. In addition, these networks only use one attention layer between exercises and answers, which limits their ability to understand the complexity of learning activities. At the same time, different feature combinations of queries, keys and values ​​in the attention mechanism for knowledge tracing have not been fully explored. For example, EKT (Exercise-aware Knowledge Tracing), SAKT, etc. all use exercises to construct queries and use answers to construct keys and values. Other possible combinations have not been tried, and perhaps other combinations can provide substantial improvements. On the other hand, learners' exercises and answers involve a lot of information. Currently, most knowledge tracking models only consider time and accuracy, and do not use auxiliary information contained in exercises such as knowledge points, question types, question difficulty, etc., and auxiliary information in learners' answers such as number of interactions, time spent, accuracy, and other side information into knowledge tracking. Previous teaching research has shown that learning behavior has a complex mechanism of influence on knowledge acquisition. If these side information can be fully utilized, it will greatly improve the accuracy of tracking and evaluating learners' learning status in learning activities, and solve the problem of non-generalization when dealing with sparse data.

[0031] Example 2 The present invention provides a student side information fusion knowledge tracking method based on Transformer, specifically: first, the student's exercise sequence and answer sequence are processed by an embedding layer to obtain an exercise embedding sequence and an answer embedding sequence, and the obtained embedding sequences are embedded with side information in the exercises and side information in the student's answers. The output of the embedding layer is the sum of all embedded vectors that constitute the attribute. Then the embedded sequence is input into the encoder and decoder respectively, the encoder uses the exercise embedding sequence as a query, key and value and generates an output O through a self-attention mechanism, the decoder applies the first attention layer to the answer embedding sequence, and calculates the complex relationship between the exercise and the answer by using the output O received from the encoder and the output of the first attention layer through a second attention layer, and finally, the output of the decoder passes through a sigmoid activated linear layer to obtain the final prediction result.

[0032] Example 3 The present invention provides a student-side information fusion knowledge tracking method based on Transformer, which solves the problem that the existing technology cannot fully capture the complexity of students' learning activities after a period of time; and does not make full use of the rich learning trajectory information retained by the online platform, that is, does not evaluate students' answers in a fine-grained manner, resulting in an incomplete description of students' learning results. Specifically: (1) In the process of inputting the exercise sequence and the answer sequence, the side information related to them is taken into consideration, such as the question type, difficulty, and students’ answer speed, etc. The learning trajectory information retained by the online platform is fully utilized to portray the students’ learning outcomes more completely and comprehensively.

[0033] (2) Using the self-attention mechanism to capture the complex relationship between students and answers, when students perform different exercises, the skills they master are related to their responses to previous exercises. Therefore, the effect of the attention mechanism is reflected because the attention mechanism learns to capture the interdependence between exercises and answers and gives more weight to exercises that are related to prediction.

[0034] (3) Transformer is used to build the entire model framework because the Transformer model does not rely on the RNN framework, so there is no long-term dependency problem; at the same time, it has an encoder-decoder structure, which allows the student's exercise sequence and answer sequence to be input into the encoder and decoder respectively. This input separation structure enables the attention layers to be stacked multiple times, thereby improving the accuracy of the prediction.

[0035] Example 4 The student-side information fusion knowledge tracing method based on Transformer provided by the present invention is improved on the existing Transformer model, and a knowledge tracing model based on learner-side information fusion (SIFKT) is proposed, which consists of an encoder and a decoder, wherein the encoder and the decoder are stacked by multiple identical layers, including a multi-head self-attention layer and a feedforward network. The model separates the exercise sequence and the answer sequence, and inputs them into the encoder and the decoder respectively, so that the encoder focuses on the overall structure and internal association of the exercise, and the model can deeply understand the exercise; the decoder combines the student's answer record and the exercise characteristics to complete the prediction task. The structure ensures that the model can learn the complex relationship between the exercise and the answer, and can generate personalized predictions for each student; at the same time, the importance of side information in knowledge tracing is explored, and the side information contained in the exercise and the answer is input into the model through the embedding layer, so that the model has higher flexibility and attention representation ability, makes full use of the student's learning behavior trajectory, and more comprehensively constructs the student's knowledge level, thereby improving the accuracy of the prediction of the student's skills and abilities.

[0036] Example 5 The present invention provides a student-side information fusion knowledge tracking method based on Transformer, such as Figure 1 As shown, the following steps are included: S1: Separation of student learning records based on Seq2Seq model A method based on a sequence-to-sequence (Seq2Seq) model is used to process the student's learning record sequence and separate it into an exercise sequence and an answer sequence. The details are as follows: The goal of the sequence-to-sequence model is to obtain the student's exercise sequence and answer sequence by analyzing the previous interaction records, so as to provide support for the subsequent input separation. First, it is defined formally, and the student's interaction record is an interaction sequence , the model takes as input a sequence tuple ,here It not only indicates the first i Exercises, and with information related to the exercise such as the knowledge points of the question. Indicates that students Answer , and also with information related to the answer, such as the time it took the student to answer. Student's answer If his first i The answer is correct, then ,otherwise Since the goal of the model is to make a series of correctness estimates for the learner's exercises, that is, to calculate , indicating whether the student solved the next exercise correctly Therefore, the student correctly answered The probability calculation formula for each exercise is as follows:

[0037] S2: Capturing the complex relationship between students’ questions and responses Use self-attention mechanism to capture the complex relationship between students and answers; specifically: S21: embedding side information into the sequence; specifically including: The embedding layer transforms each and Mapped to a vector in the latent space, generating a sequence of exercise embeddings And answer embedding sequence This layer embeds the following attributes into and The specific properties are shown in Table 1. The calculation formula is as follows: First, learners interact Encoding as index , and then in the mapping layer we get:

[0038] Here, Represents a learner interaction The vector representation of Represents the weights of all potential side information. The matrix Covers all relevant potential side information, Each column of is a vector representation of potential side information. Therefore, It is actually the weighted sum of all potential side information.

[0039] Finally, the exercise sequence and answer sequence After being processed by the embedding layer, the obtained exercise embedding sequence Embedded sequence with answer is the sum of all embedding vectors of its constituent attributes. Specifically, the exercise information includes the exercise ID, knowledge point and its ID, and position. Therefore, the exercise embedding is constructed by adding the corresponding exercise ID embedding vector, knowledge point embedding vector, knowledge point ID embedding vector, and position embedding vector. The student's answer information includes accuracy, time, and position. Similarly, the accuracy embedding vector, time embedding vector, and position embedding vector are added together to construct the answer embedding The resulting embedding sequence is as follows:

[0040]

[0041] in, represents the corresponding embedding layer, encoding the question sequence and answer sequence as well as the side information contained in them into vectors. The embedding matrix can be expressed as , Indicates the total number of various side information corresponding to different projects, Represents the embedding dimension of the item.

[0042] Table 1 Attributes and descriptions of exercises and answers

[0043] S22: Using the input separation method, the exercise sequence and the answer sequence are input into the encoder and decoder respectively. Specifically including: SIFKT will exercise sequence and answer sequence Separately, after being processed by the embedding layer, they are input into the encoder and decoder respectively. The encoder receives the embedded sequence of the exercise , through the combination of multiple layers of masked attention mechanism and feedforward network to process the exercise embedding sequence, capture the complex dependencies between exercises, such as the knowledge point association or difficulty similarity of different exercises, the output of each layer is improved through residual connection and layer normalization to improve training stability and information flow, and then output a sequence Contains the context information of each exercise in the entire sequence, where Represents the characteristic representation of each exercise, and the calculation formula is as follows:

[0044] The decoder receives the output sequence generated by the encoder and answer embedding sequence ,in (Start Token) is the embedding of the start token. The decoder first responds to the embedding sequence Use the multi-head attention mechanism to generate the intermediate representation of the current layer , and then The output of the encoder Combined with the multi-head attention mechanism, the relationship between the answer sequence and the exercise sequence is captured to generate , by stacking the layers of the decoder multiple times, the final output of the decoder is obtained The calculation formula is as follows:

[0045]

[0046]

[0047] Therefore, the encoder extracts the correlation of each element in the input question sequence through a multi-layer stacked masked multi-head attention network, while the decoder captures the relationship between the question sequence and the answer sequence.

[0048] In the above process, the masked multi-head attention mechanism adopted by the encoder and decoder is as follows: The biggest difference between the SIFKT proposed in this invention and the traditional Transformer architecture is that in order to prevent invalid attention (for example, to predict the learner's i The answer to each exercise should be based on the learner's previous i -1 exercise results to predict, and i The results of the subsequent exercises are irrelevant). SIFKT performs a mask operation on the input of all multi-head attention networks involved in the encoder and decoder, resetting the attention weight after the current position to 0.

[0049] Multi-head attention network adopts , which represent the query, key, and value sequences respectively. The multi-head attention network actually applies the attention network multiple times to the same input sequence, using a different projection matrix each time. In the attention layer, the output of the embedding layer is first and Passed to the masked attention layer, through the matrix , and Each , and Projected into the latent space, where , the calculation formula is as follows:

[0050]

[0051]

[0052] The attention layer first starts from the input Extract query from ,key Sum It then classifies past interactions based on two factors: Distribute your attention ; The details are as follows: (1) Consistency between queries and keys Right now and The query and key similarity between them can be understood as interaction and The degree of overlap of potential knowledge points between them.

[0053] (2) Time interval deviation , according to the interaction and The time interval between adjusts the attention weight, where Indicates a timestamp.

[0054] At the same time, the attention network needs a masking mechanism to prevent the current position from paying attention to the subsequent position. The masking mechanism transforms the matrix The upper triangular part of yes Function, through After the operation, the attention network will reset the attention weights of subsequent positions to zero. is a scaling factor, that is, and The dimension is divided by To scale, the attention head is value Multiply by the weighted sum of the past masked attention weights. Finally, the output of the multi-head attention network is obtained by The concatenation result of the attention heads is multiplied by the matrix To summarize the outputs of different attention heads; the calculation formula is as follows:

[0055] S23: Adding nonlinear features to the model In a neural network, linear operations refer to simple weighted sum operations. Without nonlinear operations, a multi-layer neural network will be equivalent to a single-layer network, making it unable to capture and learn complex relationships and patterns in the data, limiting the model's expressive power. Therefore, in order to enable the model to better adapt to and learn various complex patterns and features, a position-aware feedforward network is applied to the output of the multi-head attention network to add nonlinear features to the model; the calculation formula is as follows:

[0056]

[0057] here represents the output of the attention head, , , as well as Respectively represent different The weight matrix and bias vector shared between .

[0058] S25: Setting of loss function Before calculating the final loss function, the output of the attention head after being processed by the feed-forward layer is fed to the connection layer. and is to embed the learner interaction The vector representation is applied to and ; The calculation formula is as follows:

[0059]

[0060] because is the joint feature representation of a series of time steps by the decoder. For a more detailed representation, define Indicates that the decoder i The feature representation of each time step includes: (1) contextual information in the answer sequence; (2) the interaction between the question and the answer in the encoder output; and (3) feature enhancement after nonlinear processing.

[0061] S3: Prediction layer After processing in step S2, This output is mapped to the (0,1) interval through the sigmoid activation function, which represents the predicted probability value, that is, the probability of whether the student can correctly answer a certain exercise. The calculation formula is as follows:

[0062] Through the calculation of the prediction layer, the predicted answer to the student is finally generated , which represents the predicted probability that the student gets each exercise right.

[0063] Example 6 The present invention provides a student-side information fusion knowledge tracking method based on Transformer, which divides the data set into a training set and a test set in a ratio of 8:2. Figure 1 As shown, the method comprises the following steps: S1: Data collection and preprocessing The datasets in this example are all public real datasets, which are obtained by collecting and recording the interactions between students and real online tutoring systems, such as ASSISTment2017, ASSISTment2015, ASSISTment2009, STATICS2011, etc. These datasets are widely used in KT tasks.

[0064] Take ASSISTment2017 as an example to illustrate the data processing process. The data in this dataset comes from the ASSISTments online learning platform. After deleting empty and duplicate records in the original dataset, it contains 943K answer records of 1709 students on 102 skills.

[0065] S2: Modeling student knowledge levels S21: First, let’s define the problem. The Transformer-based student-side information fusion knowledge tracking model is a variant of the Transformer structure. It consists of an encoder and a decoder. Both the encoder and the decoder are stacked with multiple identical layers, including a multi-head self-attention mechanism and a feedforward network. This input separation structure allows the student’s exercise sequence and answer sequence to be input into the encoder and decoder respectively. Specifically, it is defined formally, recording the student’s interaction as an interaction sequence , the model takes as input a sequence tuple ,here It not only indicates the first i Exercises, and with information related to the exercise such as the knowledge points of the question. Indicates that students Answer , and also with information related to the answer, such as the time it took the student to answer. Student's answer If his first i The answer is correct, then ,otherwise Since the goal of the model is to make a series of correctness estimates for the students’ exercises, that is, to calculate , indicating whether the student solved the next exercise correctly .

[0066] S22: embedding the side information into the sequence; specifically including: The embedding layer transforms each and Mapped to a vector in the latent space, generating a sequence of exercise embeddings And answer embedding sequence This layer embeds the following attributes into and The specific properties are shown in Table 1. The calculation formula is as follows: First, learners interact Encoding as index , and then in the mapping layer we get:

[0067] Here, Represents a learner interaction The vector representation of Represents the weights of all potential side information. The matrix Covers all relevant potential side information, Each column of is a vector representation of potential side information. Therefore, It is actually the weighted sum of all potential side information.

[0068] Finally, the exercise sequence and answer sequence After being processed by the embedding layer, the obtained exercise embedding sequence Embedded sequence with answer is the sum of all embedding vectors of its constituent attributes. Specifically, the exercise information includes the exercise ID, knowledge point and its ID, and position. Therefore, the exercise embedding is constructed by adding the corresponding exercise ID embedding vector, knowledge point embedding vector, knowledge point ID embedding vector, and position embedding vector. The student's answer information includes accuracy, time, and position. Similarly, the accuracy embedding vector, time embedding vector, and position embedding vector are added together to construct the answer embedding The resulting embedding sequence is as follows:

[0069]

[0070] in, represents the corresponding embedding layer, encoding the question sequence and answer sequence as well as the side information contained in them into vectors. The embedding matrix can be expressed as , Indicates the total number of various side information corresponding to different projects, Represents the embedding dimension of the item.

[0071] S23: Using the input separation method, the exercise sequence and the answer sequence are input into the encoder and decoder respectively. Specifically including: SIFKT will exercise sequence and answer sequence Separately, after being processed by the embedding layer, they are input into the encoder and decoder respectively. The encoder receives the embedded sequence of the exercise , through the combination of multiple layers of masked attention mechanism and feedforward network to process the exercise embedding sequence, capture the complex dependencies between exercises, such as the knowledge point association or difficulty similarity of different exercises, the output of each layer is improved through residual connection and layer normalization to improve training stability and information flow, and then output a sequence Contains the context information of each exercise in the entire sequence, where Represents the characteristic representation of each exercise, and the calculation formula is as follows:

[0072] The decoder receives the output sequence generated by the encoder and answer embedding sequence ,in (Start Token) is the embedding of the start token. The decoder first responds to the embedding sequence Use the multi-head attention mechanism to generate the intermediate representation of the current layer , and then The output of the encoder Combined with the multi-head attention mechanism, the relationship between the answer sequence and the exercise sequence is captured to generate , by stacking the layers of the decoder multiple times, the final output of the decoder is obtained The calculation formula is as follows:

[0073]

[0074]

[0075] Therefore, the encoder extracts the correlation of each element in the input question sequence through a multi-layer stacked masked multi-head attention network, while the decoder captures the relationship between the question sequence and the answer sequence.

[0076] In the above process, the masked multi-head attention mechanism adopted by the encoder and decoder is as follows: Multi-head attention network adopts , which represent the query, key, and value sequences respectively. The multi-head attention network actually applies the attention network multiple times to the same input sequence, using a different projection matrix each time. In the attention layer, the output of the embedding layer is first and Passed to the masked attention layer, through the matrix , and Each , and Projected into the latent space, where , the calculation formula is as follows:

[0077]

[0078]

[0079] The attention layer first starts from the input Extract query from ,key Sum It then classifies past interactions based on two factors: Distribute your attention ; The details are as follows: (1) Consistency between queries and keys Right now and The query and key similarity between them can be understood as interaction and The degree of overlap of potential knowledge points between them.

[0080] (2) Time interval deviation , according to the interaction and The time interval between adjusts the attention weight, where Indicates a timestamp.

[0081] At the same time, the attention network needs a masking mechanism to prevent the current position from paying attention to the subsequent position. The masking mechanism transforms the matrix The upper triangular part of yes Function, through After the operation, the attention network will reset the attention weights of subsequent positions to zero. is a scaling factor, that is, and The dimension is divided by To scale, the attention head is value Multiply by the weighted sum of the past masked attention weights. Finally, the output of the multi-head attention network is obtained by The concatenation result of the attention heads is multiplied by the matrix To summarize the outputs of different attention heads; the calculation formula is as follows:

[0082] S3: Adding nonlinear features to the model In a neural network, linear operations refer to simple weighted sum operations. Without nonlinear operations, a multi-layer neural network will be equivalent to a single-layer network, making it unable to capture and learn complex relationships and patterns in the data, limiting the model's expressive power. Therefore, in order to enable the model to better adapt to and learn various complex patterns and features, a position-aware feedforward network is applied to the output of the multi-head attention network to add nonlinear features to the model; the calculation formula is as follows:

[0083]

[0084] here represents the output of the attention head, , , and Respectively represent different The weight matrix and bias vector shared between .

[0085] S4: Loss function setting Before calculating the final loss function, the output of the attention head after being processed by the feed-forward layer is fed to the connection layer. and is to embed the learner interaction The vector representation is applied to and ; The calculation formula is as follows:

[0086]

[0087] because is the joint feature representation of a series of time steps by the decoder. For a more detailed representation, define Indicates that the decoder i The feature representation of each time step includes: (1) contextual information in the answer sequence; (2) the interaction between the question and the answer in the encoder output; and (3) feature enhancement after nonlinear processing.

[0088] S5: Output prediction results The final output of the decoder is , and this output result is mapped to the (0,1) interval through the sigmoid activation function, which represents the predicted probability value, that is, the possibility of whether the student can correctly answer a certain exercise. The calculation formula is as follows:

[0089] Through the calculation of the prediction layer, the predicted answer to the student is finally generated , which represents the predicted probability that the student gets each exercise right.

[0090] S6: Set parameters for the model Embedding Matrix , The problem embedding matrix and the answer embedding matrix are randomly initialized from a Gaussian distribution with a mean of 0 and a standard deviation of 0.1. and the answer embedding matrix The dimensions of are set to 50 and 100 respectively. Every 25 attempts of a student are regarded as a time interval. For the dataset, the batch size is set to 32, corresponding to 32 split sequences for each student. The Adam optimizer with a learning rate of 0.01 is used to train the model. In order to improve the stability and smoothness during training, the cropping domain is limited to 50. At the same time, a dropout of 0.1 is used to alleviate overfitting.

Claims

1. A student-side information fusion knowledge tracking method based on Transformer, characterized by: The following steps are involved: Step 1: Build a sequence-to-sequence model to separate the student's learning record sequence into an exercise sequence and an answer sequence; Step 2: embed the sequence obtained in step 1 into the side information through the embedding layer of the sequence-to-sequence model to obtain the question embedding sequence and the answer embedding sequence; Step 3: Build a Transformer model based on the masked attention mechanism. The exercises obtained in step 2 are embedded in the sequence and input into the encoder of the Transformer model. The encoder captures the relationship between exercises through the multi-head attention mechanism, and then adds nonlinear features through the feedforward network to obtain the output. The decoder embeds the answer obtained in step 2 into the sequence and outputs through a multi-head attention mechanism Combine and capture the relationship between the exercise and the answer, and then add nonlinear features through the feedforward network to obtain the decoder output ; Step 4: Decoder output according to step 3 Generates the predicted probability that the student will answer the question correctly.

2. The student-side information fusion knowledge tracking method based on Transformer as claimed in claim 1, characterized in that: In step 1, the student's learning record is recorded as an interaction sequence , where the sequence tuple , Indicates the student's answer i Exercises and related information, response Indicates that students Answer and answer relevant information, , i If the answer is correct, ,otherwise .

3. The student-side information fusion knowledge tracking method based on Transformer as claimed in claim 2 is characterized in that: The step 2 specifically includes the following steps: Step 2.1: Substitute the sequence tuple Encoding as index , the vector representation of the side information is obtained by calculating the mapping layer of the sequence-to-sequence model. The calculation formula is as follows: In the formula, Represents a sequence tuple The side information vector representation of ; Represents the weights of all potential side information; the matrix Covers all relevant potential side information, each column is a vector representation of potential side information; Step 2.2: Use the embedding layer of the sequence-to-sequence model to embed the exercise sequence and answer sequence Embed the side information vector to get the problem embedding sequence Embedded sequence with answer , specifically expressed as follows: In the formula, Represents the embedding layer, which is used to embed the exercise sequence and answer sequence And the side information contained is encoded as a vector; the embedding matrix is ​​expressed as , Indicates the total number of various side information corresponding to different projects, Represents the embedding dimension of the item.

4. The student-side information fusion knowledge tracking method based on Transformer as claimed in claim 3 is characterized in that: The step 3 specifically comprises the following steps: Step 3.1: Embed the exercises obtained in step 2.2 into the sequence In the encoder of the input Transformer model, the encoder uses the masked attention layer to extract Extract query from ,key Sum , and then through the matrix , and Each , and Projection to latent space: In the formula, ; Step 3.2: According to the query and key The consistency between and The similarity between the query and key is the past interaction Distribute your attention , the calculation formula is as follows: In the formula, the matrix is ​​transformed into The upper triangular part of ; Time interval deviation , according to the interaction and The time interval between adjusts the attention weight, where Indicates a timestamp; is the scaling factor, i.e. and Dimensions; Step 3.3: Multiply the weighted sum of past mask attention weights by the value Get the attention head : In the formula, yes Function, through After the operation, the attention network resets the attention weights of subsequent positions to zero; Step 3.4: The concatenation result of the attention heads is multiplied by the matrix To summarize the output of different attention heads, the calculation formula is as follows: Step 3.5: Add nonlinear features to the attention head output obtained in step 3.4 by the following formula: In the formula, represents the output of the attention head, , , as well as Respectively represent different The weight matrix and bias vector shared between them; Step 3.6: Output the attention head with nonlinear features added in step 3.5 The output sequence of the encoder is obtained through residual connection and layer normalization in Represents the characteristic representation of each exercise, and the calculation formula is as follows: Step 3.7: Embed the answer obtained in step 2.2 into the sequence The decoder is input into the Transformer model, and the decoder embeds the start token through the same multi-head attention mechanism as the encoder. The answer embedding sequence Generates the intermediate representation of the current layer : In the formula, The decoder uses the masked attention layer to obtain the input The query extracted from ,key Sum ; Step 3.8: The output sequence of the encoder Combined, use a multi-head attention mechanism to capture the embedding start token The answer embedding sequence Embedding sequence with exercises The relationship between : Step 3.9: Add nonlinear features to get the final output of the decoder : 。 5. The student-side information fusion knowledge tracking method based on Transformer as claimed in claim 4 is characterized in that: The decoder output obtained in step 3 is Defined as , indicating that the decoder i The feature representation of the time step, then the student correctly answers the The probability of an exercise is expressed as: In the formula, and is the interaction sequence tuple The vector representation is applied to and .

6. The student-side information fusion knowledge tracking method based on Transformer as claimed in claim 5, characterized in that: Said It includes the contextual information in the answer sequence, the interaction between questions and answers in the encoder output, and feature enhancement after nonlinear processing.

7. The student-side information fusion knowledge tracking method based on Transformer as claimed in claim 6, characterized in that: It also includes the students' correct answers to the The probability setting loss function for each exercise , expressed as: 。 8. The student-side information fusion knowledge tracking method based on Transformer as claimed in claim 7, characterized in that: The predicted probability of the student doing the exercise correctly in step 4 is expressed as: 。

Images