Transformer-based student-side information fusion knowledge tracing method

Abstract

The application discloses a student-side information fusion knowledge tracking method based on a Transformer, which comprises the following steps: constructing a sequence-to-sequence model, separating a learning record sequence of a student into a problem sequence and an answer sequence; embedding side information of the obtained sequence to obtain a problem embedding sequence and an answer embedding sequence; constructing a Transformer model based on a masking attention mechanism, inputting the problem embedding sequence into an encoder, capturing the relationship between problems through a multi-head attention mechanism of the encoder, adding nonlinear features to obtain an output, combining the answer embedding sequence with the output through a multi-head attention mechanism of a decoder, capturing the relationship between the problems and the answers, adding nonlinear features to obtain a decoder output, and generating a prediction probability of problems solved by the student. The application comprehensively constructs the knowledge level of the student and improves the accuracy of the prediction of the knowledge state of the student.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent education, and particularly relates to a student-side information fusion knowledge tracking method based on a Transformer. BACKGROUND

[0002] Knowledge tracking (KT) is a task of modeling each learner's mastery of knowledge concepts while participating in a series of learning activities. Specifically, the goal of knowledge tracking is to predict whether students can correctly answer the next question according to their learning history.

[0003] Traditional knowledge tracking methods include the methods of Bayes knowledge tracing (BKT) and collaborative filtering (CF), wherein the BKT is the first model to solve the problem of knowledge tracking. The BKT adopts a framework of a hidden Markov model (HMM), and the hidden state of the HMM represents the knowledge state of a learner, which is described by a set of binary variables: mastered or not mastered. Specifically, the BKT independently models each knowledge concept state, however, the traditional method has some obvious shortcomings such as the binary representation of the understanding of a learner may be unrealistic.

[0004] In recent years, with the successful application of deep learning in various fields, many deep learning-based knowledge tracing models have been proposed and achieved good performance. Neural network structures such as recurrent neural networks (RNNs) and Transformers have become common building blocks for knowledge tracing models. Over the years, traditional methods have been improved and extended many times, such as Deep Knowledge Tracing (DKT) and its variants. DKT is the first to use recurrent neural networks to model the dynamic knowledge state of students in a total hidden vector. Dynamic Key-Value Memory Networks (DKVMN) is a KT development using memory-enhanced neural networks, which includes static and dynamic matrices. The static matrix is the key matrix, which is used to store potential knowledge concepts (KCs), while the dynamic matrix is the value matrix, which is used to store the primary level of the corresponding KCs. They have learned the correlation between the three of them, namely the exercises, the basic knowledge points and the knowledge state of the students. A Self-Attentive model for Knowledge Tracing (SAKT) is a Transformer-based knowledge tracing model that uses a pure attention mechanism to model the learning process of learners. The DKT model faces the problem that its parameters are not interpretable. DKVMN is more interpretable than DKT because it explicitly maintains a knowledge point representation matrix (key) and a knowledge state representation matrix (value). However, since all these deep learning models are based on RNNs, they all face the problem of not generalizing when dealing with sparse data. At the same time, there is a lot of information involved in the exercises and behaviors of learners, among which the most important is the knowledge points, and there are many other important auxiliary information such as disciplines, question types, question difficulties, etc., and the auxiliary information in the behavior such as the number of interactions, the speed of doing the questions, the accuracy, etc. If these side information can be fully utilized, it will greatly improve the accuracy of tracing and evaluating the learning state of learners in learning activities, and at the same time solve the problem of not generalizing when dealing with sparse data. SUMMARY

[0005] The purpose of the present application is to provide a Transformer-based student-side information fusion knowledge tracing method, which solves the problem of low prediction accuracy of the knowledge state of students in the prior art.

[0006] The technical scheme adopted by the present application is: a Transformer-based student-side information fusion knowledge tracing method, comprising the following steps:

[0007] Step 1, constructing a sequence-to-sequence model, separating the student's learning record sequence into a problem sequence and an answer sequence;

[0008] Step 2, embedding the sequence obtained in step 1 through the embedding layer of the sequence-to-sequence model to obtain a problem embedding sequence and an answer embedding sequence;

[0009] Step 3, constructing a Transformer model based on a masking attention mechanism, inputting the problem embedding sequence obtained in step 2 into the encoder of the Transformer model, capturing the relationship between problems through multi-head attention mechanism, and then increasing the nonlinear features through a feedforward network to obtain the output The decoder combines and captures the relationship between the answer embedding sequence obtained in step 2 and the output , and then increases the nonlinear features through a feedforward network to obtain the decoder output ;

[0010] Step 4, generating the prediction probability of the student solving the problem according to the decoder output obtained in step 3.

[0011] The features of the present application also include

[0012] In step 1, the student's learning record is recorded as an interactive sequence , wherein the sequence tuple , represents the student's answer to the i th problem and the problem-related information, and the response represents the student's answer and the answer-related information, , if the th answer is correct, then i , otherwise .

[0013] Step 2 specifically includes the following steps:

[0014] Step 2.1, encoding the sequence tuple into an index , obtaining a vector representation of side information through the mapping layer of the sequence-to-sequence model, and the calculation formula is as follows:

[0015]

[0016] In the formula, represents the vector representation of side information of the sequence tuple ; represents the weight of all potential side information; the matrix ​All relevant potential side information is covered, and each column is a vector representation of potential side information;

[0017] Step 2.2, embedding layers of the sequence-to-sequence model are respectively inputted to the question sequence and the answer sequence to embed side information vectors, obtaining question embedding sequence and answer embedding sequence , which are specifically represented as follows:

[0018]

[0019]

[0020] In the formula, represents an embedding layer, which is used to encode the question sequence and the answer sequence and the side information contained into vectors; the embedding matrix is represented as , represents the total number of various side information corresponding to different items, represents the embedding dimension of the item.

[0021] Step 3 specifically includes the following steps:

[0022] Step 3.1, input the question embedding sequence obtained in step 2.2 into the encoder of the Transformer model, and the encoder extracts query , key and value from the input through the masked attention layer, and then projects each , and to the latent space through matrices , and :

[0023]

[0024] In the formula, ;

[0025] Step 3.2, according to the consistency between the query and the key , i.e. the similarity between the query and the key and , the attention is allocated to the past interaction , and the calculation formula is as follows:

[0026] ​

[0027] where the upper triangular part of the matrix is replaced by ; the time interval bias adjusts the attention weights according to the time interval between the interactions and , where denotes the time stamp; is a scale factor, i.e., the dimension of and ;

[0028] Step 3.3, multiply the weighted sum of past mask attention weights by the value to obtain the attention head :

[0029]

[0030] where is the function, and after the operation, the attention network sets the attention weights of subsequent positions to zero;

[0031] Step 3.4, multiply the concatenated results of attention heads by the matrix to aggregate the outputs of different attention heads, and the calculation formula is as follows:

[0032]

[0033] Step 3.5, add nonlinear features to the attention head output obtained in step 3.4 by the following formula:

[0034]

[0035]

[0036] where denotes the output of the attention head, , , and denote different weight matrices and bias vectors shared between ;

[0037] Step 3.6, obtain the output sequence of the encoder by residual connection and layer normalization processing of the attention head output obtained in step 3.5 with added nonlinear features where denotes the feature representation of each exercise, and the calculation formula is as follows:

[0038]

[0039] Step 3.7. Embed the answer sequence obtained in Step 2.2 into the decoder of the Transformer model. The decoder captures the relationship between the answer embedding sequence and the question embedding sequence by the same multi-head attention mechanism as the encoder to generate the intermediate representation for the current layer :

[0040]

[0041] where are the query , key and value extracted by the masked attention layer from the input ;

[0042] Step 3.8. Combine with the output sequence of the encoder to capture the relationship between the answer embedding sequence and the question embedding sequence using the multi-head attention mechanism to generate :

[0043]

[0044] Step 3.9. Add nonlinear features to to obtain the final output of the decoder :

[0045] .

[0046] Define the decoder output obtained in Step 3 as , which represents the feature representation of the decoder at the th time step. Then the probability that the student correctly answers the i th question is represented as:

[0047]

[0048] where and are the vector representations of the interaction sequence tuple applied to and .

[0049] ​​The context information in the answer sequence, the interaction between the exercises and the answers in the encoder output, and the feature enhancement after the nonlinear processing are included.

[0050] The loss function is also set according to the probability of the student correctly answering the first , and is expressed as:

[0051] .

[0052] The prediction probability of the student correctly answering the exercises in step 4 is expressed as:

[0053] .

[0054] The beneficial effects of the present application are: the student side information fusion knowledge tracking method based on the Transformer of the present application separates the exercise sequence and the answer sequence and inputs them into the improved Transformer model respectively, so that the model can calculate and capture the complex relationship between the exercises and the answers through the deep attention mechanism, the importance of the side information in the knowledge tracking is considered in the input process, the model has higher flexibility and attention representation ability, fully utilizes the student learning behavior trajectory, more comprehensively constructs the knowledge level of the student, and thus improves the accuracy of the prediction of the knowledge state of the student. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a flowchart of the student side information fusion knowledge tracking method based on the Transformer of the present application. DETAILED DESCRIPTION

[0056] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0057] Example 1

[0058] ​The application provides a student-side information fusion knowledge tracing method based on a Transformer, and solves two defects of the prior art: on the one hand, the attention layers of these models are relatively shallow, so they cannot fully capture the complexity of the learning activities of the learners after a certain period of time, and these learning activities are usually in the form of high-dimensional and sequenced data, and the current models often have difficulty in processing these data. For example, the network structure mentioned in SAKT does not apply a self-attention mechanism to the exercises and answers, but directly uses the latent features of the embedding layer as input to pass to the attention layer. In addition, these networks only use one layer of attention layer between exercises and answers, which limits their understanding ability of the complexity of learning activities, and different feature combinations of queries, keys and values in the attention mechanism for knowledge tracing have not been fully explored, for example, EKT (Exercise-aware Knowledge Tracing), SAKT, etc. all use exercises to construct queries and use answers to construct keys and values, and other possible combinations have not been tried, and other combinations may provide substantial improvements. On the other hand, a large amount of information is involved in the exercises and answers of the learners, and most of the current knowledge tracing models only consider time and accuracy, and do not use the auxiliary information such as knowledge points, question types and question difficulties contained in the exercises and the auxiliary information such as interaction times, time spent, accuracy in the answers to the knowledge tracing. Past teaching research shows that learning behavior has a complex influence mechanism on knowledge acquisition. If these side information can be fully utilized, the accuracy of tracing and evaluating the learning state of the learners in the learning activities will be greatly improved, and the problem of not generalizing when processing sparse data will be solved.

[0059] Embodiment 2

[0060] The application provides a student-side information fusion knowledge tracing method based on a Transformer, and specifically, first, the exercise sequence and the answer sequence of the student are processed through an embedding layer to obtain an exercise embedding sequence and an answer embedding sequence, and the obtained embedding sequences respectively embed the side information in the exercises and the side information in the answers of the student. The output of the embedding layer is the sum of all embedding vectors constituting the attribute. Then the embedding sequences are respectively input into an encoder and a decoder, the encoder takes the exercise embedding sequence as the query, the key and the value and generates the output O through a self-attention mechanism, and the decoder applies the first attention layer to the answer embedding sequence, and calculates the complex relationship between the exercises and the answers through the second attention layer by taking the output O received from the encoder and the output of the first attention layer, finally, the output of the decoder is taken through a sigmoid-activated linear layer to obtain the final prediction result.

[0061] Embodiment 3

[0062] The application provides a student-side information fusion knowledge tracking method based on a Transformer, solves the problem that the complexity of learning activities of students after a period of time cannot be fully captured in the prior art, and the problem that rich learning track information retained by an online platform is not fully utilized, that is, the problem that the learning results of students are not completely described due to not evaluating the answers of students in a fine-grained manner, in particular:

[0063] (1) In the process of inputting the exercise sequence and the answer sequence, the side information related to them, such as the question category, the difficulty, and the student answer speed and the like, is considered, the learning track information retained by the online platform is fully utilized, and the learning results of students are described more completely and comprehensively.

[0064] (2) The self-attention mechanism is used to obtain the complex relationship between students and answers, when students perform different exercises, the skills they master are related to the reactions of the previous exercises. Therefore, the effect of the attention mechanism is embodied, because the attention mechanism learns to capture the mutual dependence between exercises and answers, and gives more weight to and predicts the related exercises.

[0065] (3) The Transformer is used to build the entire model framework, because the Transformer model does not depend on the RNN framework, so there is no long-term dependence problem; at the same time, it has an encoder-decoder structure, which can make the exercise sequence and the answer sequence of students input into the encoder and the decoder respectively, and this input separation structure can stack the attention layer multiple times, thereby improving the prediction accuracy.

[0066] Embodiment 4

[0067] This invention provides a Transformer-based student-side information fusion knowledge tracing method, which improves upon existing Transformer models by proposing a Knowledge Tracing Model Based on Learner-Side Information Fusion (SIFKT). This model consists of an encoder and a decoder, both composed of multiple stacked identical layers, including multi-head self-attention layers and feedforward networks. The model separates the question sequence and the answer sequence, inputting them separately into the encoder and decoder. This allows the encoder to focus on the overall structure and internal relationships of the questions, enabling the model to gain a deep understanding of the questions. The decoder combines student answer records and question features to complete the prediction task. This structure ensures the model can learn the complex relationships between questions and answers and generate personalized predictions for each student. Furthermore, the importance of side information in knowledge tracing is explored by embedding the side information contained in the questions and answers into the model through an embedding layer. This gives the model greater flexibility and attention representation capabilities, fully utilizing student learning behavior trajectories and more comprehensively constructing students' knowledge levels, thereby improving the accuracy of predicting student skills and abilities.

[0068] Example 5

[0069] This invention provides a student-side information fusion and knowledge tracking method based on Transformer, such as... Figure 1 As shown, it includes the following steps:

[0070] S1: Student learning record separation based on the Seq2Seq model

[0071] A sequence-to-sequence (Seq2Seq) model is used to process student learning record sequences, separating them into question sequences and answer sequences. Details are as follows:

[0072] The goal of sequence-to-sequence models is to obtain students' question and answer sequences by analyzing preceding interaction records, thus supporting subsequent input separation. Firstly, it is formally defined that students' interaction records are an interaction sequence. The model requires a sequence tuple as input. ,here This not only indicates the student's answer number i Each exercise includes relevant information such as the knowledge points covered in the question. Similarly, the response... Indicates students' opinion The answer It also includes information related to the answer, such as the time the student took to answer. (Student's answer) If his answer to the i th question is correct, then , otherwise . Since the goal of the model is to make a series of correctness estimates for the learner on the practice problems, that is, to compute , which represents whether the student correctly solved the next problem . Thus, the probability that the student correctly answered the th problem is computed as follows:

[0073]

[0074] S2: Capture the complex relationship between student problems and answers

[0075] Use self-attention mechanism to capture the complex relationship between student and answer; Specifically, including:

[0076] S21: Embed side information into the sequence; Specifically, including:

[0077] The embedding layer maps each and to a vector in the latent space, generating the problem embedding sequence and the answer embedding sequence . This layer embeds the following attributes into and , as shown in Table 1. The formula is as follows:

[0078] First, the learner interaction is encoded as an index , and then the mapping layer is obtained:

[0079]

[0080] Here, represents a vector representation of a learner interaction , indicates the weight of all latent side information. The matrix covers all relevant latent side information, each column of which is a vector representation of the latent side information. Therefore, is actually the weighted sum of all latent side information.

[0081] Finally, the problem sequence and the answer sequence are processed by the embedding layer to obtain the problem embedding sequence and the answer embedding sequence is the sum of all embedding vectors of its constituent attributes. Specifically, the question information includes question ID, knowledge point and its ID, position, so by adding the corresponding question ID embedding vector, knowledge point embedding vector, knowledge point ID embedding vector and position embedding vector, the question embedding is constructed. Similarly, the answer information of the student includes accuracy, time, position, and by adding the accuracy embedding vector, time embedding vector, and position embedding vector, the answer embedding is constructed. The obtained embedding sequence is as follows:

[0082]

[0083]

[0084] wherein, represents the corresponding embedding layer, which encodes the question sequence and answer sequence and the side information contained into vectors. The embedding matrix can be represented as , represents the total number of various side information corresponding to different items, represents the embedding dimension of the item.

[0085] Table 1 Attributes and descriptions of questions and answers

[0086]

[0087] S22: In the input separation mode, the question sequence and the answer sequence are respectively input into the encoder and the decoder. Specifically, it includes:

[0088] SIFKT separates the question sequence and the answer sequence , and after processing by the embedding layer, they are respectively input into the encoder and the decoder. The encoder receives the question embedding sequence , and processes the question embedding sequence through a combination of multiple levels of masking attention mechanism and feedforward network, to capture the complex dependency between questions, such as the association of knowledge points of different questions or the similarity of difficulty. The output of each layer is connected through residual connection and layer normalization to improve the training stability and information flow. Then, an output sequence is output, which contains the context information of each question in the whole sequence, wherein represents the feature representation of each question, and the calculation formula is as follows:

[0089]

[0090] The decoder receives the output sequence generated by the encoder and the answer embedding sequence , wherein The (Start Token) is an embedding of a start token, and the decoder first generates the answer embedding sequence using multi-head attention mechanism to generate the intermediate representation of the current layer , and then combines the output of the encoder with the output of the decoder using multi-head attention mechanism to capture the relationship between the answer sequence and the question sequence, and generate , by stacking each layer of the decoder multiple times, the final output of the decoder is obtained . The calculation formula is as follows:

[0091]

[0092]

[0093]

[0094] Therefore, the encoder extracts the relevance of each element in the input question sequence through the multi-layer stacked masked multi-head attention network, and the decoder captures the relationship between the question sequence and the answer sequence.

[0095] In the above process, the masked multi-head attention mechanism used by the encoder and the decoder is as follows:

[0096] The biggest difference between the SIFKT proposed in the application and the traditional Transformer architecture is that in order to prevent invalid attention (for example, to predict the learner's answer to the first i question, the learner's previous i -1 question result is used to predict, and the result of the question after i is irrelevant), the SIFKT performs a mask operation on the input of all multi-head attention networks involved in the encoder and the decoder, and sets the attention weight after the current position to 0.

[0097] The multi-head attention network uses , which respectively represent the query, key and value sequences. The multi-head attention network actually applies the attention network to the same input sequence multiple times, each time using a different projection matrix. In the attention layer, first pass the output of the embedding layer and to the masked attention layer, project each , and to the latent space through matrices , and , where , the calculation formula is as follows:

[0098]

[0099]

[0100]

[0101] The attention layer first extracts the query , key and value from the input . Then, it assigns attention to past interactions according to two factors; specifically as follows: (1) Consistency between query and key

[0102] , i.e., the similarity between the query and key , can be understood as the degree of overlap between the potential knowledge points between the interactions and . (2) Time interval deviation , the attention weight is adjusted according to the time interval between the interactions

[0103] and , where represents the timestamp. At the same time, the attention network needs a masking mechanism to prevent the current position from paying attention to subsequent positions. The masking mechanism replaces the upper triangular part of the matrix with

[0104] is the function, after the operation, the attention network will set the attention weight of the subsequent position to zero. Where is a scale factor, that is and , the reason for dividing by is to scale, the attention head is the weighted sum of the value multiplied by the past mask attention weight. Finally, the output of the multi-head attention network is the output of different attention heads by multiplying the concatenated results of attention heads by the matrix ; the calculation formula is as follows:

[0105]

[0106] S23: Add nonlinear features to the model

[0107] ​​In neural networks, linear operations refer to simple weighted sum operations. Without nonlinear operations, a multi-layer neural network would be equivalent to a single-layer network, making it unable to capture and learn complex relationships and patterns in data, limiting the model's expressive power. Therefore, in order to enable the model to better adapt to and learn various complex patterns and features, a position-aware feedforward network is applied to the output of the multi-head attention network to add nonlinear features to the model; the calculation formula is as follows:

[0108]

[0109]

[0110] Here represents the output of the attention head, , , and represent different weight matrices and bias vectors shared between them.

[0111] S25: Setting of loss function

[0112] Before calculating the final loss function, the output of the attention head after the feedforward layer is fed to the connection layer. and are the vector representations of the embedding layer learner interaction applied to and ; the calculation formula is as follows:

[0113]

[0114]

[0115] Since is the joint feature representation of the decoder for a series of time steps, in order to represent it in more detail, define to represent the feature representation of the decoder for the i time step, which includes: (1) the context information in the answer sequence; (2) the interaction between the exercises and the answers in the encoder output; (3) the enhanced features after nonlinear processing.

[0116] S3: Prediction layer

[0117] After step S2, the output result is mapped to the (0, 1) interval through the sigmoid activation function, representing the predicted probability value, i.e. the likelihood of the student being able to correctly answer a certain exercise, the calculation formula is as follows:

[0118]

[0119] Through calculations in the prediction layer, a predicted response for the student is ultimately generated. , where represents the predicted probability that a student answers each exercise correctly.

[0120] Example 6

[0121] This invention provides a student-side information fusion knowledge tracking method based on Transformer, which divides the dataset into training and test sets in an 8:2 ratio, such as... Figure 1 As shown, the method includes the following steps:

[0122] S1: Data Collection and Preprocessing

[0123] The datasets in this example are all public, real-world datasets. They were obtained by collecting and recording interactions between students and real online tutoring systems, such as ASSISTment2017, ASSISTment2015, ASSISTment2009, and STATICS2011. These datasets are widely used in KT tasks.

[0124] Using ASSISTment2017 as an example, the data processing procedure is explained. The data in this dataset comes from the ASSISTments online learning platform. After removing empty and duplicate records from the original dataset, it contains 943K response records from 1709 students across 102 skills.

[0125] S2: Modeling students' knowledge levels

[0126] S21: First, let's define the problem. The student-side information fusion knowledge tracking model based on Transformer is a variant of the Transformer architecture. It consists of an encoder and a decoder, both of which are stacks of multiple identical layers, including a multi-head self-attention mechanism and a feedforward network. This input-separated structure allows the student's exercise sequence and answer sequence to be input into the encoder and decoder respectively. Specifically, formally, the student's interactions are recorded as an interaction sequence. The model requires a sequence tuple as input. ,here This not only indicates the student's answer number i Each exercise includes relevant information such as the knowledge points covered in the question. Similarly, the response... Indicates students' opinion The answer It also includes information related to the answer, such as the time the student took to answer. (Student's answer) If his firsti is correct, then , otherwise . Since the goal of the model is to make a series of correctness estimates for the student's practice, that is, to compute , which indicates whether the student correctly solved the next practice .

[0127] S22: Embed side information into the sequence; specifically including:

[0128] The embedding layer maps each and to a vector in the latent space, generating a practice embedding sequence and an answer embedding sequence . This layer embeds the following properties into and , as shown in Table 1. The calculation formula is as follows:

[0129] First, the learner interaction is encoded into an index , and then the mapping layer obtains:

[0130]

[0131] Here, represents a vector representation of a learner interaction , indicates the weight of all latent side information. The matrix covers all relevant latent side information, each column of which is a vector representation of the latent side information. Therefore, is actually the weighted sum of all latent side information.

[0132] Finally, the practice sequence and the answer sequence are processed by the embedding layer to obtain the practice embedding sequence and the answer embedding sequence , which are the sum of all embedded vectors of their constituent attributes. Specifically, the practice information includes the practice ID, the knowledge point and its ID, and the position, so by adding the corresponding practice ID embedding vector, knowledge point embedding vector, knowledge point ID embedding vector, and position embedding vector, the practice embedding is constructed; and the student's answer information includes the correctness, time, and position, so by adding the correctness embedding vector, time embedding vector, and position embedding vector, the answer embedding is constructed. The obtained embedding sequence is as follows:

[0133]

[0134]

[0135] wherein, denote the corresponding embedding layers, which encode the question sequence and the answer sequence and the side information contained into vectors. The embedding matrices can be denoted as , denote the total number of various side information corresponding to different items, denote the embedding dimension of the items.

[0136] S23: The question sequence and the answer sequence are input into the encoder and the decoder respectively in a way of input separation. Specifically, it includes:

[0137] The SIFKT separates the question sequence and the answer sequence , and inputs them into the encoder and the decoder respectively after processing by the embedding layers. The encoder receives the embedding sequence of the questions , processes the question embedding sequence by combining the masking attention mechanism with the feedforward network at multiple levels, captures the complex dependency between the questions, such as the correlation of knowledge points or the similarity of difficulty of different questions, and improves the training stability and information flow through residual connection and layer normalization at each level. Then, it outputs a sequence which contains the context information of each question in the whole sequence, wherein denote the feature representation of each question, and the calculation formula is as follows:

[0138]

[0139] The decoder receives the output sequence generated by the encoder and the answer embedding sequence , wherein (Start Token) is the embedding of the start token. The decoder first uses the multi-head attention mechanism on the answer embedding sequence to generate the intermediate representation of the current layer , and then combines with the output of the encoder to capture the relationship between the answer sequence and the question sequence using the multi-head attention mechanism, and generates . By stacking each layer of the decoder multiple times, the final output of the decoder is obtained . The calculation formula is as follows:

[0140]

[0141]

[0142]

[0143] Therefore, the encoder extracts the relevance of each element in the input question sequence by the multi-layer stacked masked multi-head attention network, and the decoder captures the relationship between the question sequence and the answer sequence.

[0144] In the above process, the masked multi-head attention mechanism adopted by the encoder and the decoder is as follows:

[0145] The multi-head attention network adopts , which respectively represent the query, key and value sequences. The multi-head attention network actually applies the attention network to the same input sequence multiple times, each time using a different projection matrix. In the attention layer, the output of the embedding layer is first passed to the masked attention layer , and each , and is projected into the latent space by the matrices , and , where , and the calculation formula is as follows:

[0146]

[0147]

[0148]

[0149] The attention layer first extracts the query , key and value from the input . Then, it assigns attention to past interactions according to two elements; specifically as follows:

[0150] (1) Consistency between query and key , i.e., the similarity between the query and the key and , can be understood as the degree of overlap of the underlying knowledge points between the interactions and .

[0151] (2) Time interval bias , the attention weight is adjusted according to the time interval between the interactions and , where represents the timestamp.

[0152] At the same time, the attention network needs a masking mechanism to prevent the current position from paying attention to the subsequent position. The masking mechanism projects the matrix Replace the upper triangle part with yes Function, after After the operation, the attention network resets the attention weights of subsequent positions to zero. It is a scaling factor, that is... and The dimension, the reason for dividing by This is for scaling, pay attention to the head It is worth Multiply by the weighted sum of past mask attention weights. Ultimately, the output of the multi-head attention network is obtained by... The concatenation result of each attention head multiplied by the matrix To summarize the outputs of different attention heads; the calculation formula is as follows:

[0153]

[0154] S3: Add nonlinear features to the model

[0155] In neural networks, linear operations refer to simple weighted sum operations. Without nonlinear operations, a multi-layer neural network would be equivalent to a single-layer network, making it unable to capture and learn complex relationships and patterns in the data, thus limiting the model's expressive power. Therefore, to enable the model to better adapt to and learn various complex patterns and features, a position-aware feedforward network is applied to the output of a multi-head attention network, adding nonlinear features to the model; the calculation formula is as follows:

[0156]

[0157]

[0158] here This indicates the output of the attention head. , , and They represent different The weight matrix and bias vector are shared between them.

[0159] S4: Setting the loss function

[0160] Before calculating the final loss function, the output of the attention head, which has been processed by the feedforward layer, is fed into the connection layer. and It involves the interaction of learners in the embedded layer. Vector representation applied to and The calculation formula is as follows:

[0161]

[0162]

[0163] Since is the joint feature representation of the decoder for a series of time steps, in order to represent in more detail, define represent the feature representation of the decoder for the first i time step, including: (1) the context information in the answer sequence; (2) the interaction between the exercises and the answers in the encoder output; (3) the enhanced features after nonlinear processing.

[0164] S5: Output the prediction result

[0165] For the final output of the decoder , this output result is mapped to the interval (0, 1) through the sigmoid activation function, representing the predicted probability value, that is, the possibility of whether the student can correctly answer a certain exercise, and the calculation formula is as follows:

[0166]

[0167] Through the calculation of the prediction layer, the final prediction answer of the student is generated, representing the predicted probability of the student answering each exercise correctly.

[0168] S6: Parameter setting for the model

[0169] Embedding matrix , , the exercise embedding matrix and the answer embedding matrix are randomly initialized from the Gaussian distribution with mean 0 and standard deviation 0.1, and the dimensions of the exercise embedding matrix and the answer embedding matrix are set to 50 and 100 respectively. The student's every 25 attempts are regarded as a time interval. For the data set, the batch size is set to 32, corresponding to 32 segmented sequences for each student. The Adam optimizer with a learning rate of 0.01 is used to train the model. In order to improve the stability and smoothness in the training process, the clipping domain is limited within 50. At the same time, 0.1 dropout is used to alleviate overfitting.

Claims

1. A Transformer-based student-side information fusion knowledge tracing method, characterized in that, The method comprises the following steps: Step 1, constructing a sequence-to-sequence model to separate the student's learning record sequence into a problem sequence and an answer sequence; Step 2, embedding the sequence obtained in step 1 through the embedding layer of the sequence-to-sequence model to obtain a problem embedding sequence and an answer embedding sequence; Step 3, construct a Transformer model based on a masking attention mechanism, embed the problem sequence obtained in step 2 into the encoder of the Transformer model, and capture the relationship between the problems through the multi-head attention mechanism, and then pass through the feedforward network to increase the nonlinear features to obtain the output The decoder combines and captures the relationship between the problems and the answers through the multi-head attention mechanism, and then passes through the feedforward network to increase the nonlinear features to obtain the decoder output ;​ Step 4, generating a prediction probability that the student got the exercise right based on the decoder output from step 3 Generate a prediction probability that the student got the exercise right. 2.The Transformer-based student-side information fusion knowledge tracing method of claim 1, wherein, The student's learning record is recorded as an interaction sequence in step 1 , where the sequence tuple , represents the student's answer to the i th exercise and the exercise-related information, response represents the student's answer to the th exercise and the answer-related information, if the i th answer is correct, then , otherwise . 3.The Transformer-based student-side information fusion knowledge tracing method of claim 2, wherein, The step 2 specifically comprises the following steps: Step 2.1, the sequence tuple Encoding as index The side information is calculated by computing a vector representation of the side information at the mapping layer of the sequence-to-sequence model, with the following formula: wherein a side information vector representation of a sequence tuple a weight of all potential side information; matrix covers all relevant potential side information, each column being a vector representation of a potential side information​ Step 2.

2. Embedding the question sequence and the answer sequence by a sequence-to-sequence model respectively and the answer sequence to get the question embedding sequence and the answer embedding sequence and the answer embedding sequence , which is specifically represented as follows: wherein denotes the question embedding layer, which encodes the question sequence and the contained side information into a vector, denotes the answer embedding layer, which encodes the answer sequence and the contained side information into a vector; the embedding matrix is denoted as , wherein denotes the total number of question side information, denotes the total number of answer side information, denotes the embedding dimension of the item. 4.The Transformer-based student-side information fusion knowledge tracing method of claim 3, wherein, The step 3 specifically comprises the following steps: Step 3.1: Embed the exercises obtained in Step 2.2 into the sequence. In the encoder of the Transformer model, the encoder receives input through a masked attention layer. Extract query from ,key Sum Then through the projection matrix , and Each Projected into the potential space: In the formulae, ; Step 3.2, consistency between query and key is query, key similarity between past interactions assign attention , the calculation formula is as follows: where the upper triangular part of the matrix is replaced by through a masking mechanism; the time interval bias adjusts the attention weights according to the time interval between the interactions and , where denotes the time stamp; is a scale factor, i.e. the dimension of and ; Step 3.3, multiply the weighted sum of past mask attention weights by a value get attention head : In the formula, is function, after After the operation, the attention network sets the attention weight of the subsequent position to zero; Step 3.4, multiply the concatenated results of the attention heads by a matrix to aggregate the outputs of the different attention heads, computed as follows: to aggregate the outputs of the different attention heads, computed as follows: Step 3.5, adding a nonlinear feature to the attention head output obtained in step 3.4 by the following formula: wherein denotes the output of the attention head, , , and denote different weight matrices and bias vectors shared between Step 3.6, output the attention head with added nonlinear features from step 3.5 obtaining an output sequence of the encoder by residual connection and layer normalization processing wherein denotes the feature representation of each exercise, and the calculation formula is as follows: Step 3.

7. Embedding the answer sequence resulting from step 2.2 In the decoder of the Transformer model, the decoder embeds the start token by the same multi-headed attention mechanism as the encoder to generate an intermediate representation for the current layer : wherein are respectively extracted from the input query , key and value by the decoder through the masked attention layer Step 3.8, to the output sequence of the encoder In combination, the multi-headed attention mechanism captures relationships between the answer embedding sequence and the question embedding sequence and the problem embedding sequence and generates : Step 3.9, OR Increasing the non-linear feature to get the final output of the decoder : 。 5.The Transformer-based student-side information fusion knowledge tracing method of claim 4, wherein, The decoder output from step 3 is denoted as is defined as , where x^t denotes the feature representation of the decoder at time step t, then the probability that the student correctly answers the t-th exercise is given by: i ​​ wherein and is a vector representation of the interaction sequence tuple is applied to and . 6.The Transformer-based student-side information fusion knowledge tracing method of claim 5, wherein, The Contextual information in the answer sequence, interaction between the exercises and answers in the encoder output, and feature enhancement after nonlinear processing are included. 7.The Transformer-based student-side information fusion knowledge tracing method of claim 6, wherein, Also included is a probability of a student answering a given question correctly Setting a loss function is represented as: 。 8.The Transformer-based student-side information fusion knowledge tracing method of claim 7, wherein, In the step 4, the prediction probability of the student solving the problem is represented as: 。

Images