A multi-modal feature fusion knowledge tracing method for computer programming test questions

Abstract

The application discloses a kind of multi-modal feature fusion knowledge tracking methods for computer programming test, belong to the field of intelligent education.There is following step: (1) collection and arrangement university computer programming test and student's test record, artificial marking label, constructs programming test knowledge tracking dataset Buddy.(2) design hypergraph network structure fusion test rich text information and multi-knowledge point label information.(3) utilize relationship graph network to depict the educational relationship between student, test and knowledge point, and obtain the feature representation of the three kinds of objects fused dependence by network iteration update.(4) effective fusion of multi-modal features is carried out using self-attention mechanism, and integrated into the knowledge tracking model designed by oneself, analyze student knowledge state and predict test performance.The method of the application fuses multi-modal information in computer programming test and student test record to help the model improve performance, and has high use value.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent education, and particularly relates to a multi-modal feature fusion knowledge tracing method for computer programming test questions. BACKGROUND

[0002] The rapid development of intelligent education technology promotes the gradual transformation of traditional education to personalized intelligent education, which requires timely understanding of the knowledge state level of each student, and knowledge tracing (KT) emerges as the times require. Knowledge tracing is to model the test records of a student in a period of time to depict the mastery of knowledge points of the student in this period of time. In recent years, in the field of intelligent education, many researchers at home and abroad have proposed many knowledge tracing models, which are mainly divided into two categories: one is based on traditional machine learning method, and the other is based on deep learning method.

[0003] In the traditional method, the knowledge tracing model based on Bayesian network (Bayesian Knowledge Tracing, BKT) is relatively widely used. By constructing a hidden Markov model, the latent variables are learned to represent the changing knowledge state of the student, the change of the knowledge state of the student is tracked, and the probability that the student masters the knowledge point is predicted. The BKT model assumes that the knowledge state of the student is a set of binary variables, which represents whether the student masters a specific knowledge point or not, and the update of the variable depends on the update of the test behavior of the student, that is, the student correctly answers the question of the given knowledge point or incorrectly answers the question of the given knowledge point.

[0004] Deep learning technology has developed rapidly, and many deep learning-based models have emerged in the field of knowledge tracing. In the knowledge tracing task, the student's knowledge state is modeled through the student's test records. The difficulty lies in the high complexity of the relationship between the student and the test knowledge points, and the student's test behavior is a very complex process. Deep learning-based models are more suitable for the establishment of complex models due to their large number of parameters and deeper and more complex models. DKT is the first knowledge tracing model based on deep learning. The DKT model is based on RNN (Recurrent Neural Networks), which takes the student's test records in time series as input, converts them into feature vector representations through one-hot encoding or compressed sensing algorithms, and models the student's changing knowledge state through the hidden layer. Finally, it outputs the predicted student test results. It fully utilizes the time series information. In addition, compared with the hidden Markov model, the RNN-based model has high-dimensional, continuous, and potential state feature representation, so it is very suitable for handling knowledge tracing tasks. EERNN is a knowledge tracing model based on LSTM (Long Short-Term Memory) with an attention mechanism. It calculates the similarity between the current input and the historical input through the attention mechanism, aiming to capture the relevance of the current test and the test in the historical test records. By giving different weights to the similarity scores of the previous different time inputs, it fuses the information representation to help the model predict the student's performance. Because in the field of education, the student's test performance on training tests is largely related to their performance on similar tests. DKVMN is a memory-enhanced neural network knowledge tracing model that uses a static matrix key to store the feature representation of knowledge points and a dynamic matrix value to store the updated student's mastery of the corresponding knowledge points. The model describes the student's mastery of each concept. SAKT is a knowledge tracing model that uses a self-attention layer. In the calculation of each self-attention layer, the test information embedding vector is used as the query vector, and the test information and the student's test record embedding vector are used as the key vector and the value vector. SAINT is a knowledge tracing model based on Transformer. Its structure is composed of stacked encoders-decoders containing self-attention layers. The test information is used as the input of the encoder, and the encoder output, student test results, etc. are used as the input of the decoder. Finally, the probability of the student answering the test is output as the prediction value.

[0005] In summary, the main task of knowledge tracing is to track the knowledge state of students according to their test records, so as to predict their test performance in the future. The basic idea of this task is similar to typical sequence mining in various fields. In addition, in the modeling process, a method that can better integrate these features is needed to help the model improve the prediction accuracy and better depict the students' mastery of knowledge points in the face of a large amount of time series data of different types or different modalities. SUMMARY

[0006] The present application is proposed to solve the problem that the existing knowledge tracing model cannot well depict the characteristics of programming test questions, and it is difficult to effectively utilize and fully integrate the multi-modal data to improve the prediction accuracy of the model in the face of different types and different modalities of data. The present application proposes a multi-modal feature fusion knowledge tracing method (Transformer based Knowledge Tracing using Multi-modality Fusion, TKTMF) for computer programming test questions, which effectively integrates the multi-modal information in computer programming test questions and students' test records, helps the knowledge tracing model to improve the performance, analyzes the knowledge state of students and predicts the test performance of students.

[0007] Technical scheme: To achieve the purpose of the present application, the technical scheme adopted by the present application is as follows: a multi-modal feature fusion knowledge tracing method for computer programming test questions, comprising the following steps:

[0008] Step 1, constructing a data set: collecting and organizing the college computer programming test questions and students' test records, manually marking labels, and constructing a programming test knowledge tracing data set Buddy;

[0009] Step 2, fusion of multi-modal information of computer programming test questions: design a hypergraph network structure to fuse the semantic information of test question text and the multi-knowledge point label information contained in the test question;

[0010] Step 3, fusion of the educational dependence relationship information in the test records of students in programming test questions: use a relational graph network to depict the educational relationship among students, test questions and knowledge points, and obtain the feature representation of the three objects fused with the dependence relationship through iterative updating of the network;

[0011] Step 4, constructing a knowledge tracing model fused with multi-modal features: use the self-attention mechanism to effectively fuse the multi-modal features, and integrate them into the knowledge tracing model to analyze the knowledge state of students and predict the test performance.

[0012] Further, in step 4, a knowledge tracking model based on a self-attention transformation network (Transformer) is constructed in an encoder-decoder structure, including the following steps:

[0013] Step 4-1, input data: the original input of the encoder part is an embedding vector composed of test item number, knowledge point, student number and student answering situation; the original input of the decoder part is an embedding vector of student answering situation;

[0014] Step 4-2, multi-head self-attention mechanism with relative position encoding: the relative position encoding information of the input part at each time is directly input into the multi-head attention mechanism module of the encoder and the decoder through linear projection;

[0015] Step 4-3, predicting student answering performance and analyzing student knowledge state: the output feature of the decoder part is converted into a student knowledge state feature vector through a linear layer, and the knowledge state feature vector is dot multiplied with the Multi-hot encoding information of the computer programming test knowledge point to predict the student's performance on the specific knowledge point, which is expressed as follows:

[0016]

[0017] where h t is the knowledge state vector, T represents transposition, s t is the Multi-hot encoding of the test knowledge point at the current t time, is the predicted probability of the student answering the test correctly at t time.

[0018] Further, in step 2, when fusing the multi-modal information of computer programming test, a hypergraph network structure is designed to obtain test feature representation, including the following steps:

[0019] Step 2-1, hypergraph construction: in the Buddy dataset, the computer programming test is described from two angles: test text information and contained knowledge point set, i.e. the Bert vector of test text information and the multi-knowledge point information feature vector of test, which are used as modal information from different angles to construct a hypergraph, generate two association matrices H1 and H2 of the test, and obtain the final form H of the hypergraph by splicing the two modal association matrices H1 and H2;

[0020] Step 2-2, hypergraph network convolution: after the formation of the data hypergraph structure, the hypergraph association matrix H and the node feature Q = [q1, …, q N ] T are input into the graph convolution operation to process the data correlation in the feature learning process; the formula of the convolution layer is as follows:

[0021]

[0022] where Q (l) and Q (l+1) are the question features of the l-th and l+1-th layer respectively, σ denotes a nonlinear activation function, H is the hypergraph adjacency matrix, Θ is the learnable filter matrix; D v and D e are the diagonal matrices of hypergraph vertices and hyperedges respectively, W is an identity matrix, and denotes that all hyperedges have the same weight.

[0023] Further, in step 3, when fusing the educational dependency relationship information in the student's record on the programming questions, the embedding representations of the students, questions and knowledge points are obtained using the relational graph network structure, including the following steps:

[0024] Step 3-1, initialize the node embedding representations of students, questions and knowledge points: the embedding layer uses a d-dimensional trainable matrix to encode the students and knowledge points, where N and K are the number of students and knowledge points respectively; the feature representation of the question after updating by the hypergraph network structure in step 2 is used as the embedding representation of the question node in the relational graph;

[0025] Step 3-2, describe three local maps: construct a knowledge point dependency graph, a question-knowledge point correlation graph and a student-question interaction graph according to the Buddy dataset;

[0026] Step 3-3, update the node embedding representations of students, questions and knowledge points: use the fusion layer in the relational graph network that realizes the multi-level attention structure to balance information between multiple layers of relational graphs and learn the embedding representations.

[0027] Further, the self-attention mechanism is used to fuse the multi-modal features and integrate them into the knowledge tracking model, which specifically includes:

[0028] The original numbers of students, questions and knowledge points in the dataset are obtained through a learnable matrix to obtain their embedding representations 0, q i and c i ; the updated embedding representations of students, questions and knowledge points obtained through the relational graph network are 0', q1' and c1';

[0029] 0, q1, c1, u', q1' and c i ' are concatenated with the embedding representation of the answer to serve as the input for feature fusion, and then go through linear layer mapping, residual connection, layer normalization and multi-head attention mechanism with triangular mask to finally obtain the input embedding representation Input t that fuses different modal information at each time t in the knowledge tracking model.

[0030] Beneficial effects: compared with the prior art, the technical scheme of the present application has the following beneficial technical effects:

[0031] Firstly, the model proposed by the present application can be used in the knowledge tracking task of computer programming test questions. By modeling the test records of students on computer programming test questions, the knowledge state of students can be analyzed and the test performance of students can be predicted;

[0032] Then, the present application mines the multi-modal information in computer programming test questions and students' test records, designs applicable hypergraph network structure and relationship graph network structure, and obtains effective feature representation;

[0033] Finally, the present application integrates the multi-modal information of students, test questions and knowledge points into the knowledge tracking model through the design of multi-modal feature fusion network, realizes better model performance, and has good use value. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is the overall flowchart of the present application;

[0035] Figure 2 is the basic knowledge tracking model schematic diagram of the present application;

[0036] Fig. 3(a) is a schematic diagram of the construction module of the hypergraph network structure of the present application;

[0037] Fig. 3(b) is a schematic diagram of the construction module of the hypergraph network structure of the present application;

[0038] Figure 4 is a schematic diagram of the convolution module of the hypergraph network structure of the present application;

[0039] Figure 5 is a schematic diagram of the multi-modal feature fusion network of the present application;

[0040] Figure 6 is a schematic diagram of the multi-modal feature fusion knowledge tracking method for computer programming test questions of the present application;

[0041] Figure 7 is a radar chart of the knowledge tracking model analyzing the ability of students;

[0042] Figure 8 is a heat map of the knowledge tracking model analyzing the ability of students. DETAILED DESCRIPTION

[0043] The technical scheme of the present application will be further described below in combination with the drawings and examples.

[0044] As Figure 1As shown, the present application discloses a kind of multi-modal feature fusion knowledge tracking method for computer programming test questions, comprising the following steps:

[0045] Step 1, construct dataset: collect and arrange university computer programming test questions and student test records, artificial mark labels, and construct dataset Buddy.

[0046] The information used by the dataset of the present application is obtained from an online programming learning website. The text information of the test questions is obtained by using crawler technology, and the test question knowledge point label information is artificially marked. The test records of students are made into time series data by the data provided by the background administrator.

[0047] Each test question is composed of question description, input format and output format, and the text information of each test question is composed of the three parts of literal content.

[0048] The present application divides the following 18 knowledge point types by arranging the knowledge point contents examined by all programming test questions: array, string, sorting algorithm, dynamic programming, search, greedy algorithm, search algorithm, tree, stack and queue, heap, graph, linked list, set, bit operation, mathematics, game, class and program basic structure.

[0049] The test records of each student are presented in Json format data, which contains the number of students, the number of test records of students and the test records of students. The test records include the number of each test question, the correctness of the answer, the knowledge point label array and the submission time.

[0050] The number of test questions in Buddy dataset is 274, which contains 18 computer programming knowledge points. There are 1016 students with test records of computer programming test questions, a total of 82011 test records, and the average length of test records of each student is 80.7, so the input length of the model is uniformly set to 80 in the subsequent experimental setting. For input data less than 80, fill in the blank, for input data more than 80, divide the long sequence input into multiple inputs with a maximum of 80.

[0051] The basic knowledge tracking model of the present application models the test records of students, analyzes the knowledge state of students and predicts the test performance of students. The schematic diagram is as shown in Figure 2 .

[0052] Step 2, fuse multi-modal information of computer programming test questions: design hypergraph network structure to fuse the test text semantic information and the multi-knowledge point label information contained in the test questions.

[0053] The hypergraph construction module is shown in Figure 3 (a) and Figure 3 (b). Hypergraph is defined as Include vertex collection Hyperedge set ε, weight matrix W. The hypergraph G can be used The incidence matrix H represents that if the vertex is in this hyperedge, the value of the corresponding item in the incidence matrix is ​​equal to 1, otherwise it is equal to 0. The calculation of each element is shown in the following formula:

[0054]

[0055] As shown in Figure 3(a), in the knowledge tracing task, each question can be regarded as a vertex, and a question can be represented as Q = [q1,…,q N ] T , according to the Euclidean distance d(q i ,q j ) can establish hyperedges = 1, each hyperedge is composed of a vertex and its K nearest neighbors, and the items that are 1 in the hyperedge indicate that the corresponding node is in the hyperedge. Its degree is defined as d(v) = ∑ e∈ε ω(e)g(v,e), for a hyperedge = ∈ε, its degree is defined as δ(e) = ∑ v∈v g(v,e).

[0056] In the Buddy dataset, computer programming test questions can be described from two perspectives: the test text and the set of knowledge points contained in the test. These two components, the Bert vector of the test text and the label vector of the test's multiple knowledge points, serve as distinct modal information to construct a hypergraph structure. This allows us to generate two association matrix structures, H1 and H2, for the test questions. These are formed by combining hyperedges. To leverage the multimodal data characteristics of the test questions, we then concatenate the two association matrices, H1 and H2, to obtain the final hypergraph representation, H, as shown in Figure 3(b).

[0057] Hypergraph convolution module such as Figure 4 As shown. After the hypergraph structure of the data is formed, the hypergraph association matrix H and node features Q = [q1,…,q N ] T As input, graph convolution operations are used to process data correlation in the feature learning process. The formula of the convolution layer is as follows:

[0058]

[0059] where Q (l) and Q (l+1) are the test question features of the lth layer and the l+1th layer, σ represents the nonlinear activation function, H is the hypergraph association matrix, Θ is the learnable filter matrix; D v and D eThe diagonal matrices representing the hypergraph vertices and hyperedges respectively play a normalization role in the equation. W is an identity matrix, indicating that all hyperedges have the same weight.

[0060] Through graph convolution calculation, we obtain a feature representation that integrates the different modal information of each test question, effectively extracting the high-order correlation of computer programming test question data.

[0061] Step 3: Integrate the pedagogical dependency information from students' programming test records: Use a relationship graph network to characterize the pedagogical relationship between students, test questions, and knowledge points. Through iterative updates of the network, obtain a feature representation of these three objects that integrates the dependency relationship.

[0062] The embedding layer uses a d-dimensional trainable matrix Encode students and knowledge points. Where N and K are the number of students and knowledge points respectively. For student s z , whose embedding representation Is to pass his one-hot vector x z The result obtained by multiplying the trainable matrix S is the transpose of the zth row of the student embedding matrix S. Similarly, the relevant knowledge point c k Embed is the transpose of the kth row of matrix C. The embedding layer modeling formula is as follows:

[0063]

[0064] In the embedding layer of the present invention, the encoding of the two objects, student nodes and knowledge point nodes, is obtained through a trainable matrix. Considering the rich test text information and multiple knowledge point labels in the computer programming test question dataset Buddy, which is suitable for knowledge tracking, the test questions obtained in step 2 are updated with feature representations through a hypergraph network and used as the embedded representations of the test question nodes in the relationship graph, further enhancing the effect. After obtaining the embedded representations of students, test questions, and knowledge points, a fusion layer with a multi-level attention structure implemented in the relationship graph network is used to balance information between the multi-layer relationship graph to learn effective embedding representations, ultimately obtaining updated node embedding representations of students, test questions, and knowledge points.

[0065] Step 4: Build a knowledge tracking model that integrates multimodal features: Use the self-attention mechanism to effectively integrate multimodal features and integrate them into the knowledge tracking model to analyze students' knowledge status and predict their answering performance. Based on the self-attention transformer network, build an encoder-decoder knowledge tracking model, specifically including:

[0066] Step 4-1, input data: the original input of the encoder part is an embedding vector composed of test question numbers, knowledge points, student numbers and student answering conditions; the original input of the decoder part is an embedding vector of student answering conditions;

[0067] Step 4-2, multi-head self-attention mechanism with relative position encoding: the relative position encoding information of the input part input at each time is directly input into the multi-head attention mechanism module of the encoder and the decoder through linear projection;

[0068] Step 4-3, predicting student answering performance and analyzing student knowledge state: the output features of the decoder part are converted into student knowledge state feature vectors through a linear layer, and the knowledge state feature vectors are dot multiplied with the Multi-hot encoding information of the computer programming test knowledge points to predict the test performance of the student on the specific knowledge point, which is expressed as follows:

[0069]

[0070] where h t is a knowledge state vector, T represents transposition, s t is the Multi-hot encoding of the test knowledge point at the current t time, is the predicted probability of the student answering the test correctly at t time.

[0071] The three different objects of students, test questions and knowledge points can obtain respective embedding representations u, q1 and c1 through their original numbers through a learnable matrix. In step 2, the multi-modal information existing in the computer programming test is extracted by using the hypergraph network structure, and the updated test feature representation is used as the feature representation of the test question node in the relationship graph network used in step 3. In step 3, the relationship between students, test questions and knowledge points in computer programming education is considered, and a relationship graph network is used for modeling, obtaining the feature representations of the three different objects fused with the relationship between batches, which are u', q1' and c1', respectively. In the designed multi-modal feature fusion network, the embedding representations of the three different objects of students, test questions and knowledge points obtained through the learnable matrix and the enhanced embedding representations obtained through the hypergraph network and the relationship graph network are spliced with the embedding representations of the answering conditions as the input of feature fusion, and then mapped through a linear layer, processed through residual connection, layer normalization and multi-head attention mechanism with a triangular mask, etc., to finally obtain the model input embedding representation Inpu4 t as shown in Figure 5 .

[0072] as shown in Figure 6As shown, the present application models the multi-modal information existing in the data set through the hypergraph network structure and the relational graph network structure, and obtains the embedding representation after fusion through the designed multi-modal feature fusion network, and inputs it into the encoder and decoder part of the designed knowledge tracking model as the original input, obtains the knowledge tracking method finally fused with multi-modal features, and better analyzes the student knowledge state and predicts the probability of the student answering the test questions.

[0073] In the experimental setup, the AdamW optimizer is used, and the batch size is set to 32. The input sequence length of the model is uniformly set to 80, and the minimum length of the original input sequence is set to 10. The original input sequence less than 80 is filled with blank fillers, and the input sequence longer than 80 is divided into multiple input sequences with a maximum length of 80. The data set is divided into training set and test set according to the ratio of 7:3. The encoder and the decoder are stacked with 4 blocks, the number of heads of the multi-head attention mechanism is set to 8, the dropout method is used, the value is set to 0.1, the dimension of the model feature vector is set to 256, and the learning rate is set to 0.0002. In the hypergraph network, the parameter K of the K nearest neighbor algorithm for constructing hyperedges is set to 5.

[0074] Table 1 Comparison experiment

[0075]

[0076] In Table 1, the DKT model based on RNN (LSTM), the EERNNM model, the SAINT model based on the Transformer model and the TKTMF model of the present application are compared. It can be seen that the prediction effect of the TKTMF model in the knowledge tracking task is better than that of other models. The AUC of TKTMF is higher, which is improved by 2.41 percentage points compared with the TKT model in the previous chapter, and the MSE and MAE are the lowest. It can be shown that the fusion of multi-modal information helps the model to obtain better feature representation and helps the model to obtain higher accuracy to help predict the student's answering situation at the current time.

[0077] The visualization results of the model output are mainly the radar chart representation and the heat map representation of the student knowledge state. As shown in Figure 7 and Figure 8The radar chart mainly depicts a phased comparison of the student's behavior sequence at the beginning and end of the problem. Each axis of the polar coordinate of the radar chart represents a knowledge point. The solid line in the radar chart represents the student's initial mastery of each knowledge point, and the dashed line represents the student's latest mastery of each knowledge point. The values are between 0 and 1, and the larger the value, the higher the degree of mastery. The heat map mainly shows the continuous change of the student's mastery of knowledge points in a continuous period of time. Each student can understand the level of each knowledge point, and then conduct targeted practice. Each row of the heat map shows the time change of the student's mastery of a knowledge point, and each column of the heat map shows the student's mastery of all knowledge points at a time. The markers on the vertical axis represent the serial number of the knowledge point, and the markers (c, r) on the horizontal axis represent that the main knowledge point of the problem at the current time is c, and the answer is r.

[0078] The above describes in detail the multi-modal feature fusion knowledge tracking method for computer programming problems provided by the present application. It is worth noting that there are many methods and ways to implement this technical solution. The above description is only the preferred embodiment of the present application, and is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, modifications and adjustments made on the basis of the core idea of the present application will be considered as the protection scope of the present application. In summary, the content of the specification should not be understood as a limitation of the present application, and the protection scope of the present application should be defined by the appended claims.

Claims

1.A computer programming question oriented multi-modal feature fusion knowledge tracing method, characterized in that, Comprising the following steps: Step 1, constructing a data set: collecting and organizing college computer programming test questions and students' test records, manually labeled tags, and constructing a data set Buddy; Step 2, fusion of computer programming test multi-modal information: design a hypergraph network structure to fuse the semantic information of the test text and the multi-knowledge point label information contained in the test; Step 2-1, hypergraph construction: in the data set Buddy, the computer programming test is described from two angles: the Bert vector of the test text information and the multi-knowledge point information feature vector of the test. As the modal information described from different angles, the two kinds of association matrices H1 and H2 of the test are generated, and the final form H of the hypergraph is obtained by splicing the two modal association matrices H1 and H2; Step 2-2, Hypergraph network convolution: After the hypergraph structure of the data is formed, the hypergraph adjacency matrix H and the node features Q = [q1, …, qN] are input into the hypergraph convolution layer. The hypergraph convolution formula is as follows: N ] T As input, the data correlation in the feature learning process is processed by the graph convolution operation; the formula of the convolution layer is as follows: where Q (l) and Q (l+1) are the question features of the l-th and l+1-th layers, respectively, σ denotes a nonlinear activation function, H is the hypergraph adjacency matrix, Θ is a learnable filter matrix; D v and D e are the diagonal matrices of hypergraph vertices and hyperedges, respectively, and W is an identity matrix, indicating that all hyperedges have the same weight. Step 3, fusion of the educational dependence relationship information in the students' test records on the programming test: use a relational graph network to describe the educational relationship between students, tests and knowledge points, and obtain the feature representation of the three objects fused with the dependence relationship through the iterative update of the network; Step 3-1, initialize the node embedding representation of students, test questions and knowledge points: the embedding layer uses a d-dimensional trainable matrix Encode the students and knowledge points, where N and K are the number of students and knowledge points, respectively; the feature representation of the test questions after updating through the hypergraph network structure in step 2 is used as the embedding representation of the test question nodes in the relationship graph; Step 3-2, describe three local maps: according to the Buddy data set, construct a knowledge point dependence graph, a test-knowledge point correlation graph and a student-test interaction graph; Step 3-3, update the node embedding representation of students, tests and knowledge points: use the fusion layer with multi-level attention structure in the relational graph network to balance information between multiple layers of relational graphs and learn the embedding representation; Step 4, constructing a knowledge tracking model that fuses multi-modal features: use self-attention mechanism to fuse multi-modal features and integrate them into the knowledge tracking model to analyze the student's knowledge state and predict the test performance. 2.The computer programming test question multi-modal feature fusion knowledge tracking method according to claim 1, characterized in that, In step 4, based on the Transformer, a knowledge tracking model with encoder-decoder structure is constructed, including the following steps: Step 4-1, input data: the original input of the encoder part is an embedding vector composed of test number, knowledge point, student number and student test condition; the original input of the decoder part is an embedding vector of student test condition; Step 4-2, multi-head self-attention mechanism with relative position encoding: the relative position encoding information of the input part at each time is directly input into the multi-head attention mechanism module of the encoder and the decoder through linear projection; Step 4-3, predict student test performance and analyze student knowledge state: the output feature of the decoder part is converted into a knowledge state feature vector of the student through a linear layer, and the dot product of the knowledge state feature vector and the Multi-hot encoding information of the computer programming test knowledge point is calculated to predict the test performance of the student on the specific knowledge point, which is represented by the following formula: where h t is the knowledge state vector, T denotes the transpose, s t is the Multi-hot encoding of the knowledge point of the question at the current time t, is the predicted probability of the student answering the question correctly at time t. 3.The computer programming test question multi-modal feature fusion knowledge tracking method of claim 1, wherein, Use self-attention mechanism to fuse multi-modal features and integrate them into the knowledge tracking model, which specifically includes; The original numbers of students, test questions and knowledge points in the dataset are obtained through a learnable matrix to obtain respective embedding representations u, q i and c i ; the embedding representations of students, test questions and knowledge points after updating obtained through the relational graph network u', q' i and c ′ ' i ; u, q1, c i , u', q' i , and c' i are spliced as a feature fusion input, and then mapped through a linear layer, residual connection, layer normalization, and multi-head attention mechanism with a triangular mask to obtain the input embedding representation Input t that integrates different modal information at each time t in the knowledge tracking model.

Images