Knowledge tracking method and system based on multi-modal auxiliary information extraction

Abstract

The invention relates to the technical field of knowledge tracking, in particular to a knowledge tracking method and system based on multi-modal auxiliary information extraction, and the method comprises the steps: obtaining learning interaction data, test question data containing graphic and text information of test questions, and a step-by-step solving process corresponding to each test question, and carrying out the auxiliary feature extraction to obtain an auxiliary information set; constructing a knowledge tracking model embedded based on an auxiliary information set, obtaining question embedding of test questions from learning interaction data of students, and embedding of answer activities covering knowledge concepts and answers of the students; establishing an attention network model, and based on question embedding and answering activity embedding, determining the degree of mastering the question and the knowledge state of the student; and performing data enhancement on the learning interaction data, enhancing representation consistency of knowledge states before and after by utilizing comparison loss constraint data, and predicting subsequent answering performance based on the current knowledge state of the student. Through multi-modal information extraction and an attention network model, the knowledge state of the student is accurately evaluated, and the answering performance is predicted.

Description

Technical Field

[0001] The present application relates to the technical field of knowledge tracing, and more specifically, to a knowledge tracing method and system based on multimodal auxiliary information extraction. Background Art

[0002] Knowledge tracing, as one of the core tasks of educational data mining, has gradually become a key technical support for achieving digital and personalized education. Its goal is to dynamically assess students' knowledge mastery and predict their future learning performance by modeling their historical learning behavior data, providing a scientific basis for optimizing teaching strategies and adapting resources.

[0003] The data foundation for knowledge tracking comes from the multi-dimensional interaction records generated by students on the intelligent education platform, including but not limited to: the questions answered by students and their associated knowledge concepts, the correctness of the answers, the time spent answering the questions, the trajectory of repeated practice, and auxiliary information (such as the difficulty of the questions, the relevance of the knowledge graph, the use of multimedia resources, etc.). These data depict the dynamic evolution of the learning process from multiple levels such as time, cognition, and behavior. Current mainstream knowledge tracking methods mainly build models by counting the explicit features of students' historical answer sequences (such as accuracy rate and knowledge point coverage), or rely on expert-defined prior knowledge (such as association rules between knowledge points) to infer knowledge status.

[0004] However, these approaches face two key challenges: first, cold start and data sparsity. The lack of sufficient interactive data for new students or new knowledge points limits model generalization. Second, there is insufficient utilization of multimodal information. Existing models focus on temporal modeling of question sequences, but ignore the potential impact of semantic information implicit in question text / image content, as well as side information such as individual student attributes and question contextual features, on the evolution of knowledge states. Therefore, addressing the shortcomings of existing intelligent online education platforms in terms of resource utilization and personalized services has become a pressing technical challenge. Summary of the Invention

[0005] In view of this, the present application provides a knowledge tracking method and system based on multimodal auxiliary information extraction to solve the problems of traditional knowledge tracking methods such as ignoring the multimodal content semantics of test questions, difficulty in handling cold start and data sparsity, and inaccurate evaluation of dynamic knowledge status.

[0006] This application provides the following technical solutions:

[0007] In a first aspect, the present application provides a knowledge tracking method based on multimodal auxiliary information extraction, comprising:

[0008] Obtaining student learning interaction data, test question data including graphic and text information of the test questions, and the step-by-step solution process corresponding to each test question, and performing auxiliary feature extraction to obtain an auxiliary information set; the auxiliary information set includes the student's learning ability, the difficulty of the question, and the inherent complexity of the knowledge concept;

[0009] Constructing a knowledge tracking model based on the embedding of the auxiliary information set, obtaining the question embedding of the test questions and the embedding of the answering activities covering the knowledge concepts and the students' answers from the students' learning interaction data;

[0010] Establishing an attention network model to determine the student's mastery of the question and knowledge status based on the question embedding and the question-answering activity embedding;

[0011] Data enhancement is performed on the learning interaction data, and the representation consistency of the knowledge states before and after the data enhancement is constrained by contrast loss. The answering performance in the next time step is predicted based on the student's current knowledge state.

[0012] In one possible implementation, the learning interaction data consists of students' answer records, including basic information about the test questions, knowledge concepts involved in the test questions, and the content of the students' answers.

[0013] In one possible implementation, obtaining the step-by-step solution process for each test question includes:

[0014] The test data containing graphic and text information of the test questions are divided into subcategories, and the step-by-step solution process of the test questions is obtained based on the thought chain prompt method and the multimodal large language model.

[0015] In one possible implementation, the question embedding of the test questions is obtained from the students' learning interaction data, which is expressed as:

[0016]

[0017] Where qt represents the embedding of the question answered by the student at time t, is the inherent complexity of the knowledge concept corresponding to the question answered by the student at time t, and K represents the knowledge concept k corresponding to the question answered by the student at time t t Embedded into a vector, Represents the knowledge concept k covered t The problem changes, μ is the difficulty scalar parameter, which represents the difficulty deviation of a specific problem and its knowledge concept, P t represents the ability requirement corresponding to the question answered by the student at time t;

[0018] The question-answering activity embedding covering knowledge concepts and student responses is expressed as:

[0019]

[0020] Where, G represents the answer r to the student's question at time t t Embedded into a vector, Represents the knowledge concept k covered t Changes in students' learning activities.

[0021] In one possible implementation, the attention network model includes two self-attention encoder attention network models for respectively estimating the student's mastery of the problem and the student's knowledge status;

[0022] Embed q with questions t and answer questions embedded in a t As the input of the attention network model, we get the student’s question embedding q at each time step t t The mastery level m t , expressed as:

[0023]

[0024] Among them, q τ is the question answered at the current time step τ, is the transposed representation of the question answered at the total time step t, α t，τ represents the time effect attention weight score of the question answered at the current time step τ relative to the total time step t, θ represents the parameter controlling the intensity of the time effect, and d(t-τ) represents the time distance function; d k represents the dimension of the key vector used to scale the dot product; d(tl) represents the temporal distance from the lth time step to the total time step t;

[0025] Determine each knowledge concept k for time step t t The knowledge state h distributed in each head t , expressed as:

[0026]

[0027] Where, β t，z Represents the time effect attention weight score of the knowledge concept corresponding to the question answered at the current time step τ relative to the total time step t, is the transposed representation of the knowledge concept corresponding to the question answered in the total time step t, m τ is the student’s mastery of the question at the current time step τ.

[0028] In one possible implementation, performing data enhancement on the learning interaction data includes:

[0029] exchanging positions of adjacent interaction records in the learning interaction data;

[0030] Deleting at least one interaction record in the learning interaction data;

[0031] The correct or incorrect label of the answer of at least one interaction record in the learning interaction data is flipped.

[0032] In one possible implementation, after performing data enhancement on the learning interaction data, the method further includes:

[0033] Get the enhanced question embedding q + and student answering activities embedded in a + , the knowledge state representation h of the augmented sequence is obtained through the multi-head attention mechanism network t + ;

[0034] For each knowledge state h of each student i at time step t t i , construct positive and negative sample pairs for student i in the current batch, and use contrast loss constraint data to enhance the representation consistency of the previous and next knowledge states;

[0035] Among them, the contrast loss of student i at time step t is expressed as:

[0036]

[0037] Among them, h t +j represents each knowledge state of each student j at time step t after data augmentation, δ is the temperature hyperparameter, and sim() is the cosine similarity function.

[0038] In one possible implementation, the student's performance in answering questions at the next time step is predicted based on their current knowledge state, which can be expressed as:

[0039]

[0040] Among them, γ i Represented as the current prediction of student i for the new question q t+1 The attention weight score is used to predict the answering performance at time step t+1, and It represents the transpose of the new question answered at time step t+1, h i，t represents the total knowledge state of student i at the time step, h j，t represents the knowledge state of student i for the jth knowledge concept at time step t, For students on question q t+1 The state of knowledge, To predict the student's answering performance in the next time step, σ represents the Sigmoid function, W Ais a trainable parameter, and N is the total number of knowledge concepts contained in the knowledge concept K.

[0041] In a second aspect, the present application provides a knowledge tracking system based on multimodal auxiliary information extraction, comprising an auxiliary information extraction module, an auxiliary information embedding module, an attention network module, and a model prediction module;

[0042] The auxiliary information extraction module is used to obtain students' learning interaction data, test question data including graphic information of the test questions, and the step-by-step solution process corresponding to each test question, and perform auxiliary feature extraction to obtain an auxiliary information set; the auxiliary information set includes students' learning ability, the difficulty of the questions, and the inherent complexity of the knowledge concepts;

[0043] The auxiliary information embedding module includes a knowledge tracking model based on the embedding of the auxiliary information set, which is used to obtain the question embedding of the test questions and the answering activity embedding covering the knowledge concepts and the students' answers from the students' learning interaction data;

[0044] The attention network module is used to determine the student's mastery of the question and knowledge status based on the question embedding and the question-answering activity embedding;

[0045] The model prediction module is used to perform data enhancement on the learning interaction data, use contrast loss to constrain the representation consistency of the knowledge states before and after data enhancement, and predict the answering performance in the next time step based on the student's current knowledge state.

[0046] Compared with the existing technology, the technical solution provided by this application has the following beneficial effects:

[0047] By integrating the graphic and text content of multimodal test questions with the step-by-step solution process for auxiliary feature extraction, combined with the knowledge tracking model to dynamically model the difficulty of the questions, students' abilities and the complexity of knowledge concepts, and using the attention network to accurately capture the differences in students' mastery of knowledge points; further through data enhancement strategies and contrast loss optimization, the robustness of the knowledge state representation to interactive noise is effectively improved, thereby significantly enhancing the cold start adaptability to new questions or low-activity students on the basis of ensuring the interpretability of the model, and at the same time achieving more accurate personalized learning path prediction and resource recommendation through fine-grained knowledge state perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flowchart of a knowledge tracking method based on multimodal auxiliary information extraction provided in Example 1 of the present application.

[0049] Figure 2 This is a flowchart of a knowledge tracking method based on multimodal auxiliary information extraction provided in Example 2 of the present application.

[0050] Figure 3 This is a flowchart of a method for estimating a student's level of mastery of a question and a student's cognitive state, provided in Example 2 of the present application.

[0051] Figure 4 This is a schematic diagram of the changes in student knowledge status provided in Example 2 of this application.

[0052] Figure 5 This is a flowchart of the method for performing data enhancement operations on students' learning interaction data provided in Example 2 of the present application.

[0053] Figure 6 A structural diagram of a knowledge tracking system based on multimodal auxiliary information extraction provided in Example 3 of the present application. DETAILED DESCRIPTION

[0054] The following will combine the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0055] Example 1

[0056] See also Figure 1 , is a flow chart of a knowledge tracking method based on multimodal auxiliary information extraction provided in Example 1 of this application. Figure 1 As shown in , the specific implementation steps of the above method include:

[0057] Step 101: Obtain learning interaction data, test question data including graphic and text information of the test questions, and a step-by-step solution process corresponding to each test question.

[0058] In the embodiment of the present application, the above learning interaction data is collected from the intelligent online education platform and is recorded as X={x1, x2, x3, ..., x t}. Among them, each learning interaction data x i It consists of a record of answering questions, i.e. a series of basic information about the questions i and k i And the corresponding answer content i For each student i at time step t, it will answer a question that reflects the knowledge concept The problem and get an answer in, and The answers to the corresponding questions are respectively represented by the students’ correct and incorrect answers. Therefore, the answer record of each student can be recorded as:

[0059]

[0060] The above test data containing the graphic information of the test questions are divided into several subcategories to facilitate more thorough auxiliary information extraction. The thought chain prompts the content answer of the multimodal large language model q a , generating a step-by-step solution process for the text and image content of the test question, so that the multimodal large language model can fully understand the question. q Denotes a dataset containing question text and images, D a represents student interaction data, D s Represents the final set of auxiliary information. LLM Represents the large language model LLM, and the LLM prompt is represented as and Represent the thought chain prompt and auxiliary information extraction prompt respectively. The distribution of the corresponding output of LLM is expressed as

[0061] Step 102: Based on the graphic and text information of the test questions, the learning interaction data, and the step-by-step solution process corresponding to each test question, auxiliary feature extraction is performed to obtain an auxiliary information set.

[0062] Among them, the extracted auxiliary features include students’ learning ability P i 、Difficulty of the test questions D i , the inherent complexity of knowledge concepts C i .

[0063] Step 103: Construct a knowledge tracking model based on the embedding of the auxiliary information set, and obtain the question embedding q of the test questions and the student's answering activity embedding a from the students' learning interaction data to distinguish individual differences between different test questions.

[0064] Specifically, simply embedding all questions, knowledge concepts, and extracted auxiliary information into separate vectors would incur high computational costs. To avoid excessive parameterization and achieve effective feature embedding, the Rasch model (hereafter referred to as the Rasch model) is used to embed questions and student answering behaviors. The standard Rasch model describes the probability of a student answering correctly based on question difficulty and student ability, which is highly consistent with the application of the auxiliary feature information used.

[0065] Among them, the above problem is embedded in the Qt structure as follows:

[0066]

[0067] Where qt represents the embedding of the question answered by the student at time t, is the inherent complexity of the knowledge concept corresponding to the question answered by the student at time t, and K represents the knowledge concept k corresponding to the question answered by the student at time t t Embedded into a vector, Represents the knowledge concept k covered t The problem changes, μ is the difficulty scalar parameter, which represents the difficulty deviation of a specific problem and its knowledge concept, P t Represents the ability requirement corresponding to the question answered by the student at time t.

[0068] Will cover knowledge concept k t Questions and students' answers t Incorporate it into students' answering activities t Specifically, it is expressed as:

[0069]

[0070] Where, G represents the answer r to the student's question at time t t Embedded into a vector, Represents the knowledge concept k covered t Changes in students' learning activities.

[0071] Step 104: Build an attention network model and embed q based on the above problem t , students' answering activities are embedded in a t and various knowledge concepts k t , determine the students’ mastery of the problem and their knowledge status.

[0072] The above attention network model includes two self-attention encoders, which are used to estimate the student's mastery of the problem m and m respectively. t and students' level of knowledge and concepts t In order to keep the students’ learning behavior consistent over time, both parts are constructed based on the multi-head attention mechanism.

[0073] Embed q with the above problem t and students' answering activities embedded in a t As input to the first part, we get the student’s question embedding q at each time step t t The mastery level m t After obtaining each student's problem mastery, each knowledge concept k based on time step t t Estimate the knowledge state h distributed over each head t Finally, h obtained from each head t Combined together as the student's current state of knowledge.

[0074] Step 105: Perform data enhancement on the learning interaction data, use contrast loss constraints to enhance the consistency of the knowledge state representation of the previous and subsequent data, and predict the answering performance in the next time step based on the student's current knowledge state.

[0075] Among them, the above-mentioned data enhancement of the learning interaction data includes but is not limited to exchanging the positions of adjacent interaction records in the learning interaction data, deleting at least one interaction record in the learning interaction data, and flipping the correct or incorrect answer label of at least one interaction record in the learning interaction data.

[0076] Based on the above data augmentation operation, the enhanced question embedding and student answering activity embedding are obtained, and the knowledge state representation h of the augmented sequence is obtained through the multi-head attention mechanism network. t + For each batch, the contrast loss is calculated separately, that is, for each knowledge state h of each student i at time step t t i , construct positive and negative sample pairs for student i in the current batch, and determine the contrast loss of student i at time step t to ensure the consistency of knowledge state representation.

[0077] In this embodiment of the application, the student's current knowledge state is used to predict the answering performance of the next time step. In other words, in order to predict the student's answer to the next question q t+1 Whether it is correct or not, we only need to focus on the current knowledge state of the student. According to the student’s knowledge state h at time step t t Get the final prediction

[0078] Compared with the prior art, the technical solution provided in Example 1 of the present application has the following beneficial effects:

[0079] This application extracts fine-grained auxiliary features by integrating multimodal data such as graphic information of test questions and step-by-step solution processes, and dynamically constructs knowledge states by combining the improved Rasch model and attention network. It not only effectively distinguishes the differentiated examination dimensions of test questions of the same difficulty, but also enhances the model's robustness to learning noise (such as accidental errors and sequential disturbances) through data enhancement and comparative learning mechanisms. This enables knowledge state assessment to accurately capture students' long-term mastery trends of knowledge concepts, and to stably support personalized learning resource recommendations and weakness diagnosis. This significantly improves the reliability and adaptability of learning predictions in online education scenarios while reducing the traditional method's dependence on massive historical data.

[0080] Example 2

[0081] See also Figure 2 , is a flow chart of a knowledge tracking method based on multimodal auxiliary information extraction provided in the second embodiment of the present application. Figure 2 As shown in , the above-mentioned knowledge tracking method specifically includes the following steps:

[0082] Step 201: Acquire learning interaction data and construct a data set including question text and images.

[0083] Step 202: The above dataset D q The problem data in subclasses.

[0084] In the entire multimodal auxiliary information extraction process, the embodiment of the present application first performs the multimodal auxiliary information extraction on the dataset D. q The total number of questions in the dataset is divided. q There are Q questions in total, so the question data will be divided into subclasses to facilitate more thorough information extraction.

[0085] Step 203: Apply the thought chain prompting method to generate a step-by-step solution process for the problem contained in the data set.

[0086] In the embodiment of the present application, before extracting auxiliary information, a step-by-step solution process is generated for the text and image content of the question based on the multimodal language model, so that the multimodal language model can fully understand the question. The thought chain prompts the content of the multimodal large language model to answer questions.

[0087] Step 204: Integrate the graphic information of the problem, the learning interaction data and the step-by-step problem solving process to obtain the final auxiliary information set D s .

[0088] Specifically, after using prompt learning and thought chain methods to extract feasible problem-solving methods, the learning interaction data is input into the multimodal large language model, and the Auxiliary information extraction prompts, integrate the graphic information of the problem and the students' interactive data and the problem-solving process to obtain the auxiliary information content q s Then the normalization process regularization function g is applied to prevent the parameter space from being too sparse and to standardize the extracted auxiliary information to obtain the auxiliary information set D s The above comprehensive extraction of auxiliary information is based on each x i The multi-dimensional information of the three basic elements of student ID, question ID and knowledge concept ID. Among them, the extracted multi-dimensional auxiliary information includes the student’s learning ability P i 、Difficulty of the problem D i , the inherent complexity of knowledge concepts C i .

[0089] The learning ability of the above students Pi , which is auxiliary information extracted from the student's perspective. Inspired by traditional educational cognitive theory and the examination of knowledge tracking datasets, evaluating students' learning ability requirements for specific problems and knowledge concepts helps to more accurately predict students' learning status. In the process of extracting students' learning ability requirements, prior knowledge or constraints can be combined to improve the accuracy of the results generated by the multimodal large language model. Therefore, before extraction, combined with the ability requirements of the basic data, the possible ability requirements are divided into five specified items to obtain the student's learning ability representation P i .

[0090] The difficulty of the above problem is D i , is auxiliary information extracted from the perspective of the question. In the concept of traditional education, there is a close connection between the difficulty of the question and the knowledge level of the student it reflects. Students with higher knowledge levels can answer more difficult questions more accurately than students with lower abilities. Therefore, students' answers to questions of different difficulty levels directly reflect their cognitive status. Using a multimodal large language model, through prompt learning and thinking chain methods, combined with question text, images and the problem-solving process, the difficulty index of the problem is generated and limited to a reasonable range to obtain the difficulty expression D of the problem. i .

[0091] The inherent complexity C of the above knowledge concept i , is auxiliary information extracted from the perspective of knowledge concepts, and its role is similar to that of difficulty information. The assessment of students' cognitive level is based on knowledge concepts, and the level of cognitive level directly indicates the students' proficiency in specific knowledge concepts. Therefore, when the knowledge concepts involved in the questions answered by students are more complex, the change in their cognitive level should also be greater. Based on the multimodal large language model, combined with the problem-solving process and the level of knowledge concepts, the inherent complexity expression of knowledge concepts C is obtained. i .

[0092] Step 205: Based on the extracted auxiliary information set D s , construct a Rasch model based on auxiliary information embedding.

[0093] In view of the fact that existing knowledge tracking methods mainly rely on statistical data and prior knowledge defined by teachers, ignoring the implicit information of the graphic content of the question itself and the impact of other rich side information based on students, knowledge and questions on the knowledge tracking model, this application uses a large language model to mine the multi-dimensional information implicit in the data, and uses the Rasch model, an item response theory model widely used in the field of psychology, to embed an auxiliary information set containing multi-dimensional feature information into the knowledge tracking model to improve its efficiency and performance.

[0094] Specifically, the Rasch model embedding is combined with the students' learning interaction data to obtain the problem representation embedding sequence q and the student's answering activity embedding a to distinguish individual differences between different problems.

[0095] Since the standard Rasch model describes the probability of students answering questions correctly by the difficulty of the question and the student's ability, this application introduces the previously obtained auxiliary information based on the original model embedding. Specifically, question q t The embedding structure is expressed as:

[0096]

[0097] Where q t represents the question answered by the student at time t, is the inherent complexity of the knowledge concept corresponding to the question answered by the student at time t, and K represents the knowledge concept k corresponding to the question answered by the student at time t t Embedded into a vector, Represents the knowledge concept k covered t The problem changes, μ is the difficulty scalar parameter, which represents the difficulty deviation of a specific problem and its knowledge concept, P t Represents the ability requirement corresponding to the question answered by the student at time t.

[0098] In addition, the present application embodiment will cover the knowledge concept k t Questions and students' answers t Incorporate it into students' answering activities t Specifically, it is expressed as:

[0099]

[0100] Where, G represents the answer r to the student's question at time t t Embedded into a vector, Represents the knowledge concept k covered t Changes in students' learning activities.

[0101] Step 206: Based on the self-attention network, obtain the student's mastery of the problem and the student's cognitive level of the knowledge concept.

[0102] In the embodiments of the present application, in order to achieve more stable knowledge tracking, the present application uses a network model based on the self-attention mechanism to capture deeper learning dynamics in the student's learning process, thereby diagnosing the student's knowledge status from their sequential learning activities.

[0103] Specifically, an attention network model containing two self-attention encoders is pre-established. The two self-attention encoders are used to estimate the student’s mastery of the problem m and m respectively.t and students' level of knowledge and concepts t In the embodiment of the present application, in order to maintain a consistent time effect for students' learning behavior, both parts are constructed based on the multi-head attention mechanism.

[0104] Considering the sequential position of the student in the interaction sequence, that is, the influence of the questions answered by the student at each time step t on the overall interaction process. Since the early learning experience has little influence on the current learning state during the student's learning process, the initial self-attention mechanism is adjusted as follows:

[0105]

[0106] Among them, θ ＞0 represents the time decay rate parameter that can be learned and trained, d(Δt) represents the time distance between time steps, d k is the dimension of the key vector used to scale the dot product. According to the adjusted self-attention mechanism, the attention weight of the current answer to the previously answered question depends not only on the similarity between the query and the key, but also on the relative temporal distance between them, thus reflecting the temporal effect of the interaction process.

[0107] Then, we scale the click attention to multiple heads to get more robust results when aggregating. In multi-head attention, the query, key, and value are linearly transformed and then divided into H heads, which can be expressed as:

[0108] MultiHead(Q,K,V)=Concat(head i ,…head H )W O

[0109]

[0110] W Q , W K , W V , W O It is a learnable parameter matrix. The output obtained by the multi-head attention mechanism, after residual links and layer normalization, finally obtains the complete network layer architecture MMIEKTLayer, which is expressed as:

[0111] MMIEKTLayer(Q,K,V)=LayerNorm(MultiHead(Q,K,V)+Q)

[0112] The self-attention output of each head is followed by a residual connection and then layer normalization, where Q is the output of the sub-layer itself.

[0113] like Figure 3As shown in , based on the multi-head attention network layer defined above, the student's mastery of the problem and the student's cognitive state are estimated respectively. Specifically, the following steps are included:

[0114] Step 2061: Embed q using question t and students’ responses are embedded in r t As input to the first part, we get the student’s question embedding q at each time step t t The mastery level m t .

[0115] Specifically, in order to estimate the students’ mastery of the questions, we need to accurately understand the students’ performance on each question. t The mastery level m t Specifically expressed as:

[0116]

[0117] Among them, q τ is the question answered at the current time step τ, is the transposed representation of the question answered at the total time step t, α t，τ represents the time effect attention weight score of the question answered at the current time step τ relative to the total time step t, θ represents the parameter controlling the intensity of the time effect, and d(t-τ) represents the time distance function. k represents the dimension of the key vector used to scale the dot product; d(tl) represents the temporal distance from the lth time step to the total time step t.

[0118] Finally, the m obtained by each head t Combined, and then applying residual connection and layer normalization. Finally, we get the student's answer to each question q t The mastery level m t .

[0119] Step 2062: Determine each knowledge concept k for time step t t The knowledge state h distributed in each head t .

[0120] After obtaining each student's mastery of the question, the embodiment of the present application estimates the knowledge state of each student. In this part, the question is no longer used as the attention query, but the query of the knowledge concept should be ensured to be used consistently throughout the entire sequence in order to obtain a more accurate understanding of the student's knowledge state change process. Therefore, the embodiment of the present application estimates the knowledge state of each student for each knowledge concept k at time step t. t The knowledge state h distributed in each head t The estimates are as follows:

[0121]

[0122] Where, β t，z Represents the time effect attention weight score of the knowledge concept corresponding to the question answered at the current time step τ relative to the total time step t, is the transposed representation of the knowledge concept corresponding to the question answered in the total time step t, m τ is the student’s mastery of the question at the current time step τ.

[0123] Similarly, the h obtained for each head t are combined together as the student’s current state of knowledge, e.g. Figure 4 As shown in .

[0124] Step 207: perform data enhancement on the above learning interaction data, and obtain the current knowledge state h of each student through the multi-head attention mechanism network. t + , to predict the answering performance at the next time step.

[0125] To mitigate information bias in the task of predicting student interaction sequences, this application introduces a contrastive learning model training method to produce stable knowledge tracking results. Specifically, a contrastive loss function is used to maintain consistency in the evaluation of the student's knowledge state, and the student's knowledge state is directly used in the prediction phase to predict the student's future performance.

[0126] The first part of the training process involves contrastive loss for knowledge state assessment. According to common educational practice, if a student gives different answers to a given question multiple times during the learning process, this situation will not significantly affect their final knowledge state. Based on this, this application introduces contrastive learning techniques during the training process to maintain the consistency and robustness of knowledge tracking results.

[0127] First, data augmentation is performed on the students’ learning interaction data. Then, the multi-head attention mechanism is used to evaluate the students’ knowledge status in the new sequence. Finally, combined with contrast loss, the original interaction sequence and the enhanced interaction sequence of the students are used as positive samples, thereby eliminating negative samples. Figure 5 As shown in , the specific data enhancement operation includes the following steps:

[0128] Step 2071: swap the positions of adjacent interaction records in the above learning interaction data.

[0129] In the student's learning process, the overall learning order is very important, but the knowledge state obtained in the overall learning process should not be affected by smaller sequence disturbances. Therefore, the embodiment of the present application randomly selects and exchanges adjacent interaction records in the student interaction sequence to obtain similar student knowledge states.

[0130] Step 2072: Delete at least one interaction record in the above learning interaction data.

[0131] In the long-term learning process, the student's knowledge status will not be affected by a single question, but depends on the entire learning process. Therefore, the embodiment of the present application randomly selects and deletes a certain interaction record to maintain a similar knowledge status.

[0132] Step 2073: Flip the correct or incorrect label of the answer of at least one interaction record in the above learning interaction data.

[0133] Similar to the concepts of the two enhancement strategies in step 2071 and step 2072, occasional mistakes and guesswork will not have much impact on the student's final knowledge status. Therefore, this application randomly selects some interaction records and flips the correctness of the student's answers to simulate minor glitches in the learning process.

[0134] Based on the above data enhancement operation, we finally get a new input sequence, including the enhanced question embedding q + and student answering activities embedded in a + , the knowledge state representation h of the augmented sequence is obtained through the multi-head attention mechanism network t + For each batch, the contrast loss is calculated separately, that is, for each knowledge state h of each student i at time step t t i , construct positive and negative sample pairs for student i in the current batch. The contrast loss of student i at time step t is expressed as:

[0135]

[0136] Among them, h t +j represents each knowledge state of each student j at time step t after data augmentation, τ is the temperature hyperparameter, which is set to 0.05 in this embodiment. sim() is the cosine similarity function. In the model prediction part, the student’s current knowledge state is used to predict the answering performance in the next time step. In other words, in order to predict the student’s answer to the next new question q t+1 Whether it is correct or not, we only need to focus on the current knowledge state of the student. According to the student’s knowledge state h at time step t t Get the final prediction Here, the knowledge state h t Involving all knowledge concepts k t ∈K. Specifically expressed as:

[0137]

[0138] Among them, γi Represented as the current prediction of student i for the new question q t+1 The attention weight score is used to predict the answering performance at time step t+1, and It represents the transpose of the new question answered at time step t+1, h i，t represents the total knowledge state of student i at the time step, h j，t represents the knowledge state of student i for the jth knowledge concept at time step t, For students on question q t+1 The state of knowledge, To predict the student's answering performance in the next time step, σ represents the Sigmoid function, W A is a trainable parameter, and N is the total number of knowledge concepts contained in the knowledge concept K.

[0139] Based on this, the overall objective loss function of the model is obtained, which can be defined as a linear combination of two loss functions, specifically expressed as:

[0140]

[0141] Among them, λ is used to control the impact of contrast loss and is set to 0.1 in this embodiment. t is the cross entropy loss at each time step t, expressed as:

[0142]

[0143] Compared with the prior art, the technical solution provided in Example 2 of the present application has the following beneficial effects:

[0144] The embodiment of the present application proposes a knowledge tracking framework (MM-IEKT) based on multimodal auxiliary information extraction. First, a multimodal large language model is used to extract auxiliary information from the existing public learning interaction data set, and the prompt learning and thinking chain technology are used to extract multi-dimensional implicit information based on students, questions and knowledge concepts in the data content. The extracted implicit information includes the students' mastery of knowledge, the difficulty level of the questions and the inherent complexity of the knowledge concepts; then the Rasch model is used to obtain the representation embedding based on the students' original interaction data and auxiliary information, and finally the neural network based on the self-attention mechanism is used to obtain the student's current knowledge state, predict their future answering performance, and introduce contrastive learning technology to calculate the training loss, so as to improve the performance of the knowledge tracking model.

[0145] The model architecture of the above-mentioned knowledge tracking framework MM-IEKT is described in detail below with reference to specific embodiments.

[0146] Example 3

[0147] See also Figure 6 , is a structural diagram of a knowledge tracking model based on multimodal auxiliary information extraction provided in Example 3 of this application. Figure 6 As shown in , the above-mentioned knowledge tracking model includes an auxiliary information extraction module, an auxiliary information selection module, an auxiliary information embedding module, an attention network module and a model prediction module.

[0148] The aforementioned auxiliary information extraction and selection modules are specifically used to process the text and image content of questions in the learning interaction data collected by the intelligent online education platform using a multimodal large language model. Using the large language model's application of prompt learning and thought chaining, feasible problem-solving solutions are extracted from the data content. These solutions are then fed into the multimodal large language model, combining the student's learning interaction data and outputting specific auxiliary information. In a practical application scenario, the specific multimodal auxiliary information extraction algorithm is shown in Table 1.

[0149]

[0150] Among them, D q Denotes a dataset containing question text and images, D a represents student interaction data, D s Represents the final set of auxiliary information. Represents the large language model LLM, and the LLM prompt is represented as and Represent the thought chain prompt and auxiliary information extraction prompt respectively. The distribution of the corresponding output of LLM is expressed as g is a regularization function that prevents the parameter space from being too sparse and standardizes the extracted auxiliary information.

[0151] Assume that the student interaction data set is X = {x1, x2, x3, ..., x t}, the comprehensive extracted auxiliary information should be based on each learning interaction data x i Student stu_id=s i 、Question q_id = q i And knowledge concept k_id=k i Multi-dimensional information of the three basic elements; the extracted multi-dimensional auxiliary information includes students’ learning ability Pi, the difficulty of the problem D i and the inherent complexity of knowledge concepts C i .

[0152] The auxiliary information embedding module includes a knowledge tracking model based on auxiliary information set embedding, which is used to obtain question embedding of test questions and answering activity embedding covering knowledge concepts and student answers from students' learning interaction data.

[0153] The above-mentioned attention network module is used to determine the students' mastery of the questions and their knowledge status based on the deployed question embedding and question-answering activity embedding.

[0154] The above-mentioned model prediction module is used to perform data enhancement on the above-mentioned learning interaction data, use contrast loss to constrain the data to enhance the representation consistency of the knowledge states before and after, and predict the answering performance in the next time step based on the student's current knowledge state.

[0155] It should be noted that the detailed implementation steps of each module in Example 3 of the present application can be found in Example 1 and Example 2, and will not be repeated here.

[0156] Example 4

[0157] In Example 4 of this application, a large dataset from a real-world education platform is selected as a knowledge tracking experimental dataset, named XES3G5M, which provides high-quality question content. The dataset is collected from an online mathematics learning platform and is widely used to simulate and evaluate students' learning process. XES3G5M contains data from thousands of students and questions, as well as millions of interactions, which is very suitable for benchmarking various knowledge tracking models and effectively demonstrating the impact of question semantics. Table 2 lists the distribution of the dataset.

[0158] Table 2: Dataset distribution statistics

[0159]

[0160] Auxiliary information extraction is implemented on the public dataset XES3G5M. Then, combined with the constructed model architecture, comparative experiments are carried out with baseline models such as DKT, DKT+, DKVMN, SAKT and SAINT to verify the effectiveness of the proposed multimodal auxiliary information extraction module and self-attention neural network encoding knowledge tracking module.

[0161] The knowledge tracing task can be viewed as a binary classification problem, i.e. predicting the correctness of the student's answer to the question (correct or incorrect). Therefore, referring to most previous existing works, the accuracy (ACC) and the area under the receiver operating characteristic (ROC) curve (AUC) are selected to evaluate the prediction performance of the model.

[0162] All experiments were conducted using PyTorch and an A40 GPU with 48GB of memory. Multiple baseline models were trained using the PyTorch integrated code library, with the same training, validation, and test sets. For this model architecture, the initial random seed was fixed at 42, the batch size varied depending on the model, and the dropout rate was set to 0.2. Adam was used for optimization, with a learning rate of 1e-3. The hyperparameters τ and λ were set to 0.05 and 0.1, respectively.

[0163] The overall comparative experimental results are listed in Table 3. The results reveal several key points: First, our MMIEKT model outperforms all other models in both metrics on the public dataset XES3G5M. This shows that extracting multi-dimensional information from question text and image data through a large language model and incorporating it into the knowledge tracking model is valuable and meaningful.

[0164] Table 3: Experimental results on the dataset XES3G5M

[0165]

[0166] Finally, we conducted ablation experiments to verify the effectiveness of the auxiliary information extraction module and the auxiliary information embedding module, as well as the necessity of introducing contrastive learning. Taking the dataset XES3G5M as an example, the two modules corresponding to the MMIEKT model were deleted, denoted as MMIEKT (w / o Extract) and MMIEKT (w / o Embed), and a comparison model without the contrast loss function was added, denoted as MMIEKT (w / o CL). According to the results of the ablation experiment in Table 4, it can be clearly seen that, first of all, after removing the two modules, the performance of the model is significantly reduced. More specifically, when the auxiliary information extraction module is removed, the performance of the model is more significantly reduced, which shows that in the framework proposed in this application, the auxiliary information extraction module is more important. In addition, the effect of the model without the contrast loss function is more significantly reduced than that of the model with the contrast loss introduced, which proves the necessity of introducing contrastive learning technology.

[0167] Table 4: Ablation experiment results

[0168]

[0169] This application focuses on improving the versatility and reliability of intelligent online education platforms in the context of digital transformation of education. It adopts the basic framework of multimodal information extraction to complete the knowledge tracking task. By introducing technologies such as large language models in the knowledge tracking task, it effectively captures the student learning status based on the student learning data, and at the same time makes up for the shortcomings of existing intelligent online education platforms in educational resource utilization and personalized services.

[0170] Although the embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A knowledge tracking method based on multimodal auxiliary information extraction, characterized in that: include: Obtaining student learning interaction data, test question data including graphic and text information of the test questions, and the step-by-step solution process corresponding to each test question, and performing auxiliary feature extraction to obtain an auxiliary information set; the auxiliary information set includes the student's learning ability, the difficulty of the question, and the inherent complexity of the knowledge concept; Constructing a knowledge tracking model based on the embedding of the auxiliary information set, obtaining the question embedding of the test questions and the embedding of the answering activities covering the knowledge concepts and the students' answers from the students' learning interaction data; Establishing an attention network model to determine the student's mastery of the question and knowledge status based on the question embedding and the question-answering activity embedding; Data enhancement is performed on the learning interaction data, and the representation consistency of the knowledge states before and after the data enhancement is constrained by contrast loss. The answering performance in the next time step is predicted based on the student's current knowledge state.

2. The knowledge tracking method based on multimodal auxiliary information extraction according to claim 1 is characterized in that: The learning interaction data consists of students' answer records, including basic information about the test questions, the knowledge concepts involved in the test questions, and the content of the students' answers.

3. The knowledge tracking method based on multimodal auxiliary information extraction according to claim 1 is characterized in that: Get step-by-step solutions for each question, including: The test data containing graphic and text information of the test questions are divided into subcategories, and the step-by-step solution process of the test questions is obtained based on the thought chain prompt method and the multimodal large language model.

4. The knowledge tracking method based on multimodal auxiliary information extraction according to claim 1 is characterized in that: The question embedding of the test questions is obtained from the students' learning interaction data and is expressed as: Where q t represents the question embedding answered by the student at time t, is the inherent complexity of the knowledge concept corresponding to the question answered by the student at time t, and K represents the knowledge concept k corresponding to the question answered by the student at time t t Embedded into a vector, Represents the knowledge concept k covered t The problem changes, μ is the difficulty scalar parameter, which represents the difficulty deviation of a specific problem and its knowledge concept, P t represents the ability requirement corresponding to the question answered by the student at time t; The question-answering activity embedding covering knowledge concepts and student responses is expressed as: Where, G represents the answer r to the student's question at time t t Embedded into a vector, Represents the knowledge concept k covered t Changes in students' learning activities.

5. The knowledge tracking method based on multimodal auxiliary information extraction according to claim 1 is characterized in that: The attention network model includes two self-attention encoder attention network models, which are used to estimate the student's mastery of the problem and the student's knowledge status respectively; Embed q with questions t and answer questions embedded in a t As the input of the attention network model, we get the student’s question embedding q at each time step t t The mastery level m t , expressed as: Among them, q τ is the question answered at the current time step τ, is the transposed representation of the question answered at the total time step t, α t，τ represents the time effect attention weight score of the question answered at the current time step τ relative to the total time step t, θ represents the parameter controlling the intensity of the time effect, and d(t-τ) represents the time distance function; d k represents the dimension of the key vector used to scale the dot product; d(tl) represents the temporal distance from the lth time step to the total time step t; Determine each knowledge concept k for time step t t The knowledge state h distributed in each head t , expressed as: Where, β t，z Represents the time effect attention weight score of the knowledge concept corresponding to the question answered at the current time step τ relative to the total time step t, is the transposed representation of the knowledge concept corresponding to the question answered in the total time step t, m τ is the student’s mastery of the question at the current time step τ.

6. The knowledge tracking method based on multimodal auxiliary information extraction according to claim 1, characterized in that: Performing data enhancement on the learning interaction data, including: exchanging positions of adjacent interaction records in the learning interaction data; Deleting at least one interaction record in the learning interaction data; The correct or incorrect label of the answer of at least one interaction record in the learning interaction data is flipped.

7. The knowledge tracking method based on multimodal auxiliary information extraction according to claim 1 is characterized in that: After performing data enhancement on the learning interaction data, the method further includes: Get the enhanced question embedding q + and student answering activities embedded in a + , the knowledge state representation h of the augmented sequence is obtained through the multi-head attention mechanism network t + ; For each knowledge state h of each student i at time step t t i , construct positive and negative sample pairs for student i in the current batch, and use contrast loss constraint data to enhance the representation consistency of the previous and next knowledge states; Among them, the contrast loss of student i at time step t is expressed as: Among them, h t +j represents each knowledge state of each student j at time step t after data augmentation, δ is the temperature hyperparameter, and sim() is the cosine similarity function.

8. The knowledge tracking method based on multimodal auxiliary information extraction according to claim 1 is characterized in that: The prediction of the answering performance at the next time step based on the student’s current knowledge state is expressed as: Among them, γ i Represented as the current prediction of student i for the new question q t+1 The attention weight score is used to predict the answering performance at time step t+1, and It represents the transpose of the new question answered at time step t+1, h i，t represents the total knowledge state of student i at the time step, h j，t represents the knowledge state of student i for the jth knowledge concept at time step t, For students on question q t+1 The state of knowledge, To predict the student's answering performance in the next time step, σ represents the Sigmoid function, W A is a trainable parameter, and N is the total number of knowledge concepts contained in the knowledge concept K.

9. A knowledge tracking system based on multimodal auxiliary information extraction, characterized in that: Includes auxiliary information extraction module, auxiliary information embedding module, attention network module and model prediction module; The auxiliary information extraction module is used to obtain students' learning interaction data, test question data including graphic information of the test questions, and the step-by-step solution process corresponding to each test question, and perform auxiliary feature extraction to obtain an auxiliary information set; the auxiliary information set includes students' learning ability, the difficulty of the questions, and the inherent complexity of the knowledge concepts; The auxiliary information embedding module includes a knowledge tracking model based on the embedding of the auxiliary information set, which is used to obtain the question embedding of the test questions and the answering activity embedding covering the knowledge concepts and the students' answers from the students' learning interaction data; The attention network module is used to determine the student's mastery of the question and knowledge status based on the question embedding and the question-answering activity embedding; The model prediction module is used to perform data enhancement on the learning interaction data, use contrast loss to constrain the representation consistency of the knowledge states before and after data enhancement, and predict the answering performance in the next time step based on the student's current knowledge state.

Images