Multi-modal large model knowledge tracking method for new problem scene

By using the MKT-LLM method and employing perceptual coordination and cognitive synthesis modules, zero-sample knowledge tracking of unseen problems is achieved, solving the problem of dependence on predefined problem IDs in existing technologies and improving the accuracy and generalization ability of knowledge state prediction.

CN121882230APending Publication Date: 2026-04-17EAST CHINA NORMAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610008035.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-01-06
Publication Date
2026-04-17

AI Technical Summary

Technical Problem

Existing knowledge tracing models rely heavily on predefined question IDs, making it impossible to generalize to unseen questions in an open world. This hinders zero-shot knowledge tracing for unseen questions and accurate prediction of students' knowledge status.

Method used

The method employs a multimodal large model knowledge tracing approach (MKT-LLM), which achieves modal-level fusion and alignment through a perceptual coordination module, processes problem images using a CLIP visual encoder, and combines this with a cognitive synthesis module for problem-level fusion and alignment. It dynamically integrates multimodal cues with learner interaction history to achieve zero-shot transfer.

Benefits of technology

It enables zero-shot knowledge tracking in open-ended and real-world educational content, and can understand and predict unseen problems, improving the accuracy and generalization ability of knowledge state prediction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121882230A_ABST
    Figure CN121882230A_ABST
Patent Text Reader

Abstract

The invention discloses a new problem scene-oriented multi-modal large model knowledge tracking method, which is characterized in that modal level fusion is realized by adopting a perception coordination module, vision, text and context input are aligned, and rich problem characterization is constructed; problem-level fusion is realized through a cognitive integration module, multi-modal clues and learner interaction history are dynamically integrated, and a potential knowledge state is deduced under the condition of not depending on a question ID; and through an adaptive generalization module, representation is dynamically generated for unseen problems based on semantic similarity, and zero sample migration is realized. Compared with the prior art, the method has the advantages that zero sample knowledge without problems is tracked, so that the MKT-LLM can truly realize zero sample generalization to open and real world education content, and the limitation that an existing knowledge tracking method cannot be generalized to a new problem is effectively solved; the method can be widely applied to an intelligent education system, a self-adaptive learning platform and an interactive tutoring system, and provides technical support for personalized education in an open learning environment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of personalized education and artificial intelligence technology, specifically a multimodal large model knowledge tracking method for new problem scenarios. Background Technology

[0002] Knowledge tracing (KT) is a fundamental task in intelligent education systems. Its goal is to model learners' evolving knowledge status over time based on their historical interactions with learning materials. By accurately estimating which concepts students have mastered and which remain challenging, knowledge tracing enables personalized learning recommendations, adaptive assessments, and timely interventions. The mainstream paradigm in knowledge tracing employs sequence models to capture the temporal dynamics of student performance. Deep Knowledge Tracing (DKT) pioneered the use of LSTM to model question-answer sequences as a prediction task. Dynamic Key-Value Memory Networks (DKVMN) introduce different key-value memory networks to explicitly track concept mastery. Subsequent advancements include: Attentional Knowledge Tracing (AKT) applying Transformers to capture long-range dependencies; temporal cross-effects models incorporating Hawkes processes to model knowledge decay; and Learning Process Consistency Knowledge Tracing (LPKT) decomposing student behavior into learning and forgetting processes, aligning model dynamics with cognitive theory. Furthermore, graph neural networks are also used in knowledge tracing to model relationships between questions or knowledge concepts, and large language models such as BERT and RoBERTa have been used to enhance semantic representations.

[0003] Despite these advances, most existing knowledge tracing models heavily rely on predefined question IDs to learn ID-specific embeddings from a fixed set. This ID-dependent paradigm works in closed environments (such as standardized tests) but fails in open-world settings—where learners face new questions without IDs or historical data, a common scenario in open learning or interactive tutoring (such as scenarios with diagrams or images). The challenge of knowledge tracing that does not rely on question identifiers lies in the need to go beyond memory embeddings to understand the semantic meaning of the question. Previous cold-start approaches, such as introducing cone attention, using LLM to generate concept-based sequences, and leveraging LLM to estimate properties, still rely on predefined IDs or auxiliary metadata and mostly operate in a plain text space. These methods address a symptom, not the root cause: over-reliance on symbolic identifiers. In the real world, new questions often appear in multimodal forms (such as diagrams with hints) without IDs or labels. To achieve true generalization, the model must understand the question like the learner, constructing meaning directly from the raw multimodal input.

[0004] In summary, existing knowledge tracking models rely on predefined question IDs and cannot generalize to unseen questions, thus failing to achieve zero-sample knowledge tracking of unseen questions and accurate prediction of students' knowledge status. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a multimodal large-scale model knowledge tracing method for novel problem scenarios. It employs a "fusion-alignment-adaptation" approach to integrate multimodal information with a large language model, achieving zero-shot knowledge tracing for unseen problems and accurate prediction of student knowledge status. This method moves from ID-based referencing to perception-driven reasoning. MKT-LLM does not rely on any form of pre-registered item identity or structured metadata, but instead directly builds knowledge tracing on the multimodal content of the problem. Through a perception coordination module, a unified representation is constructed from the original input (text, image, ID) using modal-level fusion and alignment. Through a cognitive synthesis module, these representations are dynamically integrated with the learner's history. Knowledge status is inferred through problem-level fusion and alignment, enabling MKT-LLM to truly achieve zero-shot generalization to open-ended and real-world educational content. This effectively solves the limitation of existing knowledge tracing methods in generalizing to new problems, achieving zero-shot knowledge tracing for unseen problems. This invention can be applied to intelligent education systems, adaptive learning platforms, and interactive tutoring systems, providing technical support for personalized education in open learning environments, and has good application prospects and commercial development value.

[0006] The specific technical solution to achieve the purpose of this invention is: a multimodal large model knowledge tracking method for new problem scenarios. Its features include using a large language model (LLaMA2-7B) combined with LoRA fine-tuning, using the CLIP visual encoder to process problem images, achieving modal-level fusion through a perceptual coordination module to align visual, textual, and contextual inputs, constructing rich problem representations, and then achieving problem-level fusion through a cognitive synthesis module to dynamically integrate multimodal cues and learner interaction history, inferring potential knowledge states without relying on problem IDs, and dynamically generating representations for unseen problems based on semantic similarity through an adaptive generalization mechanism to achieve zero-shot transfer.

[0007] The multimodal large model knowledge tracing method for new problem scenarios specifically includes the following steps:

[0008] Step 1: Perceptual Harmonization

[0009] The perceptual coordination module is a core component of the MKT-LLM framework. Its purpose is to construct problem representations directly from multimodal content without relying on problem IDs. This module learns to generate coherent representations that can generalize to unseen problems by using IDs and image embeddings during training through modal-level fusion and alignment.

[0010] The modal-level fusion: For each problem q, obtain the image embedding from the visual encoder. Obtain the ID embedding from the embedding layer A multi-head cross-attention mechanism is employed to enable ID embedding to focus on relevant parts of the image embedding: Adaptively combine different modalities using a gating fusion mechanism: , The fused features are then combined and normalized: .

[0011] Modal-level alignment: To align the semantics between different modalities, a contrastive InfoNCE loss is used for optimization.

[0012] .

[0013] Where τ is the temperature parameter. This loss function encourages consistency in the representation of different modalities of the same problem.

[0014] Step 2: Cognitive Synthesis Module Processing

[0015] The cognitive synthesis module performs advanced reasoning based on the multimodal representations generated by the perceptual coordination module. It models the knowledge evolution process by integrating temporal dynamics and cross-problem semantic similarity. This module generalizes to new problems by integrating historical interactions and recognizing analogical patterns, specifically including:

[0016] 1) Problem-level fusion: Given in-problem representations from t historical interactions We apply a causal self-attention mechanism to model sequence dependencies, ensuring that each step only focuses on past and current inputs: The output is processed by a feedforward network and then normalized using residual connections and layer normalization. , obtained Context-aware encoding that includes students’ learning trajectories.

[0017] 2) Problem-level alignment: To improve generalization ability, representations of semantically similar problems are aligned in the latent space. For the fusion embedding of the i-th problem, Let be the similarity between the i-th and j-th questions.

[0018] Define normalized weights The problem-level alignment loss is: This loss encourages the invariance of representations for function-related problems and supports analogical reasoning.

[0019] Step 3: Adaptive Generalization Module

[0020] The adaptive generalization module completes the "fusion-alignment-adaptation" paradigm, achieving zero-shot transfer by dynamically generating representations of unseen problems. Unlike using randomly initialized embeddings, this module utilizes semantic similarity to known problems to synthesize representations of new problems, specifically including:

[0021] 1) Similarity-based adaptation: For unseen questions q, extract their multimodal features. The similarity to all training problems i is calculated using inverse Euclidean distance: ,in To ensure numerical stability, select the top-k most similar training problems to form a neighborhood. ,set up For problem i, the ID representation is the adapted representation. Weighted average of similarity: ,in This ensures that new questions are embedded in a semantically coherent space and can be meaningfully integrated into the student's interaction history.

[0022] Step 4: Model Training and Inference

[0023] 1) Model training phase: fusing the historical interaction vectors Embedded with the corresponding answer Add them together and input them into a large language model to obtain a trajectory representation. and its fusion vector with the target problem The final vector z is computed through multiplicative combination. An optimized composite objective function is used to support fusion, alignment, and adaptation. The main task loss is the binary cross-entropy of the predicted logits.

[0024] .

[0025] in, For the sigmoid function, This is a real label.

[0026] Two auxiliary alignment losses support adaptation: To align the ID and the InfoNCE loss of the image embedding, To align the weighted MSE loss of the fusion representation of similar problems, its total loss combines all components and is expressed as follows:

[0027] .

[0028] in, , Balance the contributions of each part.

[0029] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:

[0030] 1) It formalizes the multimodal knowledge tracing problem in new problem scenarios, and pushes knowledge tracing beyond the closed predefined itemset when no problem ID is available, thus achieving open-world applicability.

[0031] 2) Two principle modules, perceptual coordination and cognitive synthesis, are proposed to enable the large language model to understand new problems through multimodal integration and to dynamically track knowledge through integrated perception and prior experience.

[0032] 3) Numerous experiments have shown that MKT-LLM achieves superior performance in zero-shot and few-shot settings, establishing a new direction for adaptive and generalizable educational AI systems. Attached Figure Description

[0033] Figure 1 This is an overall framework diagram of the MKT-LLM of the present invention. Detailed Implementation

[0034] This invention proposes a paradigm shift from ID-based referencing to perception-driven reasoning. MKT-LLM does not rely on any form of pre-registered item identity or structured metadata. Instead, it directly builds knowledge tracking on the multimodal content of questions. Through a perception coordination module, it constructs a unified representation from raw inputs (text, images, IDs) using modal-level fusion and alignment. A cognitive synthesis module dynamically integrates these representations with the learner's history, and question-level fusion and alignment infer the knowledge state. This enables MKT-LLM to truly achieve zero-shot generalization to open-ended and real-world educational content.

[0035] See Figure 1 The MKT-LLM framework constructed in this invention demonstrates the structure and interrelationships of the Perceptual Harmonization, Cognitive Synthesis, and Adaptive Generalization modules. This invention integrates multimodal information with a large language model through a "fusion-alignment-adaptation" paradigm to achieve knowledge tracking for new problem scenarios.

[0036] The present invention will be further described in detail below through specific embodiments.

[0037] Example 1

[0038] This embodiment is based on two knowledge tracing datasets: Junyi Academy and NeurIPS 2020 Education Challenge (NIPS2020). It aims to demonstrate in detail the specific operation process and application effects of the MKT-LLM method. The specific steps are as follows:

[0039] Step 1: Sensing and Coordination Module Processing

[0040] 1) Modal-level fusion: CLIP-ViT-B / 32 is selected as the visual encoder. For each question q, the question image is input into the visual encoder to obtain the image embedding. Simultaneously, obtain the ID embedding from the embedding layer. (The ID representation of the new problem is generated through the adaptive generalization mechanism in step 3); a multi-head cross-attention mechanism (setting 8 attention heads) is adopted to embed the ID into the relevant parts of the image embedding; a gated fusion mechanism is used to adaptively combine the two modalities through a learnable gate parameter g; finally, the in-problem representation is obtained through layer normalization. .

[0041] 2) Modal alignment: Contrastive InfoNCE loss is used to align ID and image embeddings, with a temperature parameter τ=0.07 and a batch size N=32. By minimizing this loss, ID and image embeddings for the same problem are brought closer together in the vector space, while embeddings for different problems are kept further apart, thereby learning cross-modal semantic consistency.

[0042] Step 2: Cognitive Integration Module Processing

[0043] 1) Problem-level fusion: Assuming a student has t=5 historical interaction records, represent these interactions within the question level. (Including the problem to be predicted) Input is a causal self-attention layer. A causal mask ensures that the i-th position can only focus on information from position 1 to i, simulating the accumulation of knowledge during actual learning. The self-attention output passes through two layers of feedforward network (with a hidden dimension of 4096), and is then processed using residual connections and layer normalization to obtain a fused representation. .

[0044] 2) Question-level alignment: Calculate the semantic similarity of all question pairs within a batch. For semantically similar problems (such as different problems examining the same knowledge points), the weighted MSE loss is used to bring their fusion representations closer together, enabling the model to learn invariant representations of functionally equivalent problems and support analogical reasoning on new problems.

[0045] Step 3: Adaptive Generalization Mechanism

[0046] For unseen problems encountered during testing (such as problems specific to the NIPS2020 dataset test set), perform the following: Extract image features of the new problem. The inverse Euclidean distance between the new problem and the image features of all training problems is calculated as the similarity. The top-k (k=5) most similar training problems are selected, and the weighted average of their ID embedding similarity is used as the ID embedding of the new problem. This method ensures that the representation of the new problem is consistent with the semantically relevant known problems and avoids the performance degradation caused by random initialization.

[0047] Step 4: Model Training and Inference

[0048] 1) Model Training: MKT-LLM is built on LLaMA2-7B and uses LoRA for efficient parameter fine-tuning (rank=32, alpha=32, dropout=0.1). The dataset is divided into training and test sets in an 8:2 ratio. Training employs an early stopping strategy (based on validation set AUC) for a maximum of 10 epochs. The Adam optimizer is used (learning rate lr=3×10). -4 Weight decay = 1 × 10 -5 This is used in conjunction with a cosine learning rate scheduler. In the total loss function, λ1=0.1 and λ2=0.1 are set to balance the main task loss and the auxiliary alignment loss.

[0049] 2) Model Reasoning: The students' historical interaction sequences are processed through the perceptual coordination and cognitive integration modules to obtain trajectory representations. The fusion vector of the problem to be predicted (which may be a new problem, with its representation obtained through an adaptive generalization mechanism) and Combined with the input classification header, the predicted probability is obtained. If the probability is greater than 0.5, the student's answer is predicted to be correct; otherwise, the prediction is incorrect.

[0050] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements made without departing from the principle of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A multimodal large-scale model knowledge tracing method for new problem scenarios, characterized in that, This method employs a perceptual coordination module, a cognitive synthesis module, and an adaptive generalization module to integrate multimodal information from a large language model through a "fusion-alignment-adaptation" process, enabling zero-shot knowledge tracking of unseen problems. The specific steps include: Step 1: Fusion and Alignment of the Perception Coordination Module The perception coordination module performs modal-level fusion and alignment on the large language model to obtain multimodal representations. The large language model uses ID and image embedding during training to enable the large language model to learn to generate coherent representations that can generalize to unseen problems. The modal-level fusion specifically includes: 1) For each problem q, obtain the image embedding from the visual encoder. and obtain the ID embedding from the embedding layer. ; 2) Employ a multi-head cross-attention mechanism to enable ID embedding. Focus on image embedding The relevant parts are obtained by using the following vector representation of the features after multi-head cross-fusion. : ; in, Image embedding, Embedded for ID; 3) Using a gating fusion mechanism to adaptively combine different modalities, the following formula represents the fused features: , ; in, Image embedding, For ID embedding, g is the weight parameter; The fused features, after combination and normalization, yield a multimodal representation expressed by the following equation: ; in, This is the vector representation obtained in step 2); This is the vector representation obtained in step 3); The modal-level alignment uses contrastive InfoNCE loss to perform image embedding. and ID embedding Perform optimization The resulting loss is expressed by the following function: ; Where τ is the temperature parameter; Step 2: Reasoning in the Cognitive Synthesis Module The cognitive synthesis module performs reasoning based on the multimodal representations generated by the perception coordination module. By integrating temporal dynamics and cross-problem semantic similarity, it constructs a knowledge evolution process, enabling the cognitive synthesis module to generalize to new problems by integrating historical interactions and recognizing analogy patterns. The reasoning in the cognitive synthesis module specifically includes: 1) Problem-level fusion: Given in-problem representations from t historical interactions We apply a causal self-attention mechanism to construct sequence dependencies, so that each step only focuses on the past and current inputs represented by the following formula. : ; in, The feature fusion representation obtained by the perception coordination module; Its output is processed by a feedforward network and then subjected to residual connections and layer normalization to obtain a context-aware code containing the student's learning trajectory, represented by the following formula. : ; in, This is the vector representation after problem-level fusion in step 1); 2) Problem-level alignment: Let For the fusion embedding of the i-th problem, Let the similarity between the i-th and j-th questions be defined; define the normalized weights. The loss caused by problem-level alignment is expressed by the following function: ; in, , These are the fusion embeddings of the i-th and j-th questions, respectively; Step 3: Zero-shot transfer of the adaptive generalization module The adaptive generalization module achieves zero-sample transfer by dynamically generating representations of unseen problems and synthesizing representations of new problems using semantic similarity to known problems. Specifically, it includes: 1) Similarity-based adaptation: For unseen questions q, extract their multimodal features. Use the following formula The inverse Euclidean distance is used to calculate the similarity with all training problems i: ; in, Ensure numerical stability. , These are the multimodal features of questions q and i, respectively; Select the top-k most similar training problems to form a neighborhood. ,set up For problem i, the ID representation is the adapted representation. The weighted average of similarity is expressed by the following formula: ; in, Let be the similarity between problem q and problem j; Step 4: Model Training and Inference 1) During the training phase, the fused vectors of historical interactions are... Embedded with the corresponding answer Add them together and input them into a large language model to obtain a trajectory representation. ; 2) The fusion vector of the target problem and The final vector z is calculated through multiplication and combination; 3) Optimize the composite objective function to support fusion, alignment, and adaptation: The main task loss is the binary cross-entropy of predicting logits: ; in, For the sigmoid function; This is a real label; Two auxiliary alignment losses support adaptation: Let To align the ID and the InfoNCE loss of the image embedding, The weighted MSE loss for aligning similar problems and fusing representations is expressed by the following formula: ; in, , To balance the contributions of each part; To align the ID and the InfoNCE loss of the image embedding, A weighted MSE loss is used to fuse representations of similar problems.