An intelligent teaching interaction method based on a multi-modal knowledge graph

Abstract

The application belongs to the field of education, and specifically relates to an intelligent teaching interaction method based on a multi-modal knowledge graph. The method constructs a multi-modal knowledge graph infrastructure oriented to the field of education, which includes an entity layer, a relationship layer, a modal layer and a value layer. The original teaching resources are subjected to multi-modal analysis and semantic alignment to generate standardized knowledge units. Based on a preset education ontology model, the knowledge units are uniformly coded and associated modeled to form a multi-modal knowledge graph with logical coherence and value explicitness. According to the real-time interactive behavior and historical learning track of students, the relevant sub-graph structures in the graph are dynamically activated, and adaptive teaching responses are generated in combination with a context perception module. Through a multi-channel feedback mechanism, an interactive closed loop is completed, and the student cognitive state vector, emotion state vector and graph association weight are continuously updated. The method ensures that the teaching content is high-fidelity, high-coherence and value-oriented controllable.

Description

Technical Field

[0001] This invention belongs to the field of education, and specifically relates to an intelligent teaching interaction method based on multimodal knowledge graphs. Background Technology

[0002] Against the backdrop of the current in-depth advancement of educational informatization, education, as a crucial link in fulfilling the fundamental task of cultivating morality and character, is undergoing a systematic transformation from traditional lecture-based teaching to digital and intelligent teaching methods. Education not only carries the core function of value guidance but also needs to achieve precise content delivery, interactive teaching processes, and quantifiable learning outcomes in the complex and ever-changing social context of the new era. In this process, building an intelligent interactive system that supports high-quality teaching based on information technology has become an important path to improve the effectiveness of education. Especially under the policy drive of the national strategy of vigorously promoting digital education, how to effectively integrate diverse teaching resources through technological means, dynamically adapt to students' cognitive characteristics, and achieve the organic unity of knowledge transmission and value guidance constitutes the core proposition of educational informatization development.

[0003] Currently, mainstream educational informatization solutions primarily rely on structured course databases and rule-based teaching management systems. These systems typically use textbook chapters as basic units, linearly organizing resources such as text lesson plans, video lectures, and question banks, supplemented by simple user behavior logs and standardized testing and evaluation mechanisms. Their design logic aims to ensure the standardization of teaching content and the consistency of its dissemination through pre-defined knowledge paths and fixed interaction flows. In the early stages of informatization construction, such solutions did indeed play a positive role in expanding the coverage of high-quality resources and improving the efficiency of teaching management. Specifically, their technical architecture often uses relational databases to store knowledge point tags, combined with a front-end interface for resource retrieval and playback, while the back-end completes the learning feedback loop through pre-defined answer logic. This model, centered on "content distribution + result verification," ensures standardized teaching while also initially achieving partial traceability of the teaching process.

[0004] However, as education places higher demands on personalized guidance, contextualized experiences, and the cultivation of higher-order thinking, the inherent structural limitations of the aforementioned technological solutions at the principle level are becoming increasingly apparent. Fundamentally, this stems from the fact that their knowledge representation methods and interaction logic are based on a single modality and static associations, making it difficult to address the multidimensionality of knowledge elements, the contextual dependence of value judgments, and the dynamic evolution of students' cognitive states in teaching. Furthermore, existing systems generally treat teaching materials such as text, images, and audio-visual content as isolated, independent resources, relying solely on manually annotated keywords or course catalogs for coarse-grained associations. This results in the inability to effectively model the deep semantic connections, historical evolution, and value connotation mapping between knowledge. Consequently, when students raise complex questions across chapters and media during the learning process, the system often only returns isolated fragments, failing to construct a comprehensive response with logical coherence and value orientation. Based on this, teaching interaction is forced to degenerate into one-way information delivery or mechanical question-and-answer matching, failing to capture students' emotional inclinations in understanding specific historical events, and also struggling to dynamically reorganize multimodal teaching materials based on their cognitive blind spots to form targeted guidance. This technological paradigm of "emphasizing resource accumulation while neglecting semantic integration" actually creates a profound contradiction between the richness of teaching content and the superficiality of teaching interaction—the more diverse the resources, the more difficult it is for the system to achieve organic integration; the more frequent the interaction, the more fragmented and decontextualized the feedback becomes.

[0005] Crucially, the essence of education is not merely the transmission of knowledge, but the internalization and construction of values. This process highly relies on a multi-dimensional, collaborative interpretation of historical context, contemporary relevance, and theoretical logic. Current technologies cannot automatically extract supporting arguments from multimodal evidence chains and generate persuasive interactive responses. This lack of capability not only weakens the depth and impact of teaching but may also lead to cognitive biases due to one-sided or disconnected responses. Therefore, constructing a new method that deeply integrates heterogeneous teaching resources from multiple sources, such as text, images, and audio / video, achieves dynamic association of knowledge elements and explicit expression of value logic based on a unified semantic framework, and supports high-fidelity, highly adaptive teaching interactions, has become key to breaking through current bottlenecks in educational informatization and achieving a leap from "resource digitization" to "intelligent education." Therefore, designing an intelligent teaching interaction method that aligns with the specific laws of education and is supported by multimodal knowledge graphs, without relying on general artificial intelligence paradigms, while balancing knowledge systematization, value orientation, and interactive adaptability, has become a key challenge and an urgent technical problem for those skilled in the art. Summary of the Invention

[0006] This invention provides an intelligent teaching interaction method based on multimodal knowledge graphs, aiming to resolve the structural contradiction between the richness of teaching content and the depth of education caused by the singularity of knowledge representation, fragmented resources, and superficial interaction in existing educational informatization systems. To achieve the above-mentioned objective, this invention constructs an intelligent teaching interaction system with multimodal knowledge graphs as the core support, integrating semantic modeling, dynamic reasoning, and context adaptation mechanisms. This system, without relying on general artificial intelligence paradigms, strictly adheres to the value orientation, systematic nature of knowledge, and cognitive adaptability requirements of education. By structurally integrating heterogeneous teaching resources such as text, images, audio, and video, it achieves high-fidelity, high-coherence, and highly targeted teaching interaction.

[0007] An intelligent teaching interaction method based on multimodal knowledge graphs includes the following steps:

[0008] S1. Construct a multimodal knowledge graph infrastructure for the education field, which includes an entity layer, a relation layer, a modal layer, and a value layer.

[0009] S2. Perform multimodal analysis and semantic alignment on the original teaching resources to generate standardized knowledge units;

[0010] S3. Based on the pre-set educational ontology model, knowledge units are uniformly encoded and associated with each other to form a multimodal knowledge graph with logical coherence and explicit value.

[0011] S4. Based on students' real-time interactive behavior and historical learning trajectory, dynamically activate relevant subgraph structures in the graph and generate adaptive teaching responses in conjunction with the context awareness module.

[0012] S5. The interaction loop is completed through a multi-channel feedback mechanism, and the student's cognitive state vector, emotional state vector and graph association weights are continuously updated.

[0013] Preferably, in step S1, the entity layer is used to represent core concepts, historical events, policy documents, characters and social cases in teaching, and each entity is given a globally unique identifier.

[0014] The relation layer is used to define the semantic associations between entities, including causal relationships, temporal relationships, unity of opposites relationships, value mapping relationships, and theoretical evolution relationships. Each type of relationship is configured with corresponding logical axioms and constraint rules.

[0015] The modal layer is used to bind multimodal teaching materials associated with entities or relationships, including text paragraphs, static images, dynamic videos, audio explanations, and interactive cases. Each material is generated into a structured description vector by a content feature extractor and linked to the corresponding graph node through a modal binding index.

[0016] The value layer is used to explicitly label the core value dimensions carried by each entity and relationship, including national identity, social responsibility, historical perspective, rule of law awareness, and moral judgment.

[0017] Preferably, in step S2, the multimodal parsing and semantic alignment process includes: performing syntactic dependency analysis and keyword extraction on text resources to identify the core concepts and argument logic involved; performing object detection and scene understanding on image resources, using a pre-trained multimodal visual model based on the Transformer architecture to identify and extract historical symbols, character identities, and spatial context information contained in the image; performing speech transcription and temporal segmentation on audio and video resources, simultaneously extracting speech content, background music emotional features, and keyframe semantics of the image; and mapping the parsing results of all modalities to a unified semantic space through a cross-modal alignment engine, and determining semantic consistency based on a cosine similarity threshold to generate cross-modal aligned knowledge units.

[0018] Preferably, in step S3, the unified coding and association modeling process includes: importing aligned knowledge units into the educational ontology model; assigning an entity category to each knowledge unit according to the entity type labeling system in the ontology model; automatically deriving the semantic relationships that should exist between knowledge units according to the logical rules in the relation axiom set; labeling the value transmission direction and value transmission intensity of each relation path according to the value constraint rules; and persistently storing all knowledge units and their association relationships in a graph database.

[0019] Preferably, in step S4, the context awareness module includes a central collaborative controller, a physiological state analysis engine, a cognitive state tracking module, and a multimodal response generator; the central collaborative controller receives student questions or operation instructions and extracts a set of keywords; the cognitive state tracking module constructs a cognitive state vector containing knowledge mastery, value identification tendency, and thinking development stage based on the student's historical answer records, dwell time, click path, and interaction frequency; the physiological state analysis engine acquires facial expression, eye movement trajectory, or heart rate variability data through standard multimedia acquisition devices or optional wearable devices, and converts it into emotional arousal and attention concentration indicators; the central collaborative controller integrates the above information to determine the representation of the current teaching context, and retrieves the optimal subgraph structure that satisfies semantic relevance, cognitive matching degree, and emotional adaptability in the multimodal knowledge graph.

[0020] More preferably, the multimodal response generator generates a teaching response based on the activated subgraph structure, including: extracting the main logical chain from the subgraph, which connects the starting entity to the target entity through several relational edges, with each edge having a value-oriented label; selecting text, image, and video material clips in priority order according to the modal binding information in the logical chain, prioritizing materials that can simultaneously cover multiple entities and whose emotional tone is consistent with the student's current emotional state; performing temporal arrangement and semantic connection processing on the selected materials to ensure narrative logic coherence and value expression consistency; and encapsulating the arranged multimodal content into a structured response package and presenting it through the front-end interface.

[0021] Preferably, in step S5, the multi-channel feedback mechanism includes two subsystems: explicit feedback acquisition and implicit feedback analysis. The explicit feedback acquisition subsystem records students' actions such as liking, asking follow-up questions, skipping, or repeating playback in response to the teaching response. The implicit feedback analysis subsystem monitors the distribution of students' gaze focus, the content of their voice response, the operation delay time, and the relevance of subsequent questions during the response presentation. Both types of feedback data are transmitted back to the central collaborative controller, the state vector, and the emotional state vector in the physiological state analysis engine, and the access weights and relationship confidence of relevant nodes in the graph are adjusted.

[0022] Preferably, the multimodal knowledge graph adopts an incremental update mechanism: when new teaching resources are imported, preliminary knowledge units are first generated; then, a nearest neighbor search is performed in the existing graph to identify potential related nodes; if the semantic similarity between the new unit and the existing nodes exceeds a preset threshold, the relational reasoning engine is triggered to automatically establish new relations based on the axiomatic rules in the ontology model; if an effective relation cannot be established, the new unit is temporarily stored in the review area, and after manual review to confirm its educational suitability and correct value orientation, it is included in the main graph.

[0023] The beneficial effects of this invention are as follows:

[0024] This invention achieves deep semantic fusion of teaching resources by constructing a four-layer multimodal knowledge graph; solves the problem of fragmented heterogeneous materials through multimodal parsing and cross-modal alignment; ensures the systematic and standardized nature of knowledge expression through unified encoding based on ontology models; realizes personalized and adaptive teaching interaction through a context-aware dynamic activation mechanism; ensures the system's continuous evolution capability through multi-channel feedback and incremental updates; and upholds the fundamental attributes of education through the explicit design of value throughout the entire process. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating an intelligent teaching interaction method based on multimodal knowledge graphs according to the present invention. Detailed Implementation

[0027] This invention provides an intelligent teaching interaction method based on a multimodal knowledge graph. Its technical implementation strictly adheres to the specific requirements of educational information systems, taking localized deployment, controllable value orientation, and systematic knowledge representation as its core principles. By constructing a four-layer multimodal knowledge graph and integrating semantic modeling, dynamic reasoning, and context adaptation mechanisms, a closed-loop, evolvable intelligent teaching interaction system is formed. The specific embodiments of this invention will be described in detail below with reference to the accompanying drawings.

[0028] like Figure 1 As shown, an intelligent teaching interaction method based on multimodal knowledge graphs includes the following steps:

[0029] S1. Construct a multimodal knowledge graph infrastructure for the education field, which includes an entity layer, a relation layer, a modal layer, and a value layer.

[0030] S2. Perform multimodal analysis and semantic alignment on the original teaching resources to generate standardized knowledge units;

[0031] S3. Based on the pre-set educational ontology model, knowledge units are uniformly encoded and associated with each other to form a multimodal knowledge graph with logical coherence and explicit value.

[0032] S4. Based on students' real-time interactive behavior and historical learning trajectory, dynamically activate relevant subgraph structures in the graph and generate adaptive teaching responses in conjunction with the context awareness module.

[0033] S5. The interaction loop is completed through a multi-channel feedback mechanism, and the student's cognitive state vector, emotional state vector and graph association weights are continuously updated.

[0034] First, this invention constructs a multimodal knowledge graph infrastructure for the education field. This architecture consists of an entity layer, a relation layer, a modal layer, and a value layer, with these four layers logically nested. For data storage, a unified graph database model is used for persistent management.

[0035] The entity layer is used to represent core concepts, historical events, policy documents, figures, and social cases in teaching. Each entity is assigned a globally unique identifier and is classified according to a pre-defined type label system. Entity types include, but are not limited to, "theoretical concepts", "historical events", "policy texts", "typical figures", and "social phenomena".

[0036] The relation layer is used to define the semantic associations between entities. Its relation types are formally defined in strict accordance with the norms of educational disciplines, including causal relations, temporal relations, relations of unity of opposites, value mapping relations, and theoretical evolution relations. Each relation type is configured with corresponding logical axioms and constraint rules to ensure the legitimacy and educational appropriateness of relational reasoning.

[0037] The modality layer is used to bind multimodal teaching materials associated with entities or relationships, including text paragraphs, static images, dynamic videos, audio explanations, and interactive cases. After each material is imported into the educational intelligent teaching interaction system based on multimodal knowledge graph, a structured description vector is generated by the content feature extractor. This structured description vector includes dimensions such as semantic theme, emotional tone, visual composition, and voice intonation, and is linked to the corresponding graph node through modality binding index.

[0038] The value layer is used to explicitly label the core value dimensions carried by each entity and relationship, including national identity, social responsibility, historical perspective, rule of law awareness, and moral judgment. Each value label corresponds one-to-one with the specific items in the core competency framework of the curriculum issued by the education authorities, and also supports the quantitative labeling of the direction and intensity of value transmission.

[0039] Secondly, this invention performs multimodal parsing and semantic alignment on the original teaching resources to generate standardized knowledge units. This process is executed by a multimodal parsing module, whose inputs are teaching resources from official textbooks, authoritative publications, policy documents, and publicly reported mainstream media, and whose output is a set of structured knowledge units.

[0040] For text resources, the multimodal parsing module performs syntactic dependency analysis and keyword extraction to identify the core concepts and argument logic involved, including subject-verb-object structure extraction, logical connector identification, and determination of the relationship between arguments and evidence.

[0041] For image resources, the multimodal parsing module performs object detection and scene understanding, using a pre-trained multimodal vision model based on the Transformer architecture to identify historical symbols, personal identities, and spatial context information contained in the image. The pre-trained multimodal vision model based on the Transformer architecture can be a multimodal visual language model based on the ViT architecture.

[0042] For audio and video resources, the system performs speech transcription and temporal segmentation, simultaneously extracting speech content, background music emotional features, and keyframe semantics from the video. Speech transcription employs a pre-trained deep learning speech recognition model to ensure accurate recognition of technical terms, while emotional feature extraction quantifies the emotional tone of the audio through spectral analysis and rhythmic variation modeling. The pre-trained speech recognition model can utilize a pre-trained acoustic-language joint model based on a sequence-to-sequence architecture.

[0043] All parsing results from all modalities are mapped to a unified semantic embedding space through a cross-modal alignment engine. This cross-modal alignment engine is based on a pre-trained cross-modal semantic embedding model, which adopts an encoder structure based on the Transformer architecture. It can project feature vectors from different modalities such as text, audio, and images to the same 512-dimensional to 768-dimensional semantic embedding space. The projected feature vectors are used to determine semantic consistency through cosine similarity calculation. When the cosine similarity exceeds a preset threshold of 0.80 to 0.85, the cross-modal alignment engine merges the parsing results from multiple modalities into a cross-modal aligned knowledge unit. This knowledge unit contains multimodal feature vectors, semantic summaries, timestamp ranges, and source metadata, thereby achieving accurate cross-modal semantic alignment and knowledge fusion.

[0044] Furthermore, based on a pre-defined educational ontology model, this invention performs unified encoding and relational modeling of knowledge units to form a multimodal knowledge graph with logical coherence and explicit value.

[0045] In a preferred embodiment of the present invention, the unified coding and association modeling process specifically includes: importing aligned knowledge units into an educational ontology model, which is formed after formal modeling and includes an entity type label system, a set of relational axioms, and value constraint rules. The entity type label system, the set of relational axioms, and the value constraint rules serve as the modeling basis for the entity layer, relation layer, and value layer in the multimodal knowledge graph, respectively; the educational intelligent teaching interaction system based on the multimodal knowledge graph assigns an entity category to each knowledge unit according to the entity type label system in the ontology model; automatically derives the semantic relationships that should exist between knowledge units according to the logical rules in the set of relational axioms; and labels the value transmission direction and value transmission intensity for each relation path according to the value constraint rules, with the value constraint rules corresponding one-to-one with the core value dimensions in the value layer. Finally, all knowledge units and their associations are persistently stored in a graph database, forming a queryable, reasonable, and scalable multimodal knowledge graph. The graph database adopts an attribute graph model, where node attributes include entity identifiers, type labels, value labels, and modality binding indexes, and edge attributes include relation type, logical axiom references, value transmission direction, and value transmission intensity.

[0046] Subsequently, based on students' real-time interactive behavior and historical learning trajectory, the present invention dynamically activates relevant subgraph structures in the graph and generates adaptive teaching responses in conjunction with the context awareness module.

[0047] The context awareness module comprises four functional components: a central collaborative controller, a physiological state analysis engine, a cognitive state tracking module, and a multimodal response generator.

[0048] The central collaborative controller receives student questions or operation instructions from the front-end interactive interface, parses their semantic intent, and extracts a set of keywords. This parsing process is based on an education-specific intent recognition model built from a pre-trained language model that has been fine-tuned for the domain. This intent recognition model is based on BERT or RoBERTa and is trained under supervision on a classroom question-and-answer corpus that includes factual knowledge, value discrimination, and case studies. It can accurately distinguish different types of interactive intents such as "factual inquiry," "value discussion," and "case analysis," and outputs the semantic vector Q of the student's question. The semantic vector Q is a pooled representation of the last hidden state of the model.

[0049] The cognitive state tracking module constructs a current cognitive state vector based on students' historical answer records, dwell time, click paths, and interaction frequency. This cognitive state vector includes three dimensions: knowledge mastery, value identification tendency, and thinking development stage. Knowledge mastery is calculated by weighting knowledge point coverage and answer accuracy. Value identification tendency is modeled by students' interaction preferences for content with different value tags. The thinking development stage is discretized and graded according to Piaget's cognitive development theory and educational goals. The above three dimensions are encoded by an embedding layer and then concatenated to form the cognitive state vector C.

[0050] The physiological state analysis engine acquires students' facial expressions, eye movements, or heart rate variability data through standard multimedia acquisition devices or optional wearable devices. After feature extraction, it is transformed into indicators of emotional arousal and attention concentration. For example, facial expressions and eye movements are captured by video streams from a camera and estimated in real time through a lightweight convolutional neural network. Heart rate variability is acquired through a standard interface connecting to a smart bracelet or watch. The above data is analyzed in the time and frequency domains to quantify the level of autonomic nervous activity. The indicators of emotional arousal and attention concentration are linearly transformed to form an emotional state vector E.

[0051] The central collaborative controller integrates the above three types of information to determine the complete representation of the current teaching context. Based on this, it retrieves the optimal subgraph structure in the multimodal knowledge graph that satisfies semantic relevance, cognitive matching degree, and emotional adaptability. The retrieval process employs a multi-objective optimization algorithm, and the utility function is as follows:

[0052]

[0053] Where Q is the semantic vector of the student's question output by the intent recognition model, C is the cognitive state vector generated by the cognitive state tracking module, E is the emotional state vector output by the physiological state parsing engine, and S is the feature representation of the candidate subgraph. All of these vectors are mapped to a unified latent space through a shared projection layer to eliminate dimensional differences. The feature representation of the candidate subgraph is generated by fusing the type labels of the entity layer, the logical axioms of the relation layer, and the value labels of the value layer within the subgraph. Sim, Match, and Adapt are the calculation functions for semantic similarity, cognitive matching, and emotional adaptation, respectively, and their output values ​​are normalized to the [0,1] interval by the Sigmoid function. α, β, and γ are weight coefficients, dynamically adjusted according to the teaching stage; for example, the weight of α is increased during the knowledge introduction stage, and the weight of β is increased during the value enhancement stage. The initial values ​​of each weight coefficient are 0.4, 0.3, and 0.3, respectively, and the adjustment step size does not exceed 0.1.

[0054] As a key technical feature of this invention, the multimodal response generator generates teaching responses according to the activated subgraph structure, following a deterministic process: First, it extracts the main logical chain from the subgraph. This logical chain connects the starting entity to the target entity via several relational edges, and each edge is labeled with a value orientation. The extraction of the logical chain uses an optimal path algorithm based on value transmission strength. The optimal path algorithm uses value transmission strength as the edge weight and employs an improved Dijkstra algorithm to solve for the maximum weighted path from the starting entity to the target entity. The value transmission strength is calculated based on the attribute weighting of the edges, ensuring the coherence and orientation of the output content in terms of value expression. For example, assuming there is a starting entity and a target entity in the subgraph, and multiple candidate paths exist between them, the algorithm calculates the sum of the value transmission strength of each path and selects the path corresponding to the maximum value as the main logical chain. Second, based on the modalities in the logical chain... The system binds information and selects text, image, and video clips in priority order, prioritizing those that cover multiple entities and whose emotional tone aligns with the student's current emotional state. For example, when students' emotional arousal and attention span are low, the multimodal response generator prioritizes video clips with motivating background music and positive color tones. Next, the selected materials are arranged chronologically and semantically to ensure the output content is coherent in narrative logic and consistent in value expression. The arrangement process employs a script template-based generation strategy, with a template library containing various commonly used teaching structures such as "problem-analysis-conclusion" and "case-principle-inspiration." Finally, the arranged multimodal content is packaged into a structured response package and presented to students through a front-end interface in the form of synchronized playback, step-by-step guidance, or interactive Q&A. The presentation format is dynamically adjusted based on students' cognitive development stages and interaction habits.

[0055] Finally, this invention completes the interactive closed loop through a multi-channel feedback mechanism and continuously updates the student's cognitive state vector, emotional state vector, and graph association weights.

[0056] Furthermore, the multi-channel feedback mechanism comprises two subsystems: explicit feedback collection and implicit feedback analysis. The explicit feedback collection subsystem records students' direct evaluations of the teaching response, including actions such as "like," "follow-up question," "skip," or "replay." Each action is assigned a different feedback weight; for example, the weight of "follow-up question" is higher than "like," indicating a deeper level of interest in the content. Feedback weights are preset based on the student's level of engagement reflected by the action. The implicit feedback analysis subsystem continuously monitors students' behavioral data during the response presentation, including eye focus distribution, voice response content, operation delay time, and the relevance of subsequent questions. Eye focus distribution is estimated using camera data to calculate the proportion of gaze duration in each modality. Voice response content is extracted using speech recognition and sentiment analysis, extracting keywords and emotional tendencies and mapping them to an emotional state vector E. Operation delay time reflects cognitive load levels and is mapped to a cognitive state vector C. The relevance of subsequent questions is measured by semantic distance with other nodes in the graph.

[0057] Both types of feedback data are transmitted back to the central collaborative controller in real time and used to update the state vector in the cognitive state tracking module and the emotional state vector in the physiological state parsing engine. At the same time, the access weights and relationship confidence of relevant nodes in the multimodal knowledge graph are adjusted. The access weights are updated using an exponential decay model, while the relationship confidence is dynamically calibrated through Bayesian inference, thereby realizing online optimization of teaching strategies and gradual evolution of the graph structure.

[0058] The construction and maintenance of the multimodal knowledge graph adopts an incremental update mechanism. When new teaching resources are imported into the system, the multimodal parsing module first generates preliminary knowledge units. Subsequently, the system performs nearest neighbor search in the existing graph to identify potential related nodes. The nearest neighbor search adopts a vector retrieval method based on graph embedding. The embedding model is periodically retrained on the entire graph to maintain semantic consistency. Specifically, the embedding model is retrained every 24 hours or after every 1000 new knowledge units. If the semantic similarity between a new unit and an existing node exceeds a preset threshold of 0.8, the relation reasoning engine is triggered, and a new relation is automatically established based on the axiomatic rules in the ontology model. If a valid relation cannot be established, the new unit is temporarily stored in the review area. After a human reviewer confirms its educational appropriateness and the correctness of its value orientation, it is then included in the main graph. All update operations are logged, and version rollback and consistency verification are supported. Version rollback adopts a snapshot mechanism, and consistency verification is performed through integrity checks using ontology constraint rules.

[0059] As another key feature of this invention, the intelligent teaching interaction method is strictly limited to educational application scenarios, and the design of all its technical components is based on ensuring the correct value orientation. Entities and relationships in the multimodal knowledge graph are derived solely from officially approved textbooks, authoritative publications, policy documents, and publicly reported mainstream media; the value layer labeling system is directly mapped and generated from the core competency framework of the curriculum issued by the education authorities; the response generation process prohibits the introduction of any external information sources not verified by the graph, ensuring that all teaching outputs are within a controllable, auditable, and traceable technical closed loop. The system operates in a privately deployed server cluster, without relying on any general-purpose large models or cloud-based intelligent services. All algorithm modules are executed locally, and data does not leave the domain, complying with national regulations regarding educational data security.

[0060] To verify the technical effects of the present invention, the following embodiments and comparative examples were designed for comparative experiments.

[0061] In one specific embodiment, the method of this invention was deployed on a university teaching platform to construct a multimodal knowledge graph. The graph contains 12,876 entity nodes, 34,521 relation edges, and 28,943 bound multimodal materials, including 12,000 text segments, 8,500 images, 6,200 videos, and 2,243 audio segments. The system runs on a local server cluster, equipped with modules such as a central collaborative controller, a graph database, and a multimodal parsing engine. Two hundred first-year university students were selected as the experimental group to use the system for 16 weeks of supplementary teaching. The system dynamically generates personalized teaching responses based on student interaction behavior.

[0062] In the comparative study, a traditional educational information system was used. This system only provides a static multimedia courseware library and simple keyword search functionality, lacking knowledge graph support, context awareness capabilities, and dynamic response generation mechanisms. Another 200 students of the same grade were selected as a control group, and the same content was taught using this traditional system.

[0063] After the experiment, standardized tests were used to assess the knowledge mastery, value identification, and higher-order thinking skills of the two groups of students. The test content, designed by a team of educational experts, included three types of questions: objective questions, case analysis questions, and value judgment questions. The experimental results are shown in the table below:

[0064]

[0065] Experimental data show that the experimental group using the method of this invention significantly outperformed the control group in all evaluation indicators, especially in higher-order thinking skills, demonstrating that this invention effectively promotes students' deep understanding and internalization of knowledge through multimodal knowledge graphs and context-adaptation mechanisms. Furthermore, measuring the average interaction depth of students by the number of graph nodes activated in a single session, system log analysis shows that the average interaction depth of the experimental group was 12.4, far exceeding the 3.2 of the control group, indicating that this invention successfully achieves deeper and more personalized teaching interaction.

[0066] In summary, this invention achieves deep semantic fusion of teaching resources by constructing a four-layer multimodal knowledge graph; solves the problem of fragmented heterogeneous materials through multimodal parsing and cross-modal alignment; ensures the systematic and standardized nature of knowledge expression through unified encoding based on ontology models; realizes personalized and adaptive teaching interaction through a context-aware dynamic activation mechanism; guarantees the system's continuous evolution capability through multi-channel feedback and incremental updates; and upholds the fundamental attributes of education through the explicit design of value throughout the entire process.

[0067] Those skilled in the art can make adaptive adjustments to the specific implementation details based on this embodiment without departing from the spirit and scope of the invention. For example, they can expand the value tag system, optimize the cross-modal alignment algorithm, or add new modal types. All of these should be considered as equivalent implementations of the invention.

Claims

1. An intelligent teaching interaction method based on multimodal knowledge graphs, characterized in that, Includes the following steps: S1. Construct a multimodal knowledge graph infrastructure for the education field, which includes an entity layer, a relation layer, a modal layer, and a value layer. S2. Perform multimodal analysis and semantic alignment on the original teaching resources to generate standardized knowledge units; S3. Based on the pre-set educational ontology model, knowledge units are uniformly encoded and associated with each other to form a multimodal knowledge graph with logical coherence and explicit value. S4. Based on students' real-time interactive behavior and historical learning trajectory, dynamically activate relevant subgraph structures in the graph and generate adaptive teaching responses in conjunction with the context awareness module. S5. The interaction loop is completed through a multi-channel feedback mechanism, and the student's cognitive state vector, emotional state vector and graph association weights are continuously updated.

2. The intelligent teaching interaction method based on multimodal knowledge graph according to claim 1, characterized in that, In step S1, the entity layer is used to represent core concepts, historical events, policy documents, characters and social cases in teaching, and each entity is given a globally unique identifier. The relation layer is used to define the semantic associations between entities, including causal relationships, temporal relationships, unity of opposites relationships, value mapping relationships, and theoretical evolution relationships. Each type of relationship is configured with corresponding logical axioms and constraint rules. The modal layer is used to bind multimodal teaching materials associated with entities or relationships, including text paragraphs, static images, dynamic videos, audio explanations, and interactive cases. Each material is generated into a structured description vector by a content feature extractor and linked to the corresponding graph node through a modal binding index. The value layer is used to explicitly label the core value dimensions carried by each entity and relationship, including national identity, social responsibility, historical perspective, rule of law awareness, and moral judgment.

3. The intelligent teaching interaction method based on multimodal knowledge graphs according to claim 1, characterized in that, In step S2, the multimodal parsing and semantic alignment process includes: performing syntactic dependency analysis and keyword extraction on text resources to identify the core concepts and argument logic involved; performing object detection and scene understanding on image resources, using a pre-trained multimodal visual model based on the Transformer architecture to identify and extract historical symbols, character identities, and spatial context information contained in the image; performing speech transcription and temporal segmentation on audio and video resources, simultaneously extracting speech content, background music emotional features, and keyframe semantics of the image; and mapping the parsing results of all modalities to a unified semantic space through a cross-modal alignment engine, and determining semantic consistency based on a cosine similarity threshold to generate cross-modal aligned knowledge units.

4. The intelligent teaching interaction method based on multimodal knowledge graph according to claim 1, characterized in that, In step S3, the unified encoding and association modeling process includes: importing aligned knowledge units into the educational ontology model; assigning an entity category to each knowledge unit according to the entity type labeling system in the ontology model; automatically deriving the semantic relationships that should exist between knowledge units according to the logical rules in the relation axiom set; labeling the value transmission direction and value transmission intensity of each relation path according to the value constraint rules; and persistently storing all knowledge units and their associations in the graph database.

5. The intelligent teaching interaction method based on multimodal knowledge graph according to claim 1, characterized in that, In step S4, the context awareness module includes a central collaborative controller, a physiological state analysis engine, a cognitive state tracking module, and a multimodal response generator. The central collaborative controller receives student questions or operation instructions and extracts a set of keywords. The cognitive state tracking module constructs a cognitive state vector containing knowledge mastery, value identification tendency, and thinking development stage based on the student's historical answer records, dwell time, click path, and interaction frequency. The physiological state analysis engine acquires facial expression, eye movement trajectory, or heart rate variability data through standard multimedia acquisition devices or optional wearable devices and converts it into emotional arousal and attention concentration indicators. The central collaborative controller integrates the above information to determine the representation of the current teaching context and retrieves the optimal subgraph structure that satisfies semantic relevance, cognitive matching degree, and emotional adaptability in the multimodal knowledge graph.

6. The intelligent teaching interaction method based on multimodal knowledge graph according to claim 5, characterized in that, The multimodal response generator generates a teaching response based on the activated subgraph structure, including: extracting the main logical chain from the subgraph, which connects the starting entity to the target entity through several relational edges, with each edge having a value-oriented label; selecting text, image, and video clips in priority order according to the modal binding information in the logical chain, prioritizing materials that can simultaneously cover multiple entities and whose emotional tone is consistent with the student's current emotional state; performing temporal arrangement and semantic connection processing on the selected materials to ensure narrative logic coherence and value expression consistency; and encapsulating the arranged multimodal content into a structured response package and presenting it through the front-end interface.

7. The intelligent teaching interaction method based on multimodal knowledge graph according to claim 1, characterized in that, In step S5, the multi-channel feedback mechanism includes two subsystems: explicit feedback acquisition and implicit feedback analysis. The explicit feedback acquisition subsystem records students' actions such as liking, asking follow-up questions, skipping, or repeating playback in response to the teaching. The implicit feedback analysis subsystem monitors the distribution of students' gaze focus, the content of their voice responses, the operation delay time, and the relevance of subsequent questions during the response presentation. Both types of feedback data are transmitted back to the central collaborative controller, the state vector, and the emotional state vector in the physiological state analysis engine, and the access weights and relationship confidence of relevant nodes in the graph are adjusted.

8. The intelligent teaching interaction method based on multimodal knowledge graph according to claim 1, characterized in that, The multimodal knowledge graph adopts an incremental update mechanism: when new teaching resources are imported, preliminary knowledge units are first generated; then, a nearest neighbor search is performed in the existing graph to identify potential related nodes; if the semantic similarity between the new unit and the existing nodes exceeds a preset threshold, the relation reasoning engine is triggered to automatically establish new relations based on the axiomatic rules in the ontology model; if an effective relation cannot be established, the new unit is temporarily stored in the review area, and after manual review to confirm its educational suitability and correct value orientation, it is included in the main graph.

Images