Online teaching interaction methods, media, and devices based on multimodal knowledge graphs

Abstract

This invention discloses an online teaching interaction method, medium, and device based on a multimodal knowledge graph. The method first collects raw teaching information and performs cross-modal semantic alignment processing to obtain structured teaching information. Then, it constructs a multimodal knowledge graph containing textual concept entities, video keyframe feature vectors, and transcribed audio text, forming concept-visual feature association edges, concept-audio segment association edges, and cross-modal similarity association edges. User interaction behavior data is converted into knowledge graph query vectors, and a three-dimensional teaching context subgraph is extracted from the multimodal knowledge graph. Progressive or comparative teaching content is generated based on user operation types. Finally, a cognitive state tracking matrix is ​​constructed to dynamically adjust teaching strategies and generate a personalized learning navigation graph. This invention solves the problems of fragmented knowledge and insufficient intelligent interaction in virtual simulation teaching, improving the teaching effectiveness of ideological and political education.

Description

Technical Field

[0001] This invention relates to the field of ideological and political education, specifically to an online teaching interaction method, medium, and device based on multimodal knowledge graphs. Background Technology

[0002] In the field of ideological and political education informatization, the main teaching model adopted is a combination of video lectures, courseware presentations, and online tests. With the development of educational informatization, virtual simulation technology has become an important means to improve teaching effectiveness. Various online platforms conduct teaching activities through 3D scene simulation and interactive operations. Currently, mainstream virtual teaching systems mainly rely on pre-set programs to achieve basic teaching functions. While they can provide a certain degree of immersive experience, they have significant shortcomings in the systematic construction of knowledge systems and the intelligentization of teaching interactions. Teaching elements in virtual scenes often lack deep semantic connections, making it difficult to construct a systematic knowledge system. Especially in handling multimodal teaching resources and achieving personalized teaching, existing virtual simulation systems struggle to organically integrate knowledge elements, generally exhibiting insufficient adaptability. Summary of the Invention

[0003] In view of the above problems, the present invention provides an online teaching interaction method, medium and device based on multimodal knowledge graph, which solves the problems of knowledge fragmentation and insufficient interactive intelligence in virtual simulation teaching.

[0004] To achieve the above objectives, in a first aspect, this application provides an online teaching interaction method based on a multimodal knowledge graph, comprising:

[0005] Collect raw teaching information, perform cross-modal semantic alignment processing on the raw teaching information, and obtain structured teaching information;

[0006] A multimodal knowledge graph is constructed, and textual concept entities, video keyframe feature vectors, and transcribed speech text content are deeply fused through graph neural networks to obtain cross-modal semantic association edges, which include concept-visual feature association edges, concept-speech segment association edges, and cross-modal similarity association edges.

[0007] Receive user interaction behavior data, which includes voice question content, virtual teaching operation sequence and interface click trajectory, and convert user interaction behavior data into knowledge graph query vectors;

[0008] Based on the query vector of the knowledge graph, a three-dimensional teaching context subgraph is extracted from the multimodal knowledge graph. When a standard teaching operation is detected, the forward reasoning path of the three-dimensional teaching context subgraph is activated to generate progressive teaching content. When an incorrect teaching operation is detected, the counterfactual reasoning mechanism of the three-dimensional teaching context subgraph is triggered to generate comparative teaching content.

[0009] Real-time recording of user interaction behavior data in the 3D teaching context subgraph, denoted as context interaction data, matching the mapping relationship between context interaction data and multimodal knowledge graph, and constructing a cognitive state tracking matrix;

[0010] Teaching strategies are dynamically adjusted based on the cognitive state tracking matrix to strengthen cross-modal presentation combinations of low-mastery concept nodes and extend higher-order knowledge associated with high-proficiency operation sequences.

[0011] Furthermore, based on the group behavior analysis model, the cognitive state tracking matrix is ​​aggregated, the teaching path recommendation weights in the multimodal knowledge graph are adjusted, and a personalized learning navigation graph is generated. The learning navigation graph includes optimal learning path planning and multimodal resource recommendation schemes.

[0012] Furthermore, the original teaching information includes textbook text information, teaching video information, and classroom audio information;

[0013] Raw teaching information is collected and subjected to cross-modal semantic alignment processing to obtain structured teaching information, including:

[0014] Obtain the text information of the teaching materials, extract the teaching concept entities from it through named entity recognition, and generate a concept feature vector by word embedding encoding for each teaching concept entity;

[0015] In addition, the teaching video information is obtained, and the video stream is sampled for keyframes to obtain a video keyframe sequence, which includes multiple first video keyframes.

[0016] In addition, classroom audio information is acquired, and the audio stream is segmented to obtain an audio segment set, which includes multiple first audio segments;

[0017] Constructing a cross-modal semantic alignment model includes:

[0018] Calculate the first attention weight between the concept feature vector and the visual feature vector of all video keyframes, and calculate the second attention weight between the concept feature vector and the acoustic feature vector of all speech segments;

[0019] Based on the first attention weight and the second attention weight, video keyframes and audio segments that are strongly associated with the teaching concept entities are selected and denoted as the second video keyframe and the second audio segment.

[0020] Structured instructional information is generated based on a cross-modal semantic alignment model. The structured instructional information includes instructional concept entities and concept feature vectors, second video keyframes and first attention weights, second speech segments and second attention weights, visual association between instructional concept entities and second video keyframes, and speech association between instructional concept entities and second speech segments.

[0021] Furthermore, a multimodal knowledge graph is constructed, and textual concept entities, video keyframe feature vectors, and transcribed speech text content are deeply fused using graph neural networks to obtain cross-modal semantic association edges, including:

[0022] The initial node set for constructing a multimodal knowledge graph includes:

[0023] The teaching concept entity is used as the concept node, and the node feature is used as the concept feature vector;

[0024] The second video keyframe is used as a visual node, and the node features are used as visual feature vectors.

[0025] The second speech segment is taken as a speech node, and the node features are speech feature vectors;

[0026] Establishing cross-modal semantic association edges includes:

[0027] Based on the first attention weight value of the concept node, a concept-visual association edge is established with the visual node. The weight value of the concept-visual association edge is the first attention weight after normalization.

[0028] Based on the second attention weight value of the concept node, a concept-speech association edge is established with the speech node. The weight value of the concept-speech association edge is the normalized second attention weight.

[0029] For visual nodes and audio nodes that have a temporal correspondence, a visual-audio temporal edge is established. The weight value of the visual-audio temporal edge is calculated based on the temporal overlap between the second video keyframe and the second audio segment.

[0030] Deep fusion of multimodal features using graph neural networks includes:

[0031] Input the initial set of nodes into the graph attention network;

[0032] Feature propagation is performed along concept-visual association edges and concept-speech association edges, and cross-modal similarity scores between nodes are calculated through a multi-head attention mechanism;

[0033] The cross-modal feature representation of the initial node set is updated based on the cross-modal similarity score to generate knowledge graph nodes with fusion features;

[0034] The final multimodal knowledge graph is obtained.

[0035] Furthermore, user interaction behavior data is converted into knowledge graph query vectors, including:

[0036] The voice query content is converted into query text by a speech recognition engine;

[0037] Use named entity recognition to extract core concept entities from query text;

[0038] The core concept entities are mapped to concept nodes in a multimodal knowledge graph, and the feature vectors of the successfully mapped concept nodes are used as the basis vectors for text queries.

[0039] Furthermore, the system analyzes the operation objects and actions in the virtual teaching operation sequence to obtain the analysis results;

[0040] Based on the analysis results, locate the corresponding teaching process node in the multimodal knowledge graph, and associate the feature vector of the teaching process node as the operation query basis vector;

[0041] Additionally, the feature vectors of knowledge graph nodes associated with the click areas of the interface click trajectory are identified, denoted as click feature vectors, and popularity weights are generated based on click frequency.

[0042] Generate a click query basis vector based on popularity weights and click feature vectors;

[0043] The text query basis vector, operation query basis vector, and click query basis vector are weighted and fused to obtain a fused feature vector. The fused feature vector is then input into the query encoder to generate the final knowledge graph query vector.

[0044] Furthermore, based on the knowledge graph query vector, a three-dimensional teaching context subgraph is extracted from the multimodal knowledge graph, including:

[0045] Based on knowledge graph query vectors, node matching is performed in a multimodal knowledge graph to locate the initial query node;

[0046] Traverse the graph along the concept-visual association edges and the concept-speech association edges to extract the associated visual query nodes and speech query nodes;

[0047] Based on the visual-speech temporal sequence edge, a spatiotemporal relationship is established to form a three-dimensional teaching context subgraph;

[0048] When a standard teaching operation is detected, the forward reasoning path of the 3D teaching context subgraph is activated to generate progressive teaching content, including:

[0049] Starting from the initial query node, reasoning is performed along the standard teaching path in the knowledge graph;

[0050] The concept query nodes, visual query nodes, and voice query nodes are grouped and associated according to the logical order of teaching.

[0051] Generate a progressive teaching sequence that includes explanations of basic concepts, example demonstrations, and voice guidance;

[0052] When an erroneous teaching operation is detected, the counterfactual reasoning mechanism of the 3D teaching context subgraph is triggered to generate comparative teaching content, including:

[0053] Identify the abnormal nodes in the knowledge graph corresponding to erroneous operations;

[0054] Disable standard teaching path edges that are directly associated with abnormal nodes;

[0055] Activate the counterfactual reasoning edge to generate teaching content that includes error analysis, correct demonstrations, and comparative explanations.

[0056] Furthermore, contextual interaction data includes the frequency of visits and dwell time of concept query nodes, the viewing completeness of visual query nodes, the number of times voice query nodes are repeated, and the selection ratio of standard teaching paths and counterfactual reasoning paths.

[0057] The mapping relationship between contextual interaction data and multimodal knowledge graphs is matched to construct a cognitive state tracking matrix, including:

[0058] An initial cognitive state tracking matrix is ​​established. The rows of the initial cognitive state tracking matrix represent concept nodes in the multimodal knowledge graph, and the columns include proficiency indicators, operational proficiency indicators, error pattern indicators, and attention distribution indicators.

[0059] Map the access records of concept query nodes to concept nodes, calculate the average test accuracy and access time scores of concept nodes, and obtain the proficiency index.

[0060] The viewing records of visual query nodes are mapped to visual nodes, and the success rate and average time of operation of teaching process nodes are statistically analyzed to obtain the operation proficiency index.

[0061] The teaching path selection records are mapped to the standard teaching path or the counterfactual reasoning path. The number of times the counterfactual reasoning path is triggered and the degree of improvement in subsequent operations are recorded to obtain the error pattern index.

[0062] The playback records of the voice query node are mapped to the voice node, and the attention distribution index is obtained based on the frequency of interaction events of the voice node, concept node, and visual node within a unit of time.

[0063] The proficiency index, operational proficiency index, error pattern index, and attention distribution index are updated to the columns of the initial cognitive state tracking matrix to obtain the updated cognitive state tracking matrix.

[0064] Furthermore, teaching strategies are dynamically adjusted based on the cognitive state tracking matrix to strengthen cross-modal presentation combinations of low-mastery concept nodes and extend higher-order knowledge associated with high-proficiency operation sequences, including:

[0065] Extract the set of low-mastery concept nodes whose proficiency index is below a first preset threshold from the cognitive state tracking matrix;

[0066] Perform a cross-modal reinforcement step for each low-mastery concept node in the set of low-mastery concept nodes:

[0067] Increase the display duration and frequency of visual nodes associated with low-level concept nodes;

[0068] Increase the playback priority of voice nodes associated with low-level concept nodes;

[0069] Generate a reinforced instructional combination that includes text explanations, visual demonstrations, and voice guidance;

[0070] Extract a set of highly proficient concept nodes from the cognitive state tracking matrix whose operational proficiency index is higher than a second preset threshold;

[0071] Perform a higher-order knowledge extension step for each high-proficiency concept node in the set of high-proficiency concept nodes:

[0072] Extract higher-order visual nodes along the concept-visual association edge;

[0073] Extract higher-order speech nodes along the concept-speech association edge;

[0074] Generate advanced teaching combinations that include advanced knowledge explanations, in-depth case analyses, and extended exercises;

[0075] Based on the error pattern indicators in the cognitive state tracking matrix, adjust the triggering conditions of the counterfactual reasoning path, including:

[0076] Add warning notification nodes for high-frequency error patterns;

[0077] Optimize the way incorrect operations are displayed compared with correct examples;

[0078] Based on the attention distribution indicators in the cognitive state tracking matrix, the teaching pace is dynamically adjusted, including:

[0079] Simplify teaching content for areas of low attention;

[0080] Add interactive elements to areas of high attention;

[0081] The existing teaching strategy configuration is updated to obtain an updated teaching configuration, which is then implemented. The updated teaching configuration includes reinforcement teaching plans for low-mastery concept nodes, advanced extension plans for high-proficiency concept nodes, optimization parameters for counterfactual reasoning paths, and adjustment parameters for teaching pace.

[0082] Furthermore, based on the aggregated cognitive state tracking matrix of the group behavior analysis model, the teaching path recommendation weights in the multimodal knowledge graph are adjusted to generate a personalized learning navigation graph, including:

[0083] Obtain the cognitive state tracking matrix of multiple users, denoted as the group cognitive state tracking matrix. The group cognitive state tracking matrix includes group concept nodes. Calculate the average mastery index of each group concept node and count high-frequency error patterns and group-related nodes.

[0084] After analyzing the common features of the optimal learning path, hot spots in the distribution of group attention are identified.

[0085] The average mastery index, high-frequency error patterns, and optimal learning path features of the group cognitive state tracking matrix are calculated. The weights for recommending teaching paths in the multimodal knowledge graph are then adjusted based on the hotspots of group attention distribution, including:

[0086] Reduce the weight of basic teaching for conceptual nodes with high group mastery, and increase the weight of counterfactual reasoning for nodes with high group error rates.

[0087] Increase the recommendation priority of related edges in the optimal path of the group, and optimize the resource display weight of the hot spots of group attention;

[0088] Based on the adjusted teaching path recommendation weights, extract subgraph structures related to the current user from the multimodal knowledge graph;

[0089] After recalculating the path score of the teaching path, the key concept nodes on the recommended learning path are marked, and the error-prone node areas that need to be focused on are marked.

[0090] Generate a learning navigation map that includes a main learning path and alternative paths with personalized recommendations, recommended concept nodes, recommended visual nodes, recommended audio node combinations, recommended path weight scores and estimated learning time, and key reinforcement area prompts.

[0091] In a second aspect, the present invention also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method described in the first aspect.

[0092] In a third aspect, the present invention also provides an electronic device including a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect.

[0093] Unlike existing technologies, the above technical solution has the following beneficial effects:

[0094] The above technical solution provides an online teaching interaction method, medium, and device based on a multimodal knowledge graph. The method first collects raw teaching information and performs cross-modal semantic alignment processing to obtain structured teaching information. Then, it constructs a multimodal knowledge graph containing textual concept entities, video keyframe feature vectors, and transcribed audio text, forming concept-visual feature association edges, concept-audio segment association edges, and cross-modal similarity association edges. User interaction behavior data is converted into knowledge graph query vectors, and a three-dimensional teaching context subgraph is extracted from the multimodal knowledge graph. Progressive or comparative teaching content is generated based on user operation types. Finally, a cognitive state tracking matrix is ​​constructed to dynamically adjust teaching strategies and generate a personalized learning navigation graph. This technical solution addresses the problems of fragmented knowledge and insufficient interactive intelligence in virtual simulation teaching, improving the teaching effectiveness of ideological and political education.

[0095] The above description of the invention is merely an overview of the technical solution of this application. In order to enable those skilled in the art to better understand the technical solution of this application and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of this application easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of this application. Attached Figure Description

[0096] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of the present invention and other related contents, and should not be considered as limitations on this application.

[0097] In the accompanying drawings of the instruction manual:

[0098] Figure 1 This is a flowchart illustrating steps S101 to S106 of the online teaching interaction method described in a specific implementation.

[0099] Figure 2 This is a flowchart illustrating steps S201 to S203 of the online teaching interaction method described in a specific implementation.

[0100] Figure 3 This is a flowchart illustrating steps S301 to S306 of the online teaching interaction method described in a specific implementation.

[0101] Figure 4 This is a flowchart illustrating steps S401 to S405 of the online teaching interaction method described in a specific implementation.

[0102] Figure 5 This is a schematic diagram of the structure of the electronic device described in a specific embodiment.

[0103] The reference numerals used in the above figures are explained as follows:

[0104] 1. Electronic equipment;

[0105] 11. Memory;

[0106] 12. Processor. Detailed Implementation

[0107] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.

[0108] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0109] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0110] In the description of this application, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" logical relationship.

[0111] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.

[0112] Without further limitations, the use of terms such as “comprising,” “including,” “having,” or other similar open-ended expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0113] As understood in the Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceeding" are understood to exclude the stated number; expressions such as "above," "below," and "within" are understood to include the stated number. Furthermore, in the description of the embodiments in this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times," unless otherwise explicitly specified.

[0114] Please see Figure 1 In a first aspect, this embodiment provides an online teaching interaction method based on a multimodal knowledge graph, including:

[0115] S101. Collect raw teaching information, perform cross-modal semantic alignment processing on the raw teaching information, and obtain structured teaching information;

[0116] S102. Construct a multimodal knowledge graph. Deeply fuse text concept entities, video keyframe feature vectors, and transcribed speech text content through graph neural networks to obtain cross-modal semantic association edges. Cross-modal semantic association edges include concept-visual feature association edges, concept-speech segment association edges, and cross-modal similarity association edges.

[0117] S103. Receive user interaction behavior data, which includes voice question content, virtual teaching operation sequence and interface click trajectory, and convert the user interaction behavior data into knowledge graph query vectors.

[0118] S104. Extract a three-dimensional teaching context subgraph from the multimodal knowledge graph based on the knowledge graph query vector. When a standard teaching operation is detected, activate the forward reasoning path of the three-dimensional teaching context subgraph to generate progressive teaching content. When an incorrect teaching operation is detected, trigger the counterfactual reasoning mechanism of the three-dimensional teaching context subgraph to generate comparative teaching content.

[0119] S105. Record user interaction behavior data in the three-dimensional teaching context sub-graph in real time, denoted as context interaction data, match the mapping relationship between context interaction data and multimodal knowledge graph, and construct a cognitive state tracking matrix.

[0120] S106. Dynamically adjust teaching strategies based on the cognitive state tracking matrix, strengthen cross-modal presentation combinations of low-mastery concept nodes, and extend higher-order knowledge related to high-proficiency operation sequences.

[0121] Furthermore, based on the group behavior analysis model, the cognitive state tracking matrix is ​​aggregated, the teaching path recommendation weights in the multimodal knowledge graph are adjusted, and a personalized learning navigation graph is generated. The learning navigation graph includes optimal learning path planning and multimodal resource recommendation schemes.

[0122] In step S101, the original teaching information includes text textbooks, teaching videos, experimental operation videos, and classroom audio explanations. Preferably, the cross-modal semantic alignment processing employs a Transformer-based multimodal encoder to map heterogeneous teaching resources to a unified semantic representation space, ensuring semantic consistency across different modalities. The structured teaching information includes textual conceptual entities, video keyframe feature vectors, and transcribed audio content. This step achieves standardized processing of heterogeneous data through a cross-modal feature extraction network. Preferably, textual conceptual entities are extracted using named entity recognition technology, video keyframe feature vectors are extracted using a convolutional neural network, and transcribed audio content is obtained through a speech recognition model.

[0123] In step S102, the constructed multimodal knowledge graph employs a graph attention network to achieve feature fusion. The concept-visual feature association edges include spatial attention weights, and the concept-speech segment association edges include temporal alignment information. The spatial attention weights are calculated using a visual-text alignment network, reflecting the correspondence between text concepts and video regions; the temporal alignment information is established using a dynamic time warping algorithm to ensure the temporal synchronization of speech segments and text concepts. Cross-modal similarity association edges are constructed using a contrastive learning framework to quantify the semantic consistency between features from different modalities.

[0124] In step S103, user interaction behavior data is converted into a knowledge graph query vector using a multimodal encoder. Preferably, the voice question content is encoded using a pre-trained language model, the virtual teaching operation sequence has features extracted using a spatiotemporal convolutional network, and the interface click trajectory is converted into a spatial distribution vector of knowledge nodes. The fused query vector of these three interaction features—voice question content, virtual teaching operation sequence, and interface click trajectory—provides a retrieval basis for subsequent contextual subgraph extraction.

[0125] In step S104, the three-dimensional teaching context subgraph includes concept nodes, associated edge weights, and cross-modal resource links. The subgraph extraction process preferably employs an attention sampling algorithm based on query vectors to dynamically construct a knowledge subnetwork related to the current teaching context. Preferably, the counterfactual reasoning mechanism of the three-dimensional teaching context subgraph generates contrasting contexts that violate teaching principles through conditional masking of knowledge graph edges. The counterfactual reasoning mechanism is implemented using a trainable edge mask matrix, which can selectively mask specific semantic relationships to construct teaching counterexamples. Preferably, the counterfactual reasoning mechanism implements path redirection through a graph convolutional network, specifically including: constructing a trainable edge mask matrix; when an abnormal node is detected, calculating the path blocking probability based on a graph attention mechanism and dynamically masking standard teaching path edges; simultaneously, constructing a contrasting teaching segment generator through a pre-trained adversarial generative network, adversarially fusing the feature vectors of incorrect operations with the feature vectors of correct demonstrations to generate multimodal contrasting cases with difference annotations.

[0126] In step S105, the cognitive state tracking matrix includes a proficiency score, an operational proficiency index, and an error pattern classification. The matrix element values ​​are jointly determined by the knowledge node access depth and the activation frequency of associated edges. Preferably, the access depth is calculated based on the node's position in the interaction path, and the activation frequency is calculated by counting the number of times associated edges are triggered per unit time. Further, the proficiency score is calculated based on the knowledge node's dwell time and access path depth; the operational proficiency index is based on the completion time of the operation sequence and the standard process matching degree; and the error pattern classification is achieved by comparing the difference patterns between user operations and standard operations.

[0127] In step S106, the teaching strategy adjustment module implements differentiated teaching based on the cognitive state tracking matrix. For concept nodes with low mastery, visual salience and verbal explanations are enhanced in their cross-modal presentation combinations; for operation sequences with high proficiency, the reasoning path associated with higher-order knowledge nodes is extended. Preferably, the generation of the personalized learning navigation map adopts a collaborative filtering algorithm based on swarm intelligence, which optimizes the recommended path by integrating individual cognitive characteristics and group learning patterns.

[0128] This embodiment achieves deep fusion of textual concept entities, video keyframe feature vectors, and transcribed audio text content through cross-modal semantic alignment processing of structured teaching information. The constructed multimodal knowledge graph establishes a cross-modal semantic association network through concept-visual feature association edges, concept-audio segment association edges, and cross-modal similarity association edges. Voice question content, virtual teaching operation sequences, and interface click trajectories are converted into knowledge graph query vectors, achieving accurate mapping between user interaction behavior data and the knowledge graph. Through the forward reasoning path and counterfactual reasoning mechanism of the 3D teaching context subgraph, the system can dynamically generate progressive and comparative teaching content. The construction of the cognitive state tracking matrix is ​​based on the mapping relationship between contextual interaction data and the multimodal knowledge graph, accurately reflecting the user's mastery score, operation proficiency index, and error pattern classification. Adjustments to differentiated teaching strategies implemented according to the cognitive state tracking matrix effectively strengthen the cross-modal display combination of low-mastery concept nodes and extend higher-order knowledge associated with high-proficiency operation sequences. By combining a learning navigation graph generated from a group behavior analysis model with optimal learning path planning and multimodal resource recommendation schemes, personalized adaptation of the teaching process is achieved. This embodiment, through the construction and application of a multimodal knowledge graph, forms a complete closed loop from teaching resource organization and learning status assessment to teaching strategy optimization, significantly improving the intelligence level and teaching effectiveness of the online teaching system.

[0129] In some embodiments, the original teaching information includes textbook text information, teaching video information, and classroom audio information;

[0130] Raw teaching information is collected and subjected to cross-modal semantic alignment processing to obtain structured teaching information, including:

[0131] Obtain the text information of the teaching materials, extract the teaching concept entities from it through named entity recognition, and generate a concept feature vector by word embedding encoding for each teaching concept entity;

[0132] In addition, the teaching video information is obtained, and the video stream is sampled for keyframes to obtain a video keyframe sequence, which includes multiple first video keyframes.

[0133] In addition, classroom audio information is acquired, and the audio stream is segmented to obtain an audio segment set, which includes multiple first audio segments;

[0134] Constructing a cross-modal semantic alignment model includes:

[0135] Calculate the first attention weight between the concept feature vector and the visual feature vector of all video keyframes, and calculate the second attention weight between the concept feature vector and the acoustic feature vector of all speech segments;

[0136] Based on the first attention weight and the second attention weight, video keyframes and audio segments that are strongly associated with the teaching concept entities are selected and denoted as the second video keyframe and the second audio segment.

[0137] Structured instructional information is generated based on a cross-modal semantic alignment model. The structured instructional information includes instructional concept entities and concept feature vectors, second video keyframes and first attention weights, second speech segments and second attention weights, visual association between instructional concept entities and second video keyframes, and speech association between instructional concept entities and second speech segments.

[0138] In this embodiment, the teaching concept entities are extracted from the textbook text information using named entity recognition technology. Preferably, each entity is encoded by word embedding through a pre-trained language model to generate a concept feature vector, which represents the distribution characteristics of the teaching concept in the semantic space.

[0139] In the process of processing teaching video information, keyframe sampling is performed on the video stream to obtain a video keyframe sequence, including:

[0140] Visual feature vectors are extracted using a pre-trained convolutional neural network;

[0141] Detect and recognize text annotation information in the image;

[0142] Establish a temporal correspondence between video keyframes and teaching concept entities.

[0143] In classroom speech information processing, the speech stream is segmented to obtain a set of speech segments, including:

[0144] Converted into speech-text content using a speech recognition engine;

[0145] Extract the acoustic feature vectors of speech segments;

[0146] Establish semantic correspondences between audio segments and teaching concept entities.

[0147] The construction process of the cross-modal semantic alignment model achieves multimodal feature association through an attention mechanism. The first attention weight calculates the correlation between the concept feature vector and the visual feature vector of the video keyframe, reflecting the degree of matching between the teaching concept and the visual content. The second attention weight measures the correlation strength between the concept feature vector and the acoustic feature vector of the speech segment, characterizing the semantic consistency between the speech content and the teaching concept. The selected second video keyframe and second speech segment both maintain a strong correlation with the teaching concept entity, ensuring the accuracy of cross-modal alignment.

[0148] The generation process of structured instructional information establishes a systematic association between instructional concept entities and multimodal resources. Visual association quantifies the matching degree between instructional concepts and video keyframes through a first attention weight; while phonological association reflects the correspondence strength between instructional concepts and phonological segments through a second attention weight. Visual and phonological associations together constitute the semantic network foundation of cross-modal instructional knowledge.

[0149] This embodiment transforms originally discrete teaching resources into a structured information system with clear semantic relationships through systematic cross-modal alignment processing. When processing a teaching concept, the system automatically associates keyframes of relevant scenes in the video with audio segments explaining the concept, forming a multi-dimensional combination of teaching resources. This not only preserves the richness of the original teaching resources but also establishes a systematic connection between knowledge elements through semantic alignment, providing a standardized data foundation for the subsequent construction of a multimodal knowledge graph. This embodiment realizes the transformation process from original teaching resources to a structured knowledge system, ensuring a high degree of semantic consistency among teaching elements of different modalities.

[0150] Please see Figure 2 In some embodiments, a multimodal knowledge graph is constructed, and textual concept entities, video keyframe feature vectors, and transcribed speech text content are deeply fused using a graph neural network to obtain cross-modal semantic association edges, including:

[0151] S201. Construct the initial node set for a multimodal knowledge graph. The initial node set includes:

[0152] The teaching concept entity is used as the concept node, and the node feature is used as the concept feature vector;

[0153] The second video keyframe is used as a visual node, and the node features are used as visual feature vectors.

[0154] The second speech segment is taken as a speech node, and the node features are speech feature vectors;

[0155] S202. Establish cross-modal semantic association edges, including:

[0156] Based on the first attention weight value of the concept node, a concept-visual association edge is established with the visual node. The weight value of the concept-visual association edge is the first attention weight after normalization.

[0157] Based on the second attention weight value of the concept node, a concept-speech association edge is established with the speech node. The weight value of the concept-speech association edge is the normalized second attention weight.

[0158] For visual nodes and audio nodes that have a temporal correspondence, a visual-audio temporal edge is established. The weight value of the visual-audio temporal edge is calculated based on the temporal overlap between the second video keyframe and the second audio segment.

[0159] S203. Deep fusion of multimodal features using graph neural networks, including:

[0160] Input the initial set of nodes into the graph attention network;

[0161] Feature propagation is performed along concept-visual association edges and concept-speech association edges, and cross-modal similarity scores between nodes are calculated through a multi-head attention mechanism;

[0162] The cross-modal feature representation of the initial node set is updated based on the cross-modal similarity score to generate knowledge graph nodes with fusion features;

[0163] The final multimodal knowledge graph is obtained.

[0164] In step S201, constructing the initial node set of the multimodal knowledge graph involves transforming the elements in the structured teaching information into graph-structured data. Specifically, concept nodes are based on teaching concept entities, and their concept feature vectors are obtained through pre-trained language models, representing the semantic features of the teaching concepts; visual nodes correspond to the second video keyframes, and their visual feature vectors are extracted through convolutional neural networks, containing visual semantic information of the video content; speech nodes come from the second speech segment, and their speech feature vectors are extracted through acoustic models, reflecting the acoustic features and semantic content of the speech. These initial nodes provide the basic data units for subsequent cross-modal feature fusion.

[0165] In step S202, establishing cross-modal semantic association edges involves constructing the topological structure of a multimodal knowledge graph using attention weights and temporal relationships. Specifically, the weights of concept-visual association edges are determined by normalized first attention weights, reflecting the semantic relevance between the teaching concept and the video content; the weights of concept-speech association edges are derived from normalized second attention weights, characterizing the matching strength between the teaching concept and the speech explanation. The establishment of visual-speech temporal edges is based on multimedia synchronization principles, and their weights are obtained by calculating the temporal overlap between the second video keyframe and the second speech segment, ensuring the consistency of audiovisual content in the temporal dimension.

[0166] In step S203, deep fusion of multimodal features via graph neural networks refers to utilizing the characteristics of graph attention networks to achieve cross-modal knowledge representation learning. Specifically, the feature propagation process proceeds along concept-visual and concept-speech association edges, and a multi-head attention mechanism is used to calculate cross-modal similarity scores between nodes. Preferably, the cross-modal similarity scores comprehensively consider semantic relevance and modal complementarity. Feature updates of knowledge graph nodes employ a gating mechanism to dynamically adjust the fusion ratio of each modality feature, resulting in fused features that retain original modal characteristics while also containing cross-modal interaction information.

[0167] The resulting multimodal knowledge graph includes concept nodes, visual nodes, and speech nodes with fusion features, concept-visual association edges and their weights, concept-speech association edges and their weights, and visual-speech temporal edges and their weights, providing users with cross-modal association evidence.

[0168] This embodiment realizes multimodal structured representation and deep fusion of teaching knowledge. When processing a certain concept node, the system uses a graph attention network to fuse its features with relevant theoretical explanation video keyframes (visual nodes) and audio narration segments (audio nodes), generating a unified knowledge representation that includes textual concepts, visual examples, and audio explanations. This not only establishes explicit associations between teaching elements, but also achieves complementary enhancement of knowledge representation through deep fusion at the feature level, providing a rich semantic foundation for subsequent contextualized teaching interactions, enabling the system to understand and present teaching content from different perspectives.

[0169] Please see Figure 3 In some embodiments, converting user interaction behavior data into knowledge graph query vectors includes:

[0170] S301. The content of the voice question is converted into query text by the voice recognition engine;

[0171] S302. Use named entity recognition to extract core concept entities from query text;

[0172] S303. Map the core concept entities to concept nodes in the multimodal knowledge graph, and use the feature vectors of the successfully mapped concept nodes as the basis vectors for text queries.

[0173] Furthermore, the system analyzes the operation objects and actions in the virtual teaching operation sequence to obtain the analysis results;

[0174] S304. Based on the parsing results, locate the corresponding teaching process node in the multimodal knowledge graph, and associate the feature vector of the teaching process node as the operation query basis vector.

[0175] Additionally, the feature vectors of knowledge graph nodes associated with the click areas of the interface click trajectory are identified, denoted as click feature vectors, and popularity weights are generated based on click frequency.

[0176] S305. Generate a click query basis vector based on the popularity weight and click feature vector;

[0177] S306. The text query basis vector, operation query basis vector and click query basis vector are weighted and fused to obtain the fused feature vector. The fused feature vector is then input into the query encoder to generate the final knowledge graph query vector.

[0178] In step S301, the voice question is converted into query text by a speech recognition engine, transforming the user's voice interaction input into a processable text format. The speech recognition engine preferably employs an end-to-end deep learning model, capable of adapting to specialized terminology and diverse pronunciations in teaching scenarios. The converted query text retains the semantic integrity of the original voice question, providing an input basis for subsequent concept entity extraction.

[0179] In step S302, named entity recognition is used to extract core concept entities from the query text, and a pre-trained language model is used to identify key concepts related to teaching. These core concept entities correspond to concept nodes in the multimodal knowledge graph, ensuring that user queries are accurately mapped to the knowledge system. The named entity recognition model is optimized for the teaching domain and can accurately identify professional terms and core concepts in ideological and political education.

[0180] In step S303, semantic similarity calculation is used to map core concept entities to concept nodes in the multimodal knowledge graph. The feature vectors of successfully mapped concept nodes serve as text query basis vectors, which contain deep semantic features of the teaching concepts. Simultaneously, an action semantic analysis model is used to parse the operation objects and actions in the virtual teaching operation sequence, and the parsing results reflect the user's teaching behavioral intentions in the virtual environment.

[0181] In step S304, locating the corresponding teaching process node in the multimodal knowledge graph based on the parsing results is achieved through behavior-knowledge association rules. The feature vector of the associated teaching process node serves as the basis vector for the operation query, representing the knowledge content corresponding to the teaching operation. Identifying the feature vector of the knowledge graph node associated with the click area of ​​the interface click trajectory is accomplished through a spatial mapping algorithm, and the popularity weight generated by the click frequency reflects the user's interest preferences.

[0182] In step S305, the generation of the click query basis vector based on the popularity weight and click feature vector is achieved through a weighted aggregation algorithm. The popularity weight is normalized to ensure a balanced contribution from different interactive behaviors. The click query basis vector comprehensively reflects the user's explicit interest direction.

[0183] In step S306, the weighted fusion of text query basis vectors, operation query basis vectors, and click query basis vectors is achieved through an attention mechanism. The fused feature vector is input into the query encoder to generate the final knowledge graph query vector. Here, the weights of the text query basis vectors are the confidence scores of the voice questions; the weights of the operation query basis vectors are the standardization scores of the operation steps; and the weights of the click query basis vectors are the normalized popularity weights. The query encoder employs a multilayer perceptron structure, enabling it to learn deep associations between different interaction features.

[0184] This embodiment achieves intelligent conversion from multimodal interactive behaviors to knowledge graph queries. When a user asks a question via voice and clicks on a relevant case area, the system converts the voice content into a text query basis vector. This, combined with the click query basis vector generated from the clicked area, ultimately forms a comprehensive knowledge graph query vector that reflects the user's intent. This embodiment fully considers the semantic contribution of different interactive behaviors, ensuring that the query results not only meet the user's explicit needs but also uncover potential learning interests. By constructing an intelligent bridge from user behavior to knowledge retrieval, this embodiment provides a precise query foundation for personalized teaching.

[0185] Please see Figure 4 In some embodiments, extracting a three-dimensional teaching context subgraph from a multimodal knowledge graph based on a knowledge graph query vector includes:

[0186] S401. Based on the knowledge graph query vector, perform node matching in the multimodal knowledge graph to locate the initial query node;

[0187] S402. Traverse the graph along the concept-visual association edge and the concept-speech association edge to extract the associated visual query nodes and speech query nodes;

[0188] S403. Establish spatiotemporal relationships based on the visual-speech temporal sequence edges to form a three-dimensional teaching context sub-graph;

[0189] S404. When a standard teaching operation is detected, the forward reasoning path of the three-dimensional teaching context subgraph is activated to generate progressive teaching content, including:

[0190] Starting from the initial query node, reasoning is performed along the standard teaching path in the knowledge graph;

[0191] The concept query nodes, visual query nodes, and voice query nodes are grouped and associated according to the logical order of teaching.

[0192] Generate a progressive teaching sequence that includes explanations of basic concepts, example demonstrations, and voice guidance;

[0193] S405. When an erroneous teaching operation is detected, the counterfactual reasoning mechanism of the three-dimensional teaching context subgraph is triggered to generate comparative teaching content, including:

[0194] Identify the abnormal nodes in the knowledge graph corresponding to erroneous operations;

[0195] Disable standard teaching path edges that are directly associated with abnormal nodes;

[0196] Activate the counterfactual reasoning edge to generate teaching content that includes error analysis, correct demonstrations, and comparative explanations.

[0197] In step S401, node matching is performed in the multimodal knowledge graph based on the knowledge graph query vector, and the initial query node most relevant to the query intent is located by calculating vector similarity. The initial query node serves as the starting point for extracting the context subgraph, and its matching accuracy directly affects the accuracy of subsequent teaching content. Preferably, the node matching process employs an approximate nearest neighbor search algorithm to ensure both real-time performance and the reliability of the query results.

[0198] In step S402, preferably, the graph traversal along the concept-visual association edges and the concept-speech association edges is implemented using a breadth-first search algorithm. The extracted visual query nodes may contain visual examples related to the teaching concept, while the speech query nodes provide corresponding audio explanations. During the traversal, path filtering is performed based on the weight values ​​of the association edges to ensure that the extracted nodes have sufficient semantic relevance.

[0199] In step S403, preferably, the establishment of spatiotemporal correlation based on the visual-speech temporal edge is accomplished through a time alignment algorithm. The spatiotemporal correlation ensures that the video presentation and audio explanation remain synchronized, forming a three-dimensional teaching context subgraph with spatiotemporal consistency. The establishment of the spatiotemporal correlation takes into account the logical coherence and temporal continuity requirements of the teaching content.

[0200] In step S404, preferably, the generation of progressive teaching content through the forward reasoning path of activating the 3D teaching context subgraph is achieved through a teaching logic engine. The standard teaching path is predefined by domain experts and reflects the progressive relationship of knowledge points. Optionally, the generation of the progressive teaching sequence follows a "concept explanation - example demonstration - practical guidance" teaching model to ensure the systematic and coherent learning process.

[0201] In step S405, abnormal nodes in the knowledge graph identify knowledge blind spots related to erroneous operations. Disabling standard teaching path edges prevents the propagation of incorrect demonstrations. Optionally, a graph anomaly detection model is used to identify abnormal nodes corresponding to erroneous operations, and time-series analysis is combined to verify abnormal paths. The activation of counterfactual reasoning edges is based on the teaching rule base, and the generated comparative teaching content highlights the differences between incorrect and correct operations, strengthening the cognitive correction effect.

[0202] This embodiment accurately locates the initial query node using knowledge graph query vectors, extracts associated nodes along concept-visual and concept-speech association edges, and constructs a spatiotemporally consistent 3D teaching context subgraph based on visual-speech temporal edges. When a standard teaching operation is detected, the system generates a progressive teaching sequence along a preset teaching path, including concept explanations, example demonstrations, and voice guidance; when an incorrect operation is identified, comparative teaching content is generated through a counterfactual reasoning mechanism. This embodiment achieves intelligent organization and dynamic generation of teaching content, ensuring the systematic nature of standard teaching and the targeted nature of error correction. By accurately matching user query intent, maintaining the spatiotemporal association of multimodal teaching resources, and supporting bidirectional teaching reasoning, this embodiment significantly improves the accuracy and adaptability of online teaching, ensuring the systematic integrity of knowledge transfer while effectively intervening in learning misconceptions, forming a closed-loop optimized intelligent teaching mechanism.

[0203] In some embodiments, contextual interaction data includes the access frequency and dwell time of concept query nodes, the viewing completeness of visual query nodes, the number of times voice query nodes are repeated, and the selection ratio of standard teaching paths and counterfactual reasoning paths.

[0204] The mapping relationship between contextual interaction data and multimodal knowledge graphs is matched to construct a cognitive state tracking matrix, including:

[0205] An initial cognitive state tracking matrix is ​​established. The rows of the initial cognitive state tracking matrix represent concept nodes in the multimodal knowledge graph, and the columns include proficiency indicators, operational proficiency indicators, error pattern indicators, and attention distribution indicators.

[0206] Map the access records of concept query nodes to concept nodes, calculate the average test accuracy and access time scores of concept nodes, and obtain the proficiency index.

[0207] The viewing records of visual query nodes are mapped to visual nodes, and the success rate and average time of operation of teaching process nodes are statistically analyzed to obtain the operation proficiency index.

[0208] The teaching path selection records are mapped to the standard teaching path or the counterfactual reasoning path. The number of times the counterfactual reasoning path is triggered and the degree of improvement in subsequent operations are recorded to obtain the error pattern index.

[0209] The playback records of the voice query node are mapped to the voice node, and the attention distribution index is obtained based on the frequency of interaction events of the voice node, concept node, and visual node within a unit of time.

[0210] The proficiency index, operational proficiency index, error pattern index, and attention distribution index are updated to the columns of the initial cognitive state tracking matrix to obtain the updated cognitive state tracking matrix.

[0211] In this embodiment, contextual interaction data refers to the behavioral records generated during user interaction with the 3D teaching context sub-graph, including key indicators such as the frequency and duration of visits to concept query nodes, the viewing completeness of visual query nodes, and the number of times audio query nodes are repeated. Contextual interaction data is collected through multimodal sensors, reflecting the user's cognitive processing of different teaching elements. The selection ratio between standard teaching paths and counterfactual reasoning paths characterizes the user's learning strategy preferences and is an important basis for evaluating teaching effectiveness.

[0212] The cognitive state tracking matrix is ​​constructed by quantifying the user's learning status through multi-dimensional indicators. The matrix includes a mastery score, operational proficiency indicators, and error pattern classification. Matrix element values ​​are jointly determined by the knowledge node access depth and the activation frequency of associated edges. The initial row and column structure of the cognitive state tracking matrix ensures the systematic organization of various evaluation indicators. Mastery is based on the test accuracy and access depth of concept query nodes; operational proficiency is based on the operational accuracy and completion speed of teaching process nodes; error patterns are based on the trigger frequency and correction effect of counterfactual reasoning paths; and attention distribution is based on the interaction duration and hotspot click data of each query node. This structured design enables a multi-faceted portrayal of cognitive states.

[0213] Preferably, the indicator calculation process uses a mapping statistical method to achieve data transformation. The average test accuracy and access duration scores of concept nodes are calculated using a weighted algorithm to reflect the degree of knowledge mastery; the operation success rate statistics of teaching process nodes take into account the normalization of task complexity; the trigger count records of counterfactual reasoning paths are combined with time decay factors to highlight the impact of recent behavior; and the attention distribution indicator identifies the user's focus of attention through spatiotemporal clustering analysis of interactive events.

[0214] This embodiment achieves precise assessment of the learning process through systematic cognitive state modeling. For example, when a student exhibits high access frequency but low test accuracy at a certain concept node, the system determines that they are superficially familiar but lack substantial understanding; when a certain counterfactual reasoning path is frequently triggered, it is identified that the method contains cognitive biases. This not only captures explicit behavioral characteristics but also uncovers deeper cognitive states, providing a reliable basis for adjusting personalized teaching strategies. This embodiment constructs a complete analysis chain from interactive behavior to cognitive assessment, supporting intelligent decision-making in the teaching system.

[0215] In some embodiments, teaching strategies are dynamically adjusted based on a cognitive state tracking matrix to strengthen cross-modal presentation combinations of low-mastery concept nodes and extend higher-order knowledge associated with high-proficiency operation sequences, including:

[0216] Extract the set of low-mastery concept nodes whose proficiency index is below a first preset threshold from the cognitive state tracking matrix;

[0217] Perform a cross-modal reinforcement step for each low-mastery concept node in the set of low-mastery concept nodes:

[0218] Increase the display duration and frequency of visual nodes associated with low-level concept nodes;

[0219] Increase the playback priority of voice nodes associated with low-level concept nodes;

[0220] Generate a reinforced instructional combination that includes text explanations, visual demonstrations, and voice guidance;

[0221] Extract a set of highly proficient concept nodes from the cognitive state tracking matrix whose operational proficiency index is higher than a second preset threshold;

[0222] Perform a higher-order knowledge extension step for each high-proficiency concept node in the set of high-proficiency concept nodes:

[0223] Extract higher-order visual nodes along the concept-visual association edge;

[0224] Extract higher-order speech nodes along the concept-speech association edge;

[0225] Generate advanced teaching combinations that include advanced knowledge explanations, in-depth case analyses, and extended exercises;

[0226] Based on the error pattern indicators in the cognitive state tracking matrix, adjust the triggering conditions of the counterfactual reasoning path, including:

[0227] Add warning notification nodes for high-frequency error patterns;

[0228] Optimize the way incorrect operations are displayed compared with correct examples;

[0229] Based on the attention distribution indicators in the cognitive state tracking matrix, the teaching pace is dynamically adjusted, including:

[0230] Simplify teaching content for areas of low attention;

[0231] Add interactive elements to areas of high attention;

[0232] The existing teaching strategy configuration is updated to obtain an updated teaching configuration, which is then implemented. The updated teaching configuration includes reinforcement teaching plans for low-mastery concept nodes, advanced extension plans for high-proficiency concept nodes, optimization parameters for counterfactual reasoning paths, and adjustment parameters for teaching pace.

[0233] In this embodiment, the dynamic adjustment of teaching strategies is based on a deep analysis of the cognitive state tracking matrix. The set of low-mastery concept nodes refers to knowledge nodes whose mastery index is below a first preset threshold. This first preset threshold is determined by analyzing the mastery distribution of typical learning difficulties in historical teaching data. The cross-modal reinforcement step constructs a multi-dimensional reinforcement teaching combination by increasing the display duration of visual nodes and raising the playback priority of audio nodes, thereby specifically improving students' understanding and memory of low-mastery knowledge points.

[0234] The advanced knowledge extension steps are implemented for high-proficiency concept nodes whose operational proficiency index exceeds a second preset threshold, which is set according to the skill mastery standards required by the teaching syllabus. The implementation of advanced knowledge extension relies on the hierarchical topological characteristics of the knowledge graph, performing multi-hop reasoning along the concept association edges to discover knowledge expansion paths that conform to the laws of cognitive development. Specifically, advanced visual and advanced phonological nodes are acquired by expanding along the association edges of the knowledge graph, including deeper teaching content and more complex application cases. The extension process maintains the core concepts unchanged, only increasing the depth of argumentation and the breadth of application, ensuring the coherence of the knowledge system, and strengthening the teaching combination and advanced teaching to form a differentiated teaching plan, achieving individualized instruction.

[0235] The optimization of the counterfactual reasoning path focuses on improving the immediacy and relevance of error correction. The location of the warning prompt nodes is determined based on the knowledge graph location of the error. Furthermore, the preferred location for embedding the warning prompt nodes is the nearest common ancestor node in the knowledge graph, ensuring the relevance of the prompt content to the root cause of the error. Preferably, the optimization of the comparison display method employs a dual-channel presentation technology, maintaining the integrity of the standard teaching content while highlighting key differences.

[0236] The dynamic adjustment of the teaching pace is based on attention distribution indicators; that is, it is grounded in trend analysis of attention distribution patterns and achieves gradual optimization through the coordinated adjustment of content density and interaction frequency. Specifically, a content simplification strategy is adopted for low-attention areas, preferably using concept decomposition and example-focused strategies. Interactive design is added to high-attention areas, preferably implementing cognitive challenges and transfer training. Furthermore, the above-mentioned dynamic adjustment of the teaching pace can be achieved through a content presentation engine.

[0237] This embodiment achieves precise personalized teaching by dynamically optimizing the teaching strategy through a cognitive state tracking matrix. For concepts with low mastery, cross-modal reinforcement is implemented, constructing a multi-dimensional teaching combination by increasing the display time of associated visual nodes and increasing the playback priority of audio nodes. For concepts with high proficiency, higher-order knowledge is extended, extracting in-depth teaching content along the knowledge graph's related edges. Simultaneously, the counterfactual reasoning path and teaching pace are optimized. Through a differentiated adjustment mechanism based on cognitive state, both solid mastery of basic knowledge and support for the continuous development of higher-order abilities are ensured. This embodiment continuously optimizes the counterfactual reasoning path and teaching pace, forming a progressive teaching loop from assessment to low-mastery concepts, then to the extension of higher-order knowledge, and finally to the optimization of error-prone operations, ensuring that the teaching strategy always maintains an optimal match with the learner's cognitive state.

[0238] In some embodiments, based on a group behavior analysis model, the cognitive state tracking matrix is ​​aggregated, and the teaching path recommendation weights in the multimodal knowledge graph are adjusted to generate a personalized learning navigation graph, including:

[0239] Obtain the cognitive state tracking matrix of multiple users, denoted as the group cognitive state tracking matrix. The group cognitive state tracking matrix includes group concept nodes. Calculate the average mastery index of each group concept node and count high-frequency error patterns and group-related nodes.

[0240] After analyzing the common features of the optimal learning path, hot spots in the distribution of group attention are identified.

[0241] The average mastery index, high-frequency error patterns, and optimal learning path features of the group cognitive state tracking matrix are calculated. The weights for recommending teaching paths in the multimodal knowledge graph are then adjusted based on the hotspots of group attention distribution, including:

[0242] Reduce the weight of basic teaching for conceptual nodes with high group mastery, and increase the weight of counterfactual reasoning for nodes with high group error rates.

[0243] Increase the recommendation priority of related edges in the optimal path of the group, and optimize the resource display weight of the hot spots of group attention;

[0244] Based on the adjusted teaching path recommendation weights, extract subgraph structures related to the current user from the multimodal knowledge graph;

[0245] After recalculating the path score of the teaching path, the key concept nodes on the recommended learning path are marked, and the error-prone node areas that need to be focused on are marked.

[0246] Generate a learning navigation map that includes a main learning path and alternative paths with personalized recommendations, recommended concept nodes, recommended visual nodes, recommended audio node combinations, recommended path weight scores and estimated learning time, and key reinforcement area prompts.

[0247] In this embodiment, the group cognitive state tracking matrix is ​​a group learning feature database formed by aggregating the individual cognitive state tracking matrices of multiple users. Statistical analysis methods are used to extract learning patterns at the group level. Specifically, the average mastery index of group concept nodes reflects the general difficulty level of knowledge points, high-frequency error patterns reveal common cognitive misconceptions, and group association nodes represent the inherent connections between knowledge points. These group feature data provide an objective basis for optimizing teaching paths.

[0248] The analysis of common features of optimal learning paths is achieved by mining cognitive state tracking matrix data of high-achieving users. Specifically, data mining techniques are used to extract common access sequences of high-achieving learners in the knowledge graph. These sequences typically exhibit optimized combinations of specific concept nodes, visual nodes, and phonological nodes. Combined with path evaluation metrics (such as learning efficiency improvement rate and knowledge transfer effect), the node distribution characteristics of universally applicable high-quality paths are selected. Furthermore, by integrating interaction heatmap data from all users, regions that consistently generate high levels of attention within the knowledge graph are identified as hotspots of group attention distribution. These ultimately identified hotspots of group attention distribution usually correspond to core concept groups or typical application scenarios within the knowledge system, providing crucial information for the subsequent optimization and allocation of teaching resources.

[0249] The process of adjusting the weights of the teaching path recommendations takes into account multiple factors such as group mastery, error patterns, and attention distribution, to ensure that the recommendation strategy not only conforms to general learning patterns but also optimizes for common problems.

[0250] The adjustment process for the recommended weights of teaching paths employs a multi-dimensional group learning characteristic comprehensive analysis mechanism. Specifically, it identifies the knowledge difficulty gradient by calculating the average mastery index of the group, appropriately simplifying basic teaching content for nodes with high mastery; it analyzes high-frequency error patterns to identify cognitive weaknesses, and specifically enhances the display intensity of counterfactual reasoning paths. Simultaneously, it improves the recommendation priority of core knowledge links based on the correlation edge features of the optimal learning path, and optimizes the spatial layout weight of teaching resources according to the group's attention hotspots. By dynamically labeling key concept nodes and error-prone areas, it forms a visual learning guidance scheme that is clear in its priorities and highlights key points.

[0251] The generation process of the personalized learning navigation graph employs a hierarchical recommendation strategy. The main learning path is generated based on adjusted recommendation weights, prioritizing coverage of core knowledge points; alternative paths provide alternatives for different learning styles. The combination of recommended concept nodes, recommended visual nodes, and recommended audio nodes follows multimodal matching principles to ensure the coordinated presentation of various teaching resources. The recommended path weight score is calculated by comprehensively considering knowledge importance, learning difficulty, and user suitability, while the estimated learning time is estimated based on the average time spent in historical learning data. By adjusting the teaching path recommendation weights in the multimodal knowledge graph in the above manner, the topological structure of the knowledge graph is preserved while incorporating group learning behavior characteristics, enabling the final generated learning navigation graph to intelligently balance the integrity of the knowledge system with the personalized needs of the learning path.

[0252] This embodiment organically combines collective intelligence with personalized learning. For example, when most students make frequent errors at a certain point, the system automatically increases the recommendation weight of related counterfactual reasoning paths; for a concept point that the group has a good grasp of, the display intensity of basic teaching resources is appropriately reduced. By dynamically optimizing the recommendation weight of teaching paths and marking key concept points and error-prone areas, the generated learning navigation map can reflect collective learning experience while meeting individual differentiated needs, effectively improving learning efficiency and quality.

[0253] In a second aspect, this embodiment also provides a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the method described in the first aspect.

[0254] The computer program involved in this embodiment can be stored in a computer device readable storage medium, which includes, but is not limited to, disks, magnetic tapes, magnetic cards, floppy disks, flash memory, optical disks, optical cards, read-only memory (ROM), random access memory (RAM), erasable programmable ROM (EPROM), and electrically erasable programmable ROM (EEPROM), etc. It also includes other biological, physical, or chemical structures capable of performing similar or equivalent functions to the storage media listed above, such as DNA, RNA, proteins, and other units with information storage capabilities. In specific embodiments, the storage medium involved can be one of the above-mentioned media types or a combination of the above media types. In different embodiments, the computer program involved in the embodiment can be centrally stored in a single medium or distributed across multiple media. The memory containing the computer device readable storage medium can be non-volatile memory or random access memory. These computer device readable storage media can be built into the device or connected to the device involved in the embodiment as an external device or part of an external device. In some embodiments, the memory having a computer device readable storage medium is deployed locally; in other embodiments, the memory may be deployed remotely from the processor, for example, as a network-attached memory accessed via RF circuitry or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, a local area network (LAN), a wide area network (WLAN), a storage area network (SAN), or a suitable combination thereof, as long as computer device access to the memory is enabled. Furthermore, the computer program involved in the embodiments may be stored in plaintext / ciphertext form, or it may be designed as training data, integrated and recombined through model training and implicitly stored in the parameter states of a deep neural network or other machine learning model.

[0255] Please see Figure 5 In a third aspect, this embodiment also provides an electronic device 1, including a memory 11 and a processor 12, wherein the memory 11 is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor 12 to implement the method described in the first aspect.

[0256] The processor described in this embodiment can be implemented by hardware, firmware, software, or a combination thereof. It can be a circuit, one or more of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, or a microprocessor. It also includes other physical, biological, or chemical structures that can implement the same or equivalent functions as the processors listed above, such as biological neurons, quantum computing units, DNA computing units, etc., so that the processor can execute some or all of the steps in the computer program or method involved in the various embodiments of this application, or any combination of the steps mentioned therein.

[0257] Unlike existing technologies, the above technical solution has the following beneficial effects:

[0258] By constructing a multimodal knowledge graph, the system achieves deep integration and intelligent organization of teaching resources. It dynamically generates progressive and comparative teaching content based on a three-dimensional teaching context subgraph, accurately assesses learning outcomes using a cognitive state tracking matrix, and optimizes teaching strategies, ultimately forming a closed-loop adaptive intelligent teaching system. The method ensures consistency of heterogeneous teaching resources through cross-modal semantic alignment, achieves multimodal fusion of knowledge representation using graph neural networks, generates accurate knowledge graph query vectors based on user interaction data, and optimizes personalized learning navigation graphs based on group behavior analysis. This enables the system to implement differentiated teaching for different cognitive states, strengthen cross-modal presentation of low-mastery concept nodes, extend higher-order knowledge for high-proficiency operation sequences, and continuously optimize counterfactual reasoning paths and teaching pace, achieving a complete teaching closed loop from knowledge presentation to learning assessment and strategy adjustment. The above technical solutions effectively improve the accuracy, adaptability, and personalization of online teaching, ensuring the systematic integrity of the knowledge system while effectively intervening in learning misconceptions, providing reliable technical support for intelligent education.

[0259] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.

Claims

1. An online teaching interaction method based on multimodal knowledge graphs, characterized in that, include: Collect raw teaching information, perform cross-modal semantic alignment processing on the raw teaching information, and obtain structured teaching information; A multimodal knowledge graph is constructed, and textual concept entities, video keyframe feature vectors, and transcribed speech text content are deeply fused through graph neural networks to obtain cross-modal semantic association edges, which include concept-visual feature association edges, concept-speech segment association edges, and cross-modal similarity association edges. Receive user interaction behavior data, which includes voice question content, virtual teaching operation sequence and interface click trajectory, and convert user interaction behavior data into knowledge graph query vectors; Based on the query vector of the knowledge graph, node matching is performed in the multimodal knowledge graph to locate the initial query node. Graph traversal is performed along the concept-visual association edge and the concept-speech association edge to extract the associated visual query node and speech query node. Spatiotemporal association relationship is established according to the visual-speech temporal edge to form a three-dimensional teaching context subgraph. The three-dimensional teaching context subgraph includes concept nodes, association edge weights, and cross-modal resource links. The three-dimensional teaching context subgraph extraction process adopts an attention sampling algorithm based on query vector to dynamically construct a knowledge subnetwork related to the current teaching context. When a standard teaching operation is detected, the forward reasoning path of the three-dimensional teaching context subgraph is activated to generate progressive teaching content. Specifically, this includes: starting from the initial query node, reasoning along the standard teaching path in the knowledge graph, combining related concept query nodes, visual query nodes, and voice query nodes according to the teaching logic order, and generating a progressive teaching sequence that includes explanations of basic concepts, example demonstrations, and voice guidance. When an erroneous teaching operation is detected, the counterfactual reasoning mechanism of the three-dimensional teaching context subgraph is triggered to generate comparative teaching content. Specifically, this includes: identifying the abnormal node of the knowledge graph corresponding to the erroneous operation, disabling the standard teaching path edge directly associated with the abnormal node, activating the counterfactual reasoning edge, and generating teaching content that includes error analysis, correct demonstration and comparative explanation. Real-time recording of user interaction behavior data in the three-dimensional teaching context sub-graph, denoted as context interaction data, matching the mapping relationship between the context interaction data and the multimodal knowledge graph, and constructing a cognitive state tracking matrix; Teaching strategies are dynamically adjusted based on the cognitive state tracking matrix to strengthen cross-modal presentation combinations of low-mastery concept nodes and extend higher-order knowledge associated with high-proficiency operation sequences. Furthermore, based on the group behavior analysis model, the cognitive state tracking matrix is ​​aggregated, the teaching path recommendation weights in the multimodal knowledge graph are adjusted, and a personalized learning navigation graph is generated. The learning navigation graph includes optimal learning path planning and multimodal resource recommendation schemes.

2. The online teaching interaction method based on multimodal knowledge graphs according to claim 1, characterized in that, The original teaching information includes textbook text information, teaching video information, and classroom audio information; Raw teaching information is collected and subjected to cross-modal semantic alignment processing to obtain structured teaching information, including: Obtain the text information of the teaching materials, extract the teaching concept entities from it through named entity recognition, and generate a concept feature vector by word embedding encoding for each teaching concept entity; In addition, the teaching video information is obtained, and the video stream is sampled for keyframes to obtain a video keyframe sequence, wherein the video keyframe sequence includes multiple first video keyframes; In addition, classroom audio information is acquired, and the audio stream is segmented to obtain an audio segment set, which includes multiple first audio segments; Constructing a cross-modal semantic alignment model includes: Calculate the first attention weight between the concept feature vector and the visual feature vector of all video keyframes, and calculate the second attention weight between the concept feature vector and the acoustic feature vector of all speech segments; Based on the first attention weight and the second attention weight, video keyframes and audio segments that are strongly associated with the teaching concept entities are selected and denoted as the second video keyframe and the second audio segment. Structured teaching information is generated based on the cross-modal semantic alignment model. The structured teaching information includes teaching concept entities and concept feature vectors, second video keyframes and first attention weights, second speech segments and second attention weights, visual association between teaching concept entities and second video keyframes, and speech association between teaching concept entities and second speech segments.

3. The online teaching interaction method based on multimodal knowledge graphs according to claim 2, characterized in that, A multimodal knowledge graph is constructed by deeply fusing textual concept entities, video keyframe feature vectors, and transcribed speech text content using a graph neural network to obtain cross-modal semantic association edges, including: An initial node set is constructed for a multimodal knowledge graph, the initial node set including: The teaching concept entity is used as the concept node, and the node feature is used as the concept feature vector; The second video keyframe is used as a visual node, and the node features are used as visual feature vectors. The second speech segment is taken as a speech node, and the node features are speech feature vectors; Establishing cross-modal semantic association edges includes: Based on the first attention weight value of the concept node, a concept-visual association edge is established with the visual node, and the weight value of the concept-visual association edge is the first attention weight after normalization. Based on the second attention weight value of the concept node, a concept-speech association edge is established with the speech node, and the weight value of the concept-speech association edge is the normalized second attention weight; For visual nodes and audio nodes that have a temporal correspondence, a visual-audio temporal edge is established. The weight value of the visual-audio temporal edge is calculated based on the temporal overlap between the second video keyframe and the second audio segment. Deep fusion of multimodal features using graph neural networks includes: Input the initial set of nodes into the graph attention network; Feature propagation is performed along concept-visual association edges and concept-speech association edges, and cross-modal similarity scores between nodes are calculated through a multi-head attention mechanism; The cross-modal feature representation of the initial node set is updated based on the cross-modal similarity score to generate knowledge graph nodes with fusion features; The final multimodal knowledge graph is obtained.

4. The online teaching interaction method based on multimodal knowledge graphs according to claim 1, characterized in that, Convert user interaction behavior data into knowledge graph query vectors, including: The voice query content is converted into query text by a speech recognition engine; Use named entity recognition to extract core concept entities from query text; The core concept entities are mapped to concept nodes in the multimodal knowledge graph, and the feature vectors of the successfully mapped concept nodes are used as the text query basis vectors. Furthermore, the system analyzes the operation objects and actions in the virtual teaching operation sequence to obtain the analysis results; Based on the analysis results, the corresponding teaching process node is located in the multimodal knowledge graph, and the feature vector of the teaching process node is used as the operation query basis vector. Additionally, the feature vectors of knowledge graph nodes associated with the click areas of the interface click trajectory are identified, denoted as click feature vectors, and popularity weights are generated based on click frequency. Generate a click query basis vector based on the popularity weight and the click feature vector; The text query basis vector, operation query basis vector, and click query basis vector are weighted and fused to obtain a fused feature vector. The fused feature vector is then input into the query encoder to generate the final knowledge graph query vector.

5. The online teaching interaction method based on multimodal knowledge graphs according to claim 1, characterized in that, The contextual interaction data includes the access frequency and dwell time of concept query nodes, the viewing completeness of visual query nodes, the number of times voice query nodes are repeated, and the selection ratio of standard teaching path and counterfactual reasoning path. Matching the mapping relationship between the contextual interaction data and the multimodal knowledge graph to construct a cognitive state tracking matrix includes: An initial cognitive state tracking matrix is ​​established, wherein the rows of the initial cognitive state tracking matrix represent concept nodes in the multimodal knowledge graph, and the columns include proficiency indicators, operational proficiency indicators, error pattern indicators, and attention distribution indicators; Map the access records of concept query nodes to concept nodes, calculate the average test accuracy and access time scores of concept nodes, and obtain the proficiency index. The viewing records of visual query nodes are mapped to visual nodes, and the success rate and average time of operation of teaching process nodes are statistically analyzed to obtain the operation proficiency index. The teaching path selection records are mapped to the standard teaching path or the counterfactual reasoning path. The number of times the counterfactual reasoning path is triggered and the degree of improvement in subsequent operations are recorded to obtain the error pattern index. The playback records of the voice query node are mapped to the voice node, and the attention distribution index is obtained based on the frequency of interaction events of the voice node, concept node, and visual node within a unit of time. The proficiency index, operational proficiency index, error pattern index, and attention distribution index are updated to the columns of the initial cognitive state tracking matrix to obtain the updated cognitive state tracking matrix.

6. The online teaching interaction method based on multimodal knowledge graph according to claim 5, characterized in that, Teaching strategies are dynamically adjusted based on the cognitive state tracking matrix to strengthen cross-modal presentation combinations of low-mastery concept nodes and extend higher-order knowledge associated with high-proficiency operation sequences, including: Extract a set of low-mastery concept nodes whose proficiency index is below a first preset threshold from the cognitive state tracking matrix; Perform a cross-modal reinforcement step for each low-mastery concept node in the set of low-mastery concept nodes: Increase the display duration and frequency of visual nodes associated with low-level concept nodes; Increase the playback priority of voice nodes associated with low-level concept nodes; Generate a reinforced instructional mix that includes text explanations, visual demonstrations, and voice guidance; Extract a set of highly proficient concept nodes whose operational proficiency index is higher than a second preset threshold from the cognitive state tracking matrix; Perform a higher-order knowledge extension step for each high-proficiency concept node in the set of high-proficiency concept nodes: Extract higher-order visual nodes along the concept-visual association edge; Extract higher-order speech nodes along the concept-speech association edge; Generate advanced teaching combinations that include advanced knowledge explanations, in-depth case analyses, and extended exercises; Based on the error pattern indicators in the cognitive state tracking matrix, adjust the triggering conditions of the counterfactual reasoning path, including: Add warning notification nodes for high-frequency error patterns; Optimize the way incorrect operations are displayed compared with correct examples; Based on the attention distribution indicators in the cognitive state tracking matrix, the teaching pace is dynamically adjusted, including: Simplify teaching content for areas of low attention; Add interactive elements to areas of high attention; The existing teaching strategy configuration is updated to obtain an updated teaching configuration, which is then executed. The updated teaching configuration includes a reinforcement teaching plan for low-mastery concept nodes, a higher-order extension plan for high-proficiency concept nodes, optimization parameters for counterfactual reasoning paths, and adjustment parameters for teaching pace.

7. The online teaching interaction method based on multimodal knowledge graphs according to claim 1, characterized in that, Based on the group behavior analysis model, the cognitive state tracking matrix is ​​aggregated, and the teaching path recommendation weights in the multimodal knowledge graph are adjusted to generate a personalized learning navigation graph, including: Obtain the cognitive state tracking matrix of multiple users, denoted as the group cognitive state tracking matrix. The group cognitive state tracking matrix includes group concept nodes. Calculate the average mastery index of each group concept node and count high-frequency error patterns and group-related nodes. After analyzing the common features of the optimal learning path, hot spots in the distribution of group attention are identified. The average mastery index, high-frequency error patterns, and optimal learning path features of the group cognitive state tracking matrix are calculated. The weights for recommending teaching paths in the multimodal knowledge graph are then adjusted based on the hotspots of group attention distribution, including: Reduce the weight of basic teaching for conceptual nodes with high group mastery, and increase the weight of counterfactual reasoning for nodes with high group error rates. Increase the recommendation priority of related edges in the optimal path of the group, and optimize the resource display weight of the hot spots of group attention; Based on the adjusted teaching path recommendation weights, extract subgraph structures related to the current user from the multimodal knowledge graph; After recalculating the path score of the teaching path, the key concept nodes on the recommended learning path are marked, and the error-prone node areas that need to be focused on are marked. Generate a learning navigation map that includes a main learning path and alternative paths with personalized recommendations, recommended concept nodes, recommended visual nodes, recommended audio node combinations, recommended path weight scores and expected learning duration, and key reinforcement area prompts.

8. A computer-readable storage medium storing computer program instructions thereon, characterized in that, The computer program instructions, when executed by a processor, implement the method as described in any one of claims 1 to 7.

9. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1 to 7.

Images