Teaching and learning knowledge graph and mind mapping construction system based on multi-modal digital intelligence

Abstract

The invention discloses a teaching and learning knowledge graph and mind mapping construction system based on multi-modal digital intelligence, relates to the technical field of education informatization, and is used for solving the problem of knowledge logic extraction and visualization of teaching multi-modal data. The method comprises the following steps: firstly, establishing a space-time three-dimensional coordinate system, locking an effective time period by utilizing the interaction density of writing actions and voice accent, and intercepting an audio-visual synchronous data packet; then, geometric coordinates and indentation features of the text entities are analyzed, and an initial structured entity tree representing the hierarchical relation of the static space is constructed; the analyzed teaching operator constructs a cross-space implicit association edge through a logic tension coefficient and executes reverse pruning to generate a dynamic semantic topology network; and finally, executing a minimum spanning tree algorithm based on the multi-modal reproduction weighted value, reserving a backbone path, converting the backbone path into hierarchical tree data, and generating a mind map, thereby realizing intelligent construction from unstructured teaching field data to a structured knowledge graph.

Description

Technical Field

[0001] This invention relates to the field of educational informatization technology, specifically to a teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence. Background Technology

[0002] With the explosive growth of digital educational resources, online teaching platforms and smart classrooms have accumulated massive amounts of teaching video data. When faced with these unstructured video streams, learners often struggle to quickly grasp the core knowledge system, requiring significant time for manual analysis. Simultaneously, educators lack automated tools to transform fragmented classroom recordings into structured knowledge assets when reflecting on their teaching and reorganizing resources. How to automatically extract knowledge logic from multimodal teaching data, construct clear knowledge graphs, and generate mind maps to aid cognition has become a pressing common need in the field of educational technology.

[0003] Existing knowledge graph construction methods suffer from significant shortcomings when processing teaching scenario data. First, current technologies typically treat blackboard images and teacher audio explanations in videos as independent data streams, ignoring the strong temporal correlation between writing actions and vocal stress. This leads to the incorrect input of numerous invalid blackboard drafts or meaningless background images, increasing the data processing load and reducing the purity of knowledge entities. Second, traditional layout analysis techniques rely heavily on the linear arrangement of text, failing to effectively analyze the common non-linear spatial layouts in teaching blackboards. This results in the misclassification of logically related but physically distant knowledge points as unrelated, or the misclassification of parallel knowledge points at the same level as having a parent-child relationship. Furthermore, existing systems lack the ability to understand causal deductions and logical inversions in speech semantics. The generated graphs are often static text stacks, failing to reflect the dynamically generated implicit logical connections and real-time error correction processes during teaching. The resulting mind maps are overly complex and lack a logical thread, failing to effectively reduce learners' cognitive load. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence, thus solving the problems mentioned above.

[0005] To achieve the above objectives, this invention employs the following technical solution: a teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence, comprising the following modules: a trajectory audiovisual alignment module, used to establish a spatiotemporal three-dimensional coordinate system in the teaching scenario, synchronously collect the dynamic trajectory flow of the teacher's writing strokes and the prosodic features of the speech signal, calculate the interaction density of writing actions and speech stress on the time axis, and lock the time period when the interaction density exceeds a threshold to extract the audiovisual synchronous data package containing the complete handwriting generation process and the corresponding explanation audio; a layout semantic parsing module, used to perform optical character recognition to extract discrete text entities from the audiovisual synchronous data package, analyze the geometric coordinates of the text entities on the blackboard plane, font indentation, and writing area clustering features, and construct a representation of static spatial hierarchical relationships. The initial structured entity tree; the causal operator injection module is used to parse the audio explanation to extract teaching operator vectors containing derivation and inversion attributes, project them onto the node space of the initial structured entity tree, calculate the logical tension coefficient between the speech logic and the blackboard layout, and when the logical tension coefficient exceeds the preset threshold, implicit association edges are forcibly constructed between non-adjacent nodes, and reverse pruning is performed on the spatial hierarchy of the initial structured entity tree according to the inversion operator to generate a dynamic semantic topology network; the topology dimensionality reduction and rendering module is used to calculate the recurrence weight and centrality of each node in the dynamic semantic topology network in the multimodal data stream, execute the minimum spanning tree algorithm to retain the backbone paths with the highest weight, and convert the complex network topology into hierarchical tree-like visual data to generate a mind map.

[0006] Furthermore, the specific process of establishing a spatiotemporal three-dimensional coordinate system in the teaching scenario, simultaneously collecting the dynamic trajectory flow of the teacher's writing strokes and the prosodic features of the speech signal, and calculating the interaction density of writing actions and speech stress on the time axis is as follows: The normal vector and depth threshold of the blackboard plane are calibrated using a depth vision sensor, constructing a three-dimensional Euclidean space with the upper left corner of the blackboard plane as the origin, and tracking the spatial displacement of key points of the teacher's hand in real time. By calculating the displacement velocity along the tangential direction and the normal pressing depth of the key points within a unit time window, a dynamic curve of writing force is generated. Simultaneously, a short-time Fourier transform is performed on the collected speech signal to extract the fundamental frequency trajectory in the frequency domain and the short-time energy amplitude in the time domain, and the two are weighted and fused to generate a speech emphasis envelope. Normalization processing is performed on the dynamic curve of writing force and the speech emphasis envelope, and the covariance of the two curves on the same time axis is calculated using a sliding window cross-correlation algorithm. This covariance is used as the interaction density value characterizing the degree of synchronous excitement of visual and auditory perception.

[0007] Furthermore, the specific process of capturing the audiovisual synchronized data packet containing the complete handwriting generation process and corresponding audio explanation during the period when the interaction density exceeds the threshold is as follows: Real-time monitoring of the rate of change of the interaction density value, identifying the time point when the rising edge of the interaction density curve crosses the preset trigger threshold as the candidate start frame, and the time point when the falling edge crosses the trigger threshold as the candidate end frame; Based on the candidate start frame, tracing back the dynamic curve of writing force along the time axis to locate the most recent pen stroke when the speed is zero and the normal depth is detached from the blackboard plane, and correcting the candidate start frame; Based on the candidate end frame, extending the speech emphasis envelope along the time axis in the forward direction to locate the most recent semantic pause when the short-term energy amplitude is lower than the silence threshold, and correcting the candidate end frame. The final extraction time window is determined based on the corrected pen stroke time and semantic pause time. Image sequences and audio segments within the time window are extracted from the original multimodal stream and encapsulated into audiovisual synchronization data packets.

[0008] Furthermore, the specific process of performing optical character recognition to extract discrete text entities from the audiovisual synchronization data packet, and analyzing the geometric coordinates, font indentation, and writing area clustering features of the text entities on the writing plane is as follows: Adaptive binarization and connected component analysis are performed on the image sequence in the audiovisual synchronization data packet to extract the minimum bounding rectangle of each independent character, and the characters are merged into text line entities according to the horizontal spacing and vertical overlap rate of adjacent rectangles; the two-dimensional plane coordinates of the geometric center of each text line entity are calculated, and the horizontal Euclidean distance from the leftmost pixel of the text line entity to the baseline is calculated as the font indentation; the average pixel height of the bounding rectangle of the text line entity is calculated as the font size feature, and the two-dimensional plane coordinates of all text line entities are clustered using a density-based spatial clustering algorithm, and text line entities with a spatial distance less than a preset paragraph threshold are classified into the same writing area cluster, and the centroid coordinates of each cluster are calculated.

[0009] Furthermore, the specific process of constructing the initial structured entity tree representing the static spatial hierarchy is as follows: traverse all writing area clusters, select the region whose centroid coordinates are located at the top of the writing plane and whose text line entity within the cluster has the largest font size feature, and define it as the root node region; establish a hierarchy mapping rule based on font indentation and font size features, and recursively traverse the text line entities in non-root node regions: if the font size of the current entity is smaller than that of the parent node and the font indentation is greater than that of the parent node, then create a parent-child connection and mount the current entity as a child node of the parent node; if the font size of the current entity is at the same level as that of the parent node and the difference in font indentation is within a preset tolerance range, then create a sibling connection and mount the current entity as a parallel node of the parent node; if the current entity is located at the edge of the writing area cluster and has no clear parent-child indentation feature, then calculate its semantic vector similarity with the nearest node, and only mark it as an isolated note node when the similarity is lower than the threshold, generating an initial structured entity tree containing the root node, child nodes, sibling nodes, and note nodes.

[0010] Furthermore, the specific process of extracting teaching operator vectors containing derivation and reversal attributes from the audio explanation, projecting them onto the node space of the initial structured entity tree, and calculating the logical tension coefficient between the speech logic and the blackboard layout is as follows: Semantic role labeling and dependency parsing are performed on the audio explanation; keywords representing causal progression are identified to generate forward derivation operators, and keywords representing negation, transition, or modification are identified to generate reverse reversal operators. These two types of operators are encapsulated into teaching operator vectors. Using the timestamp index of the audio-visual synchronization data packet, the teaching operator vectors are mapped to the target nodes in the initial structured entity tree corresponding to their occurrence time period, establishing a temporal mapping relationship between speech instructions and blackboard nodes. The Euclidean spatial distance between the target node and the semantically pointed node on the blackboard plane is calculated, along with the cosine similarity between the two in the pre-trained semantic vector space. A weighted fusion calculation is performed on the cosine similarity and Euclidean spatial distance to obtain the logical tension coefficient, which represents the deviation ratio between the semantic association strength and the physical spatial adjacency.

[0011] Furthermore, when the logical tension coefficient exceeds a preset crossing threshold, implicit connection edges are forcibly constructed between non-adjacent nodes, and reverse pruning is performed on the spatial hierarchy of the initial structured entity tree according to the inversion operator. The specific process for generating a dynamic semantic topology network is as follows: traverse all node pairs in the initial structured entity tree. When the logical tension coefficient exceeds the preset crossing threshold, it is determined that there is a strong logical connection across the physical layout. Directed implicit connection edges are directly established between two spatially non-adjacent nodes, and the edges are assigned semantic connection attributes with the logical tension coefficient as the weight. When the inverse operator is detected to be mapped to a target node, the parent node connection of the node in the initial structured entity tree is retrieved. If the semantic pointer of the inverse operator is logically mutually exclusive with the current parent-child hierarchy, a logical blocking operation is performed to cut off the physical connection edge between the node and the original parent node. The node after the connection is cut off is reattached to the new target node pointed to by the semantic pointer of the inverse operator, or it is downgraded to an independent free node. The adjacency matrix between all nodes is updated, thereby reconstructing the static tree structure into a dynamic semantic topology network containing explicit physical edges and implicit logical edges.

[0012] Furthermore, the specific process of calculating the recurrence weights and centralities of each node in the dynamic semantic topology network in the multimodal data stream, and executing the minimum spanning tree algorithm to retain the backbone paths with the highest weights, is as follows: The cumulative pixel duration of each node in the image sequence and the keyword recurrence frequency in the speech signal are statistically analyzed, and the multimodal recurrence weights of each node are calculated by weighted summation; based on the adjacency matrix of the dynamic semantic topology network, the degree centrality and betweenness centrality of each node are calculated using graph theory algorithms to characterize the hub status of the node in the knowledge transfer network; the multimodal recurrence weights and centrality indices are fused to generate a comprehensive importance score, and the reciprocal of the comprehensive importance score is defined as the traversal cost of the edges in the dynamic semantic topology network; taking the node with the highest comprehensive importance score as the starting root node, the minimum spanning tree algorithm is executed to traverse the dynamic semantic topology network, prioritizing the retention of the connected paths with the lowest traversal cost while eliminating network loops, removing redundant weakly related paths, and generating a core backbone path set.

[0013] Furthermore, the process of converting the complex network topology into hierarchical tree-like visual data to generate a mind map is as follows: A hierarchical traversal of the core backbone path set is performed, and nodes located on the main path are marked as mind maps. Figure 1Secondary branches are marked as secondary branches directly connected to the main node; a cognitive load aggregation mechanism is introduced to calculate the sum of the comprehensive importance scores of all child nodes under the same parent node. If the sum of scores is lower than the preset cognitive display threshold, the child nodes in this group are automatically encapsulated into a collapsed attribute list, retaining only the direct display of the parent node; the processed node hierarchy is converted into a standard tree data structure, and the original whiteboard screenshots and audio clips corresponding to the nodes are embedded in the metadata fields of the data structure as interactive multimedia indexes to render and generate a visual mind map.

[0014] The present invention has the following beneficial effects: (1) The teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence effectively solves the problems of information redundancy and structural misalignment caused by heterogeneous multimodal data through the collaborative work of the trajectory audiovisual alignment module and the page semantic parsing module. The system uses the interaction density of writing actions and speech stress as the filtering basis to accurately lock high-value teaching time periods and eliminate invalid blackboard writing processes and redundant backgrounds from the source. At the same time, by analyzing the geometric coordinates of text entities on the blackboard plane and the font indentation, the system can break through the limitations of linear text recognition and automatically construct an initial structured entity tree that conforms to the teaching blackboard writing logic based on the physical spatial distribution characteristics, ensuring the accuracy of knowledge extraction and the physical authenticity of hierarchical relationships.

[0015] (2) The teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence solves the problems of lack of deep logical connections and chaotic visualization effects in existing graphs through the deep processing of the causal operator injection module and the topology dimensionality reduction rendering module. The system projects the derivation and inversion operators in speech onto the static entity space, calculates the logical tension coefficient, and thus establishes implicit strong logical connections between physically non-adjacent nodes, and dynamically corrects the erroneous hierarchical structure according to speech commands. Furthermore, by calculating the multimodal reproduction weights and centrality of nodes and executing the minimum spanning tree algorithm, the system can transform complex mesh semantic networks into hierarchical tree structures, automatically retain the core backbone paths and fold secondary branches, and generate mind maps that conform to human cognitive laws, significantly improving the efficiency of knowledge transfer.

[0016] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0017] Figure 1 This is a flowchart of the teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence, as described in this invention. Detailed Implementation

[0018] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0019] This application's embodiments address the problems of low accuracy in knowledge extraction, lack of logical connections, and excessive cognitive load in visualization results in complex teaching scenarios by constructing a teaching and learning knowledge graph and mind map system based on multimodal digital intelligence.

[0020] The overall approach of the scheme in this application is as follows: First, a spatiotemporal three-dimensional coordinate system is established, and the effective teaching time period is locked by the interaction density of writing actions and speech signals. Audiovisual synchronous data packets are captured to achieve accurate data alignment. Second, spatial geometric analysis is performed on the extracted text entities, and an initial structured entity tree reflecting the physical blackboard layout is constructed based on coordinates and indentation features. Third, causal and inversion operators in the speech are analyzed, and implicit related edges are constructed or reverse pruning is performed in the entity tree by calculating the logical tension coefficient, forming a dynamic semantic topology network. Finally, based on the reproduction weights and centrality of multimodal data, the minimum spanning tree algorithm is used to optimize the structure and reduce the dimension of the topology network, generating a visual mind map that highlights the core path.

[0021] Please see Figure 1 This invention provides a technical solution: a teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence, comprising the following modules: a trajectory audiovisual alignment module, used to establish a spatiotemporal three-dimensional coordinate system in the teaching scenario, synchronously collect the dynamic trajectory flow of the teacher's writing strokes and the prosodic features of the speech signal, calculate the interaction density of writing actions and speech stress on the time axis, and lock the time period when the interaction density exceeds a threshold to extract the audiovisual synchronous data package containing the complete handwriting generation process and the corresponding explanation audio; a layout semantic parsing module, used to perform optical character recognition to extract discrete text entities from the audiovisual synchronous data package, analyze the geometric coordinates, font indentation, and writing area clustering features of the text entities on the blackboard plane, and construct an initial structure representing the static spatial hierarchy. The system includes a structured entity tree and a causal operator injection module. The causal operator injection module parses the audio explanations to extract teaching operator vectors containing derivation and inversion attributes, projects them onto the node space of the initial structured entity tree, calculates the logical tension coefficient between the speech logic and the blackboard layout, and forcibly constructs implicit association edges between non-adjacent nodes when the logical tension coefficient exceeds a preset threshold. It also performs reverse pruning on the spatial hierarchy of the initial structured entity tree based on the inversion operator, generating a dynamic semantic topology network. The topology dimensionality reduction and rendering module calculates the recurrence weights and centrality of each node in the dynamic semantic topology network within the multimodal data stream, executes the minimum spanning tree algorithm to retain the top-ranked backbone paths, and transforms the complex network topology into hierarchical tree-like visual data, generating a mind map.

[0022] In this implementation plan, the trajectory audiovisual alignment module is mainly used to achieve precise spatiotemporal synchronization and noise reduction of teaching data. This module establishes a unified spatiotemporal three-dimensional coordinate system to bidirectionally lock the teacher's writing actions in physical space with the speech rhythm on the time axis. In this process, interaction density is a quantitative indicator used to characterize the frequency of simultaneous occurrence of the intensity of writing actions and the energy amplitude of speech stress within the same unit of time. By monitoring this indicator, the system can effectively distinguish between invalid silent writing or casual conversation without blackboard writing, locking only effective teaching periods with interaction density exceeding a threshold. This allows for the extraction of audiovisual synchronization data packets containing the complete handwriting generation process and corresponding audio explanations, providing a high signal-to-noise ratio multimodal data source for subsequent processing. The layout semantic parsing module is mainly used to transform unstructured image data into structured data with physical spatial logic. This module performs optical character recognition on the extracted audiovisual synchronization data packets, extracting discrete text entities and focusing on analyzing the geometric position and layout features of these entities on the blackboard plane. The initial structured entity tree refers to the preliminary logical framework constructed solely based on the physical layout rules of the blackboard, such as font size representing hierarchy, left indentation representing subordination, and top position representing precedence. This step leverages the inherent spatial attributes of the blackboard to quickly establish static spatial hierarchical relationships between knowledge points, transforming the planar blackboard image into a computer-traversable node tree structure. The causal operator injection module primarily utilizes speech semantics to logically reconstruct and correct the static spatial structure. This module parses teaching operator vectors containing derivation and inversion attributes from the audio explanation and projects them into the entity tree. Here, the logical tension coefficient is a metric used to quantify the difference between the strength of semantic associations emphasized in the speech and the physical spatial distances on the blackboard. When this coefficient exceeds a preset threshold, it indicates that the speech is emphasizing a strong association between two physically distant knowledge points, and the system accordingly forces the construction of implicit association edges between non-adjacent nodes. Simultaneously, the system utilizes inversion operators (such as instructions indicating negation or correction) to perform reverse pruning on the initial structured entity tree, i.e., deleting or modifying hierarchical relationships established based on spatial location errors, thereby generating a dynamic semantic topology network that conforms to the logic of real teaching. The topology dimensionality reduction rendering module is mainly used to transform complex semantic networks into visualization results that conform to human cognitive habits. This module calculates the multimodal features of each node in the dynamic semantic topology network, where the recurrence weighting value refers to the combined statistical measure of the node's dwell time on the whiteboard and the frequency of its verbal mention, used to measure the importance of knowledge points. The system uses the minimum spanning tree algorithm to retain the backbone paths with the highest weighting values ​​and remove redundant weak connections. This process realizes the transformation from a network to a tree-like topology structure, simplifying the intricate multidimensional semantic relationships into a hierarchical tree structure, ultimately generating a mind map that highlights the core teaching path and allows secondary information to be collapsed, effectively reducing the user's cognitive load.

[0023] Specifically, the process of establishing a spatiotemporal three-dimensional coordinate system in the teaching scenario, synchronously collecting the dynamic trajectory flow of the teacher's writing strokes and the prosodic features of the speech signal, and calculating the interaction density of writing actions and speech stress on the time axis is as follows: The normal vector and depth threshold of the blackboard plane are calibrated using a depth vision sensor, and a three-dimensional Euclidean space with the upper left corner of the blackboard plane as the origin is constructed. The spatial displacement of key points of the teacher's hand is tracked in real time. By calculating the displacement velocity along the tangential direction and the normal pressing depth of the key points along the blackboard plane within a unit time window, a dynamic curve of writing force is generated. Simultaneously, a short-time Fourier transform is performed on the collected speech signal to extract the fundamental frequency trajectory in the frequency domain and the short-time energy amplitude in the time domain. The two are weighted and fused to generate a speech emphasis envelope. Normalization processing is performed on the dynamic curve of writing force and the speech emphasis envelope. The covariance of the two curves on the same time axis is calculated using a sliding window cross-correlation algorithm. This covariance is used as the interaction density value characterizing the degree of synchronous excitement of visual and auditory perception.

[0024] In this implementation plan, a standardized three-dimensional Euclidean space is first established using a depth vision sensor. This step decouples the teacher's free hand movements in the air into tangential movements along the blackboard surface and normal movements perpendicular to the blackboard by calibrating the normal vector of the writing plane, thereby accurately capturing the writing intention. For generating the dynamic curve of writing pressure, the system does not simply record the position, but comprehensively considers the writing speed and the depth of pressure, because during teaching, teachers often use faster pen strokes or deeper chalk pressure when writing key content. The system calculates the timing based on the following formula. dynamic value of writing force ;in, :time The tangential displacement velocity of key hand points along the plane of the blackboard; :time The normal pressing depth of the hand at key points is perpendicular to the plane of the blackboard; : The preset maximum writing speed reference value, used for normalization; : Preset maximum pressing depth reference value; The weighting coefficient of tangential velocity to writing pressure ranges from 0.4 to 0.6; The weighting coefficient of normal depth on writing pressure, and satisfying This formula quantifies the intensity of the writing action through weighted summation. Simultaneously, for speech signal processing, the system performs a short-time Fourier transform to convert the time-domain signal into a frequency-domain signal to extract features reflecting the degree of emphasis. The generation of the speech emphasis envelope aims to capture the stress and intonation changes during lectures; its calculation formula is as follows: ;in, :time The speech emphasizes the envelope value; :time The fundamental frequency trajectory value of a speech signal reflects pitch changes; time The short-time energy amplitude of a speech signal reflects changes in loudness; The average fundamental frequency of the current teaching segment; The average energy value of the current teaching segment; Weighting coefficients representing fundamental frequency characteristics. The weighting coefficients represent energy characteristics; this step ensures that high-pitched or loud speech segments are identified as high-weight signals. Finally, calculating the interaction density is the core step in determining audiovisual synchronization. Simply having large movements or loud voices does not necessarily indicate effective teaching; only when the two show a high positive correlation on the timeline does it mean the teacher is explaining key points while writing. The system uses a sliding window cross-correlation algorithm to calculate the covariance between normalized writing force and the speech envelope, generating the interaction density value. ;in, : The duration of the sliding window, corresponding to the number of sampling points; : Sliding window radius; : Normalized dynamic value of writing force; The normalized speech emphasis envelope value. This formula, by calculating the integral of the product of the two within the same time window, can effectively suppress noise periods with only action and no speech or only speech and no action, and accurately quantify the synchronous excitation level of vision and hearing.

[0025] Specifically, the process of capturing a synchronized audiovisual data package containing the complete handwriting generation process and corresponding audio explanation during periods when the interaction density exceeds a threshold is as follows: Real-time monitoring of the rate of change of the interaction density value; identifying the time point when the rising edge of the interaction density curve crosses a preset trigger threshold as a candidate start frame, and the time point when the falling edge crosses the trigger threshold as a candidate end frame; using the candidate start frame as a reference, tracing back the dynamic curve of writing force along the time axis to locate the most recent pen stroke when the speed is zero and the normal depth is detached from the blackboard plane, and correcting the candidate start frame; using the candidate end frame as a reference, extending the speech emphasis envelope forward along the time axis to locate the most recent semantic pause when the short-term energy amplitude is lower than the silence threshold, and correcting the candidate end frame; determining the final capture time window based on the corrected pen stroke time and semantic pause time, extracting the image sequence and audio segments within the time window from the original multimodal stream, and encapsulating them into a synchronized audiovisual data package.

[0026] In this implementation, the occurrence range of teaching events is first roughly located by monitoring the rate of change of interaction density values. When the density value rises rapidly and exceeds a preset trigger threshold, it is marked as a candidate start frame; otherwise, it is marked as a candidate end frame. However, relying solely on the density threshold may result in the loss of silent actions at the beginning of a stroke or lingering echoes at the end. Therefore, the system needs to perform reverse backtracking correction on the start time based on physical action characteristics to calculate the corrected start time: ;in, The time variable in the backtracking process; Maximum backtracking time window; The system determines the depth threshold of the pen tip's contact with the blackboard surface. The physical meaning of this formula is to find the closest point before the approximate starting point, at which the pen tip speed is close to zero and the depth just reaches the contact threshold, thus precisely locating the moment of pen contact and ensuring the integrity of the handwriting data. Similarly, for correcting the ending time, the system needs to consider the semantic integrity of the speech, avoiding forced truncation before the teacher finishes speaking. The system searches forward along the time axis to calculate the corrected ending time. ;in, The silence threshold for ambient background noise can be obtained by statistically analyzing the average energy during non-teaching periods. This formula is used to find the moment when the energy amplitude returns to the silence level after a rough end point, thus precisely locating the semantic pause point. Finally, the system uses the corrected time interval to losslessly extract image sequences and audio segments from the original data stream, encapsulating them into synchronized audiovisual data packets, ensuring that each data packet contains a logically closed-loop teaching event.

[0027] Specifically, the process of performing optical character recognition to extract discrete text entities from audiovisual synchronous data packets, and analyzing the geometric coordinates, font indentation, and writing area clustering features of the text entities on the writing plane is as follows: Adaptive binarization and connected component analysis are performed on the image sequences in the audiovisual synchronous data packets to extract the minimum bounding rectangle of each independent character, and the characters are merged into text line entities according to the horizontal spacing and vertical overlap rate of adjacent rectangles; the two-dimensional plane coordinates of the geometric center of each text line entity are calculated, and the horizontal Euclidean distance from the leftmost pixel of the text line entity to the baseline is calculated as the font indentation; the average pixel height of the bounding rectangle of the text line entity is calculated as the font size feature; the two-dimensional plane coordinates of all text line entities are clustered using a density-based spatial clustering algorithm, and text line entities with a spatial distance less than a preset paragraph threshold are classified into the same writing area cluster, and the centroid coordinates of each cluster are calculated.

[0028] In this implementation scheme, the system preprocesses each frame of the audiovisual synchronization data packet. Adaptive binarization refers to dynamically calculating local thresholds based on the brightness distribution of different areas of the image, rather than using a globally fixed threshold, effectively addressing issues like glare or uneven lighting on the whiteboard. Connected component analysis combines adjacent pixels after binarization into independent character blocks. To reconstruct semantically complete text lines from scattered characters, the system performs a merging operation based on the principle of geometric proximity; that is, when the horizontal spacing between two character rectangles is minimal and their vertical height overlaps, they are considered to belong to the same line of text. Secondly, after obtaining the complete text line entity, its physical attributes need to be quantified to deduce logical attributes. Font indentation is a key geometric feature for determining the hierarchy of knowledge points; the system obtains this by calculating the horizontal distance between the left edge of the text line and the left boundary of the whiteboard. Simultaneously, font size features are characterized by the height of the bounding rectangle; typically, the font size of titles is significantly larger than that of body text. Finally, to identify different functional blocks on the whiteboard, such as the main whiteboard area and the secondary whiteboard area, the system uses a density-based spatial clustering algorithm to group the text lines. To quantify whether two text lines belong to the same writing region cluster, the system defines a weighted spatial distance metric formula. Assume the... The text line entity and the first The center coordinates of the text line entities are respectively ( , )and( , The weighted cluster distance between the two is calculated as follows: ;in, The x-coordinates of the two text line entities; : The y-coordinate of the two text line entities; The average pixel height of two text line entities, i.e., the font size feature; The horizontal distance weighting coefficient is usually set to a small value, considering that blackboard writing is usually horizontally formatted and has a high tolerance for horizontality. Vertical distance weighting coefficient, used to strictly limit line spacing; The font size difference penalty factor is used to distinguish text of different font sizes (i.e., different levels) that are close together. The formula works by fusing spatial Euclidean distance and font size difference to cluster lines of text that are physically close and have similar font specifications into the same semantic block, thereby determining... Whether it is less than a preset paragraph threshold, thus accurately dividing the writing area clusters.

[0029] Specifically, the process of constructing the initial structured entity tree representing the static spatial hierarchy is as follows: traverse all writing area clusters, select the region whose centroid coordinates are located at the top of the writing plane and whose text line entity within the cluster has the largest font size feature, and define it as the root node region; establish a hierarchy mapping rule based on font indentation and font size features, and recursively traverse the text line entities in non-root node regions: if the font size of the current entity is smaller than that of the parent node and the font indentation is greater than that of the parent node, then create a parent-child connection and mount the current entity as a child node of the parent node; if the font size of the current entity is at the same level as that of the parent node and the difference in font indentation is within a preset tolerance range, then create a sibling connection and mount the current entity as a parallel node of the parent node; if the current entity is located at the edge of the writing area cluster and has no clear parent-child indentation feature, then calculate its semantic vector similarity with the nearest node, and only mark it as an isolated note node when the similarity is lower than a threshold, generating an initial structured entity tree containing the root node, child nodes, sibling nodes, and note nodes.

[0030] In this implementation scheme, the system needs to establish recursive parent-child or sibling relationships. This is a crucial step in transforming planar geometric information into tree-like topological information. The core of the hierarchical mapping rule lies in comparing the relative geometric changes of the current node and its predecessor nodes. The system defines a hierarchical decision index. To quantify this relationship. Assume the font size of the parent node is... The font indentation is The font size of the currently detected entity is , font indentation is The hierarchy determination index is calculated as follows: ;in, : The font size characteristics of the parent node; : The font size characteristic of the current entity; : The font indentation of the parent node; The font indentation of the current entity; The standard indentation unit is the baseline value, usually taken as the width of a standard character on the whiteboard. Weighting factor for font size variation; : Weighting factor for the change in indentation. This formula performs a nonlinear mapping of geometric differences using a logarithmic function and a hyperbolic tangent function. When When the value is significantly greater than zero, it indicates that the font size of the current entity has become smaller and the indentation has become larger. The system determines that it is a parent-child connection, that is, the current entity is a child knowledge point belonging to the parent node. When the similarity approaches zero, it indicates that the font size and indentation of the two elements are basically the same, and the system determines them to be sibling connections, meaning they are parallel. This calculation method can mathematically simulate the intuitive human perception of typesetting hierarchy when reading blackboard writing. For text that deviates from standard typesetting, i.e., isolated annotation nodes, these are usually formulas or examples added by teachers in corners. Because their indentation and font size do not conform to the regular hierarchical rules, the system cannot simply discard them. Instead, it uses semantic vector similarity calculation to find their affiliation. The system semantically compares the isolated entity with its nearest spatial neighbor node. If the semantic similarity is below a threshold, it indicates that it is not a direct explanation of its neighbor but an independent annotation item. Therefore, it is treated as a special type of node and attached to the tree structure, ensuring the completeness of knowledge extraction.

[0031] Specifically, the process of parsing and extracting teaching operator vectors containing derivation and reversal attributes from the audio explanation, projecting them onto the node space of the initial structured entity tree, and calculating the logical tension coefficient between the speech logic and the blackboard layout is as follows: Semantic role labeling and dependency parsing are performed on the audio explanation. Keywords representing causal progression are identified to generate forward derivation operators, and keywords representing negation, transition, or modification are identified to generate reverse reversal operators. These two types of operators are encapsulated into teaching operator vectors. Using the timestamp index of the audio-visual synchronization data packets, the teaching operator vectors are mapped to the target nodes in the initial structured entity tree corresponding to their occurrence time period, establishing a temporal mapping relationship between speech instructions and blackboard nodes. The Euclidean spatial distance between the target node and the semantically pointed node on the blackboard plane is calculated, along with the cosine similarity between the two in the pre-trained semantic vector space. A weighted fusion calculation is performed on the cosine similarity and the Euclidean spatial distance to obtain the logical tension coefficient, which represents the deviation ratio between the semantic association strength and the physical spatial adjacency.

[0032] In this implementation plan, the system needs to perform deep semantic understanding on unstructured speech streams to extract logical instructions that can change the knowledge structure. The system performs semantic role labeling and dependency parsing on the audio explanations to identify the agent, patient, and core predicate in the sentences, thereby determining whether the teacher is explaining a forward deduction process (e.g., "Because A, therefore B") or performing a reverse correction or transition (e.g., "A does not belong to B," or "This should be corrected to C"). The system maps identified keywords representing causal progression to forward deduction operators and keywords representing negation, transition, or correction to reverse inversion operators, encapsulating these two types of operators into teaching operator vectors. Secondly, to apply these abstract logical instructions to specific knowledge nodes, the system uses timestamp indexes in the audiovisual synchronization data packets as a bridge. The system retrieves the time period in which the teaching operator vector appears, finds nodes in the initial structured entity tree where the teacher's body movements or eye focus occur within that time period, and identifies these nodes as target nodes, thus establishing a temporal mapping relationship between instantaneous speech instructions and static whiteboard nodes. Finally, the logical tension coefficient is the core calculation indicator of this system, used to measure the deviation ratio between the correlation strength at the speech and semantic level and the adjacency degree at the physical layout level of the blackboard. In the teaching process, situations often arise where the physical distance between elements is large (written at opposite ends of the blackboard) but the logical relationship is very close (the teacher verbally emphasizes the connection between the two); this state is called a high-tension state. The system calculates the target node using the following formula. With semantic pointer nodes Logical tension coefficient between: ;in, node With nodes Feature vectors in the pre-trained semantic vector space; The cosine similarity function is used to characterize the semantic association strength between two things; node With nodes The geometric center coordinates on the blackboard plane; Euclidean distance calculation function; The length of the diagonal of the blackboard plane is used to normalize physical distances; Basic semantic weight coefficients; Spatial span gain coefficient. The physical meaning of this formula is that when two nodes are semantically highly similar and physically far apart, the calculated logical tension coefficient is higher, thus quantifying this implicit strong association that spans physical space.

[0033] Specifically, when the logical tension coefficient exceeds a preset crossing threshold, an implicit connection edge is forcibly constructed between non-adjacent nodes, and reverse pruning is performed on the spatial hierarchy of the initial structured entity tree according to the inversion operator. The specific process for generating a dynamic semantic topology network is as follows: traverse all node pairs in the initial structured entity tree. When the logical tension coefficient exceeds the preset crossing threshold, it is determined that there is a strong logical connection across the physical layout. A directed implicit connection edge is directly established between two spatially non-adjacent nodes, and the edge is assigned a semantic connection attribute with the logical tension coefficient as the weight. When the detection... When the inverse operator is detected to be mapped to a target node, the parent node connection of the node in the initial structured entity tree is retrieved. If the semantic pointer of the inverse operator is logically mutually exclusive with the current parent-child hierarchy, a logical blocking operation is performed to cut off the physical connection edge between the node and the original parent node. The node after the connection is cut off is reattached to the new target node pointed to by the semantic pointer of the inverse operator, or it is downgraded to an independent free node. The adjacency matrix between all nodes is updated, thereby reconstructing the static tree structure into a dynamic semantic topology network containing explicit physical edges and implicit logical edges.

[0034] In this implementation, the system traverses all node pairs in the initial structured entity tree and determines whether it needs to overcome the limitations of physical space based on the calculated logical tension coefficient. When the logical tension coefficient exceeds a preset threshold, it indicates that although the two nodes are not together on the blackboard, they are closely connected in the teaching logic. The system determines that there is a strong logical connection across the physical space and directly establishes a directed implicit connection edge between the two spatially non-adjacent nodes. This edge is different from the physical edge based on the indentation relationship; it represents a leap and connection of thought. The system assigns this edge a semantic connection attribute with the logical tension coefficient as the weight, giving it a higher priority in subsequent graph traversal. Secondly, the system uses the reverse inversion operator to correct errors in the hierarchical structure established solely based on spatial location. In blackboard writing, teachers may write unrelated knowledge points below a topic due to insufficient space, causing the spatial parsing module to misjudge it as a parent-child relationship. When the system detects that the reverse inversion operator is mapped to a target node, it retrieves the parent node connection of that node in the initial structured entity tree and calculates the logical mutual exclusion to determine whether to perform pruning. ;in, time The strength value of the detected inversion operator is quantified by the intensity of the negation of the keyword; : The physical space connection weight between the current node and its original parent node; : Correction gain coefficient for the inversion operator; : The stability coefficient of the original structure. If In other words, the negation intent of the speech is stronger than the connection intent of the physical structure. The system performs a logical blocking operation, severing the physical connection edge between the node and its original parent node. Finally, for isolated nodes whose connections have been severed, the system reattaches them to a new target node based on the correct pointing information contained in the inversion operator (such as which node should belong to…); if there is no clear pointing, it is downgraded to an independent detached node. After the above connection and pruning operations, the system updates the adjacency matrix between all nodes, thereby reconstructing the original static, linear tree structure into a dynamic semantic topology network containing explicit physical edges and implicit logical edges, truly restoring the complete picture of thinking in the teaching process.

[0035] Specifically, the process of calculating the recurrence weights and centralities of each node in the dynamic semantic topology network in the multimodal data stream, and executing the minimum spanning tree algorithm to retain the backbone paths with the highest weights, is as follows: The cumulative pixel duration of each node in the image sequence and the frequency of keyword recurrence in the speech signal are statistically analyzed, and the multimodal recurrence weights of each node are calculated by weighted summation; based on the adjacency matrix of the dynamic semantic topology network, the degree centrality and betweenness centrality of each node are calculated using graph theory algorithms to characterize the hub status of the node in the knowledge transfer network; the multimodal recurrence weights and centrality indices are fused to generate a comprehensive importance score, and the reciprocal of the comprehensive importance score is defined as the traversal cost of the edges in the dynamic semantic topology network; with the node with the highest comprehensive importance score as the starting root node, the minimum spanning tree algorithm is executed to traverse the dynamic semantic topology network, prioritizing the retention of connected paths with the lowest traversal cost while eliminating network loops, removing redundant weakly related paths, and generating a core backbone path set.

[0036] In this implementation plan, the system needs to quantify the value of individual knowledge nodes, which relies on the statistical characteristics of multimodal data. The system calculates the cumulative pixel duration of each node in the image sequence to measure the time span of the knowledge point on the blackboard; a longer duration generally indicates a more fundamental or core knowledge point. The system also calculates the frequency of keyword recurrence in the speech signal to measure the degree of emphasis given by the teacher. To unify these two heterogeneous dimensions into a comparable value, the system constructs a comprehensive importance scoring model. Assuming the first node in the dynamic semantic topology network... The overall importance score of each node is: The calculation formula is as follows: ;in, :No. The cumulative pixel duration of the text entity corresponding to each node in the whiteboard image sequence; Total duration of the entire course; :No. The frequency of recurrence of keywords corresponding to each node in the speech signal; The highest recurrence frequency among all nodes, used for normalization; :No. The degree centrality of a node represents the number of direct connections between that node and other nodes; The total number of nodes in the network topology; :No. The betweenness centrality of a node represents the frequency with which that node appears in the shortest path, i.e., the degree to which it acts as an information bridge. Weighting coefficients for visual retention features; Weighting coefficients for auditory reproduction features; The system assigns weight coefficients to topological features and uses weighted fusion to ensure that the selected core nodes are both key points that teachers have spent a lot of time writing about and discussing, and also pivotal points in the logical structure. Secondly, to extract clear backbone paths from the complex network structure, the system introduces the minimum spanning tree algorithm from graph theory. Since the minimum spanning tree algorithm aims to find the path connecting all nodes with the minimum total edge weight, while the goal of this system is to retain the path with the highest importance, edge weight transformation is necessary. The system defines the reciprocal of the overall importance score as the traversal cost of the edge; that is, the more important the node, the lower the cost of traversing it. Assume the node... With nodes There are connecting edges between them. Its traversal cost is defined as: The purpose of this formula is that when the importance of the two endpoints of a connecting edge is extremely high, its traversal cost approaches zero. The system uses the node with the highest overall importance score as the starting root node and executes the Prim algorithm to traverse the dynamic semantic topology network. While eliminating logical loops in the network, it prioritizes retaining the connected paths with the lowest traversal cost, thereby eliminating redundant weakly related paths and generating a core backbone path set composed of the strongest logical chains.

[0037] Specifically, the process of converting complex network topology into hierarchical tree-like visual data to generate mind maps is as follows: A hierarchical traversal of the core backbone path set is performed, and nodes located on the main path are marked as mind maps. Figure 1 Secondary branches are marked as secondary branches directly connected to the main node; a cognitive load aggregation mechanism is introduced to calculate the sum of the comprehensive importance scores of all child nodes under the same parent node. If the sum of scores is lower than the preset cognitive display threshold, the child nodes in this group are automatically encapsulated into a collapsed attribute list, retaining only the direct display of the parent node; the processed node hierarchy is converted into a standard tree data structure, and the original whiteboard screenshots and audio clips corresponding to the nodes are embedded in the metadata fields of the data structure as interactive multimedia indexes to render and generate a visual mind map.

[0038] In this implementation scheme, the system performs a hierarchical traversal of the extracted core backbone path set to determine the display architecture of the mind map. The system uses the root node of the minimum spanning tree as the center and marks the nodes located on the main path as mind maps. Figure 1 First-level branches, typically corresponding to chapter titles or core theorems in a course; second-level branches are marked as directly connected to the main nodes, corresponding to specific knowledge points or derivation steps. This step maps undirected or complex directed graphs to a hierarchical tree structure. Secondly, to prevent excessive generated mind map nodes from causing cognitive overload, the system introduces a cognitive load aggregation mechanism. This mechanism aims to automatically identify and collapse trivial, detailed branches. The system calculates the sum of the overall importance scores of all child nodes under the same parent node as the cognitive load value for that branch. The discrimination logic is as follows: ;in, The current parent node; The total number of child nodes attached to this parent node; No. Number of child nodes; The system calculates a comprehensive importance score from the preceding steps. It then compares the calculated cognitive load value with a preset cognitive display threshold. If the cognitive load value is less than the threshold, it indicates that the overall importance of the sub-nodes in that group is low, belonging to supplementary information. The system automatically encapsulates these sub-nodes into a collapsed attribute list, retaining only the parent node when displayed on the front end; the user must click to expand it. Conversely, the system displays the collapsed list by default. The system converts the processed node hierarchy into a standard tree-like data structure, such as JSON or XML, for easy reading by the front-end rendering engine. During this process, the system performs multimedia embedding, embedding the original whiteboard screenshot (obtained from the trajectory audiovisual alignment module) and the audio clip (obtained from the causal operator injection module) corresponding to the node into the metadata field of the data structure. This step upgrades the mind map from a simple text index to an interactive multimedia index, allowing users to click on a node to review the corresponding whiteboard generation process and listen to the teacher's original explanation, achieving contextualized knowledge reproduction.

[0039] In summary, this application has at least the following effects: This teaching and learning knowledge graph and mind map construction system, based on multimodal digital intelligence, uses a trajectory audiovisual alignment module to accurately pinpoint effective teaching periods of writing and speech resonance through interaction density, fundamentally solving the problems of information redundancy and alignment deviation caused by heterogeneous multimodal data. Combined with a layout semantic parsing module, it deeply mines the geometric coordinates and indentation features of the blackboard, overcoming the limitations of traditional linear text recognition in parsing spatial hierarchy and constructing an initial structure that conforms to the logic of physical blackboard writing. Furthermore, through a causal operator injection module, it calculates the logical tension coefficient, forcibly constructing implicit related edges between nodes that are not physically adjacent but semantically close, and dynamically corrects erroneous hierarchical relationships based on a reversal operator, achieving a qualitative leap from static physical representation to dynamic semantic logic in the graph. Finally, using a topology dimensionality reduction rendering module, based on the minimum spanning tree algorithm and multimodal reproduction weight optimization, it optimizes the backbone path, automatically folding low-weight redundant branches, transforming the intricate network knowledge structure into a clearly hierarchical, low-cognitive-load visual mind map, significantly improving the efficiency of knowledge reorganization of teaching resources and the learning effect of learners.

[0040] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0041] This invention is described with reference to flowchart illustrations and / or block diagrams of systems, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0042] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0043] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0044] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0045] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence, characterized in that, Includes the following modules: The trajectory audiovisual alignment module is used to establish a spatiotemporal three-dimensional coordinate system in the teaching scenario, synchronously collect the dynamic trajectory flow of the teacher's writing strokes and the prosodic features of the speech signal, calculate the interaction density of writing actions and speech stress on the time axis, lock the time period when the interaction density exceeds the threshold, and extract the audiovisual synchronization data package containing the complete handwriting generation process and the corresponding explanation audio. The layout semantic parsing module is used to perform optical character recognition to extract discrete text entities from audiovisual synchronous data packets, analyze the geometric coordinates, font indentation, and writing area clustering features of the text entities on the whiteboard plane, and construct an initial structured entity tree representing the static spatial hierarchy. The causal operator injection module is used to parse the audio explanation to extract teaching operator vectors containing derivation and inversion attributes, project them onto the node space of the initial structured entity tree, calculate the logical tension coefficient between the speech logic and the blackboard layout, and when the logical tension coefficient exceeds the preset crossing threshold, implicit association edges are forcibly constructed between non-adjacent nodes. Inverse pruning is performed on the spatial hierarchy of the initial structured entity tree according to the inversion operator to generate a dynamic semantic topology network. The topology dimensionality reduction and rendering module is used to calculate the recurrence weight and centrality of each node in the dynamic semantic topology network in the multimodal data stream, execute the minimum spanning tree algorithm to retain the backbone paths with the highest weight, and convert the complex network topology into hierarchical tree-like visual data to generate mind maps.

2. The teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence as described in claim 1, characterized in that: The specific process of establishing a spatiotemporal three-dimensional coordinate system in the teaching scenario, synchronously collecting the dynamic trajectory flow of the teacher's writing strokes and the prosodic features of the speech signal, and calculating the interaction density of writing actions and speech stress on the time axis is as follows: By calibrating the normal vector and depth threshold of the writing plane using a depth vision sensor, a three-dimensional Euclidean space with the upper left corner of the writing plane as the origin is constructed. The spatial displacement of key points of the teacher's hand is tracked in real time. By calculating the displacement velocity of the key points along the tangential direction of the writing plane and the pressing depth along the normal direction within a unit time window, a dynamic curve of writing force is generated. Simultaneously, a short-time Fourier transform is performed on the acquired speech signal to extract the fundamental frequency trajectory in the frequency domain and the short-time energy amplitude in the time domain. The two are then weighted and fused to generate the speech emphasis envelope. Normalization is performed on the dynamic curve of writing force and the envelope of speech emphasis. The covariance of the two curves on the same time axis is calculated by the sliding window cross-correlation algorithm. This covariance is used as the interaction density value to characterize the degree of synchronous excitement of vision and hearing.

3. The teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence as described in claim 2, characterized in that: The specific process of extracting audiovisual synchronized data packets containing the complete handwriting generation process and corresponding audio explanation during periods when the interaction density exceeds a threshold is as follows: Real-time monitoring of the rate of change of interaction density value; identifying the time point when the rising edge of the interaction density curve crosses the preset trigger threshold as a candidate start frame, and the time point when the falling edge crosses the trigger threshold as a candidate end frame. Based on the candidate starting frame, trace back the dynamic curve of writing force along the time axis to locate the most recent pen stroke when the speed is zero and the normal depth is detached from the blackboard plane, and correct the candidate starting frame. Based on the candidate termination frame, the speech emphasis envelope is extended forward along the time axis to locate the most recent semantic pause when the short-term energy amplitude is lower than the silence threshold, and the candidate termination frame is corrected. The final extraction time window is determined based on the corrected pen stroke time and semantic pause time. Image sequences and audio segments within the time window are extracted from the original multimodal stream and encapsulated into audiovisual synchronization data packets.

4. The teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence as described in claim 1, characterized in that: The specific process of performing optical character recognition to extract discrete text entities from synchronized audiovisual data packets, and analyzing the geometric coordinates of the text entities on the blackboard plane, font indentation, and clustering features of the writing area is as follows: Adaptive binarization and connected component analysis are performed on the image sequences in the audiovisual synchronization data packets to extract the minimum bounding rectangle of each independent character, and the characters are merged into text line entities according to the horizontal spacing and vertical overlap rate of adjacent rectangles. Calculate the two-dimensional plane coordinates of the geometric center of each text line entity, and use the left boundary of the blackboard plane as the baseline to calculate the horizontal Euclidean distance from the leftmost pixel of the text line entity to the baseline as the font indentation. The average pixel height of the bounding rectangle of each text line entity is calculated as the font size feature. A density-based spatial clustering algorithm is used to perform cluster analysis on the two-dimensional plane coordinates of all text line entities. Text line entities with a spatial distance less than a preset paragraph threshold are classified into the same writing region cluster, and the centroid coordinates of each cluster are calculated.

5. The teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence as described in claim 4, characterized in that: The specific process of constructing an initial structured entity tree representing static spatial hierarchical relationships is as follows: Traverse all writing area clusters, select the area whose centroid coordinates are located at the top of the writing plane and whose font size feature of the text line entity within the cluster is the largest, and define it as the root node area; Based on font indentation and font size characteristics, a hierarchical mapping rule is established, and the text line entities in non-root node regions are recursively traversed: If the font size of the current entity is smaller than that of the parent node and the font indentation is greater than that of the parent node, then a parent-child connection is created, and the current entity is mounted as a child node of the parent node. If the font size of the current entity is on the same order of magnitude as the parent node and the difference in font indentation is within the preset tolerance range, then a sibling connection is created, and the current entity is mounted as a parallel node of the parent node. If the current entity is located at the edge of the writing region cluster and has no explicit parent-child indentation feature, then calculate its semantic vector similarity with the nearest node. Only when the similarity is below the threshold is it marked as an isolated memo node, generating an initial structured entity tree containing the root node, child nodes, sibling nodes, and memo nodes.

6. The teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence as described in claim 1, characterized in that: The specific process of extracting teaching operator vectors containing derivation and inversion attributes from the audio explanation, projecting them onto the node space of the initial structured entity tree, and calculating the logical tension coefficient between the speech logic and the blackboard layout is as follows: Semantic role labeling and dependency parsing are performed on the audio explanations to identify keywords representing causal and progressive relationships and generate forward derivation operators, and keywords representing negation, transition or modification relationships and generate reverse inversion operators. The two types of operators are encapsulated into teaching operator vectors. By using the timestamp index of the audio-visual synchronization data packet, the teaching operator vector is mapped to the target node in the initial structured entity tree corresponding to its occurrence time period, thus establishing a temporal mapping relationship between voice commands and blackboard nodes. Calculate the Euclidean distance between the target node and the semantically pointed node on the whiteboard plane, and simultaneously calculate the cosine similarity between the two in the pre-trained semantic vector space. Perform a weighted fusion calculation on the cosine similarity and the Euclidean distance to obtain the logical tension coefficient, which represents the deviation ratio between the semantic association strength and the physical adjacency.

7. The teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence as described in claim 6, characterized in that: When the logical tension coefficient exceeds the preset crossing threshold, implicit association edges are forcibly constructed between non-adjacent nodes, and reverse pruning is performed on the spatial hierarchy of the initial structured entity tree according to the inversion operator. The specific process of generating a dynamic semantic topology network is as follows: Traverse all node pairs in the initial structured entity tree. When the logical tension coefficient exceeds the preset crossing threshold, it is determined that there is a strong logical association that crosses the physical layout. Directed implicit association edges are directly established between two spatially non-adjacent nodes, and the edges are given semantic connection attributes with the logical tension coefficient as the weight. When the reverse inversion operator is detected to be mapped to a target node, the parent node connection of the node in the initial structured entity tree is retrieved. If the semantic pointer of the inversion operator is logically mutually exclusive with the current parent-child hierarchy, a logical blocking operation is performed to cut off the physical connection edge between the node and the original parent node. The nodes whose connections have been severed are reattached to the new target node pointed to by the inversion operator semantics, or downgraded to independent free nodes. The adjacency matrix between all nodes is updated, thereby reconstructing the static tree structure into a dynamic semantic topology network containing explicit physical edges and implicit logical edges.

8. The teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence as described in claim 1, characterized in that: The specific process of calculating the recurrence weight and centrality of each node in the multimodal data stream in a dynamic semantic topology network, and executing the minimum spanning tree algorithm to retain the backbone paths with the highest weighted values ​​is as follows: The cumulative pixel duration of each node in the image sequence and the frequency of keyword recurrence in the speech signal are statistically analyzed, and the multimodal recurrence weighted value of each node is calculated by weighted summation. Based on the adjacency matrix of dynamic semantic topology network, the degree centrality and betweenness centrality of each node are calculated by graph theory algorithm to characterize the hub degree of each node in the knowledge transfer network. A comprehensive importance score is generated by integrating the multimodal recurrence weighting value and the centrality index, and the reciprocal of the comprehensive importance score is defined as the traversal cost of the edge in the dynamic semantic topology network. Starting with the node with the highest overall importance score as the root node, the minimum spanning tree algorithm is executed to traverse the dynamic semantic topology network. While eliminating network loops, the connected paths with the lowest traversal cost are retained first, and redundant weakly related paths are eliminated to generate a core backbone path set.

9. The teaching and learning knowledge graph and mind map construction system based on multimodal digital intelligence as described in claim 8, characterized in that: The specific process of converting complex network topologies into hierarchical tree-like visual data to generate mind maps is as follows: The core backbone path set is traversed hierarchically, and nodes located on the main path are marked as first-level branches of the mind map, while secondary nodes directly connected to the main path nodes are marked as second-level branches. A cognitive load aggregation mechanism is introduced to calculate the sum of the comprehensive importance scores of all child nodes under the same parent node. If the sum of scores is lower than the preset cognitive display threshold, the child nodes in this group are automatically encapsulated into a collapsed attribute list, and only the direct display of the parent node is retained. The processed node hierarchy is converted into a standard tree data structure, and the original whiteboard screenshots and audio clips corresponding to the nodes are embedded in the metadata fields of the data structure as interactive multimedia indexes to generate a visual mind map.

Images