A teaching video mind map generation system based on visual hierarchical analysis and multi-modal fusion

Abstract

This invention discloses a teaching video mind map generation system based on visual hierarchical analysis and multimodal fusion, comprising a visual feature extraction and analysis module, an audio analysis and node extraction module, and a multimodal fusion mind map generation module. The visual feature extraction and analysis module is used for video keyframe extraction, coarse-grained extraction of visual information, and fine-grained analysis of hierarchical relationships. The audio analysis and node extraction module is used for audio transcription and adaptive granular node extraction. The multimodal fusion mind map generation module is used to perform late-stage multimodal fusion of audio transcription information, mind map nodes, and hierarchical index tree information after visual analysis to generate teaching video mind maps. This invention significantly improves the logical accuracy and semantic expressiveness of automatically generated mind maps by fusing visual analysis and audio modal information for specific teaching video scenarios.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to a teaching video mind map generation system based on visual hierarchy analysis and multimodal fusion. Background Technology

[0002] To address the problem of automated mind map generation for video, the traditional two-stage approach first transcribes the audio into text. In the first stage, named entity recognition (NER) is used to extract mind map nodes, and in the second stage, syntactic analysis or intelligent methods based on graph neural networks are used to link the nodes, thereby generating the mind map.

[0003] The existing technology has at least the following drawbacks: (1) It does not make full use of visual modal information: In the task of understanding teaching videos, although the teacher's audio explanation is rich in total semantic information, visual modal information is still indispensable. The lack of visual modal information or the processing of visual modal information is limited to simple OCR or the use of multimodal large models for basic understanding will lead to the loss of key teaching information. (2) The hierarchical relationship is wrong: The teaching video screen is usually a presentation containing a lot of text. The layout structure of the presentation often contains the speaker's understanding of the structural relationship of the teaching knowledge points. However, traditional video understanding methods cannot capture the hierarchical relationship between knowledge points. (3) The node extraction is inaccurate: Most existing methods directly use NER technology to extract mind map nodes, ignoring the uncertainty of the node granularity in the teaching video. For example, the nodes of descriptive text may be short sentences or complete sentences, resulting in inappropriate node granularity.

[0004] In summary, existing methods suffer from problems such as underutilization of visual modal information, incorrect hierarchical relationships in mind maps, and inappropriate node extraction granularity. Summary of the Invention

[0005] The purpose of this invention is to provide a teaching video mind map generation system based on visual hierarchical analysis and multimodal fusion, including a visual feature extraction and analysis module, an audio analysis and node extraction module, and a multimodal fusion mind map generation module;

[0006] The visual feature extraction and analysis module is used for video keyframe extraction, coarse-grained extraction of visual information, and fine-grained analysis of hierarchical relationships.

[0007] The audio analysis and node extraction module is used for audio transcription and adaptive granularity node extraction.

[0008] The multimodal fusion mind map generation module is used to perform late-stage multimodal fusion of audio transcription information, mind map nodes, and hierarchical index tree information after visual analysis to generate teaching video mind maps;

[0009] The steps involved in generating mind maps for instructional videos using the instructional video mind map generation system include:

[0010] Step 1) Downsample the original input video to 1FPS video, and input the 1FPS video into the audio analysis and node extraction module and the visual feature extraction and analysis module respectively;

[0011] Step 2) Use the audio analysis and node extraction module to process the 1FPS video, separate the audio information, and transcribe the separated audio information into audio text;

[0012] The visual feature extraction and analysis module is used to process 1FPS video, separate visual information, and extract key frames of visual information.

[0013] Step 3) The audio analysis and node extraction module uses a large language model that has been fine-tuned with instructions to extract mind map nodes from the audio text;

[0014] The visual feature extraction and analysis module performs coarse-grained visual analysis on the keyframes of the extracted visual information, and then performs fine-grained analysis of the hierarchical relationship to construct a hierarchical index tree;

[0015] Step 4) The multimodal fusion mind map generation module performs multimodal late-stage fusion of audio text, mind map nodes, and hierarchical index tree information after visual analysis to output a mind map.

[0016] Furthermore, the visual feature extraction and analysis module includes a keyframe recognition module and a visual hierarchy analysis module;

[0017] The keyframe recognition module extracts keyframes from the 1FPS video and outputs the extracted keyframes.

[0018] The visual hierarchy analysis module performs coarse-grained extraction of visual information and fine-grained analysis of hierarchical relationships on keyframes, and outputs hierarchical index tree information after visual analysis.

[0019] Furthermore, the keyframe recognition module processes 1FPS video in steps including reverse sampling, multi-dimensional filtering of low-quality frames, and filtering of similar frames.

[0020] Furthermore, reverse sampling refers to sampling from a video sequence... Iterate from the maximum timestamp to the minimum timestamp to obtain the sampling sequence. ;

[0021] Sampling sequence As shown below:

[0022] (1)

[0023] In the formula, Total duration This is the sampling step size;

[0024] Low-quality frame multidimensional filtering refers to removing samples from the sequence. Among blank frames, low-quality frames, and transition frames, frames that meet the edge density and image entropy thresholds are retained, and the Canny edge detection operator is used to suppress noise.

[0025] Among them, the sampling sequence medium video frames Image entropy As shown below:

[0026] (2)

[0027] In the formula, It is grayscale value The probability of occurrence It is a smoothing factor;

[0028] Similar frame filtering refers to removing low-quality frames and repeating similar frames after multidimensional filtering, extracting low-frequency features of each frame using discrete cosine transform and calculating the mean, and generating a 01 hash sequence of the frame based on the mean.

[0029] Based on the obtained hash sequence, calculate the hash of adjacent frames. Hamming distance As shown below:

[0030] (3)

[0031] In the formula, given the similarity frame threshold When the Hamming distance between two frames If the two frames are duplicated, only the frame with the later timestamp is retained.

[0032] Furthermore, the visual hierarchy analysis module includes a coarse-grained visual information extraction module and a fine-grained hierarchy relationship analysis module;

[0033] The visual information coarse-grained extraction module refers to performing coarse-grained visual information extraction on the keyframes extracted by the keyframe recognition module to obtain coarse-grained features as output.

[0034] The hierarchical relationship fine-grained analysis module takes coarse-grained visual information as input, performs fine-grained analysis, and outputs hierarchical index tree information.

[0035] Furthermore, the visual information coarse-grained extraction module refers to the keyframes extracted by the keyframe recognition module. Perform coarse-grained extraction of visual information to obtain multiple text blocks, each text block Construct a multidimensional feature vector As shown below:

[0036]

[0037] In the formula, For the identified text content, The normalized coordinates of the top-left corner of the text block. For the width and height of the text block, For The estimated font size is derived from the ratio of the font size to the image height. The color attribute is extracted from the pixel mean. For frames The corresponding timestamp;

[0038] The spatial coordinates are rescaled and updated as follows:

[0039] (4)

[0040] In the formula, This is the scale correction factor. , These are the image height and width, respectively.

[0041] For keyframes The text content in each text block Perform word segmentation to obtain a word set. At the same time, maintain a global dictionary Used to record each term Global document frequency With global word frequency ;

[0042] Iterate through the keyframe set and update , As shown below:

[0043] (5)

[0044] (6)

[0045] In the formula, This represents the total number of keyframes. For keyframes, For indicator functions, It is a counting function;

[0046] The fine-grained hierarchical relationship analysis module, based on the visual information extracted from the coarse-grained analysis, first introduces a lightweight language model to classify text blocks by function, extracts the global feature vector of the text block, and outputs the probability distribution of the text belonging to each functional category through the Softmax operator. As shown below:

[0047]

[0048] In the formula, This is a feature representation of the input text. For the model to class The original output score, It is the total number of categories. Sum the exponential scores for all categories to ensure that the sum of the output probabilities is 1.

[0049] Secondly, determine the text block. and Hierarchical relationship between Two text blocks in Relative offset in the axial direction and vertical distance is As shown below:

[0050] (7)

[0051]

[0052] In the formula, The coordinates of the left edge of the text block;

[0053] Generate a spatial dependency proposal As shown below:

[0054] (8)

[0055] In the formula, Indentation threshold The preset line spacing threshold, show It may be higher in the hierarchical structure. ;

[0056] The weighted hierarchy chain is established as follows:

[0057] (9)

[0058] In the formula, The semantic weights for the five types of text;

[0059] Hierarchical relationship As shown below:

[0060] (10)

[0061] In the formula, For text blocks and The semantic weight difference between them is calculated by the weight hierarchy chain. For a subordination function, if and only if the subordination proposal is true and At that time, the judgment for child nodes;

[0062] Finally, for each text block Construct a multidimensional salient feature vector As shown below:

[0063] (11)

[0064] In the formula, For visual saliency components, For semantic confidence components, For statistical importance components;

[0065] The hierarchical relationship of nested nodes is determined using a recursive algorithm. Let the set of nodes for all keyframes be . For any node Its dictionary-valued function The definition is as follows:

[0066] (16)

[0067] In the formula, The function directly retrieves the complete text of the node.

[0068] Furthermore, visual saliency components fractions are subject to font size with vertical coordinates The combined effects are as follows:

[0069] (12)

[0070] In the formula, A coefficient used to balance the weights of spatial location and size;

[0071] Semantic confidence component As shown below:

[0072] (13)

[0073] In the formula, It is a set of five types of teaching tags. For categorical variables, For text block content, These are the weight parameters for the text classifier;

[0074] For each node, calculate each word item after word segmentation. Statistical scores As shown below:

[0075] (14)

[0076] In the formula, It is a natural constant;

[0077] Based on the above statistical scores, node The final statistical importance components As shown below:

[0078] (15)

[0079] In the formula, The text length is used for normalization. It is a minimal smoothing constant.

[0080] Furthermore, the audio analysis and node extraction module workflow includes an audio transcription module and an adaptive granularity node extraction module;

[0081] The audio transcription module refers to taking a 1FPS video as input, separating the audio information of the 1FPS video, and then transcribing the separated audio information into text output.

[0082] The adaptive granularity node extraction module refers to the process of extracting and outputting mind map nodes from the input audio-transcribed text through prompt word engineering.

[0083] Furthermore, the adaptive granularity node extraction module uses a large model based on instruction fine-tuning to process ASR text and generate adaptive semantic units.

[0084] Furthermore, the multimodal fusion mind map generation module adopts late-stage multimodal fusion, which transforms the parsing results of different modalities into structured text representations, uses the structured text representations as context inputs to the large language model, and outputs mind maps.

[0085] Input feature vector after multimodal fusion As shown below:

[0086] (17)

[0087] In the formula, A hierarchical index tree containing visual modality information. For granularity adaptive nodes based on audio extraction, This is the original transcribed text of the audio.

[0088] The technical effects of this invention are undeniable. The teaching video mind map generation system proposed in this invention, based on visual hierarchical analysis and multimodal fusion, significantly improves the logical accuracy and semantic expressiveness of automatically generated mind maps through the fusion processing of visual analysis and audio modal information tailored to specific teaching video scenarios. The specific technical effects of this invention are as follows:

[0089] (1) Improved the completeness and processing efficiency of keyframe extraction from teaching videos. Considering the progressive display of teaching video footage, the reverse sampling algorithm proposed in this invention can prioritize acquiring the most complete image state. Combined with a joint filtering mechanism based on image entropy and edge density, the system can effectively remove redundant information such as blank frames and transition frames, significantly reducing the computational overhead of subsequent visual analysis while ensuring that key information is not lost.

[0090] (2) Significantly improves the hierarchical logic accuracy of mind maps in teaching videos. Unlike traditional methods that rely solely on textual grammar, this invention utilizes visual hierarchical analysis of teaching videos, comprehensively leveraging normalized coordinates of text blocks, estimated font size, layout indentation, and functional classification semantics (such as concepts, definitions, formulas, etc.) to construct a multi-dimensional collaborative hierarchical determination mechanism. This hierarchical relationship tree based on multi-dimensional evaluation can effectively capture the logical structure inherent in teaching presentations, solving the common hierarchical relationship errors in existing technologies.

[0091] (3) Adaptive granularity and cognitive consistency of node extraction are achieved. This invention breaks through the limitation of single granularity extraction in traditional NER technology by fine-tuning a large language model. The system can dynamically identify words, phrases, short sentences or whole sentences according to the teaching context, making the generated mind map nodes more in line with human cognitive habits, and effectively solving the problem of inappropriate granularity of descriptive text nodes that are too long or lack core key points.

[0092] (4) Achieved deeper semantic understanding and multimodal fusion in teaching. Through a late-stage multimodal fusion strategy, the system deeply integrates the visual hierarchical tree, audio-transcribed text, and adaptive nodes under the prompt word engineering. Utilizing the generative capabilities of the large model, features of different dimensions are transformed into a standardized Mermaid format, ensuring that the mind map possesses both the semantic details of the audio modality and the structural framework of the visual modality, thereby generating teaching aids with high information density and rigorous logic.

[0093] In summary, this invention solves the problems of insufficient utilization of visual modal information, incorrect hierarchical relationships of mind maps, and inappropriate node extraction granularity in traditional teaching video mind map generation methods by fine-grained mining of visual features for teaching videos and fusion of multimodal information, thus providing an efficient and accurate technical solution for automated teaching aids. Attached Figure Description

[0094] Figure 1 This is a general framework diagram of the present invention;

[0095] Figure 2 This is a flowchart of the overall algorithm of the present invention;

[0096] Figure 3 This is a flowchart of the keyframe recognition module for teaching videos of the present invention;

[0097] Figure 4 This is a flowchart of the visual hierarchy analysis module for instructional videos of the present invention. Detailed Implementation

[0098] The present invention will be further described below with reference to embodiments, but it should not be construed that the scope of the present invention is limited to the following embodiments. Various substitutions and modifications made based on ordinary technical knowledge and common practices in the art without departing from the above-described technical concept of the present invention should be included within the scope of protection of the present invention.

[0099] Example 1:

[0100] See Figures 1 to 4 A teaching video mind map generation system based on visual hierarchical analysis and multimodal fusion includes a visual feature extraction and analysis module, an audio analysis and node extraction module, and a multimodal fusion mind map generation module.

[0101] The visual feature extraction and analysis module is used for video keyframe extraction, coarse-grained extraction of visual information, and fine-grained analysis of hierarchical relationships.

[0102] The audio analysis and node extraction module is used for audio transcription and adaptive granularity node extraction.

[0103] The multimodal fusion mind map generation module is used to perform late-stage multimodal fusion of audio transcription information, mind map nodes, and hierarchical index tree information after visual analysis to generate teaching video mind maps;

[0104] The steps involved in generating mind maps for instructional videos using the instructional video mind map generation system include:

[0105] Step 1) Downsample the original input video to 1FPS video, and input the 1FPS video into the audio analysis and node extraction module and the visual feature extraction and analysis module respectively;

[0106] Step 2) Use the audio analysis and node extraction module to process the 1FPS video, separate the audio information, and transcribe the separated audio information into audio text;

[0107] The visual feature extraction and analysis module is used to process 1FPS video, separate visual information, and extract key frames of visual information.

[0108] Step 3) The audio analysis and node extraction module uses a large language model that has been fine-tuned with instructions to extract mind map nodes from the audio text;

[0109] The fine-tuning method is as follows: First, construct an instruction fine-tuning dataset. Each training sample is an "instruction-output" pair. Specifically, the "instruction" includes the role of the large model, the operation that the large model needs to perform, the content and format of the input data, and the content and format of the output data. The "output" is a list of mind map nodes in JSON format that is manually annotated. Second, fine-tune using LoRA. Finally, a small adapter of the model is trained, achieving an effect close to full parameter fine-tuning.

[0110] The visual feature extraction and analysis module performs coarse-grained visual analysis on the keyframes of the extracted visual information, and then performs fine-grained analysis of the hierarchical relationship to construct a hierarchical index tree;

[0111] Step 4) The multimodal fusion mind map generation module performs multimodal late-stage fusion of audio text, mind map nodes, and hierarchical index tree information after visual analysis to output a mind map.

[0112] Example 2:

[0113] The main structure of this embodiment is the same as that of embodiment 1. Furthermore, the visual feature extraction and analysis module includes a keyframe recognition module and a visual hierarchy analysis module.

[0114] The keyframe recognition module extracts keyframes from the 1FPS video and outputs the extracted keyframes.

[0115] The visual hierarchy analysis module performs coarse-grained extraction of visual information and fine-grained analysis of hierarchical relationships on keyframes, and outputs hierarchical index tree information after visual analysis.

[0116] Example 3:

[0117] The main structure of this embodiment is the same as any one of embodiments 1 to 2. Furthermore, the steps of the key frame recognition module in processing 1FPS video include reverse sampling, multi-dimensional filtering of low-quality frames, and filtering of similar frames.

[0118] Example 4:

[0119] The main structure of this embodiment is the same as any one of embodiments 1 to 3. Furthermore, reverse sampling refers to sampling from the video sequence... Iterate from the maximum timestamp to the minimum timestamp to obtain the sampling sequence. ;

[0120] Sampling sequence As shown below:

[0121] (1)

[0122] In the formula, Total duration This is the sampling step size;

[0123] Low-quality frame multidimensional filtering refers to removing samples from the sequence. Among blank frames, low-quality frames, and transition frames, frames that meet the edge density and image entropy thresholds are retained. The edge density threshold is 0.02, and the image entropy threshold is 3.5. The Canny edge detection operator is used to suppress noise.

[0124] Among them, the sampling sequence medium video frames Image entropy As shown below:

[0125] (2)

[0126] In the formula, It is grayscale value The probability of occurrence It is a smoothing factor;

[0127] Similar frame filtering refers to removing duplicate similar frames after multi-dimensional filtering to remove low-quality frames. It involves extracting low-frequency features of each frame using discrete cosine transform and calculating the mean, then generating a 0-1 hash sequence for that frame based on the mean. First, the preprocessed dimensions... The image I is subjected to DCT transformation to obtain the frequency domain coefficient matrix F, as shown below:

[0128]

[0129] In the formula, These are the spatial coordinates of the image domain. These are coordinates in the frequency domain. These are the normalization coefficients. Then we take one... In the low-frequency region, each DCT coefficient in the low-frequency region Compared with the global mean Compare and generate a binary matrix. As shown below:

[0130]

[0131] The final hash sequence can be represented as a binary string, as shown below:

[0132]

[0133] Based on the obtained hash sequence, calculate the hash of adjacent frames. Hamming distance As shown below:

[0134] (3)

[0135] In the formula, given the similarity frame threshold The default value is 4, which is the Hamming distance between two frames. If the two frames are duplicated, only the frame with the later timestamp is retained.

[0136] Example 5:

[0137] The main structure of this embodiment is the same as any one of embodiments 1 to 4. Furthermore, the visual hierarchy analysis module includes a visual information coarse-grained extraction module and a hierarchy relationship fine-grained analysis module.

[0138] The visual information coarse-grained extraction module refers to performing coarse-grained visual information extraction on the keyframes extracted by the keyframe recognition module to obtain coarse-grained features as output.

[0139] The hierarchical relationship fine-grained analysis module takes coarse-grained visual information as input, performs fine-grained analysis, and outputs hierarchical index tree information.

[0140] Example 6:

[0141] The main structure of this embodiment is the same as any one of embodiments 1 to 5. Furthermore, the visual information coarse-grained extraction module refers to the keyframes extracted by the keyframe recognition module. Perform coarse-grained extraction of visual information to obtain multiple text blocks, each text block Construct a multidimensional feature vector As shown below:

[0142]

[0143] In the formula, For the identified text content, The normalized coordinates of the top-left corner of the text block. For the width and height of the text block, For The estimated font size is derived from the ratio of the font size to the image height. The color attribute is extracted from the pixel mean. For frames The corresponding timestamp;

[0144] The spatial coordinates are rescaled and updated as follows:

[0145] (4)

[0146] In the formula, This is the scale correction factor. , These are the image height and width, respectively.

[0147] For keyframes The text content in each text block Perform word segmentation to obtain a word set. At the same time, maintain a global dictionary Used to record each term Global document frequency With global word frequency ;

[0148] Iterate through the keyframe set and update , As shown below:

[0149] (5)

[0150] (6)

[0151] In the formula, This represents the total number of keyframes. For keyframes, For indicator functions, It is a counting function;

[0152] The fine-grained hierarchical relationship analysis module, based on the visual information extracted in the coarse-grained stage, first introduces the lightweight language model TinyBERT to classify text blocks by function, extracts the global feature vector of the text block, and outputs the probability distribution of the text belonging to each functional category through the Softmax operator. As shown below:

[0153]

[0154] In the formula, This is a feature representation of the input text. For the model to class The original output score, It is the total number of categories. Sum the exponential scores for all categories to ensure that the sum of the output probabilities is 1;

[0155] Secondly, determine the text block. and Hierarchical relationship between Two text blocks in Relative offset in the axial direction and vertical distance is As shown below:

[0156] (7)

[0157]

[0158] In the formula, The coordinates of the left edge of the text block;

[0159] Generate a spatial dependency proposal As shown below:

[0160] (8)

[0161] In the formula, Indentation threshold The preset line spacing threshold, show It may be higher in the hierarchical structure. ;

[0162] The weighted hierarchy chain is established as follows:

[0163] (9)

[0164] In the formula, The semantic weights for the five types of text;

[0165] Hierarchical relationship As shown below:

[0166] (10)

[0167] In the formula, For text blocks and The semantic weight difference between them is calculated by the weight hierarchy chain. For a subordination function, if and only if the subordination proposal is true and At that time, the judgment for child nodes;

[0168] Finally, for each text block Construct a multidimensional salient feature vector As shown below:

[0169] (11)

[0170] In the formula, For visual saliency components, For semantic confidence components, For statistical importance components;

[0171] The hierarchical relationship of nested nodes is determined using a recursive algorithm. Let the set of nodes for all keyframes be . For any node Its dictionary-valued function The definition is as follows:

[0172] (16)

[0173] In the formula, The function directly retrieves the complete text of the node.

[0174] Example 7:

[0175] The main structure of this embodiment is the same as any one of embodiments 1 to 6. Further, the visual saliency component... fractions are subject to font size with vertical coordinates The combined effects are as follows:

[0176] (12)

[0177] In the formula, A coefficient used to balance the weights of spatial location and size;

[0178] Semantic confidence component As shown below:

[0179] (13)

[0180] In the formula, It is a set of five types of teaching tags. For categorical variables, For text block content, These are the weight parameters for the text classifier;

[0181] For each node, calculate each word item after word segmentation. Statistical scores As shown below:

[0182] (14)

[0183] In the formula, It is a natural constant;

[0184] Based on the above statistical scores, node The final statistical importance components As shown below:

[0185] (15)

[0186] In the formula, The text length is used for normalization. It is a minimal smoothing constant.

[0187] Example 8:

[0188] The main structure of this embodiment is the same as any one of embodiments 1 to 7. Furthermore, the audio analysis and node extraction module process includes an audio transcription module and an adaptive granularity node extraction module.

[0189] The audio transcription module refers to taking a 1FPS video as input, separating the audio information of the 1FPS video, and then transcribing the separated audio information into text output.

[0190] The audio separation and text transcription operations were performed by using the Python library MoviePy to separate the audio files into .wav format, and then using a pre-trained Qwen3-ASR model for text transcription. The audio text is shown below:

[0191]

[0192] In the formula, The most likely word sequence to be output, i.e., the audio text. For a single word in the set of possible word sequences, Given the input acoustic feature sequence, To provide a given acoustic feature Candidate words under the condition The conditional probability of occurrence For all possible Perform calculations and select the appropriate option. The largest word sequence is used as the final output.

[0193] The adaptive granularity node extraction module refers to the process of extracting and outputting mind map nodes from the input audio-transcribed text through prompt word engineering.

[0194] Example 9:

[0195] The main structure of this embodiment is the same as any one of embodiments 1 to 8. Furthermore, the adaptive granularity node extraction module uses a large model based on instruction fine-tuning to process ASR text and generate adaptive semantic units.

[0196] Example 10:

[0197] The main structure of this embodiment is the same as any one of embodiments 1 to 8. Furthermore, the multimodal fusion mind map generation module adopts multimodal late fusion to convert the parsing results of different modalities into structured text representations. The structured text representations are used as the context input of the large language model to output mind maps. This large language model is pre-trained using samples. The sample content includes original text, multi-level, multi-form semantic and logical structure annotations.

[0198] Input feature vector after multimodal fusion As shown below:

[0199] (17)

[0200] In the formula, A hierarchical index tree containing visual modality information. For granularity adaptive nodes based on audio extraction, This is the original transcribed text of the audio.

[0201] Example 11:

[0202] This invention presents a teaching video mind map generation system based on visual hierarchy analysis and multimodal fusion. Addressing issues in existing methods such as insufficient utilization of visual modal information, incorrect hierarchical relationships in mind maps, and inappropriate node extraction granularity, improvements are made in keyframe extraction, coarse-to-fine granular visual hierarchy analysis, and appropriately granular node extraction. The proposed method comprises a visual feature extraction and analysis module, an audio analysis and node extraction module, and a multimodal fusion mind map generation module. By combining video and audio modal information, it jointly guides mind map generation. The overall framework diagram is shown below. Figure 1 As shown.

[0203] like Figure 2 The diagram shown is the overall flowchart of the present invention, and the specific steps are as follows:

[0204] 1. Downsample the original input video to 1 FPS, input it into the model, and then proceed to steps 2 and 4;

[0205] 2. Extract audio information from the video, transcribe it into text, then proceed to step 3, while simultaneously feeding the transcribed text into step 7;

[0206] 3. Using a large language model fine-tuned with instructions, extract mind map nodes from the text transcribed from audio, and pass them to step 7;

[0207] 4. Extract visual information from the video and perform keyframe extraction;

[0208] 5. Perform coarse-grained visual analysis on the extracted keyframes;

[0209] 6. Perform fine-grained hierarchical analysis on the visual information after coarse-grained analysis preprocessing, and construct a hierarchical index tree to be passed to step 7;

[0210] 7. Integrate the audio transcription information, mind map nodes, and hierarchical index tree information from steps 2, 3, and 6, and perform multimodal late-stage fusion to jointly guide the generation of mind maps for teaching videos.

[0211] Example 12:

[0212] Key Technology 1: Keyframe Recognition for Teaching Videos

[0213] like Figure 3 The diagram shows a flowchart of the keyframe recognition module for the vertical scenario of teaching videos in this invention. This module consists of three parts: reverse sampling, multi-dimensional filtering of low-quality frames, and similar frame filtering, which will be described in detail below.

[0214] (1) Reverse sampling

[0215] In instructional videos, the content of the lecture transcripts is presented in a multi-step, progressive manner, meaning that the content of the same segment gradually becomes more complete over time. Addressing the positive correlation between timestamps and segment completeness in these videos, this invention proposes a reverse sampling algorithm. By iteratively traversing from the maximum timestamp to the minimum timestamp, the algorithm prioritizes traversing to the most complete state of each segment. The video sequence is defined as follows: The total duration is The sampling step size is Then the sampling sequence It can be represented as:

[0216] (1)

[0217] (2) Multidimensional filtering of low-quality frames

[0218] To filter out meaningless blank frames, low-quality frames, and transitional frames in teaching videos, this invention constructs a joint discrimination model that retains only frames that meet the edge density and image entropy thresholds. This invention employs the Canny edge detection operator, which effectively suppresses noise. Image entropy reflects the complexity of the image's grayscale distribution. For video frames... Its image entropy value Defined as:

[0219] (2)

[0220] in It is grayscale value The probability of occurrence This is a smoothing factor. In instructional videos, frames with higher information density have higher entropy values, while blank frames, low-quality frames, and transition frames have entropy values ​​close to 0.

[0221] (3) Similar frame filtering

[0222] To remove duplicate similar frames from the preprocessed frame set, a perceptual hashing algorithm suitable for educational video scenarios is introduced. For each frame, after scaling, low-frequency features of the image are extracted using discrete cosine transform and the mean is calculated. Based on the mean, a 0-1 hash sequence for that frame is generated. The perceptual hashing algorithm is then used to calculate the hash sequence of adjacent frames in the preprocessed frame set. Hamming distance :

[0223] (3)

[0224] when When the similarity threshold is reached, the two frames are considered to be duplicates, and only the frame with the later timestamp is retained.

[0225] Example 13:

[0226] Key Technology 2: Visual Hierarchy Analysis Module for Instructional Videos

[0227] like Figure 4 The diagram shows a flowchart of the visual hierarchy analysis module for the vertical scenario of teaching videos in this invention. This module is divided into two parts: coarse-grained extraction and fine-grained hierarchy relationship analysis, which will be described in detail below.

[0228] (1) Coarse-grained extraction of visual information

[0229] ①Text semantic capture based on multi-dimensional visual features

[0230] To accurately capture visual semantic features in vertical scenarios of teaching videos, this invention designs an OCR extraction scheme that combines multi-dimensional features. Let a keyframe... Multiple text blocks are extracted using text detection operators. For each text block... Construct a multidimensional feature vector ,in For the identified text content, The normalized coordinates of the top-left corner of the text block. For the width and height of the text block, For The estimated font size is derived from the ratio of the font size to the image height. The color attribute is extracted from the pixel mean. For frames The corresponding timestamp. To eliminate the impact of resolution differences on consistent visual understanding, the spatial coordinates are resized:

[0231] (4)

[0232] in This serves as a scale correction factor. These spatial metadata elements will become the core basis for hierarchical relationship analysis (such as determining indentation relationships).

[0233] ② Global dictionary construction

[0234] After acquiring all text blocks from all keyframes, in order to support fine-grained hierarchical analysis, the text content of each text block is first analyzed. Perform word segmentation to obtain a word set. At the same time, maintain a global dictionary Used to record each term Global document frequency ; and global word frequency Iterate through the keyframe set and update the dictionary:

[0235] (5)

[0236] (6)

[0237] in This represents the total number of keyframes. For keyframes, For indicator functions, This is a counting function.

[0238] (2) Fine-grained analysis of hierarchical relationships

[0239] This section performs fine-grained analysis on the preprocessed visual information. First, each text block is functionally classified. Then, heuristic hierarchical relationship analysis is performed based on the positional features of the text blocks. Finally, a hierarchical index tree in JSON format is recursively constructed to guide the generation of mind maps in the next stage.

[0240] Text block function classification

[0241] To perform functional analysis on text blocks (such as "definition", "background", etc.), this invention introduces a lightweight language model to classify text blocks by function, extracts the global feature vector of the text block, and outputs the probability distribution of the text belonging to each teaching function category through the Softmax operator. Based on the vertical scenarios of teaching videos, this invention categorizes the functions of text blocks into five types: "concepts", "definitions", "background", "examples", and "formulas".

[0242] Location-Semantic Hierarchical Analysis

[0243] The core of hierarchical analysis lies in identifying text blocks. and Father-son relationship Therefore, relying solely on a single spatial or semantic dimension is insufficient for accurate hierarchy determination. This invention proposes a multi-dimensional collaborative hierarchy determination mechanism. In spatial vision, teaching materials typically follow certain layout and indentation rules. Two text blocks are defined in... Relative offset in the axial direction for:

[0244] (7)

[0245] in Let these be the coordinates of the left edge of the text block. Set the indentation threshold. ,when When, that is to indicate It may be higher in the hierarchical structure. To prevent incorrect associations, the hierarchical relationship between nodes must also satisfy certain vertical position rules. Let the vertical distance between two text blocks be... Only when (Right now exist A spatial dependency proposal is generated when the following range constraints are met (below) :

[0246] (8)

[0247] in This is a preset line spacing threshold. This proposal serves as a preliminary logical basis, providing a reference for subsequent semantic verification. This invention defines semantic weights for five types of text. Establish a weighted hierarchy chain:

[0248] (9)

[0249] Therefore, hierarchical relationship Determined by both location and semantics:

[0250] (10)

[0251] Multidimensional evaluation hierarchy tree generation

[0252] After completing the hierarchical structure analysis, the abstract logical relationships need to be transformed into formatted expressions. A hierarchical index tree is recursively constructed, and knowledge nodes are encapsulated with multi-dimensional attributes based on visual, semantic, and word frequency to support multimodal fusion. For each knowledge node (text block)... Construct a multidimensional salient feature vector :

[0253] (11)

[0254] Visual salience components This reflects the importance of the knowledge points in the presentation. The score is affected by the font size. with vertical coordinates The combined effects:

[0255] (12)

[0256] in A coefficient used to balance the weights of spatial location and size.

[0257] Semantic confidence component This represents the reliability of the text classifier's classification of the node's teaching function:

[0258] (13)

[0259] in It is a set of five types of teaching tags. For categorical variables, For text block content, The weight parameters for the text classifier are: a higher score usually means that the text block has significant semantic features.

[0260] Statistical importance components This characterizes the statistical importance of nodes within the course knowledge system. For each node, it calculates the statistical importance of each word after segmentation. Statistical scores This score integrates three dimensions: local frequency, global discriminative power, and global popularity.

[0261] (14)

[0262] in This is a natural constant used to ensure that the heat enhancement factor is within... It still retains its basic weight. Based on the aforementioned statistical scores, the nodes... The final statistical importance components The calculation is as follows:

[0263] (15)

[0264] in The text length is used for normalization. As a minimal smoothing constant, this factor prevents long sentences like examples from receiving incorrectly high scores.

[0265] After obtaining the multidimensional feature vectors of all nodes, a recursive algorithm is used to nest the hierarchical relationships of the nodes. Let the set of nodes for all keyframes be . For any node Its dictionary-valued function The definition is as follows:

[0266] (16)

[0267] in The function directly retrieves the complete text of the node. This process ensures that even if the document has multiple levels of nesting, its hierarchy tree can be fully reconstructed through recursive calls.

[0268] Example 14:

[0269] Technique 3: Granular Adaptive Node Extraction for Instructional Videos

[0270] In the extraction of mind map nodes, the semantic granularity of the nodes directly affects the accuracy of the mind map. This invention employs a large model based on instruction fine-tuning to achieve node extraction with adaptive granularity that conforms to human cognitive habits. After instruction fine-tuning on a downstream task dataset of teaching videos, the large model, when processing ASR text, learns a mind map node summarization mechanism that conforms to human habits through few-shot examples in the instructions. This breaks the extreme granularity of traditional extraction paradigms and enables the generation of adaptive semantic units. For example:

[0271] "You are an expert in the field of teaching video mind map generation and mind map node recognition. You need to extract human-friendly, granular, adaptive mind map nodes from given teaching video subtitles, including words, phrases, short sentences, and complete sentences. Output in JSON list format, for example: Input: None Output: []; Input: None Output: []; Input: None Output: []."

[0272] Note: 1. The output must be a valid JSON list, where each element is a string. 2. If no mind map node is found, output an empty list []. """

[0273] Based on the context, large models can dynamically select the most appropriate node granularity and representation.

[0274] Example 15:

[0275] Technology 4: Multimodal fusion generation (based on technologies 1, 2, and 3)

[0276] Building upon the aforementioned multimodal processing module, a multimodal fusion prompt word is constructed. The multimodal generation module integrates the hierarchical tree obtained from visual analysis with semantic information extracted from audio and nodes with adaptive granularity to guide mind map generation. Leveraging the generative capabilities of a large language model, the features of visual and audio modalities are directly converted into standardized Mermaid format data to achieve mind map generation. This invention employs late-stage multimodal fusion, transforming the parsing results of different modalities into structured text representations as contextual input to the large model. The input feature vector after multimodal fusion... It can be represented as:

[0277] (17)

[0278] in A hierarchical index tree containing visual modality information. For granularity adaptive nodes based on audio extraction, The original transcribed text of the audio is used. Finally, a pre-trained large model is used to map the fused multimodal features into corresponding mind maps.

Claims

1. A teaching video mind map generation system based on visual hierarchy analysis and multimodal fusion, characterized in that: It includes a visual feature extraction and analysis module, an audio analysis and node extraction module, and a multimodal fusion mind map generation module; The visual feature extraction and analysis module is used for video keyframe extraction, coarse-grained extraction of visual information, and fine-grained analysis of hierarchical relationships. The audio analysis and node extraction module is used for audio transcription and adaptive granularity node extraction. The multimodal fusion mind map generation module is used to perform late-stage multimodal fusion of audio transcription information, mind map nodes, and hierarchical index tree information after visual analysis to generate teaching video mind maps; The steps involved in generating mind maps for instructional videos using the instructional video mind map generation system include: Step 1) Downsample the original input video to 1FPS video, and input the 1FPS video into the audio analysis and node extraction module and the visual feature extraction and analysis module respectively; Step 2) Use the audio analysis and node extraction module to process the 1FPS video, separate the audio information, and transcribe the separated audio information into audio text; The visual feature extraction and analysis module is used to process 1FPS video, separate visual information, and extract key frames of visual information. Step 3) The audio analysis and node extraction module uses a large language model that has been fine-tuned with instructions to extract mind map nodes from the audio text; The visual feature extraction and analysis module performs coarse-grained visual analysis on the keyframes of the extracted visual information, and then performs fine-grained analysis of the hierarchical relationship to construct a hierarchical index tree; Step 4) The multimodal fusion mind map generation module performs multimodal late-stage fusion of audio text, mind map nodes, and hierarchical index tree information after visual analysis to output a mind map.

2. The teaching video mind map generation system based on visual hierarchy analysis and multimodal fusion according to claim 1, characterized in that: The visual feature extraction and analysis module includes a keyframe recognition module and a visual hierarchy analysis module; The keyframe recognition module extracts keyframes from the 1FPS video and outputs the extracted keyframes. The visual hierarchy analysis module performs coarse-grained extraction of visual information and fine-grained analysis of hierarchical relationships on keyframes, and outputs hierarchical index tree information after visual analysis.

3. The teaching video mind map generation system based on visual hierarchy analysis and multimodal fusion according to claim 2, characterized in that: The keyframe recognition module processes 1FPS video in steps including reverse sampling, multi-dimensional filtering of low-quality frames, and filtering of similar frames.

4. The teaching video mind map generation system based on visual hierarchy analysis and multimodal fusion according to claim 3, characterized in that: Reverse sampling refers to sampling from a video sequence... Iterate from the maximum timestamp to the minimum timestamp to obtain the sampling sequence. ; Sampling sequence As shown below: (1) In the formula, Total duration This is the sampling step size; Low-quality frame multidimensional filtering refers to removing samples from the sampling sequence. Among blank frames, low-quality frames, and transition frames, frames that meet the edge density and image entropy thresholds are retained, and the Canny edge detection operator is used to suppress noise. Among them, the sampling sequence medium video frames Image entropy As shown below: (2) In the formula, It is grayscale value The probability of occurrence It is a smoothing factor; Similar frame filtering refers to removing duplicate similar frames after multidimensional filtering to remove low-quality frames. It involves extracting low-frequency features of each frame using discrete cosine transform and calculating the mean. Based on the mean, a 0-1 hash sequence for that frame is generated. Then, the hash sequence is used to calculate the hash sequence between adjacent frames. Hamming distance As shown below: (3) In the formula, given the similarity frame threshold When the Hamming distance between two frames If the two frames are duplicated, only the frame with the later timestamp is retained.

5. The teaching video mind map generation system based on visual hierarchy analysis and multimodal fusion according to claim 2, characterized in that: The visual hierarchy analysis module includes a coarse-grained visual information extraction module and a fine-grained hierarchy relationship analysis module; The visual information coarse-grained extraction module refers to performing coarse-grained visual information extraction on the keyframes extracted by the keyframe recognition module to obtain coarse-grained features as output. The hierarchical relationship fine-grained analysis module takes coarse-grained visual information as input, performs fine-grained analysis, and outputs hierarchical index tree information.

6. The teaching video mind map generation system based on visual hierarchy analysis and multimodal fusion according to claim 5, characterized in that: The visual information coarse-grained extraction module refers to the keyframes extracted by the keyframe recognition module. Perform coarse-grained extraction of visual information to obtain multiple text blocks, each text block Construct a multidimensional feature vector As shown below: In the formula, For the identified text content, The normalized coordinates of the top-left corner of the text block. For the width and height of the text block, For The estimated font size is derived from the ratio of the font size to the image height. The color attribute is extracted from the pixel mean. For frames The corresponding timestamp; The spatial coordinates are rescaled and updated as follows: (4) In the formula, This is the scale correction factor. , These are the image height and width, respectively. For keyframes The text content in each text block Perform word segmentation to obtain a word set. At the same time, maintain a global dictionary Used to record each term Global document frequency With global word frequency ; Iterate through the keyframe set and update , As shown below: (5) (6) In the formula, This represents the total number of keyframes. For keyframes, For indicator functions, It is a counting function; The fine-grained hierarchical relationship analysis module, based on the visual information extracted from the coarse-grained analysis, first introduces a lightweight language model to classify text blocks by function, extracts the global feature vector of the text block, and outputs the probability distribution of the text belonging to each functional category through the Softmax operator. As shown below: In the formula, This is a feature representation of the input text. For the model to class The original output score, It is the total number of categories. Sum the exponential scores for all categories to ensure that the sum of the output probabilities is 1; Secondly, determine the text block. and Hierarchical relationship between Two text blocks in Relative offset in the axial direction and vertical distance is As shown below: (7) In the formula, The coordinates of the left edge of the text block; Generate a spatial dependency proposal As shown below: (8) In the formula, Indentation threshold The preset line spacing threshold, show It may be higher in the hierarchical structure. ; The weighted hierarchy chain is established as follows: (9) In the formula, The semantic weights for the five types of text; Hierarchical relationship As shown below: (10) In the formula, For text blocks and The semantic weight difference between them is calculated by the weight hierarchy chain. For a subordination function, if and only if the subordination proposal is true and At that time, the judgment for child nodes; Finally, for each text block Construct a multidimensional salient feature vector As shown below: (11) In the formula, For visual saliency components, For semantic confidence components, For statistical importance components; The hierarchical relationship of nested nodes is determined using a recursive algorithm. Let the set of nodes for all keyframes be . For any node Its dictionary-valued function The definition is as follows: (16) In the formula, The function directly retrieves the complete text of the node.

7. The teaching video mind map generation system based on visual hierarchy analysis and multimodal fusion according to claim 6, characterized in that: Visual salience components fractions are subject to font size with vertical coordinates The combined effects are as follows: (12) In the formula, A coefficient used to balance the weights of spatial location and size; Semantic confidence component As shown below: (13) In the formula, It is a set of five types of teaching tags. For categorical variables, For text block content, These are the weight parameters for the text classifier; For each node, calculate each word item after word segmentation. Statistical scores As shown below: (14) In the formula, It is a natural constant. For the current text, Indicates the inclusion word The number of documents, Indicator The total number of times it appears in the entire document collection; Based on the above statistical scores, node The final statistical importance components As shown below: (15) In the formula, The text length is used for normalization. It is a minimal smoothing constant.

8. The teaching video mind map generation system based on visual hierarchy analysis and multimodal fusion according to claim 1, characterized in that: The audio analysis and node extraction module workflow includes an audio transcription module and an adaptive granularity node extraction module; The audio transcription module refers to taking a 1FPS video as input, separating the audio information of the 1FPS video, and then transcribing the separated audio information into text output. The adaptive granularity node extraction module refers to the process of extracting and outputting mind map nodes from the input audio-transcribed text through prompt word engineering.

9. The teaching video mind map generation system based on visual hierarchy analysis and multimodal fusion according to claim 8, characterized in that: The adaptive granularity node extraction module uses a large model based on instruction fine-tuning to process ASR text and generate adaptive semantic units.

10. The teaching video mind map generation system based on visual hierarchy analysis and multimodal fusion according to claim 1, characterized in that: The multimodal fusion mind map generation module adopts multimodal late fusion, which transforms the parsing results of different modalities into structured text representations, uses the structured text representations as context inputs to the large language model, and outputs mind maps. Input feature vector after multimodal fusion As shown below: (17) In the formula, A hierarchical index tree containing visual modal information. For granularity adaptive nodes based on audio extraction, This is the original transcribed text of the audio.

Images