Intelligent extraction and indexing system for knowledge points in computer course teaching video

Abstract

The invention discloses an intelligent extraction and indexing system for knowledge points in a computer course teaching video, and belongs to the technical field of computer data recognition. According to the system, video frames, audio streams and subtitles are separated through a video analysis module; the knowledge point extraction module adopts BiLSTM, an improved CNN and Transform to extract text, image and audio features respectively, and knowledge points are obtained through multi-modal fusion; the index construction module establishes a'knowledge point-time interval-video clip 'three-level index, and realizes standardized management in combination with a knowledge graph; the storage module adopts a mixed storage architecture to guarantee data security and access efficiency; the interaction module supports multi-mode query and visual display; according to the system, professional knowledge points in computer courses can be accurately extracted, rapid positioning and associated recommendation are realized, and the teaching resource utilization efficiency and learning experience are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer data recognition, in particular to a knowledge point intelligent extraction and indexing system in computer course teaching video. BACKGROUND

[0002] With the popularization of online education, computer course teaching video has become an important learning resource, but its linear playback characteristics make it difficult to locate knowledge points, which restricts learning efficiency. Traditional teaching video management methods mainly rely on manual annotation of knowledge points, which has the following limitations: first, the efficiency is low, a course often contains dozens of hours of video, and manual annotation consumes a lot of time; second, the accuracy is insufficient, different annotators have different understandings of knowledge points, which may lead to problems such as inconsistent terminology and timestamp deviation; third, there is a lack of correlation analysis, it is difficult to establish the hierarchical relationship between knowledge points, and it is difficult to meet the needs of personalized learning.

[0003] In the prior art, knowledge point extraction mainly relies on single modal data: text-based methods are difficult to handle oral expression and knowledge points in images; image recognition-based methods have low recognition accuracy for special content such as code screenshots; audio-based methods are easily affected by accents and noise; at the same time, simple keyword inverted index is used for index construction, without considering the special nature of computer subject knowledge points, such as the different meanings of "pointer" in C language and Python, resulting in insufficient query accuracy.

[0004] In addition, computer course knowledge points have the characteristics of dynamic updating, and existing systems lack adaptive updating mechanisms, making it difficult to cope with the addition or change of knowledge point terminology; with the explosive growth of MOOCs, micro-courses and other resources, the demand for intelligent extraction and efficient indexing of knowledge points in teaching videos is increasingly urgent, and an intelligent system that can integrate multi-modal data and adapt to the characteristics of computer disciplines is urgently needed. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a knowledge point intelligent extraction and indexing system in computer course teaching video, which solves the problems raised in the background art.

[0006] To achieve the above purpose, the present application realizes the following technical scheme: a knowledge point intelligent extraction and indexing system in computer course teaching video, comprising a video analysis module, a knowledge point extraction module, an index construction module, a storage module and an interactive module. The video parsing module extracts frames, separates audio, and parses subtitles from input computer course videos, outputting video frame sequences, audio streams, and text-based subtitles. The knowledge point extraction module, connected to the video parsing module, uses multimodal data fusion technology to identify and extract knowledge points from the video frame sequences, audio streams, and text-based subtitles. The index building module, connected to the knowledge point extraction module, associates the extracted knowledge points with video timestamps and builds a structured index. The storage module, connected to the video parsing module, knowledge point extraction module, and index building module, stores raw video data, intermediate processing results, and the structured index. The interaction module, connected to the storage module, receives user query requests and returns matching knowledge points and corresponding video clips.

[0007] Optionally, the knowledge point extraction module includes: a text feature extraction unit, which preprocesses the textual subtitles and then extracts semantic features using a bidirectional long short-term memory network. The calculation formula is as follows: in, For the first The word vector of the subtitle at any moment, , Forward and backward weight matrices, , For bias terms, , These are the forward and backward hidden states, respectively. For feature concatenation operations; the image feature extraction unit uses an improved convolutional neural network to extract features from whiteboard notes, code screenshots, and flowcharts in video frames. It strengthens key region features by introducing a spatial attention mechanism; the attention weight calculation formula is as follows: in, For the first Each convolutional feature map sigmoid Activation function The attention weights for this feature map are: The audio feature extraction unit extracts Mel-frequency cepstral coefficients from the audio stream and then obtains context-related features through a Transformer encoder; The multimodal fusion unit uses an attention mechanism to weight and fuse text, image, and audio features, outputting the final knowledge point feature vector. The fusion formula is: in, , , text, image, audio feature vector, respectively, is the modal attention weight, satisfying .

[0008] Optionally, the index construction module comprises: a knowledge point standardization unit, which matches the extracted knowledge points with a computer course knowledge graph, and unifies the knowledge point names and hierarchical relationships; a timestamp association unit, which determines the starting time and the ending time of the knowledge points in the video through video frame timestamp and audio speech pause detection, to form a time interval and an ending time . A multi-level index generation unit constructs a three-level inverted index of "knowledge point-time interval-video segment", wherein the first-level index is a knowledge point term, the second-level index is a time interval list corresponding to the knowledge point, and the third-level index is a video segment storage address associated with the time interval; an index optimization unit adopts local sensitive hashing for dimension reduction processing of the index items to reduce the query complexity, and the hash function is defined as: wherein, is a knowledge point feature vector, is a random vector, is a random offset in the interval [0, w), is the hash bucket width.

[0009] Optionally, the storage module comprises: an original data storage unit, which stores teaching video files using a distributed file system, supports breakpoint continuation and multiple copy backup; an intermediate data storage unit, which stores video frame sequences, audio waveform data and textual subtitles using MongoDB, wherein the video frames are stored at a frequency of 1-2 frames per second; a knowledge point database, which is built based on MySQL, stores knowledge point ID, name, domain, feature vector and associated timestamp information, and adopts B+ tree index to accelerate query; an index library, which stores structured index data using Elasticsearch, supports fuzzy query and semantic similarity sorting.

[0010] Optionally, the video analysis module comprises: a frame extraction unit, which adopts an adaptive key frame extraction algorithm, determines the key frame by calculating the structural similarity of adjacent frames, extracts the current frame when the value is lower than the threshold (0.6-0.8), and the calculation formula is: wherein, , are the mean values of the two frames, , are the standard deviations, ​Covariance, , Constant; audio separation unit, using Fourier transform separates speech signal and background noise, retains 16kHz sampling rate pure speech stream; subtitle analysis unit, carries out OCR identification to the subtitle embedded in the video, and compares and corrects with the speech text result, removes repeated and error text.

[0011] Optionally, the interaction module comprises: a query processing unit supporting keyword query, natural language query and image query, carrying out word segmentation and semantic expansion on the query content; a result sorting unit, according to the semantic similarity of knowledge points and queries, user historical click volume and the importance of knowledge points in courses, comprehensive sorting is carried out, and the sorting score formula is: Among them, The semantic similarity is, The normalized click volume is, The importance score is, , , The weight coefficient is; the visualization unit displays the knowledge point distribution in the form of time axis, automatically jumps to the video corresponding segment after clicking the knowledge point, and displays the associated knowledge point recommendation list.

[0012] Optionally, the multi-modal fusion unit further comprises a conflict resolution mechanism: when the knowledge points extracted from text, image and audio are different, the confidence score of each modal feature is calculated to make a weighted decision, and the confidence score calculation formula is: Among them, The probability value of knowledge point identification under the mode is, The adjustment coefficient (value 3-5) is, finally, the knowledge point with the highest confidence weighted sum is selected as the output.

[0013] Optionally, the index construction module further comprises a dynamic updating unit: when a new teaching video is added, incremental index updating is triggered, and only the newly extracted knowledge points are indexed and constructed, so that full reconstruction is avoided; when the knowledge point term is updated, the hierarchical relationship in the index is updated through the knowledge graph linkage, and the index version information is recorded to support backtracking query.

[0014] The present application provides a kind of knowledge point intelligent extraction and index system in computer course teaching video, with the following beneficial effects: The knowledge point intelligent extraction and index system in the computer course teaching video can significantly improve the accuracy of computer course knowledge point extraction through multi-modal data fusion technology; compared with single modal method, it can effectively identify the knowledge points of oral expression, the contents of image presentation and the key information implied in the voice by combining text, image and audio features for comprehensive judgment, especially for the complex knowledge points such as 'pointer operation' and'recursive call' which need to be combined with code demonstration and explanation, the recognition accuracy is substantially improved; Meanwhile, the index construction mechanism fully considers the characteristics of the knowledge system of computer discipline, realizes the standardization of knowledge points through association with knowledge graph, and avoids the problem of term confusion; the three-level index structure and dynamic updating mechanism not only ensure the efficiency of query, but also adapt to the dynamic changes of knowledge points, users can quickly locate the target content through keywords, natural language or even code screenshots, which greatly shortens the time from query to effective information acquisition, especially for students who learn independently, it can significantly improve the learning efficiency; Among them, the hybrid storage architecture of the system takes into account the data security and access efficiency, distributed storage ensures the reliable saving of a large number of teaching videos, and the cooperation of structured database and index database makes it still maintain fast response when concurrent query; the visual display function of the interactive module directly associates the knowledge points with the video timeline, helping users to build a clear knowledge context, and the associated knowledge point recommendation helps to expand the learning depth and form systematic cognition.

[0015] For teaching management, the system reduces the workload of teachers labeling knowledge points, so that they can invest more effort in teaching content design; by analyzing the query frequency and learning time of knowledge points, it can also provide data support for course optimization, such as supplementing explanation resources for high-frequency query knowledge points, improving the pertinence and effectiveness of teaching quality. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The flowchart of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments.

[0018] In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more; the orientation or positional relationship indicated by the terms "upper", "lower", "left", "right", "inner", "outer", "front end", "rear end", "head", "tail" and the like is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" and the like are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0019] In the description of the present application, it should be noted that, unless otherwise specified and limited, the terms "connected", "connected" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0020] Please refer to Figure 1 The present application provides a technical solution: a kind of knowledge point intelligent extraction and index system in computer course teaching video, including video analysis module, knowledge point extraction module, index construction module, storage module and interactive module; Video analysis module is used to carry out frame extraction, audio separation and subtitle analysis to the input computer course teaching video, and output video frame sequence, audio stream and text subtitle;Video analysis module includes: frame extraction unit, adopts adaptive key frame extraction algorithm, determines key frame by calculating the structural similarity of adjacent frames, when The value is lower than the threshold (0.6-0.8) extracts the current frame, The calculation formula is: Among them, , The mean of two frames is , The standard deviation is The covariance is , A constant; audio separation unit, Fourier transform separates speech signal and background noise, and retains 16kHz sampling rate pure speech stream;The subtitle analysis unit carries out OCR identification (adopts CRNN algorithm) to the subtitle embedded in the video, and compares and corrects the speech to text result (based on Whisper model), removes duplicate and error text; The knowledge point extraction module, connected to the video parsing module, uses multimodal data fusion technology to identify and extract knowledge points from video frame sequences, audio streams, and textual subtitles. The knowledge point extraction module includes a text feature extraction unit, which preprocesses the textual subtitles (including stop word removal and technical terminology standardization) and then uses a bidirectional long short-term memory network (BiLSTM) to extract semantic features. The calculation formula is as follows: in, For the first The word vector of the subtitle at any moment, , Forward and backward weight matrices, , For bias terms, , These are the forward and backward hidden states, respectively. For feature concatenation operations; the image feature extraction unit uses an improved convolutional neural network (CNN) to extract features from whiteboard notes, code screenshots, and flowcharts in video frames. It strengthens the features of key regions (such as code keywords and formula symbols) by introducing a spatial attention mechanism. The attention weight calculation formula is as follows: in, For the first Each convolutional feature map It is the sigmoid activation function. The attention weights for this feature map are: The audio feature extraction unit extracts Mel-frequency cepstral coefficients (MFCCs) from the audio stream and then obtains context-related features through a Transformer encoder; The multimodal fusion unit uses an attention mechanism to weight and fuse text, image, and audio features, outputting the final knowledge point feature vector. The fusion formula is: in, , , These are text, image, and audio feature vectors, respectively. For modal attention weights, satisfying ; The multimodal fusion unit also includes a conflict resolution mechanism: when there are differences in the knowledge points extracted from text, images, and audio, a weighted decision is made by calculating the confidence scores of each modality feature. The confidence calculation formula is as follows: in, The probability value of the knowledge point identification under the modal, is an adjustment coefficient (value 3-5), and finally the knowledge point with the highest confidence weighted sum is selected as the output; The index construction module is connected with the knowledge point extraction module, associates the extracted knowledge points with the video time stamp, and constructs a structured index; the index construction module comprises: a knowledge point standardization unit that matches the extracted knowledge points with a computer course knowledge graph (covering subfields such as programming languages, data structures, algorithms, etc.), unifies the knowledge point name and hierarchical relationship (such as “binary tree” belonging to “data structure-tree structure”); a time stamp association unit that determines the start time and end time of the knowledge point in the video by video frame time stamp and audio speech pause detection, forming a time interval ; a multi-level index generation unit that constructs a “knowledge point-time interval-video segment” three-level inverted index, wherein the first-level index is the knowledge point term, the second-level index is the time interval list corresponding to the knowledge point, and the third-level index is the storage address of the video segment associated with the time interval; an index optimization unit that uses local sensitive hashing (LSH) to reduce the dimension of the index item, reduces the query complexity, and the hash function is defined as: wherein, is a knowledge point feature vector, is a random vector, is a random offset in the interval [0, w), is the hash bucket width; The index construction module further comprises a dynamic updating unit: when a new teaching video is added, incremental index updating is triggered, and only the newly extracted knowledge points are indexed to avoid full reconstruction; when a knowledge point term update (such as adding a programming language syntax) is detected, the hierarchical relationship in the index is updated through the knowledge graph linkage, and index version information is recorded to support backtracking query; The storage module is connected with the video analysis module, the knowledge point extraction module and the index construction module respectively, and is used for storing original video data, intermediate processing results and structured index; the storage module comprises: an original data storage unit that uses a distributed file system (such as HDFS) to store teaching video files, supports breakpoint continuation and multiple copy backup; an intermediate data storage unit that uses MongoDB to store video frame sequences (in binary form), audio waveform data and textual subtitles, wherein the video frames are sampled and stored at a frequency of 1-2 frames per second; a knowledge point database based on MySQL, which stores knowledge point ID, name, domain, feature vector and associated time stamp information, and uses B+ tree index to speed up query; an index library that uses Elasticsearch to store structured index data, supports fuzzy query and semantic similarity sorting; An interactive module is connected with the storage module, receives a user query request and returns matched knowledge points and corresponding video clips; the interactive module comprises: a query processing unit supporting keyword query, natural language query and image query (such as uploading a code screenshot to find corresponding knowledge points), performing word segmentation and semantic expansion (calculating similarity based on Word2Vec word vector) on the query content; a result sorting unit performing comprehensive sorting according to semantic similarity of knowledge points and the query, historical click volume of the user and importance of the knowledge points in the course (obtained by teacher annotation or learning behavior data analysis), and a sorting score formula is: wherein, is semantic similarity, is normalized click volume, is importance score, , , is a weight coefficient; a visualization unit displays knowledge point distribution in the form of a time axis, automatically jumps to the video corresponding segment after clicking the knowledge point, and displays a recommended list of associated knowledge points.

[0021] Embodiment A computer college of a certain university applied the system in the course of "Data Structure and Algorithm", which contains 48 hours of teaching videos, covers 12 core knowledge points such as linked list, tree and graph, and involves a large number of code demonstrations and algorithm derivation; In the system deployment stage, the video analysis module processes the original video, extracts key frames (an average of 1 frame every 10 seconds), separates the audio and generates subtitle text, wherein the recognition accuracy of the code screenshot frame reaches 98.7%, which is significantly higher than that of the traditional OCR tool (82.3%); the knowledge point extraction module successfully identifies the easily confused knowledge points such as "binary tree traversal" and "quick sort optimization" through multi-modal fusion, wherein the extraction of the "heap sort" algorithm completely covers the three aspects of teacher explanation (audio), code demonstration (image) and subtitle summary (text), and the timestamp error is controlled within 2 seconds; After the index is built, when a student queries "balanced binary tree" through the interactive module, the system returns three segments including definition explanation (00:15:30-00:20:15), insertion operation demonstration (00:22:40-00:28:10) and complexity analysis (00:30:05-00:35:20), and recommends associated knowledge points "red-black tree" and "B-tree"; one-semester application data shows that the average knowledge point positioning time of students is shortened from 15 minutes to 1.2 minutes, and the number of questions about "how to find XX knowledge point" in the course forum is reduced by 72%; teachers find that the query frequency of "shortest path algorithm of graph" is the highest through the system, and accordingly supplement 3 extension cases, and the correct rate of examination of this knowledge point is improved by 18%; The administrator updates the "union-find set" knowledge point video in the course through the dynamic updating unit, and the system only takes 15 minutes to complete the incremental index updating, so as to ensure that the new content is available in time. The case verifies the practicability of the system in computer course teaching, and the multi-modal extraction and intelligent index ability effectively solve the retrieval problem of the traditional teaching video.

[0022] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art, according to the technical solution and inventive concept of the present application, makes equivalent replacement or change within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A system for intelligent extraction and indexing of knowledge points in computer course teaching videos, characterized in that, It includes a video parsing module, a knowledge point extraction module, an index building module, a storage module, and an interaction module; The video parsing module is used to extract frames, separate audio, and parse subtitles from input computer course teaching videos, and output video frame sequences, audio streams, and text-based subtitles. The knowledge point extraction module, connected to the video parsing module, uses multimodal data fusion technology to identify and extract knowledge points from video frame sequences, audio streams, and textual subtitles. The index building module, connected to the knowledge point extraction module, associates the extracted knowledge points with video timestamps and builds a structured index. The storage module, connected to the video parsing module, knowledge point extraction module, and index building module, stores the original video data, intermediate processing results, and structured index. The interaction module, connected to the storage module, receives user query requests and returns matching knowledge points and corresponding video segments.

2. The system according to claim 1, characterized in that, The knowledge point extraction module includes: a text feature extraction unit, which preprocesses the textual subtitles and then uses a bidirectional long short-term memory network to extract semantic features. The calculation formula is as follows: in, For the first The word vector of the subtitle at any moment, , Forward and backward weight matrices, , For bias terms, , These are the forward and backward hidden states, respectively. For feature concatenation operations; the image feature extraction unit uses an improved convolutional neural network to extract features from whiteboard notes, code screenshots, and flowcharts in video frames. It strengthens key region features by introducing a spatial attention mechanism; the attention weight calculation formula is as follows: in, For the first Each convolutional feature map sigmoid Activation function The attention weights for this feature map are: The audio feature extraction unit extracts Mel-frequency cepstral coefficients from the audio stream and then obtains context-related features through a Transformer encoder; The multimodal fusion unit uses an attention mechanism to weight and fuse text, image, and audio features, outputting the final knowledge point feature vector. The fusion formula is: in, , , These are text, image, and audio feature vectors, respectively. For modal attention weights, satisfying .

3. The system according to claim 1, characterized in that, The index construction module includes: a knowledge point standardization unit, which matches the extracted knowledge points with the computer course knowledge graph to unify the knowledge point names and hierarchical relationships; and a timestamp association unit, which determines the start time of the knowledge point in the video by using video frame timestamps and audio pause detection. and end time Forming a time interval The multi-level index generation unit constructs a three-level inverted index: "knowledge point - time interval - video clip". The first-level index represents the knowledge point terminology, the second-level index is the list of time intervals corresponding to that knowledge point, and the third-level index is the storage address of the video clip associated with each time interval. The index optimization unit uses locality-sensitive hashing to reduce the dimensionality of index items, thereby reducing query complexity. The hash function is defined as follows: in, For knowledge point feature vectors, It is a random vector. The random offset within the interval [0, w). This represents the width of the hash bucket.

4. The system according to claim 1, characterized in that, The storage module includes: a raw data storage unit, which uses a distributed file system to store teaching video files and supports breakpoint resume and multiple copy backups; an intermediate data storage unit, which uses MongoDB to store video frame sequences, audio waveform data, and textual subtitles, wherein video frames are sampled and stored at a frequency of 1-2 frames per second; a knowledge point database, built on MySQL, which stores knowledge point IDs, names, domains, feature vectors, and associated timestamp information, and uses B+ tree indexes to accelerate queries; and an index library, which uses Elasticsearch to store structured index data and supports fuzzy queries and semantic similarity sorting.

5. The system according to claim 1, characterized in that, The video parsing module includes a frame extraction unit, which employs an adaptive keyframe extraction algorithm to determine keyframes by calculating the structural similarity of adjacent frames. Value below threshold Extract the current frame at (0.6-0.8) intervals. The calculation formula is: in, , The average of the two frames. , Standard deviation For covariance, , The audio separation unit uses Fourier transform to separate the speech signal from the background noise, retaining a clean speech stream with a sampling rate of 16kHz; the subtitle parsing unit performs OCR recognition on the subtitles embedded in the video and compares and corrects them with the speech-to-text results to remove duplicate and erroneous text.

6. The system according to claim 1, characterized in that, The interactive module includes: a query processing unit that supports keyword queries, natural language queries, and image queries, and performs word segmentation and semantic expansion on the query content; and a result ranking unit that ranks results based on the semantic similarity between the knowledge points and the query, the user's historical click volume, and the importance of the knowledge points in the course. The ranking score formula is as follows: in, For semantic similarity, To normalize the click count, Assess importance by rating it. , , The weighting coefficients are used for visualization. The distribution of knowledge points is displayed in the form of a timeline. Clicking on a knowledge point will automatically jump to the corresponding video segment and display a list of recommended related knowledge points.

7. The system according to claim 2, characterized in that, The multimodal fusion unit also includes a conflict resolution mechanism: when there are differences in the knowledge points extracted from text, images, and audio, a weighted decision is made by calculating the confidence scores of each modality feature. The confidence score calculation formula is as follows: in, This represents the probability value for knowledge point recognition in this modality. The adjustment coefficient (value 3-5) is used to select the knowledge point with the highest confidence weighted sum as the output.

8. The system according to claim 3, characterized in that, The index building module also includes a dynamic update unit: when a new teaching video is added, an incremental index update is triggered, and only the newly extracted knowledge points are indexed to avoid a full reconstruction; when an update of knowledge point terminology is detected, the hierarchical relationship in the index is updated in conjunction with the knowledge graph, and the index version information is recorded to support backtracking queries.

Images