Classroom video segmentation and catalog creation method and system based on multimodal information
Through multimodal information fusion and deep learning model, the problems of classroom video segmentation and directory creation are solved, high-precision video segmentation and teaching content extraction are realized, and the learning efficiency and quality of classroom videos are improved.
Patent Information
- Application Number
- CN202410088780.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-22
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2044-01-22
AI Technical Summary
The prior art is difficult to automatically segment and create catalogs of actual classroom videos, especially because the classroom activities are rich and the audio is complex, making it difficult to ensure the efficiency and quality of video segmentation.
Using a multimodal information-based method, by obtaining classroom audio and video information, extracting teacher audio and video features, combining deep learning models such as CNN, LSTM, and Resnet, detecting classroom activity status and segmenting videos, generating knowledge points and creating teaching Table of contents.
It realizes high-precision classroom video segmentation and directory creation, eliminates teaching-independent content, and improves video quality and learning efficiency.
Smart Images
Figure CN118055266B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of smart classrooms, and more specifically, relates to a classroom video segmentation and directory creation method and system based on multimodal information. Background Art
[0002] With the continuous development of computer and network technology, online learning has become a widely adopted learning method. Teaching videos play an important role in it. It is very important to develop methods to use these teaching videos effectively. Among them, dividing teaching videos into segments by topic is a challenging research area. This kind of video division with topics can help students effectively navigate to the required part of the video. Although it is possible to manually browse each teaching video and divide the knowledge points, as the number of teaching videos increases, manually creating a catalog for all teaching videos will be an impossible task.
[0003] The existing automatic teaching video segmentation method can complete video segmentation and create each theme, and generate a teaching video directory. However, it is aimed at pre-recorded teaching videos or some open class videos, and basically cannot process actual classroom videos. This is because there are a lot of activities in the actual classroom, such as lectures, discussions, and exercises. In order to facilitate students' review after class, activities that are not related to the teacher's teaching, such as discussions and exercises, need to be eliminated, and only teaching activities such as lectures and questions are retained; at the same time, the audio collected in the classroom contains noise and the information is very complex. In order to ensure the quality of the teaching video, some sounds need to be separated. These are the difficulties in actual classroom video segmentation. Therefore, how to segment and create a directory for actual classroom videos is of great research significance. Summary of the invention
[0004] In view of the defects and improvement needs of the prior art, the present invention provides a method and system for classroom video segmentation and directory creation based on multimodal information, which aims to perform high-precision segmentation and directory creation on actual classroom videos.
[0005] To achieve the above-mentioned purpose, according to one aspect of the present invention, a method for classroom video segmentation and directory creation based on multimodal information is provided, including: S1, obtaining audio information and video information generated in the classroom, and extracting teacher audio from the audio information; S2, inputting the audio information and the video information into a classroom activity detection model, obtaining the classroom activity status and the corresponding video clips appearing in the video information, and splicing the video clips in which the classroom activity status is the teacher's explanation to obtain a classroom teaching video; S3, inputting the audio in the teacher audio corresponding to the classroom teaching video, and the classroom teaching video into a classroom video segmentation model, so as to segment the classroom teaching video into different classroom teaching video segments according to the knowledge point themes; S4, generating the knowledge point themes of each of the classroom teaching video segments and combining them to create a teaching directory of the classroom teaching video.
[0006] Furthermore, the extracting of teacher audio from the audio information includes: extracting a first audio feature from the audio information and inputting it into a CNN network to obtain a first feature sequence; using an LSTM network to sequentially perform time series processing on the first feature sequence, and using clustering classification on the processing results to separate student audio, environmental audio and the teacher audio.
[0007] Furthermore, in S2, the information processing process in the classroom activity detection model is: extracting a second audio feature from the audio information, extracting a video feature from the video information, and fusing and splicing the second audio feature and the video feature in time sequence; inputting the fused and spliced feature vector into the Resnet network to obtain the classroom activity status appearing in the video information and the corresponding video clips.
[0008] Furthermore, the extracting of video features from the video information includes: extracting key points of student head movements from the video information to generate a student head posture feature matrix; extracting key points of teacher movements from the video information to generate a teacher posture feature matrix; the video features include the student head posture feature matrix and the teacher posture feature matrix.
[0009] Furthermore, the classroom activity status includes: teacher explanation, communication and discussion, student exercises and others; teacher explanation includes: teacher lectures and teacher explanations.
[0010] Furthermore, in S3, the information processing process in the classroom video segmentation model is: extracting text features X from the audio corresponding to the classroom teaching video in the teacher audio t1 ; Extract video features X from the classroom teaching video v2 ; X by time t1 and X v2After alignment and fusion, it is converted into a bag-of-words model, and the bag-of-words model is input into the LDA model to obtain the knowledge point theme probability distribution in each key frame of the classroom teaching video; if the similarity between the knowledge point theme probability distributions of two adjacent key frames is greater than a first threshold, the two adjacent key frames are divided into the same classroom teaching video segment, otherwise, the two adjacent key frames are divided into different classroom teaching video segments.
[0011] Furthermore, the video feature X is extracted from the classroom teaching video. v2 , including: for each frame in the classroom teaching video, if the similarity between the frame and the next frame is less than a second threshold, the frame is used as a key frame; for each key frame in the classroom teaching video, the text on the presentation and the blackboard in the key frame is extracted to form corresponding text information; the text information of each key frame is combined to obtain the video feature X v2 .
[0012] Furthermore, the time-based t1 and X v2 After alignment and fusion, it is converted into a bag-of-words model, including: t1 After splicing the text corresponding to the i-th key frame in the classroom teaching video, it is spliced with the text information of the i-th key frame to obtain the i-th splicing information, i = 1, 2, ..., I, I is the number of key frames in the classroom teaching video; splicing each splicing information in chronological order to obtain fusion information, and converting the fusion information into the bag-of-words model.
[0013] According to another aspect of the present invention, a classroom video segmentation and directory creation system based on multimodal information is provided, comprising: a processor; a memory storing a computer executable program, which, when executed by the processor, enables the processor to execute the classroom video segmentation and directory creation method based on multimodal information as described above.
[0014] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for classroom video segmentation and directory creation based on multimodal information as described above is implemented.
[0015] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:
[0016] (1) A method for classroom video segmentation and directory creation based on multimodal information is provided. The method fully considers the problem in the prior art that "there are abundant activities in the actual classroom and the information generated is very complex, which makes it difficult to segment the actual classroom video". The method fully utilizes the audio data and video data generated in the classroom, adopts the multimodal information fusion method to segment the classroom video, and extracts keywords to complete the creation of the classroom video directory, which effectively realizes automatic video segmentation and improves the segmentation accuracy.
[0017] In addition, based on the teacher audio separated from the audio information, the irrelevant content in the classroom video is removed, the teacher's voice quality is improved, the quality of the classroom teaching video is improved, and it is convenient for students to learn after class;
[0018] (2) It is proposed to extract key frames from the video by detecting the similarity between adjacent frames in the video, and to segment the classroom video based on the similarity between the probability distribution of knowledge points and topics in the key frames, which effectively improves the efficiency of classroom video segmentation and directory creation. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 A flowchart of a classroom video segmentation and directory creation method based on multimodal information provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0020] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0021] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the drawings are used to distinguish similar objects but not necessarily to describe a specific order or sequence.
[0022] Figure 1 Flow chart of a method for classroom video segmentation and directory creation based on multimodal information provided by an embodiment of the present invention. Figure 1 ,The classroom video segmentation and directory creation method based on multimodal information includes operations S1-S4.
[0023] Operation S1, obtaining audio information and video information generated in the classroom, and extracting the teacher's audio from the audio information.
[0024] In this embodiment, the classroom can be an offline classroom, and the corresponding audio information and video information are the audio and video recorded during the teaching process; it can also be an online classroom, and the corresponding audio information and video information are pre-recorded audio and video. This method has great advantages when used in offline classroom scenarios.
[0025] According to an embodiment of the present invention, extracting teacher audio from audio information includes: extracting a first audio feature from the audio information and inputting it into a convolutional neural network (CNN) to obtain a first feature sequence; using a long short-term memory (LSTM) network to sequentially perform time series processing on the first feature sequence, and clustering and classifying the processed results, preferably, using K-means clustering to separate student audio, environmental audio and teacher audio.
[0026] For example, the teacher-student-environment audio separation model is used to extract the teacher audio from the audio information. The specific process is as follows:
[0027] 1) Use audio processing tools to extract the first audio feature X from the audio information S0 Preferably, the logarithmic power spectrum is extracted as X S0 The logarithmic power spectrum is the result of performing a short-time Fourier transform on the input audio, taking the square of the modulus of each coefficient, and then taking the logarithm of it.
[0028] 2) The first audio feature X S0 Enter the CNN network. As an example, the CNN network includes 3 convolutional layers and 2 pooling layers. The two pooling layers are located after the first and second convolutional layers respectively. The pooling layer size is 1*3. The first convolutional layer uses a convolution kernel size of 5*10, and the convolution kernel sizes of the second and third convolutional layers are both 3*3.
[0029] 3) The feature map after convolution pooling is input into the LSTM network, and the first feature sequence learned by the CNN network is subjected to time series processing. The output layer is classified by, for example, a spectral clustering method to separate the student audio, the environmental audio, and the teacher audio. Preferably, considering that the teacher occupies an absolute position in the classroom audio, the class with the most audio after clustering is classified as the teacher audio; when generating the training set, the pure teacher voice is used, and the environmental sound and the student voice are mixed according to different signal-to-noise ratios to obtain the mixed audio as the audio of the training set.
[0030] Operation S2, input the audio information and video information into the classroom activity detection model, obtain the classroom activity status and corresponding video clips appearing in the video information, and splice the video clips in which the classroom activity status is the teacher's explanation to obtain the classroom teaching video.
[0031] According to an embodiment of the present invention, in operation S2, the information processing process in the classroom activity detection model includes sub-operation S21-sub-operation S22.
[0032] In sub-operation S21, a second audio feature is extracted from the audio information, a video feature is extracted from the video information, and the second audio feature and the video feature are fused and spliced in time sequence.
[0033] According to an embodiment of the present invention, video features are extracted from video information, including: extracting student head movement key points from video information to generate a student head posture feature matrix; extracting teacher movement key points from video information to generate a teacher posture feature matrix; the video features include a student head posture feature matrix and a teacher posture feature matrix.
[0034] In sub-operation S22, the fused and spliced feature vectors are input into the Resnet network to obtain the classroom activity status and corresponding video clips appearing in the video information.
[0035] According to an embodiment of the present invention, the classroom activity status includes: teacher explanation, communication and discussion, student exercises and others. Teacher explanation includes: teacher lectures and teacher lecturing.
[0036] As an example, the implementation process of operation S2 is as follows:
[0037] 1) Using audio processing tools, extract the second audio feature X from the audio information S1 Preferably, the log-mel spectrogram feature is extracted. Further, the second audio feature X S1 Input into the CNN network for feature extraction.
[0038] 2) Extract video features X from video information V1 The human body posture algorithm is used to extract the key points of the teacher's movements and generate the teacher's posture feature matrix x1. The key points of the student's head movements are extracted to generate the student's head posture feature matrix x2. The video feature X V1 It can be expressed as X V1 =(x1,x2).
[0039] 3) The results extracted from the CNN network and X V1The splicing is performed according to the time series features, and the spliced feature vector is input into the Resnet network for feature vector feature extraction. Preferably, the Resnet network adopts the Resnet18 structure, the network depth is 18, and it contains 17 convolutional layers and one fully connected layer; ReLU is used as the activation function of the model, BCE is used as the loss function of the model, the initial learning rate is set to 0.005, and the Adam optimization algorithm is used to optimize the learning rate, and finally the classroom activity status and corresponding video clips appearing in the video information are output. The training set annotations required for model training are manually annotated using expert experience.
[0040] Furthermore, through the video editing tool, only the video clips corresponding to the two states of the teacher lecturing and the teacher explaining the questions are retained, and the classroom teaching video is obtained by combining them.
[0041] Operation S3, inputting the audio corresponding to the classroom teaching video in the teacher audio and the classroom teaching video into the classroom video segmentation model to segment the classroom teaching video into different classroom teaching video segments according to the knowledge point themes.
[0042] According to an embodiment of the present invention, in operation S3, the information processing process in the classroom video segmentation model includes sub-operations S31 to S33.
[0043] In sub-operation S31, the text feature X is extracted from the audio corresponding to the classroom teaching video in the teacher audio. t1 ; Extract video features X from classroom teaching videos v2 .
[0044] According to an embodiment of the present invention, video features X are extracted from classroom teaching videos. v2 , including: for each frame in the classroom teaching video, if the similarity between the frame and the next frame is less than a second threshold, the frame is taken as a key frame; for each key frame in the classroom teaching video, the text on the presentation and the blackboard in the key frame is extracted to form corresponding text information; the text information of each key frame is combined to obtain the video feature X v2 .
[0045] To ensure efficiency, in this embodiment, a similarity test is performed on each frame in the classroom teaching video and the next frame. When the similarity is less than the second threshold, the frame image is output as the key frame T i , and obtain the classroom teaching image set P1. Preferably, pHash algorithm is used for detection, and the specific process is: reduce the image size, reduce the image to a size of 32x32, remove the differences in various image sizes and image ratios, and only retain basic information such as structure and brightness; then convert the reduced image into a grayscale image, and calculate the DCT of the image. For a two-dimensional image I[i,j], its two-dimensional DCT change can be calculated by the following formula:
[0046]
[0047] Among them, M and N are the width and height of the image, and C(u) and C(v) are coefficients. After obtaining the DCT matrix, take the 8*8 matrix in the upper left corner, which presents the lowest frequency in the image, and then calculate the average value of the DCT coefficient. According to the DCT matrix, set the 64-bit hash value to 0 or 1, and set it to 1 if it is greater than or equal to the DCT mean, and set it to 0 if it is less than the DCT mean. Combined together, it constitutes a 64-bit integer. When calculating the similarity between two adjacent frames, calculate the 64-bit encoded Hamming distance of the two images. If the distance is greater than the set threshold, it means that the two images are less similar, and this frame of image is output as the key frame T i .
[0048] Afterwards, we use the text recognition algorithm to extract the text on the presentation and the blackboard text as the video feature X v2 , X v2 = {W T1 ,W T2 ,…W TI}. Ti Denotes the text on the presentation and blackboard in the i-th key frame. Extract text features X from the audio corresponding to the classroom teaching video in the teacher audio t1 ={L k1 ,L k2 ,…L kI}, L ki Represents X t1 The text corresponding to the i-th key frame in the classroom teaching video.
[0049] In sub-operation S32, X is set according to time. t1 and X v2 After alignment and fusion, it is converted into a bag-of-words model, and the bag-of-words model is input into the LDA model to obtain the probability distribution of knowledge point topics in each key frame of the classroom teaching video.
[0050] According to an embodiment of the present invention, X is converted into t1 and X v2 After alignment and fusion, it is converted into a bag-of-words model, including: t1 The text L corresponding to the i-th key frame in the classroom teaching video ki After splicing, it is combined with the text information W of the i-th key frame Ti Perform splicing to obtain the i-th splicing information F Ti , i = 1, 2, ..., I, I is the number of key frames in the classroom teaching video; splice each splicing information F in chronological order Ti Get fusion information X f , converting the fused information into a bag-of-words model.
[0051] Preferably, further preprocessing is performed, and then words that appear frequently in the document but do not affect the text semantically, such as personal terms and modal particles such as "you", "I", and "what", are filtered out to prevent them from appearing in large numbers in the text and to cover up useless words. v2 , X t1 After time alignment and fusion, it is converted into a bag-of-words model.
[0052] In sub-operation S33, if the similarity between the knowledge point theme probability distributions of two adjacent key frames is greater than a first threshold, the two adjacent key frames are divided into the same classroom teaching video segment, otherwise, the two adjacent key frames are divided into different classroom teaching video segments.
[0053] Specifically, the bag-of-words model is input into the LDA model to obtain the knowledge point topic probability distribution P(z / T i ), where z is the number of knowledge point topics corresponding to the classroom teaching video. Preferably, the value of z is determined by using a Bayesian information network; the similarity can be obtained by calculating the distance between the probability of each type of knowledge point topic in two adjacent frames and summing them up. At the same time, the LDA model outputs the keywords under each knowledge point, and uses the keywords to annotate each knowledge point topic.
[0054] Operation S4 generates and combines the knowledge point themes of each classroom teaching video segment to create a teaching catalog of the classroom teaching video.
[0055] The classroom video segmentation and directory creation method based on multimodal information provided in the embodiment of the present invention makes full use of the audio and video data generated in the classroom, adopts a multimodal information fusion method to perform classroom video segmentation, and simultaneously extracts keywords to complete the creation of the classroom video directory, thereby effectively realizing automatic video segmentation and improving segmentation accuracy.
[0056] An embodiment of the present invention also provides a classroom video segmentation and directory creation system based on multimodal information, including: a processor; a memory, which stores a computer executable program, and when the program is executed by the processor, the processor executes the above-mentioned classroom video segmentation and directory creation method based on multimodal information.
[0057] An embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned classroom video segmentation and directory creation method based on multimodal information is implemented.
[0058] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A classroom video segmentation and directory creation method based on multimodal information, characterized in that: include: S1, obtaining audio information and video information generated in the classroom, and extracting the teacher's audio from the audio information; S2, inputting the audio information and the video information into a classroom activity detection model, obtaining the classroom activity status and corresponding video clips appearing in the video information, and splicing the video clips in which the classroom activity status is the teacher's explanation to obtain a classroom teaching video; S3, inputting the audio corresponding to the classroom teaching video in the teacher audio and the classroom teaching video into a classroom video segmentation model to segment the classroom teaching video into different classroom teaching video segments according to the knowledge point themes; S4, generating knowledge point themes of each of the classroom teaching video segments and combining them to create a teaching catalog of the classroom teaching video.
2. The method for classroom video segmentation and directory creation based on multimodal information as claimed in claim 1, characterized in that: The step of extracting the teacher audio from the audio information comprises: Extracting a first audio feature from the audio information and inputting the first audio feature into a CNN network to obtain a first feature sequence; The first feature sequence is sequentially processed using an LSTM network, and the processing results are clustered and classified to separate the student audio, the environmental audio and the teacher audio.
3. The method for classroom video segmentation and directory creation based on multimodal information as claimed in claim 1, characterized in that: In S2, the information processing process in the classroom activity detection model is: Extracting a second audio feature from the audio information, extracting a video feature from the video information, and fusing and splicing the second audio feature with the video feature in time sequence; The fused and spliced feature vectors are input into the Resnet network to obtain the classroom activity status and corresponding video clips appearing in the video information.
4. The method for classroom video segmentation and directory creation based on multimodal information as claimed in claim 3, characterized in that: The extracting video features from the video information includes: The key points of student head movements are extracted from the video information to generate a student head posture feature matrix; the key points of teacher movements are extracted from the video information to generate a teacher posture feature matrix; the video features include the student head posture feature matrix and the teacher posture feature matrix.
5. The method for classroom video segmentation and directory creation based on multimodal information according to any one of claims 1 to 4, characterized in that: The classroom activity status includes: teacher explanation, communication and discussion, students doing exercises and others; teacher explanation includes: teacher lecturing and teacher explaining questions.
6. The method for classroom video segmentation and directory creation based on multimodal information as claimed in claim 1, characterized in that: In S3, the information processing process in the classroom video segmentation model is: Extract text features X from the audio corresponding to the classroom teaching video in the teacher audio t1 ; Extract video features X from the classroom teaching video v2 ; X by time t1 and X v2 After alignment and fusion, the model is converted into a bag-of-words model, and the bag-of-words model is input into an LDA model to obtain the probability distribution of the knowledge point topics in each key frame of the classroom teaching video; If the similarity between the knowledge point theme probability distributions of two adjacent key frames is greater than a first threshold, the two adjacent key frames are divided into the same classroom teaching video segment, otherwise, the two adjacent key frames are divided into different classroom teaching video segments.
7. The method for classroom video segmentation and directory creation based on multimodal information as claimed in claim 6, characterized in that: Extracting video features X from the classroom teaching video v2 ,include: For each frame in the classroom teaching video, if the similarity between the frame and the next frame is less than a second threshold, the frame is used as a key frame; For each key frame in the classroom teaching video, extract the text on the presentation and blackboard in the key frame to form corresponding text information; Combine the text information of each key frame to obtain the video feature X v2 .
8. The method for classroom video segmentation and directory creation based on multimodal information as claimed in claim 7, characterized in that: The time X t1 and X v2 After alignment and fusion, it is converted into a bag-of-words model, including: X t1 After splicing the text corresponding to the i-th key frame in the classroom teaching video, splicing it with the text information of the i-th key frame to obtain the i-th splicing information, i=1,2,…,I, I is the number of key frames in the classroom teaching video; The spliced information is spliced in chronological order to obtain fused information, and the fused information is converted into the bag-of-words model.
9. A classroom video segmentation and catalog creation system based on multimodal information, characterized in that: include: processor; A memory storing a computer executable program, which, when executed by the processor, enables the processor to execute the classroom video segmentation and directory creation method based on multimodal information as described in any one of claims 1-8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for classroom video segmentation and directory creation based on multimodal information as described in any one of claims 1-8 is implemented.