Teaching video understanding method based on large model

By establishing a large model to process the pictures and voice information in teaching videos, the in-depth understanding and annotation of teaching videos is achieved, the problem that the existing technology cannot understand the content of teaching videos is solved, the review efficiency and teaching quality are improved, and it is suitable for distance education and Adaptive learning.

CN119091925AInactive Publication Date: 2024-12-06DEQING ALPHA INNOVATION RES INST
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410982830.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing video understanding technology cannot effectively understand the language or text content in teaching videos, and the existing methods require artificial extraction of video frame images, resulting in the loss of voice and timing information in the videos, and the in-depth understanding and real-time analysis of teaching videos cannot be achieved. Inefficient.

Method used

By establishing a large model, collecting pictures and voice information in the video, processing them separately and converting them into text information, comprehension and abstract generation, combining customized prompt words for in-depth content analysis, output accurate video comprehension results, and class Videos are marked.

Benefits of technology

Achieve accurate understanding of teaching videos, improve review efficiency and teaching quality, is suitable for distance education and adaptive learning platforms, meets the learning needs of hearing-impaired people, understand accuracy reaches 90% or above, and review efficiency increases by 50%. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091925A_ABST
    Figure CN119091925A_ABST
Patent Text Reader

Abstract

The invention relates to a teaching video understanding method based on a large model, and the method comprises the following steps: 1) building a large model, and carrying out the training and fine tuning of the large model; 2) collecting information in the video, wherein the information comprises picture information and voice information; 3) independently processing the voice information and the picture information, and converting the voice information and the picture information into text information; according to the method, stable work can be achieved in a complex real teaching scene, the use value of classroom videos is improved in an automatic mode, monitoring of teaching quality is enhanced, and accurate assistance is provided for teaching and learning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of smart education and artificial intelligence technology, and specifically relates to a method for understanding teaching videos based on a large model. Background Art

[0002] Currently, there are video understanding methods based on 3D convolution, which use multiple 3D convolution layers to extract and combine more complex spatiotemporal features layer by layer, and finally integrate the features generated by the convolution layer and map them to the semantic space of the video content. In addition, there are video understanding methods based on dynamic spatiotemporal graphs, which feed video data into a trained network model to output video understanding results. These methods have good recognition effects on scenes and time sequences in videos, but cannot understand detailed language or text content in teaching videos.

[0003] At the same time, there is no large multimodal model that uses video as direct input at this stage. All of them need to be manually extracted from video frames before inputting questions and answers. However, this method loses the voice and timing information in the video, and the level of understanding cannot reach the level of language or text content, so it cannot be applied to teaching video understanding. In addition, with the digital transformation of education in recent years, the videoization of classroom content has become the norm. In order to ensure effective learning and teaching management, in-depth understanding and analysis of these teaching videos has become particularly important. However, existing classroom video analysis methods often require a lot of manual operations, are inefficient, and difficult to respond in real time.

[0004] To this end, a method for understanding teaching videos based on large models is designed to overcome the above problems. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for understanding teaching videos based on a large model. The method of the present invention can work stably in complex real teaching scenarios, improve the use value of classroom videos in an automated way, enhance the monitoring of teaching quality, and provide accurate assistance for teaching and learning.

[0006] The present invention is implemented by the following technical solution: a method for understanding teaching videos based on a large model, the method comprising the following steps: 1) Build a large model, train and fine-tune the large model; 2) Collect information in the video, including picture information and voice information; 3) Process voice information and picture information separately, and convert both voice and picture information into text information; 4) Segmental understanding generation; 5) Abstract generation; 6) Apply the output results.

[0007] As a preferred embodiment: in the step 1), a large model is established, and the large model uses an open source large model or is further fine-tuned based on an open source model. A customized large model can also be trained according to the needs of a specific teaching environment and a data set. The large models are all conventional models.

[0008] As a preferred embodiment: the voice information processing method in step 3) is: a. Speech separation and preprocessing: Use the existing standard library to separate the language in the video, convert the sampling frequency to 16kHZ, and filter the noise and blank parts in the speech through the algorithm model; b. Speech transcription model: Use an open source speech transcription algorithm model for transcription, or use an open source algorithm to collect classroom speech data and train your own model after annotation; c. Transcribed text processing: The transcribed text is stored in the order of timestamps and cleaned, such as removing and optimizing repeated text, typos, and catchphrases.

[0009] As a preferred embodiment: the text information processing method in step 3) is: a. Extraction of text information b. Transcribed text processing: The transcribed text is saved in order of timestamps, and the transcribed text is cleaned, such as removing and optimizing repeated text, typos, and catchphrases.

[0010] As a preferred embodiment, the specific method for generating segmented understanding in step 4) is: input the cleaned text into the big model in sequence according to timestamps, and set big model prompt words that can be manually changed in multiple languages. The big model will perform in-depth content analysis based on the customized prompt words, and output understanding results corresponding to the video content. The results output by the big model are combined with the timestamps to ensure that the understanding results can accurately correspond to the specific content and time nodes of the video.

[0011] As a preferred embodiment, the specific method for generating the summary in step 5) is: after the segmented understanding is performed to more than 2 segments, the understanding results of the 2 segments are input into the big model, specific prompt words are set, and a brief understanding of the 2 segment results is summarized and generated; then the understanding content and the third segment content are input into the big model for understanding, and so on, to obtain a summary of the entire classroom video content.

[0012] As a preferred embodiment, the specific method of applying the output result in step 6) is: after completing all steps of understanding the classroom video content, the classroom video is annotated with the understanding results obtained by the large model and its related timestamps to support teachers and students to intuitively understand and review the classroom content.

[0013] Compared with the existing video understanding technology, the method for teaching video understanding based on large model provided by the present invention has the following beneficial effects: 1. Combined with customized large models, in-depth analysis of teaching content can accurately abstract key knowledge points and content in teaching videos. In actual video tests of diverse teaching materials and teaching styles, the system's accuracy in understanding the overall content and summarizing key information reached 90% or more.

[0014] 2. Through the key moments analyzed by the system, students can directly replay the key parts for review, which greatly improves the relevance and efficiency of the review. In actual application, the efficiency of students' after-class review has increased by about 50%.

[0015] 3. With its high-accuracy content understanding and personalized feedback, this system is very suitable for distance education and adaptive learning platforms. It can bring a more interactive and customized experience to online learning. At the same time, it can meet the needs of hearing-impaired people to quickly match learning priorities, and has broad prospects for promotion and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the process structure of the present invention. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to more clearly understand the objectives, technical solutions and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0018] In the description of the present invention, it is necessary to understand that the orientation or position relationship indicated by the terms such as "up", "down", "left", "right", "inside", "outside", "horizontal" and "vertical" is based on the orientation or position relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention, rather than indicating or implying that the device or original referred to must have a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0019] The present invention will be described in detail below with reference to the accompanying drawings: Figure 1 As shown, a method for understanding teaching videos based on a large model comprises the following steps: 1) Build a large model, train and fine-tune the large model; 2) Collect information in the video, including picture information and voice information; 3) Process voice information and picture information separately, and convert both voice and picture information into text information; 4) Segmental understanding generation; 5) Abstract generation; 6) Apply the output results.

[0020] In the step 1), a large model is established. The large model uses an open source large model or is further fine-tuned based on an open source model. A customized large model can also be trained according to the needs of a specific teaching environment and a data set. The large models are all conventional models.

[0021] The voice information processing method in step 3) is: a. Speech separation and preprocessing: Use the existing standard library to separate the language in the video, convert the sampling frequency to 16kHZ, and filter the noise and blank parts in the speech through the algorithm model; b. Speech transcription model: Use an open source speech transcription algorithm model for transcription, or use an open source algorithm to collect classroom speech data and train your own model after annotation; c. Transcribed text processing: The transcribed text is stored in the order of timestamps and cleaned, such as removing and optimizing repeated text, typos, and catchphrases.

[0022] The text information processing method in step 3) is: a. Extraction of text information b. Transcribed text processing: The transcribed text is saved in order of timestamps, and the transcribed text is cleaned, such as removing and optimizing repeated text, typos, and catchphrases.

[0023] The specific method for generating segmented understanding in step 4) is: input the cleaned text into the big model in sequence according to timestamps, and set big model prompt words that can be manually changed in multiple languages. The big model will perform in-depth content analysis based on the customized prompt words, and output understanding results corresponding to the video content. The results output by the big model are combined with the timestamps to ensure that the understanding results can accurately correspond to the specific content and time nodes of the video.

[0024] The specific method for generating the summary in step 5) is as follows: after the segmented understanding is performed to more than 2 segments, the understanding results of the 2 segments are input into the big model, specific prompt words are set, and a brief understanding of the 2 segment results is summarized and generated; then the understanding content and the third segment content are input into the big model for understanding, and so on, a summary of the entire classroom video content can be obtained.

[0025] The specific method of applying the output result in step 6) is: after completing all steps of understanding the classroom video content, the classroom video is annotated with the understanding results obtained by the large model and its related timestamps to support teachers and students to intuitively understand and review the classroom content.

[0026] The present invention first transcribes the speech in the video into text in real time, and then inputs the transcribed text into a large language model for in-depth analysis through a specific prompt word template. The system can reliably judge the teaching content and structure in the video based on the timestamp of the transcribed text and the analysis content obtained by the large model, and provide timely feedback to teachers and students.

[0027] The present invention combines a customized large model to conduct in-depth analysis of teaching content, and can accurately abstract key knowledge points and content in teaching videos. In actual video tests of diverse teaching materials and teaching styles, the system's accuracy in understanding global content and summarizing key information reached 90% or more. The present invention marks key moments analyzed by the system, so that students can directly replay key parts for review, greatly improving the pertinence and efficiency of the review. In practical applications, students' after-class review efficiency has increased by about 50%. With its high-accuracy content understanding and personalized feedback, the present invention's system is very suitable for distance education and adaptive learning platforms. It can bring a more interactive and customized experience to online learning; at the same time, it meets the needs of hearing-impaired people to quickly match learning priorities, and has broad prospects for promotion and application.

[0028] The specific embodiments described herein are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for understanding teaching videos based on a large model, characterized by: The method comprises the following steps: 1) Build a large model, train and fine-tune the large model; 2) Collect information in the video, including picture information and voice information; 3) Process voice information and picture information separately, and convert both voice and picture information into text information; 4) Segmental understanding generation; 5) Abstract generation; 6) Apply the output results.

2. The method for understanding teaching videos based on large models according to claim 1, characterized in that: In the step 1), a large model is established. The large model uses an open source large model or is further fine-tuned based on an open source model. A customized large model can also be trained according to the needs of a specific teaching environment and a data set. The large models are all conventional models.

3. The method for understanding teaching videos based on large models according to claim 1, characterized in that: The voice information processing method in step 3) is: a. Speech separation and preprocessing: Use the existing standard library to separate the language in the video, convert the sampling frequency to 16kHZ, and filter the noise and blank parts in the speech through the algorithm model; b. Speech transcription model: Use an open source speech transcription algorithm model for transcription, or use an open source algorithm to collect classroom speech data and train your own model after annotation; c. Transcribed text processing: The transcribed text is stored in the order of timestamps and cleaned, such as removing and optimizing repeated text, typos, and catchphrases.

4. The method for understanding teaching videos based on large models according to claim 1, characterized in that: The text information processing method in step 3) is: a. Extraction of text information b. Transcribed text processing: The transcribed text is saved in order of timestamps, and the transcribed text is cleaned, such as removing and optimizing repeated text, typos, and catchphrases.

5. The method for understanding teaching videos based on large models according to claim 1, characterized in that: The specific method for generating segmented understanding in step 4) is: input the cleaned text into the big model in sequence according to timestamps, and set big model prompt words that can be manually changed in multiple languages. The big model will perform in-depth content analysis based on the customized prompt words, and output understanding results corresponding to the video content. The results output by the big model are combined with the timestamps to ensure that the understanding results can accurately correspond to the specific content and time nodes of the video.

6. The method for understanding teaching videos based on large models according to claim 1, characterized in that: The specific method for generating the summary in step 5) is as follows: after the segmented understanding is performed to more than 2 segments, the understanding results of the 2 segments are input into the big model, specific prompt words are set, and a brief understanding of the 2 segment results is summarized and generated; then the understanding content and the third segment content are input into the big model for understanding, and so on, a summary of the entire classroom video content can be obtained.

7. The method for understanding teaching videos based on large models according to claim 1, characterized in that: The specific method of applying the output result in step 6) is: after completing all steps of understanding the classroom video content, the classroom video is annotated with the understanding results obtained by the large model and its related timestamps to support teachers and students to intuitively understand and review the classroom content.