A video highlight segment extraction method, system and storage medium

By analyzing the interaction characteristics and voice recognition of online course videos, calculating the priority of video clips, the problem of inflexible editing of online course videos in the existing technology is solved, and an efficient and interpretable video editing method is realized to adapt to different needs. .

CN113920534BActive Publication Date: 2024-11-12WUHAN LINGGE EXCELLENCE EDUCATION TECHNOLOGY CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202111169577.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-08
Publication Date
2024-11-12
Estimated Expiration
2041-10-08

AI Technical Summary

Technical Problem

The existing automatic video editing technology is not flexible enough to process online videos, the end-to-end editing model is poorly interpretable, and when the demand changes, the model needs to be retrained, which is very labor-intensive and expensive.

Method used

By obtaining the teacher-student interaction characteristics of online class videos, we conduct facial expressions, eye movement attention and gesture movement analysis, combined with speech recognition to extract keywords, calculate the priority of video clips to determine exciting clips, and use multimodal information to perform videos Clip, allowing the feature weight to be adjusted according to the needs without retraining the model.

Benefits of technology

It realizes the efficiency and interpretability of video clips, and can adjust parameters according to requirements to generate video clips that meet the requirements, reducing costs and improving editing flexibility.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113920534B_ABST
    Figure CN113920534B_ABST
Patent Text Reader

Abstract

The present invention relates to a method, system and storage medium for extracting highlights from a video. The method comprises: obtaining the online course video to be processed and the teacher-student interaction features and dividing the online course video to be processed into a plurality of target videos, respectively performing facial expression analysis, eye movement attention analysis and gesture and body movement analysis on the pictures corresponding to all frames of the target video to obtain the visual feature set of the students and the visual feature set of the teachers in the pictures corresponding to each frame; determining the timeliness of the student feedback according to the time interval between the voice segments corresponding to the students and the voice segments corresponding to the teachers in the audio of the target video; performing speech recognition and extracting keywords on the voice segments corresponding to the students and the teachers to determine the fluency of the teacher's language, the fluency of the student's language and the correctness of the teaching knowledge; determining the highlights in the online course video to be processed according to the priority of each target video. The present invention has strong interpretability, and is convenient for users to adjust parameters according to needs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video analysis and editing, and in particular to a method, system and storage medium for extracting wonderful video clips. Background Art

[0002] With the development of distance learning technology, students, teachers, and course sales personnel hope to evaluate and promote the wonderful content in the classroom through short videos. Manual editing is time-consuming, and there are problems such as unclear editing standards and the quality of edited content is greatly affected by human factors. Educational institutions also hope to use automated methods to produce course highlights to promote their own courses to reduce the cost of manual editing. Existing automatic video editing technologies include: end-to-end video editing through neural network methods. This method is suitable for videos with fast lens changes, such as movies. For online course videos that emphasize course content, these end-to-end editing models are not highly interpretable. When the demand for videos changes, the model needs to be retrained, which is labor-intensive and costly. Summary of the invention

[0003] The purpose of the present invention is to provide a method, system and storage medium for extracting video highlights, which have strong interpretability and are convenient for users to adjust parameters according to needs.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A method for extracting video highlights, comprising:

[0006] Acquire the online course video to be processed and the teacher-student interaction features and divide the online course video to be processed into multiple target videos; the teacher-student interaction features include the teacher's rewards and punishments to the students at each moment in the online course video to be processed;

[0007] For any target video, facial expression analysis, eye movement attention analysis, and gesture and body movement analysis are performed on the pictures corresponding to all frames of the target video to obtain a visual feature set of students and a visual feature set of teachers in the pictures corresponding to each frame, wherein the visual feature set includes expression features, eyeball position features, eye aspect ratio features, gesture features, body movement features, and attention features;

[0008] Determining the timeliness of the student's feedback based on the time interval between the student's corresponding voice segment and the teacher's corresponding voice segment in the audio of the target video;

[0009] Perform speech recognition and extract keywords for the speech segments corresponding to the students and the teachers to determine the fluency of the teacher's language, the fluency of the student's language and the correctness of the teaching knowledge;

[0010] Determine the wonderful clips in the online course video to be processed according to the priority of each target video; the priority is the weighted sum of the frequency of occurrence of each feature in the feature set in the target video or the weighted sum of the probability density of occurrence of each feature in the feature set in the target video; the feature set includes teacher-student interaction features, teacher's visual feature set, student's visual feature set, timeliness of student feedback, fluency of teacher's language, fluency of student's language and correctness of teaching knowledge.

[0011] Optionally, the facial expression analysis, eye movement attention analysis, and gesture and body movement analysis are performed on the pictures corresponding to all frames of the target video to obtain a set of visual features of students and a set of visual features of teachers in the pictures corresponding to each frame, specifically including:

[0012] Read the target video frame by frame to obtain pictures corresponding to all frames of the target video;

[0013] For any picture, facial expression analysis, eye movement analysis, and gesture and body movement analysis are performed on a first set area in the picture to obtain a visual feature set of the teacher corresponding to the picture;

[0014] Performing facial expression analysis, eye movement attention analysis, and gesture and body movement analysis on the second set area in the image respectively to obtain a set of visual features of the student corresponding to the image.

[0015] Optionally, determining the timeliness of the student feedback according to the time interval between the voice segment corresponding to the student and the voice segment corresponding to the teacher in the audio of the target video specifically includes:

[0016] Extracting the audio from the online course video to be processed;

[0017] Performing voice endpoint detection on the audio to obtain multiple voice segments;

[0018] Perform identity recognition on each voice segment to obtain the identity corresponding to each voice segment;

[0019] The timeliness of the student's feedback is determined based on the time interval between the student's corresponding speech segment and the teacher's corresponding speech segment.

[0020] Optionally, the performing speech recognition and extracting keywords on the speech segments corresponding to the students and the teachers to determine the fluency of the teacher's language, the fluency of the student's language and the correctness of the teaching knowledge specifically includes:

[0021] Performing speech recognition on the speech segments corresponding to the students and the speech segments corresponding to the teachers respectively to obtain the speech text corresponding to the students and the speech text corresponding to the teachers;

[0022] Extracting keywords from the teacher's corresponding voice and text;

[0023] Determine the fluency of the teacher's language based on the keywords;

[0024] Determine the correctness of the teaching knowledge according to the pronunciation and text corresponding to the student;

[0025] Determine the fluency of the student's language based on the corresponding speech fragments.

[0026] Optionally, determining the highlight clips in the online course video to be processed according to the priority of each target video specifically includes:

[0027] For any target video, calculate the weighted sum of the frequencies of occurrence of each feature in the feature set in the target video or the weighted sum of the probability densities of occurrence of each feature in the feature set in the target video;

[0028] Determine that the target video corresponding to the target priority is a highlight clip; the target priority is a weighted sum of values ​​within a set range.

[0029] A video highlight segment extraction system, comprising:

[0030] An acquisition module is used to acquire the online course video to be processed and the teacher-student interaction features and divide the online course video to be processed into multiple target videos; the teacher-student interaction features include the teacher's rewards and punishments to the students at each moment in the online course video to be processed;

[0031] A visual feature set determination module is used to perform facial expression analysis, eye movement attention analysis, and gesture and body movement analysis on pictures corresponding to all frames of any target video to obtain a visual feature set of students and a visual feature set of teachers in the pictures corresponding to each frame, wherein the visual feature set includes expression features, eyeball position features, eye aspect ratio features, gesture features, body movement features, and attention features;

[0032] A timeliness determination module, used to determine the timeliness of student feedback based on the time interval between the student's corresponding voice segment and the teacher's corresponding voice segment in the audio of the target video;

[0033] A fluency correctness determination module is used to perform speech recognition and extract keywords from the speech segments corresponding to the students and the teachers to determine the fluency of the teacher's language, the fluency of the student's language and the correctness of the teaching knowledge;

[0034] A highlight segment determination module is used to determine the highlight segments in the online course video to be processed according to the priority of each target video; the priority is the weighted sum of the frequency of occurrence of each feature in the feature set in the target video or the weighted sum of the probability density of occurrence of each feature in the feature set in the target video; the feature set includes teacher-student interaction features, the teacher's visual feature set, the student's visual feature set, the timeliness of student feedback, the fluency of the teacher's language, the fluency of the student's language and the correctness of teaching knowledge.

[0035] Optionally, the visual feature set determination module specifically includes:

[0036] A picture extraction unit, used to read the target video frame by frame to obtain pictures corresponding to all frames of the target video;

[0037] A teacher's visual feature set determination unit is used to perform facial expression analysis, eye movement attention analysis, and gesture and body movement analysis on a first set area in any picture to obtain a teacher's visual feature set corresponding to the picture;

[0038] The student's visual feature set determination unit is used to perform facial expression analysis, eye movement attention analysis and gesture and body movement analysis on the second set area in the picture to obtain the student's visual feature set corresponding to the picture.

[0039] Optionally, the timeliness determination module specifically includes:

[0040] An audio extraction unit, used to extract the audio in the online course video to be processed;

[0041] A speech segment determination unit, configured to perform speech endpoint detection on the audio to obtain a plurality of speech segments;

[0042] An identity determination unit, used to perform identity recognition on each voice segment to obtain an identity corresponding to each voice segment;

[0043] The timeliness determination unit is used to determine the timeliness of the student's feedback according to the time interval between the student's corresponding voice segment and the teacher's corresponding voice segment.

[0044] Optionally, the fluency correctness determination module specifically includes:

[0045] A speech recognition unit is used to perform speech recognition on the speech segments corresponding to the students and the speech segments corresponding to the teachers to obtain the speech text corresponding to the students and the speech text corresponding to the teachers;

[0046] A keyword extraction unit, used to extract keywords from the voice and text corresponding to the teacher;

[0047] a teacher fluency determination unit, used to determine the fluency of the teacher's language according to the keywords;

[0048] A correctness determination unit, used to determine the correctness of the teaching knowledge according to the voice and text corresponding to the student;

[0049] The student fluency determination unit is used to determine the fluency of the student's language according to the student's corresponding speech segment.

[0050] A storage medium is used to store the method for extracting video highlights as described above.

[0051] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the present invention determines the wonderful clips in the online course video to be processed according to the priority of each target video, and the priority is the weighted sum of the frequency of occurrence of each feature in the feature set in the target video or the weighted sum of the probability density of occurrence of each feature in the feature set in the target video. When the demand changes, the weight of each feature in the feature set can be directly adjusted without re-training the model. It has strong interpretability and is convenient for users to adjust parameters according to needs to obtain video clips that better meet the requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 A specific flow chart of the method for extracting video highlights provided by an embodiment of the present invention;

[0054] Figure 2 A flow chart for determining the timeliness of student feedback, the fluency of teacher language, the fluency of student language and the correctness of teaching knowledge provided by an embodiment of the present invention;

[0055] Figure 3 A flowchart for obtaining teacher-student interaction features provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] In order to achieve automated highlight moment editing, it is first necessary to correctly identify the emotions of students in the course, convert the emotions into a form that can be recognized by the machine, and find the key frames in the video in combination with facial and sound features. With the widespread development and application of machine learning technology, the basic features in the video can be accurately and efficiently extracted through machine learning methods, and the time points of the key frames in the video can be calculated through the basic features. Then, the video editing tool can automatically edit the video according to these time points to generate highlights. Therefore, this embodiment proposes a video highlight segment extraction method based on multimodal information such as machine vision and speech recognition, which can efficiently and explainably generate highlights, such as Figure 1 As shown, the method includes:

[0059] Step 101: Obtain the online course video to be processed and the teacher-student interaction features and divide the online course video to be processed into multiple target videos. The teacher-student interaction features include the teacher's rewards and punishments to students at each moment in the online course video to be processed.

[0060] Step 102: For any target video, facial expression analysis, eye movement attention analysis, and gesture and body movement analysis are performed on the pictures corresponding to all frames of the target video to obtain the visual feature set of the students and the visual feature set of the teacher in the pictures corresponding to each frame. The visual feature set includes expression features, eyeball position features, eye aspect ratio features, gesture features, body movement features, and attention features.

[0061] Step 103: Determine the timeliness of the student's feedback based on the time interval between the student's corresponding voice segment and the teacher's corresponding voice segment in the audio of the target video.

[0062] Step 104: Perform speech recognition and extract keywords on the speech segments corresponding to the students and the teachers to determine the fluency of the teacher's language, the fluency of the student's language and the correctness of the teaching knowledge.

[0063] Step 105: Determine the highlights in the online course video to be processed according to the priority of each target video. The priority is the weighted sum of the frequency of each feature in the feature set appearing in the target video or the weighted sum of the probability density of each feature in the feature set appearing in the target video. The feature set includes teacher-student interaction features, teacher's visual feature set, student's visual feature set, timeliness of student feedback, fluency of teacher's language, fluency of student's language and correctness of teaching knowledge, wherein the weight of each feature is set according to the importance of the feature in the demand.

[0064] In practical applications, the pictures corresponding to all frames of the target video are analyzed for facial expression, eye movement and attention, and gesture and body movement to obtain a set of visual features of students and a set of visual features of teachers in the pictures corresponding to each frame, specifically including:

[0065] The target video is read frame by frame to obtain pictures corresponding to all frames of the target video. Specifically, the video is read frame by frame through opencv, and each frame of the video is read as a picture.

[0066] For any picture, facial expression analysis, eye movement analysis and gesture and body movement analysis are performed on a first set area in the picture to obtain a visual feature set of the teacher corresponding to the picture.

[0067] Performing facial expression analysis, eye movement attention analysis, and gesture and body movement analysis on the second set area in the image respectively to obtain a set of visual features of the student corresponding to the image.

[0068] In practical applications, facial expression analysis, eye movement analysis, and gesture and body movement analysis are performed on images to obtain the corresponding visual feature set of the image, specifically:

[0069] Face recognition: Face recognition algorithms can be used to identify faces in images and automatically annotate key points on faces. For example, the dlib algorithm can be used to output the location of faces and feature points on faces.

[0070] Facial expression analysis: Train the neural network model through the labeled data set, take the picture as input and the expression corresponding to the picture as output, use the trained neural network model to classify the pictures containing faces and output the probability of each expression. The neural network model can use ResNet or MobileNet, etc.

[0071] Attention is obtained by analyzing facial features such as the position of the eyeball and the eye aspect ratio. The eye aspect ratio EAR can be calculated from the key points in the face according to the following formula:

[0072]

[0073] Among them, P1 to P6 are the key points of the human eye: P1 and P4 are on the left and right sides of the human eye respectively, P2 and P3 are on the upper left and right sides of the human eye respectively, and P6 and P5 are on the lower left and right sides of the human eye respectively.

[0074] The position of the human eyeball is first found through the key points of the face, and then the position of the eyeball is found through the color difference between the eyeball and the white of the eye.

[0075] Mouth opening degree: find the position of the upper lip and lower lip through the key points of the face and calculate the lip opening degree. The curve of mouth opening degree changing over time can be used to evaluate the student's speaking in the course.

[0076] Recognize gestures and body movements of people through machine learning models: First, in order to save computing time, you need to identify the frames where the hands appear and the relative positions of the hands. You can use MobileNet or Mask RCNN. Then classify the pictures containing the hands and output the probability of the gestures. Gesture classification can be regarded as an image classification or video classification problem. If gesture classification is regarded as an image classification problem, the input of the classification model is a single video frame. If gesture classification is regarded as a video classification problem, the model input is a continuous segment of video frames. Neural network models such as ResNet and DenseNet can be used to classify pictures or videos.

[0077] In practical applications, the calculation formula for the probability density F(t1, t2) of a feature appearing in the target video (time period from t1 to t2) is:

[0078]

[0079] f(t) is a characteristic function that changes continuously with time t.

[0080] In practical applications, the frequency of happy expressions in a certain period of time is F. Happy For example, the calculation formula is as follows:

[0081]

[0082] Frame t1<t<t2 Indicates the total number of frames between time periods t1 and t2, is the number of frames with student smiling expressions in the time period from t1 to t2. The criterion for judging whether a student has a smiling expression in a certain frame is: the smiling probability P output by the model Happy Greater than 0.5.

[0083] In practical applications, the weighted sum The calculation formula is as follows:

[0084] in, It represents the value calculated by the above probability density formula or frequency formula during the time period from t1 to t2, W i is the weight corresponding to a feature.

[0085] In practical applications, the timeliness of student feedback is determined based on the time interval between the student's corresponding voice segment and the teacher's corresponding voice segment in the audio of the target video, such as Figure 2As shown, specifically including:

[0086] To extract the audio from the online course video to be processed, you can use tools such as ffmpeg or MoviePy to extract the sound and store it in memory or hard disk as wav or MP3.

[0087] Perform voice endpoint detection on the audio to obtain multiple voice segments.

[0088] The identities of the voice segments are identified respectively to obtain the identities corresponding to the voice segments.

[0089] The timeliness of student feedback is determined based on the time interval between the student's corresponding voice segment and the teacher's corresponding voice segment. The time period in which the teacher and the student speak can be obtained, and whether the student gives a timely response to the teacher's teaching content can be measured according to the following formula.

[0090] J = Distance(T student , T teacher )

[0091] In the above formula, the timeliness of interaction can be measured by measuring the time period T during which the teacher speaks. teacher Time period T for talking to students student The distance between the teacher and the students is J. If the distance is too far, it means that the students have not given timely feedback on the teacher's teaching content.

[0092] In practical applications, the speech recognition and keyword extraction of the speech segments corresponding to the students and the teachers are performed to determine the fluency of the teacher's language, the fluency of the student's language and the correctness of the teaching knowledge, such as Figure 2 As shown, specifically including:

[0093] Speech recognition is performed on the voice clips corresponding to the students and the voice clips corresponding to the teachers respectively to obtain the voice text corresponding to the students and the voice text corresponding to the teachers. Speech recognition can use machine learning models such as hidden Markov model, Gaussian mixture model and neural network, and can also use commercial speech recognition API.

[0094] Extract keywords from the voice and text corresponding to the teacher, perform content analysis on the converted text, extract keywords from the voice and text corresponding to the teacher, specifically, perform word segmentation on the text converted from the voice (Jieba can be used as a word segmentation tool), and obtain the keywords of the character by extracting keywords and performing text sentiment analysis (convolutional neural network can be used for text sentiment analysis), and the keywords include but are not limited to: emotion, "you are awesome", "too bad", course knowledge points, "um", "this", etc.

[0095] The fluency of the teacher's language is determined based on the keywords. Specifically, when the keywords extracted from the voice text corresponding to the teacher belong to the first preset word set, it is determined that the fluency of the teacher's language is low. For example, if there are a lot of pauses or meaningless words such as "um" and "this" during the teacher's lecture, it can be considered that the coherence of the teacher's lecture is not high and the fluency of the teacher's language is low.

[0096] The correctness of the teaching knowledge is determined based on the voice and text corresponding to the student. Specifically, the voice and text corresponding to the student are compared with the stored standard voice and text. When the comparison result shows that the voice and text corresponding to the student are not similar to the stored voice and text, it is determined that the correctness of the teaching knowledge mastered by the student is low. For example, speech recognition technology can detect the phenomenon of non-standard pronunciation in students' reading aloud, and natural language processing technology can detect incorrect sentences in students' reading aloud, and determine that the correctness of the teaching knowledge mastered by the student is low.

[0097] The fluency of the student's language is determined according to the corresponding speech segment of the student. Specifically, when the time interval between each word in the corresponding speech segment of the student exceeds the set time interval, it is determined that the fluency of the student's language is low. For example, in the reading session, if the student has a large number of pauses when reading the text, it is considered that the coherence of the student's reading is not high and the fluency of the student's language is low.

[0098] In practical applications, determining the highlights in the online course video to be processed according to the priority of each target video specifically includes:

[0099] For any target video, the weighted sum of the frequencies of occurrence of each feature in the feature set in the target video or the weighted sum of the probability densities of occurrence of each feature in the feature set in the target video is calculated.

[0100] Determine that the target video corresponding to the target priority is a highlight clip; the target priority is a weighted sum of values ​​within a set range.

[0101] In practical applications, such as Figure 3 As shown in the figure, when the interactive evaluation content is available, the interactive evaluation content is retrieved from the server, and the timestamps containing positive and negative interactive evaluations are extracted to form the teacher-student interaction features. In online classes, teachers and students interact through buttons on the video software. For example, if a student performs well, the teacher will give the student a "like" or "little red flower", and these operations and the time they occur will be recorded in the database. If the calculation of wonderful moments requires the retrieval of interactive information, the interactive information in the course and its timestamp can be retrieved from the database and used as the teacher-student interaction feature to generate wonderful moments.

[0102] In practical applications, it also includes: using video editing tools to generate several short videos based on the highlights of a long video.

[0103] This embodiment also provides a video highlight segment extraction system corresponding to the above method, the system comprising:

[0104] The acquisition module A1 is used to acquire the online course video to be processed and the teacher-student interaction features and divide the online course video to be processed into multiple target videos. The teacher-student interaction features include the teacher's rewards and punishments to students at each moment in the online course video to be processed.

[0105] The visual feature set determination module A2 is used to perform facial expression analysis, eye movement attention analysis, and gesture and body movement analysis on the pictures corresponding to all frames of any target video to obtain the student's visual feature set and the teacher's visual feature set in the pictures corresponding to each frame. The visual feature set includes expression features, eye position features, eye aspect ratio features, gesture features, body movement features, and attention features.

[0106] The timeliness determination module A3 is used to determine the timeliness of the student's feedback according to the time interval between the student's corresponding voice segment and the teacher's corresponding voice segment in the audio of the target video.

[0107] The fluency correctness determination module A4 is used to perform speech recognition and extract keywords on the speech segments corresponding to the students and the teachers to determine the fluency of the teacher's language, the fluency of the student's language and the correctness of the teaching knowledge.

[0108] The highlight segment determination module A5 is used to determine the highlight segments in the online course video to be processed according to the priority of each target video; the priority is the weighted sum of the frequency of occurrence of each feature in the feature set in the target video or the weighted sum of the probability density of occurrence of each feature in the feature set in the target video; the feature set includes teacher-student interaction features, the teacher's visual feature set, the student's visual feature set, the timeliness of student feedback, the fluency of the teacher's language, the fluency of the student's language and the correctness of teaching knowledge.

[0109] In practical applications, the visual feature set determination module specifically includes:

[0110] The picture extraction unit is used to read the target video frame by frame to obtain pictures corresponding to all frames of the target video.

[0111] The teacher's visual feature set determination unit is used to perform facial expression analysis, eye movement attention analysis and gesture and body movement analysis on a first set area in any picture to obtain the teacher's visual feature set corresponding to the picture.

[0112] The student's visual feature set determination unit is used to perform facial expression analysis, eye movement attention analysis and gesture and body movement analysis on the second set area in the picture to obtain the student's visual feature set corresponding to the picture.

[0113] In practical applications, the timeliness determination module specifically includes:

[0114] The audio extraction unit is used to extract the audio in the online course video to be processed.

[0115] The speech segment determination unit is used to perform speech endpoint detection on the audio to obtain multiple speech segments.

[0116] The identity determination unit is used to perform identity recognition on each voice segment to obtain the identity corresponding to each voice segment.

[0117] The timeliness determination unit is used to determine the timeliness of the student's feedback according to the time interval between the student's corresponding voice segment and the teacher's corresponding voice segment.

[0118] In practical applications, the fluency correctness determination module specifically includes:

[0119] The speech recognition unit is used to perform speech recognition on the speech segments corresponding to the students and the speech segments corresponding to the teachers respectively to obtain the speech text corresponding to the students and the speech text corresponding to the teachers.

[0120] The keyword extraction unit is used to extract keywords from the voice and text corresponding to the teacher.

[0121] The teacher fluency determination unit is used to determine the fluency of the teacher's language according to the keywords.

[0122] The correctness determination unit is used to determine the correctness of the teaching knowledge according to the voice and text corresponding to the student.

[0123] The student fluency determination unit is used to determine the fluency of the student's language according to the student's corresponding speech segment.

[0124] This embodiment also provides a storage medium for storing the above-mentioned method for extracting video highlights.

[0125] The beneficial effects of the present invention are as follows:

[0126] 1. Strong interpretability, which makes it convenient for relevant personnel to adjust algorithm parameters according to business needs.

[0127] 2. It is fast and can complete the analysis and editing of long videos in a short time and condense videos of several hours in length, thus saving system resources and making it convenient for users to obtain results quickly.

[0128] 3. It can help improve the quality of classroom education and help parents better understand students’ learning situation.

[0129] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0130] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for extracting highlights from a video, characterized in that: include: Acquire the online course video to be processed and the teacher-student interaction features and divide the online course video to be processed into multiple target videos; the teacher-student interaction features include the teacher's rewards and punishments to the students at each moment in the online course video to be processed; For any target video, facial expression analysis, eye movement attention analysis, and gesture and body movement analysis are performed on the pictures corresponding to all frames of the target video to obtain a visual feature set of students and a visual feature set of teachers in the pictures corresponding to each frame, wherein the visual feature set includes expression features, eyeball position features, eye aspect ratio features, gesture features, body movement features, and attention features; Determining the timeliness of the student's feedback based on the time interval between the student's corresponding voice segment and the teacher's corresponding voice segment in the audio of the target video; Perform speech recognition and extract keywords for the speech segments corresponding to the students and the teachers to determine the fluency of the teacher's language, the fluency of the student's language and the correctness of the teaching knowledge; Determine the wonderful clips in the online course video to be processed according to the priority of each target video; the priority is the weighted sum of the frequency of occurrence of each feature in the feature set in the target video or the weighted sum of the probability density of occurrence of each feature in the feature set in the target video; the feature set includes the teacher-student interaction feature, the teacher's visual feature set, the student's visual feature set, the timeliness of the student's feedback, the fluency of the teacher's language, the fluency of the student's language and the correctness of the teaching knowledge.

2. A method for extracting video highlights according to claim 1, characterized in that: The facial expression analysis, eye movement attention analysis and gesture and body movement analysis are performed on the pictures corresponding to all frames of the target video to obtain the visual feature set of students and the visual feature set of teachers in the pictures corresponding to each frame, specifically including: Read the target video frame by frame to obtain pictures corresponding to all frames of the target video; For any picture, facial expression analysis, eye movement analysis, and gesture and body movement analysis are performed on a first set area in the picture to obtain a visual feature set of the teacher corresponding to the picture; Performing facial expression analysis, eye movement attention analysis, and gesture and body movement analysis on the second set area in the image respectively to obtain a set of visual features of the student corresponding to the image.

3. A method for extracting video highlights according to claim 1, characterized in that: The determining of the timeliness of the student feedback according to the time interval between the voice segment corresponding to the student and the voice segment corresponding to the teacher in the audio of the target video specifically includes: Extracting the audio from the online course video to be processed; Performing voice endpoint detection on the audio to obtain multiple voice segments; Performing identity recognition on each of the voice segments to obtain an identity corresponding to each of the voice segments; The timeliness of the student's feedback is determined based on the time interval between the student's corresponding speech segment and the teacher's corresponding speech segment.

4. The method for extracting video highlights according to claim 1, characterized in that: The method of performing speech recognition and extracting keywords from the speech segments corresponding to the students and the teachers to determine the fluency of the teacher's language, the fluency of the student's language and the correctness of the teaching knowledge specifically includes: Performing speech recognition on the speech segments corresponding to the student and the speech segments corresponding to the teacher respectively to obtain the speech text corresponding to the student and the speech text corresponding to the teacher; Extracting keywords from the teacher's corresponding voice and text; Determine the fluency of the teacher's language based on the keywords; Determine the correctness of the teaching knowledge according to the pronunciation and text corresponding to the student; Determine the fluency of the student's language based on the corresponding speech fragments.

5. The method for extracting video highlights according to claim 1, characterized in that: Determining the highlights in the online course video to be processed according to the priority of each target video specifically includes: For any target video, calculate the weighted sum of the frequencies of occurrence of each feature in the feature set in the target video or the weighted sum of the probability densities of occurrence of each feature in the feature set in the target video; Determine that the target video corresponding to the target priority is a highlight clip; the target priority is a weighted sum of values ​​within a set range.

6. A video highlight segment extraction system, characterized in that: include: An acquisition module is used to acquire the online course video to be processed and the teacher-student interaction features and divide the online course video to be processed into multiple target videos; the teacher-student interaction features include the teacher's rewards and punishments to the students at each moment in the online course video to be processed; A visual feature set determination module is used to perform facial expression analysis, eye movement attention analysis, and gesture and body movement analysis on pictures corresponding to all frames of any target video to obtain a visual feature set of students and a visual feature set of teachers in the pictures corresponding to each frame, wherein the visual feature set includes expression features, eyeball position features, eye aspect ratio features, gesture features, body movement features, and attention features; A timeliness determination module, used to determine the timeliness of student feedback based on the time interval between the student's corresponding voice segment and the teacher's corresponding voice segment in the audio of the target video; A fluency correctness determination module is used to perform speech recognition and extract keywords from the speech segments corresponding to the students and the teachers to determine the fluency of the teacher's language, the fluency of the student's language and the correctness of the teaching knowledge; A highlight segment determination module is used to determine the highlight segments in the online course video to be processed according to the priority of each target video; the priority is the weighted sum of the frequency of occurrence of each feature in the feature set in the target video or the weighted sum of the probability density of occurrence of each feature in the feature set in the target video; the feature set includes teacher-student interaction features, the teacher's visual feature set, the student's visual feature set, the timeliness of student feedback, the fluency of the teacher's language, the fluency of the student's language and the correctness of teaching knowledge.

7. A video highlight segment extraction system according to claim 6, characterized in that: The visual feature set determination module specifically includes: A picture extraction unit, used to read the target video frame by frame to obtain pictures corresponding to all frames of the target video; A teacher's visual feature set determination unit is used to perform facial expression analysis, eye movement attention analysis, and gesture and body movement analysis on a first set area in any picture to obtain a teacher's visual feature set corresponding to the picture; The student's visual feature set determination unit is used to perform facial expression analysis, eye movement attention analysis and gesture and body movement analysis on the second set area in the picture to obtain the student's visual feature set corresponding to the picture.

8. A video highlight segment extraction system according to claim 6, characterized in that: The timeliness determination module specifically includes: An audio extraction unit, used to extract the audio in the online course video to be processed; A speech segment determination unit, configured to perform speech endpoint detection on the audio to obtain a plurality of speech segments; An identity determination unit, used to perform identity recognition on each voice segment to obtain an identity corresponding to each voice segment; The timeliness determination unit is used to determine the timeliness of the student's feedback according to the time interval between the student's corresponding voice segment and the teacher's corresponding voice segment.

9. A video highlight segment extraction system according to claim 6, characterized in that: The fluency correctness determination module specifically includes: A speech recognition unit is used to perform speech recognition on the speech segments corresponding to the students and the speech segments corresponding to the teachers to obtain the speech text corresponding to the students and the speech text corresponding to the teachers; A keyword extraction unit, used to extract keywords from the voice and text corresponding to the teacher; a teacher fluency determination unit, used to determine the fluency of the teacher's language according to the keywords; A correctness determination unit, used to determine the correctness of the teaching knowledge according to the voice and text corresponding to the student; The student fluency determination unit is used to determine the fluency of the student's language according to the student's corresponding speech fragments.

10. A storage medium, characterized in that: Used to store the video highlight segment extraction method as described in any one of claims 1-5.

Citation Information

Patent Citations

  • Network course video editing method, system and device and storage medium

    CN109889920A

  • Data acquisition method and system suitable for children language teaching scene, and server

    CN112749641A

  • Video highlight fragment recording method and device, and computing equipment

    CN113382312A