Classroom video-oriented refining method

Abstract

The invention provides a classroom video-oriented refining method, which comprises the following steps that: multi-modal feature extraction is carried out on a classroom video, and the multi-modal features comprise visual features, audio features and text features; determining knowledge point fragments based on the multi-modal features, and obtaining importance scores; determining an interactive wonderful fragment based on the multi-modal features, and obtaining a wonderful score; and according to the scores of the knowledge point fragments and the interactive wonderful fragments, selecting corresponding fragments to finish classroom video refining. Knowledge point key segments and teacher-student interaction wonderful instant segments can be automatically identified, and whole-process non-manual intervention processing from an original video to a refined video is realized.

Description

Technical Field

[0001] This invention relates to the fields of educational technology, video processing, and artificial intelligence, and in particular to a refining method for classroom videos. Background Technology

[0002] With the popularization of online education and blended learning, a large number of classroom videos are recorded and stored. However, complete classroom recordings are usually quite long (45-90 minutes) and contain a lot of redundant content. Existing video segmentation technologies do not incorporate the characteristics of teaching rhythm, resulting in an accuracy rate of less than 70% for identifying knowledge point boundaries and an inability to distinguish high-quality candidate interactive segments, leading to a high failure rate for automatic editing.

[0003] Existing video processing technologies mainly suffer from the following problems: 1. Limitations of single-dimensional analysis: Existing technologies mostly employ single image or audio analysis methods, which cannot comprehensively evaluate the value of teaching content. For example, face detection alone cannot determine whether the teacher is explaining key knowledge, and speech recognition alone cannot understand the teaching pace and the quality of interaction.

[0004] 2. Lack of awareness of teaching rhythm: Traditional methods lack an understanding of classroom teaching rhythm. Excellent classroom teaching usually has obvious rhythmic variations—the speaking speed is moderate when explaining knowledge points, there are pauses and repetitions when emphasizing key points, and the voice has rich layers during interactive sessions. Current technology cannot capture these teaching characteristics.

[0005] 3. Difficulty in identifying knowledge point boundaries: A lesson usually contains multiple knowledge points, but existing technology has difficulty accurately identifying the start and end boundaries of knowledge points, resulting in incomplete extracted segments or the inclusion of too much irrelevant content.

[0006] 4. Lack of evaluation of interactive engagement: Teacher-student interaction is an important part of the classroom, but existing technology lacks an evaluation mechanism for the quality of interaction, making it impossible to distinguish between formal question-and-answer sessions and truly inspiring in-depth interactions.

[0007] 5. Insufficient utilization of time information: Current technology rarely utilizes timetable information. In reality, the first 5-10 minutes of a class are usually for knowledge review and introduction, the middle 30-40 minutes are for core knowledge explanation, and the last 10 minutes are for summary and homework assignment. The content value of different time periods varies significantly.

[0008] Therefore, there is an urgent need for an intelligent method that can comprehensively utilize video, audio, text, and time information to accurately identify key knowledge segments and exciting interactive moments, and generate high-quality, refined videos. Summary of the Invention

[0009] This application provides a method for refining classroom videos, which can automatically identify key knowledge points and exciting moments of teacher-student interaction, and generate refined teaching videos.

[0010] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0011] According to a first aspect of the embodiments of this application, a refining method for classroom videos is provided, comprising: Multimodal features are extracted from classroom videos, including visual features, audio features, and text features; Based on the multimodal features, knowledge point fragments are determined and importance scores are obtained; Based on the multimodal features, interactive highlights are identified, and a highlight score is obtained. Based on the scores of the knowledge point segments and the interactive highlights, select the corresponding segments to complete the classroom video refinement.

[0012] In one embodiment of this application, the process of determining the knowledge point fragment includes: A teaching rhythm curve is constructed based on audio features; the audio features include audio energy, speech rate changes, and pitch changes. Based on the teaching rhythm curve, a candidate boundary set for knowledge points is generated; the candidate boundary set includes a start boundary candidate set and an end boundary candidate set. Textual features are used to correct the candidate boundary set; An LSTM time series model is used to filter the corrected candidate boundary set; The selected candidate boundary set is used to determine knowledge point fragments.

[0013] In one embodiment of this application, the step of generating a candidate boundary set of knowledge points based on the teaching rhythm curve detection specifically includes: Calculate the derivative of the teaching rhythm curve; All time points that satisfy the condition that the derivative is greater than the first threshold are taken as candidate start boundaries and form a candidate set of start boundaries; All time points where the derivative is less than the second threshold are taken as candidate termination boundaries and form a candidate set of termination boundaries.

[0014] In one embodiment of this application, the step of correcting the candidate boundary set using text features specifically includes: Each candidate start boundary in the candidate start boundary set ± Search for a preset start prompt phrase within a second range. If found, adjust the corresponding candidate start boundary to the time point of the preset start prompt phrase. Each candidate end boundary in the candidate end boundary set ± Search for a preset end prompt phrase within a time range of seconds. If found, adjust the corresponding candidate end boundary to the time point of the preset end prompt phrase.

[0015] In one embodiment of this application, the step of using an LSTM time series model to filter the corrected candidate boundary set specifically includes: Obtain the front and back of each candidate boundary in the candidate boundary set. The feature sequence is calculated in seconds and input into a pre-trained LSTM time series model for scoring, retaining candidate boundaries with scores greater than a third threshold; the feature sequence includes visual features, keyword density, and teaching rhythm curve values.

[0016] In one embodiment of this application, the knowledge point fragment is determined using the filtered candidate boundary set, specifically including: Sort the candidate start boundaries and candidate end boundaries in the candidate boundary set by time. A greedy algorithm is used to match the nearest candidate end boundary for each candidate start boundary, generating knowledge point fragments and filtering out abnormal fragments. Obtain a collection of all knowledge point segments from the class videos.

[0017] In one embodiment of this application, the specific process of obtaining the importance score includes: Calculate the average value of the rhythm curve within each knowledge point segment, the time weight corresponding to the center time of the segment, the keyword density of the ASR text, and the segment duration; the time weight corresponding to the center time of the segment is calculated based on the priority weighting of class time. The importance score of the corresponding knowledge point segment is obtained by weighting the average value of the rhythm curve within the segment, the time weight corresponding to the center time of the segment, the keyword density of the ASR text, and the segment duration.

[0018] In one embodiment of this application, determining the interactive highlights based on the multimodal features and obtaining a highlights score specifically includes: Distinguish between teacher's and student's voices based on audio features; The system detects all segments of classroom videos in which at least two different speakers appear within a 5-second window, with one of them being the teacher, and marks them as candidate interactive segments. The emotional intensity, content depth, feedback intensity, and duration factor of each interactive highlight are calculated based on multimodal features. The interaction quality score for each candidate interactive segment is obtained by weighting the emotional intensity, content depth, feedback intensity, and duration factor. Candidate interactive segments with an interactive brilliance score greater than the fourth threshold are retained as interactive brilliance segments.

[0019] In one embodiment of this application, based on the scores of knowledge point segments and engaging interactive segments, corresponding segments are selected to complete the refinement of the classroom video, specifically including: Determine the target concise video length, and select an appropriate number of knowledge point segments and interactive highlights based on the target concise video length; All clips are sorted in ascending order of start time, and a 1.5-second fade-in / fade-out transition effect is inserted between adjacent clips; Synchronized subtitles are generated for each segment based on the ASR timestamp, and a 3-second title card is overlaid at the beginning of each knowledge point segment; the title card includes the core noun phrase corresponding to the segment; Generate a JSON-formatted directory structure, including the title, timestamp, and type of each segment; Perform video compositing and output.

[0020] In one embodiment of this application, determining the target refined video length and selecting a corresponding number of knowledge point segments and interactive highlights based on the target refined video length specifically includes: Knowledge point segments and interactive highlights with time overlap exceeding the fifth threshold are merged, and only the segments with higher scores are retained; Determine the target concise video length, and based on the time allocation ratio between knowledge point segments and interactive highlights, obtain the concise video lengths for knowledge point segments and interactive highlights. Based on the duration of the refined knowledge point videos and the refined interactive segments videos, the K1 knowledge point videos and K2 interactive segments with the highest scores were selected respectively; the total duration of the K1 knowledge point videos was between 0.9 and 1.1 times the duration of the refined knowledge point videos, and the total duration of the K1 interactive segments was between 0.9 and 1.1 times the duration of the refined interactive segments videos.

[0021] Compared with existing technologies, the beneficial effects of adopting the above technical solution are as follows: 1. By extracting multimodal features from vision, audio, and text, the accuracy of identifying important classroom segments is improved, making the detection results of knowledge points and interactive scenarios more comprehensive and reliable.

[0022] 2. By constructing a teaching rhythm curve and using its peaks and valleys as a basis for judgment, the key points of classroom explanation, rhythm changes, and peak teaching positions can be accurately located, thereby improving the accuracy and stability of knowledge point identification.

[0023] 3. By combining a time-weighted model with prior information about the classroom structure, the content of different time periods is processed differently, which improves the probability of identifying core content and reduces interference from irrelevant segments.

[0024] 4. By combining rhythm features, text features, and time-series models to identify knowledge point boundaries, the starting and ending positions of knowledge points are more accurate, avoiding problems such as fragments that are too long, too short, or incomplete.

[0025] The five-dimensional quantitative evaluation method of interactive engagement enables precise differentiation of the quality of teacher-student interaction and can effectively identify in-depth candidate interactive segments with teaching value.

[0026] 6. By combining knowledge point scores and interaction scores for sorting and filtering, it can automatically generate concise videos that cover core content and exciting interactions, significantly improving the overall content density.

[0027] 7. Automatic transitions, subtitle generation, and table of contents construction make the output video structure clearer, improve the viewing experience, and make it easier for students to quickly locate and review key content.

[0028] 8. Through fully automated processing, classroom videos can be analyzed and refined without human intervention, significantly reducing processing time and making it suitable for automated processing of large-scale classroom videos. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0030] Figure 1 This is a flowchart illustrating a refined method for classroom videos according to an embodiment of this application.

[0031] Figure 2 This is a schematic diagram of the teaching rhythm curve in an embodiment of this application.

[0032] Figure 3 This is a graph showing the time weighting function and class time in an embodiment of this application.

[0033] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0034] The embodiments of this application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar modules or modules having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. Rather, the embodiments of this application include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0035] To address the shortcomings of existing technologies, this application proposes a method for refining classroom videos. This intelligent method utilizes video, audio, text, and time information to accurately identify key knowledge segments and engaging moments, and generates high-quality refined videos.

[0036] Please refer to Figure 1 This refined method for classroom videos mainly includes the following steps: S101. Extract multimodal features from the classroom video, including visual features, audio features, and text features.

[0037] Before extracting multimodal features from classroom videos, preprocessing is required. In this embodiment, the input classroom video is first decoded; then, consecutive frame images are extracted from the video at a fixed frame rate (e.g., 3 frames per second) and uniformly scaled to 1280×720 resolution; next, the audio signal is separated from the video and converted to mono, with a sampling rate set to 48kHz. Finally, the video frame sequence and audio signal are time-aligned to form the basis data for subsequent analysis.

[0038] In this embodiment, multimodal feature extraction mainly includes the extraction of visual features, audio features, and text features.

[0039] Specifically, visual features mainly include teacher actions, blackboard content, PPT slides, and student expressions. In one embodiment, a multi-stream convolutional neural network model (such as ResNet or Vision Transformer) is used to extract features from each frame of the video frame sequence to obtain visual features including the teacher's movement amplitude, movement trajectory, changes in the blackboard area, changes in the PPT page, and changes in student expressions.

[0040] Audio feature extraction involves short-time analysis of the audio signal to extract audio energy E(t), speech rate variation S(t), pitch variation P(t), volume and background noise variations, and teacher / student voiceprint distribution. In one embodiment, speech rate variation is calculated based on the number of words per unit time. Pitch variation is calculated based on the fundamental frequency difference between adjacent analysis frames.

[0041] Text feature extraction is achieved by calling an Automatic Speech Recognition (ASR) model to transcribe audio and obtain a time-aligned text sequence; keywords, technical terms, high-frequency nouns, and prompting phrases (such as "to summarize", "to introduce", "today we will talk about") are extracted to form a text feature sequence.

[0042] By extracting multimodal features from vision, audio, and text, the accuracy of identifying important classroom segments is improved, making the detection results of knowledge points and interactive scenarios more comprehensive and reliable.

[0043] S102. Based on the multimodal features, determine the knowledge point fragments and obtain the importance score.

[0044] After extracting multimodal features, knowledge point segments can be determined based on these features. In this embodiment, the determination of knowledge point segments mainly includes the following processes: (1) constructing a teaching rhythm curve based on audio features; (2) detecting and generating a candidate boundary set for knowledge points based on the teaching rhythm curve; (3) the candidate boundary set includes a start boundary candidate set and an end boundary candidate set; (4) correcting the candidate boundary set using text features; (5) filtering the corrected candidate boundary set using an LSTM time series model; and (6) determining the knowledge point segments using the filtered candidate boundary set. Each step will be explained in detail below.

[0045] In this embodiment, by constructing a teaching rhythm curve and using its peak-valley changes as a judgment criterion, the precise location of key points in class, rhythm changes, and peak teaching positions can be achieved, improving the accuracy and stability of knowledge point identification. Specifically, the teaching rhythm curve is constructed using audio energy E(t), speech rate change S(t), and pitch change P(t), with the following expression:

[0046] To facilitate subsequent calculations, the above equation is normalized to obtain:

[0047] in, The normalized audio energy is mapped to [0,1] using max-min normalization. The normalized speech rate change is calculated as follows: , for The minimum value, for The maximum value; The normalized pitch variation is calculated as follows: , for The minimum value, for The maximum value; For the weighting coefficients, satisfying The preferred value is , , ; The value range is [0,1].

[0048] Based on the constructed teaching rhythm curve, the candidate boundary set for generating knowledge points can be detected. First, the first derivative of the teaching rhythm curve is calculated. The candidate boundary set can be determined by using derivatives and preset thresholds.

[0049] In this embodiment, all those that satisfy... > The time points are used as candidate start boundaries, forming a candidate set of start boundaries. ,include{ }; for all that satisfy the derivative The time points are used as candidate termination boundaries, forming a candidate set of termination boundaries. ,include{ In this embodiment, Consider it as the first threshold, Considered as the second threshold, where, for The mean, Standard deviation This is an adjustable coefficient. .

[0050] After determining the candidate boundary set, correction is needed based on text features. Specifically, this is done by adjusting the values ​​before and after each candidate start and end boundary. Within a time window, the start and end prompt phrases are retrieved. If a corresponding prompt phrase is found within the time window, the corresponding boundary is adjusted to the timestamp corresponding to the prompt phrase. In this embodiment, ∈ [5 seconds, 15 seconds]. Start prompts can include "Next," "Next," "First," "Now we," "So," etc. End prompts can include "To summarize," "That's all," "That concludes," "Okay," etc. Start and end prompts can be added or removed as needed.

[0051] After completing the candidate boundary set correction, temporal context verification is also required to complete the boundary filtering. In this embodiment, the time-series context verification is performed on each candidate boundary in the candidate boundary set. The feature sequence of seconds is input into a pre-trained LSTM time series model for scoring, and the boundary confidence score is output. Keep score Greater than the third threshold The candidate boundary. Among them, [20 seconds, 40 seconds], depending on the average length of the clip. [0.5, 0.7]. The characteristic sequences include... ,in, This is the sequence of corresponding values ​​for the teaching rhythm curve. This is a time series of keyword density. Visual features (such as PPT page turning and blackboard updates).

[0052] For the corrected and filtered candidate boundary set, knowledge point fragments can be generated by boundary pairing. Specifically, the candidate start boundaries and candidate end boundaries in the candidate boundary set are sorted by time; a greedy algorithm is used to match the nearest candidate end boundary to each candidate start boundary to generate knowledge point fragments, thus obtaining the set of all knowledge point fragments from the classroom video. = ,in, .

[0053] This embodiment uses a knowledge point boundary identification method that combines rhythm features, textual semantics, and temporal models to make the start and end positions of knowledge points more accurate, effectively avoiding problems such as segments being too long, too short, or incomplete. For example... Figure 2 The figure shows the teaching rhythm curve of a portion of a classroom video. A peak appears at t=22.3 min (marked as 1 in the figure), with R(t)=0.99, corresponding to the teacher loudly emphasizing 'f must be continuous on the closed interval and differentiable on the open interval!'. Subsequently, a falling edge (marked as 4 in the figure) is triggered, and after verification by the semantic tag 'summarize' and an LSTM confidence score of 0.78, it is confirmed as the end boundary of the knowledge point.

[0054] In one embodiment, a segment duration constraint can be set as needed, and abnormal segments can be filtered based on this constraint. Specifically, the segment duration constraint is as follows: ,in, For the duration of the segment, Minimum duration, For the maximum duration, the preferred option is... [20 seconds, 40 seconds] [300 seconds, 600 seconds]. If the duration after pairing does not meet the constraint, try adjusting to the next nearest boundary. If it still cannot meet the constraint, discard the candidate boundary.

[0055] To ensure that all the knowledge points in the refined video are key knowledge points, this embodiment also requires scoring the importance of the acquired knowledge point segments.

[0056] Specifically, for each knowledge point segment i, the average value R(i) of the rhythm curve within the segment, the time weight W(i) corresponding to the center time of the segment, the keyword density K(i) of the ASR text, and the segment duration D(i) are calculated respectively. Among them, R(i) can be calculated by extracting the teaching rhythm curve R(t).

[0057] The time weight W(i) corresponding to the center time of the segment is calculated based on the priority weighting of class time. In this embodiment, the class time is divided into three teaching stages according to the total video duration T, and the time and weight peak settings are as follows: (1) Introduction period: [0, 0.2T], peak weight ∈[0.5,0.7].

[0058] (2) Core explanation period: [0.2T, 0.85T], peak weight =1.0.

[0059] (3) Summary period: [0.85T, T], peak weight ∈[0.4,0.6].

[0060] Based on this, a piecewise Gaussian function is used to construct the time-weighted function. :

[0061] in, For the i-th stage, μ1 = 0.1T, μ2 = 0.5T, and μ3 = 0.9T; The standard deviation is taken as 0.25 to 0.35 times the duration of this stage.

[0062] Based on the above time weighting function This allows for the calculation of the time weight W(i) corresponding to the center time of a segment. By combining a time-weighted model with prior information about the classroom structure, content from different time periods is processed differently, improving the probability of identifying core content and reducing interference from irrelevant segments. For example... Figure 3 The diagram shows the time-weighted function W(t) and the comparison with class time. The wireframe represents the typical division of a 45-minute university class: 0–9 min introduction, 9–38 min core content, and 38–45 min summary. W(t) reaches a peak of 1.0 in the core content segment, ensuring that the knowledge points in this time segment receive the highest scoring weight.

[0063] The keyword density K(i) of the ASR text is obtained by dividing the number of keywords in the fragment by the total number of words.

[0064] After normalizing the average rhythm curve R(i) within the knowledge point segment, the time weight W(i) corresponding to the segment's center time, the keyword density K(i) of the ASR text, and the segment duration D(i), the importance score of knowledge point segment i can be obtained by weighted summation. :

[0065] in, =( - / ( - ), which is the normalized value of the average rhythm intensity of knowledge point segment i; Let W(i) be the normalized value of the time weight W(i) corresponding to the center time of segment i; Let i be the normalized keyword density value of fragment i. [0.05, 0.15]; The duration factor is limited to [0,1]. . These are the weighting coefficients, satisfying... Preferred .

[0066] Finally, score the importance of all knowledge point segments. Normalize to the [0,100] interval; the normalization formula is: ,in, , These represent the minimum and maximum importance scores for knowledge point segments, respectively.

[0067] Normalized importance score of this knowledge point segment It can be used to select knowledge point segments in the subsequent video refinement stage.

[0068] By constructing a teaching rhythm curve and using its peaks and valleys as a basis for judgment, the precise positioning of key points, rhythm changes, and peak teaching positions in the classroom can be achieved, thereby improving the accuracy and stability of knowledge point identification.

[0069] S103. Based on the multimodal features, determine the interactive highlights and obtain the highlights score.

[0070] To ensure the richness of the refined videos, this embodiment also proposes extracting engaging interactive segments from classroom videos. Through a multi-dimensional quantitative evaluation method for the engaging nature of these interactions, the quality of teacher-student interactions can be accurately differentiated, effectively identifying in-depth candidate interactive segments with educational value.

[0071] First, candidate interactive segments need to be labeled. Specifically, the teacher's voice and the student's voice are distinguished by the voiceprint in the audio features. All segments in the classroom video where there are ≥2 different speakers within 5 consecutive seconds, one of whom is the teacher, are detected and labeled as candidate interactive segments.

[0072] Then, for each candidate interaction segment j, its emotional intensity is calculated based on the multimodal features extracted by S101. Content depth Feedback intensity and duration factor The specific calculation process is as follows: (1) Emotional intensity The activation value is determined by the output of the speech emotion model when extracting audio features.

[0073] (2) Content Depth The calculation is based on text features (ASR results), including: complexity_score: average sentence length and number of clauses; keyword_score: frequency of subject-specific keywords; in this embodiment, the content depth is normalized and represented as:

[0074] (3) Feedback intensity The calculation combines audio and text features, including: response_count_norm: a normalized value of the number of student speeches obtained through voiceprint recognition; and emotion_positive_ratio: the proportion of positive emotions obtained through text sentiment analysis. In this embodiment, the normalized response intensity is represented as:

[0075] (4) Duration factor : =Fragment Actual duration / ,in, The average duration of all candidate interactive segments.

[0076] Next, the emotional intensity E(j), content depth C(j), feedback intensity F(j), and duration factor L(j) are normalized, and a weighted sum is calculated to obtain the interaction excellence score of the j-th candidate interactive video. :

[0077] in, Normalized emotional intensity ; To normalize the depth of content, ; The normalized feedback strength is denoted as . For the normalized duration factor, All are weighting coefficients. Preferred .

[0078] Similar to the importance score for knowledge point segments, a score for the engaging nature of all candidate interactive segments is also required. Normalize to the [0,100] interval; the normalization formula is: ,in, , These are the minimum and maximum interactive quality scores for the candidate interactive segments, respectively.

[0079] Normalized Interaction Enthusiasm Score Based on Candidate Interaction Segments Candidate interactive segments with an interactive brilliance score greater than the fourth threshold are retained as interactive brilliance segments.

[0080] This collection provides a compilation of interactive highlights. = This corresponds to the score for the level of engagement in the interaction. .

[0081] S104. Based on the scores of the knowledge point segments and the interactive highlights, select the corresponding segments to complete the classroom video refinement.

[0082] Based on the knowledge point segments and importance scores obtained from S102 and S103, as well as the interactive highlights and interactive highlights scores, the corresponding segments can be selected to complete the refinement of the classroom video.

[0083] In this embodiment, before the video is played, an overlap check is performed on the knowledge point segments and the interactive highlights. If the time overlap between a knowledge point segment and an interactive highlight exceeds a fifth threshold, the segment with the higher score is retained, and the other segment is discarded. Preferably, the fifth threshold is set to 50%.

[0084] Then, based on the set target video length and the time allocation ratio between knowledge point segments and interactive highlights, the refined video lengths for knowledge point segments and interactive highlights are obtained. Next, based on these refined video lengths, the K1 knowledge point segments and K2 interactive highlights with the highest scores are selected. The total length of the K1 knowledge point segments is between 0.9 and 1.1 times the total length of the refined video length for knowledge point segments, and the total length of the K1 interactive highlights is also between 0.9 and 1.1 times the total length of the refined video length for interactive highlights. Through comprehensive sorting and filtering based on the importance score of knowledge points and the engagement score of interactions, refined videos covering core content and engaging interactions can be automatically generated, significantly improving the overall content density.

[0085] The selection methods for knowledge point segments and interactive highlights are the same. In this embodiment, the selection process of knowledge point segments will be specifically explained using knowledge point segments as an example.

[0086] Assuming the target refined video duration is The time allocation ratio between knowledge point segments and interactive highlights is 7:3. Therefore, the percentage of time allocated to knowledge point segments is... =0.7. The specific steps are as follows: Step 1: Refine the video length according to the target. and the proportion of knowledge points. Calculate the duration of concise video segments for key knowledge points. ,Right now = × .

[0087] Step 2: Compile knowledge point fragments The knowledge points in the text are scored according to their importance. Sort the segments from highest to lowest to form a sorted sequence.

[0088] Step 3: Use a greedy accumulation strategy to select knowledge point fragments, as follows: initialization It is an empty set, and the accumulated duration is... Set to 0; Iterate through the sorted sequence of segments in turn, when The sum of the current segment does not exceed When the value is 1.1 times, add the current segment. and add its duration to ;when Reaching or exceeding When the value is 0.9 times the value, the selection can be terminated early.

[0089] Step 4: If the final cumulative_time is less than If the number of selectable knowledge fragments is less than 0.5 times the original value, a warning message will be triggered, and the user will be advised to adjust the filtering parameters.

[0090] Step 5, Output and actual accumulated time As a result of selecting knowledge point fragments.

[0091] After selecting key knowledge segments and engaging interactive segments, the next step is to refine the video, which includes the following process: (1) Time sorting and transition processing: Arrange all segments in ascending order of start time, and insert a 1.5-second fade-in and fade-out transition effect between adjacent segments. If the segment interval is <5 seconds, connect them directly; if it is >5 seconds, display a time omission prompt.

[0092] (2) Subtitle and annotation generation: Synchronized subtitles are generated for each segment based on the ASR timestamp, and a 3-second title card (i.e., displaying the name of the knowledge point) is superimposed at the beginning of each knowledge point segment. The title text is the core noun phrases extracted from the ASR text.

[0093] (3) Table of Contents Generation: Generate a JSON format table of contents structure, including the title, timestamp, and type (knowledge point / interaction) of each segment. The table of contents index enables chapter jumping in the player.

[0094] (4) Video synthesis output: Use FFmpeg or similar tools to splice the video, and set the output format to MP4, H.264 encoding, 1280×720 resolution, and also include SRT subtitle files and JSON directory files.

[0095] Through the above-mentioned automatic transitions, subtitle generation, and table of contents construction processes, the output video structure is clearer, the viewing experience is better, and it is easier for students to quickly locate and review key content.

[0096] The fully automated processing method of this invention can complete the analysis and refinement of classroom videos without human intervention, greatly shortening the processing time and making it suitable for the automated processing of large-scale classroom videos.

[0097] To verify the effectiveness of this invention, a university advanced mathematics course was used as an example for refinement. The original video was 45 minutes long, and the method of this invention automatically selected 6 segments with a total duration of 629 seconds (≈10.5 minutes), accounting for 23.3% of the original duration. The ratio of knowledge points to interaction was approximately 70%:30%, with an average comprehensive score of 90.1, meeting the target duration requirement of 10 minutes ± 5%. Specific segments are shown in Table 1. Table 1. Clips used in the refined video

[0098] Based on the same technical concept, embodiments of this application also provide an electronic device that can implement the refined method flow for classroom videos provided in the above embodiments of the present invention. In one embodiment, the electronic device can be a server, a terminal device, or other electronic devices. Figure 4 As shown, the electronic device may include: At least one processor and a memory connected to the at least one processor. In this embodiment of the invention, the specific connection medium between the processor and the memory is not limited. Figure 4 The example used is the connection between the processor and memory via a bus. The bus... Figure 4 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. Buses can be divided into address buses, data buses, control buses, etc., but for ease of representation, [the specific bus type is not shown here]. Figure 4 The processor is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, a processor can also be called a controller; there are no restrictions on the name.

[0099] In this embodiment of the invention, the memory stores instructions executable by at least one processor. By executing the instructions stored in the memory, the at least one processor can perform the refining method for classroom videos described above. The processor can implement... Figure 4 The functions of each module in the device shown.

[0100] The processor is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory and calling data stored in memory, it can monitor the device's various functions and process data, thereby enabling overall monitoring of the device.

[0101] In an alternative design, the processor may include one or more processing units. The processor may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor. In some embodiments, the processor and memory may be implemented on the same chip; in some embodiments, they may also be implemented separately on separate chips.

[0102] The processor can be a general-purpose processor, such as a CPU, digital signal processor, application-specific integrated circuit, field-programmable array, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the refinement method for classroom videos disclosed in the embodiments of this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0103] Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited thereto. In embodiments of the present invention, memory can also be a circuit or any other device capable of implementing storage functions, used to store program instructions and / or data.

[0104] By designing and programming the processor, the code corresponding to the refined method for classroom videos described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the steps of the methods described in the foregoing embodiments during runtime. How to design and program a processor is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0105] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform a refining method for classroom videos as described above.

[0106] In some alternative embodiments, the present invention also provides a aspect of a refining method for classroom videos that can also be implemented as a program product comprising program code that, when the program product is run on a device, causes the control device to perform the steps in a refining method for classroom videos according to various exemplary embodiments of the present invention as described in this specification.

[0107] It should be noted that although several units or sub-units of the apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the invention, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units. Furthermore, although the operation of the method of the invention is described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0108] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a server, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] Program code for performing the operations of this invention can be written using any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0111] In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0112] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0113] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for refining classroom videos, characterized in that, include: Multimodal features are extracted from classroom videos, including visual features, audio features, and text features; Based on the multimodal features, knowledge point fragments are determined and importance scores are obtained; Based on the multimodal features, interactive highlights are identified, and a highlight score is obtained. Based on the scores of the knowledge point segments and the interactive highlights, select the corresponding segments to complete the classroom video refinement.

2. The method for refining classroom videos according to claim 1, characterized in that, The process of determining the knowledge point fragments includes: A teaching rhythm curve is constructed based on audio features; the audio features include audio energy, speech rate changes, and pitch changes. Based on the teaching rhythm curve, a candidate boundary set for knowledge points is generated; the candidate boundary set includes a start boundary candidate set and an end boundary candidate set. Textual features are used to correct the candidate boundary set; An LSTM time series model is used to filter the corrected candidate boundary set; The selected candidate boundary set is used to determine knowledge point fragments.

3. The method for refining classroom videos according to claim 2, characterized in that, The process of generating a candidate boundary set for knowledge points based on the teaching rhythm curve specifically includes: Calculate the derivative of the teaching rhythm curve; All time points that satisfy the condition that the derivative is greater than the first threshold are taken as candidate start boundaries and form a candidate set of start boundaries; All time points where the derivative is less than the second threshold are taken as candidate termination boundaries and form a candidate set of termination boundaries.

4. The method for refining classroom videos according to claim 3, characterized in that, The step of correcting the candidate boundary set using text features specifically includes: Each candidate start boundary in the candidate start boundary set ± Search for a preset start prompt phrase within a second range. If found, adjust the corresponding candidate start boundary to the time point of the preset start prompt phrase. Each candidate end boundary in the candidate end boundary set ± Search for a preset end prompt phrase within a time range of seconds. If found, adjust the corresponding candidate end boundary to the time point of the preset end prompt phrase.

5. The method for refining classroom videos according to claim 3, characterized in that, The step of using an LSTM time series model to filter the corrected candidate boundary set specifically includes: Obtain the front and back of each candidate boundary in the candidate boundary set. The feature sequence is calculated in seconds and input into a pre-trained LSTM time series model for scoring, retaining candidate boundaries with scores greater than a third threshold; the feature sequence includes visual features, keyword density, and teaching rhythm curve values.

6. The method for refining classroom videos according to claim 3, characterized in that, The selected candidate boundary set is used to determine knowledge point fragments, specifically including: Sort the candidate start boundaries and candidate end boundaries in the candidate boundary set by time. A greedy algorithm is used to match the nearest candidate end boundary for each candidate start boundary, generating knowledge point fragments and filtering out abnormal fragments. Obtain a collection of all knowledge point segments from the class videos.

7. The method for refining classroom videos according to claim 3, characterized in that, The specific process for obtaining the importance score includes: Calculate the average value of the rhythm curve within each knowledge point segment, the time weight corresponding to the center time of the segment, the keyword density of the ASR text, and the segment duration; the time weight corresponding to the center time of the segment is calculated based on the priority weighting of class time. The importance score of the corresponding knowledge point segment is obtained by weighting the average value of the rhythm curve within the segment, the time weight corresponding to the center time of the segment, the keyword density of the ASR text, and the segment duration.

8. The method for refining classroom videos according to claim 1, characterized in that, The process of determining engaging interactive segments based on the multimodal features and obtaining a performance score specifically includes: Distinguish between teacher's and student's voices based on audio features; The system detects all segments of classroom videos in which at least two different speakers appear within a 5-second window, with one of them being the teacher, and marks them as candidate interactive segments. The emotional intensity, content depth, feedback intensity, and duration factor of each interactive highlight are calculated based on multimodal features. The interaction quality score for each candidate interactive segment is obtained by weighting the emotional intensity, content depth, feedback intensity, and duration factor. Candidate interactive segments with an interactive brilliance score greater than the fourth threshold are retained as interactive brilliance segments.

9. The method for refining classroom videos according to claim 1, characterized in that, Based on the scores for knowledge point segments and engaging interactive segments, select the corresponding segments to complete the lesson video refinement, specifically including: Determine the target concise video length, and select an appropriate number of knowledge point segments and interactive highlights based on the target concise video length; All clips are sorted in ascending order of start time, and a 1.5-second fade-in / fade-out transition effect is inserted between adjacent clips; Synchronized subtitles are generated for each segment based on the ASR timestamp, and a 3-second title card is overlaid at the beginning of each knowledge point segment; the title card includes the core noun phrase corresponding to the segment; Generate a JSON-formatted directory structure, including the title, timestamp, and type of each segment; Perform video compositing and output.

10. The method for refining classroom videos according to claim 9, characterized in that, The process of determining the target concise video length and selecting a corresponding number of knowledge point segments and interactive highlights based on the target concise video length specifically includes: Knowledge point segments and interactive highlights with time overlap exceeding the fifth threshold are merged, and only the segments with higher scores are retained; Determine the target concise video length, and based on the time allocation ratio between knowledge point segments and interactive highlights, obtain the concise video lengths for knowledge point segments and interactive highlights. Based on the duration of the refined knowledge point videos and the refined interactive segments videos, the K1 knowledge point videos and K2 interactive segments with the highest scores were selected respectively; the total duration of the K1 knowledge point videos was between 0.9 and 1.1 times the duration of the refined knowledge point videos, and the total duration of the K1 interactive segments was between 0.9 and 1.1 times the duration of the refined interactive segments videos.

Images