An adaptive key frame extraction method for efficient and stable analysis of classroom teaching videos
Through the collaborative optimization of the MST-GRM, CHACD, and EKQI modules in the HCFS framework, the problems of stability and insufficient semantic coverage in keyframe extraction in classroom teaching videos have been solved, achieving efficient and stable keyframe extraction and complete teaching logic.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TAIYUAN UNIVERSITY OF SCIENCE AND TECHNOLOGY
- Filing Date
- 2026-04-20
- Publication Date
- 2026-07-14
Smart Images
Figure CN122391957A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of smart education and video content analysis technology, specifically relating to an adaptive keyframe extraction method for classroom teaching videos, which enables efficient and stable analysis of teaching videos. Background Technology
[0002] Classroom teaching videos, as an important carrier of modern educational informatization, play a crucial role in applications such as teaching review, intelligent summarization, and online educational resource management. With the popularization of online education and smart classrooms, massive amounts of classroom teaching video data urgently require efficient structured analysis and content understanding. However, how to stably and efficiently extract keyframes from classroom teaching videos to achieve structured summarization of video content has become a key technical bottleneck restricting the intelligent application of educational videos. Classroom teaching videos exhibit distinct domain characteristics: semantic content evolves continuously, visual differences between adjacent frames are subtle, shot transitions are sparse, and scene topology remains stable over long periods; semantic transformations between teaching behaviors show gradual characteristics. This presents a fundamental contradiction in keyframe extraction from classroom teaching videos: the continuous evolution of semantics and the discrete decision-making of keyframes are difficult to reconcile. Current mainstream keyframe extraction methods are primarily designed for general videos with short durations and frequent scene changes, focusing on capturing obvious visual changes. When these methods are directly applied to classroom teaching videos, performance bottlenecks often occur due to differences in data characteristics. Many task-specific optimization methods, such as keyframe selection strategies based on Simultaneous Localization and Mapping (SLAM), prioritize system performance such as localization accuracy and loop closure detection, without specifically addressing the continuous and smooth evolution of semantic content. In the field of video summarization and keyframe extraction, researchers have proposed various methods, such as DR-DSN based on deep reinforcement learning, AVS based on attention encoder networks, D-KTS based on temporal segmentation, DSNet driven by detection, and more recently, LMSKE and AKS based on large-scale pre-trained models, all achieving corresponding results in their respective target task scenarios. However, when these methods are applied to classroom teaching videos with continuous semantics, they generally suffer from problems such as unbalanced temporal distribution of keyframes and incomplete semantic coverage, making it difficult to meet the requirements of classroom teaching videos for stable, coherent and semantically complete keyframe sequences. A deeper analysis of the root causes of these problems reveals a common limitation in the design paradigm of existing methods: the three core components—feature representation, decision-making mechanisms, and evaluation systems—are often treated in a fragmented manner. Feature construction is systematically disconnected from task requirements; many methods directly use general pre-trained models to extract features. These models are not optimized for the domain characteristics of classroom teaching videos, resulting in limited ability to distinguish subtle semantic changes in teaching behavior. Decision-making mechanisms often rely on single clustering algorithms or fixed thresholds, leading to poor adaptability to initialization and noise, making it difficult to guarantee the consistency of semantic grouping under unsupervised conditions. Evaluation systems overly depend on the matching degree with manual annotations, neglecting the intrinsic quality attributes of keyframe sets, such as semantic coverage, temporal distribution uniformity, and redundancy control. Even in recent cutting-edge work, these fragmentation problems have not been systematically resolved. The KeyFrame Extraction via Semantic Coherence (KFE-SC) method proposed by Yan et al. is limited to a supervised framework, the medical imaging method proposed by Zhang et al. relies on domain-specific data, and the Keyframe Insertion via Neural Network Decision-making (KINND) framework proposed by Dong et al. lacks an adaptive mechanism in its composite loss function. This indicates that how to systematically and collaboratively optimize features, decisions, and evaluation to address the unique challenges of classroom teaching videos remains a bottleneck that urgently needs to be overcome in this field. In summary, while existing keyframe extraction methods perform well in general video scenarios, they face unique challenges in the intelligent analysis of classroom teaching videos. Unlike general videos such as news and vlogs, which feature frequent scene changes and significant visual shifts, classroom teaching videos are characterized by highly fixed scenes, subtle visual changes, and a gradual evolution of teaching semantics. In these videos, teaching behaviors such as writing on the blackboard line by line and gradually unfolding knowledge points exhibit continuous and gradual changes, resulting in minimal visual differences between adjacent frames while significant changes in teaching semantics may have occurred. Traditional keyframe extraction methods that rely on significant visual changes struggle to accurately capture key semantic nodes in the teaching process. Furthermore, keyframes in teaching videos must also meet the implicit constraint of fully reproducing the teaching process; that is, the selected keyframe sequence should connect a complete teaching logic chain, rather than merely pursuing visual diversity or uniform temporal distribution. Therefore, existing methods suffer from fragmented designs in feature modeling, clustering decision-making, and quality assessment, leading to an imbalance in the temporal distribution of keyframes, incomplete semantic coverage of the teaching process, and insufficient stability. How to effectively reduce the impact of gradual changes in teaching semantics and subtle visual changes on model accuracy, and design an adaptive keyframe extraction method that considers the temporal semantic characteristics and the integrity of teaching logic in classroom teaching videos, are key issues that urgently need to be addressed in this field. Summary of the Invention
[0003] To address the problems of existing keyframe extraction methods in classroom teaching video analysis, such as reliance on single feature representation, insufficient decision stability, and lack of unified quality evaluation standards, especially the failure of existing methods to fully consider the characteristics of classroom teaching video scenes being highly fixed, subtle visual changes, and gradual evolution of teaching semantics, making it difficult for keyframes to accurately capture key semantic nodes of gradual teaching behaviors such as line-by-line writing on the blackboard and the gradual unfolding of knowledge points, the purpose of this invention is to provide an adaptive keyframe extraction method for efficient and stable analysis of classroom teaching videos. This invention is achieved using the following technical solution: An adaptive keyframe extraction method for efficient and stable analysis of classroom teaching videos is proposed. A Hybrid Collaborative Framework for Stable Keyframe Extraction (HCFS) is constructed to extract keyframes from classroom teaching videos exhibiting stable semantic evolution and subtle visual differences between adjacent frames. The HCFS framework includes a multi-scale semantic-aware spatio-temporal graph representation module (MST-GRM) heterogeneous robust feature representation learning module, a clustering-based hierarchical adaptive consensus decision module (CHACD), and an enhanced keyframe quality evaluation metric. Index (EKQI) is used to learn features of the continuously evolving teaching semantics of the classroom teaching video by modeling multi-scale semantics and spatiotemporal relationships. Clustering consensus decision is used to stably divide the continuous teaching semantic flow into semantic groups corresponding to teaching events such as blackboard explanation and teacher-student interaction and generate candidate keyframes. Multi-dimensional quality assessment is used to comprehensively and quantitatively evaluate the teaching semantic coverage and logical consistency of the candidate keyframes. The collaborative optimization of feature learning and decision-making is achieved through closed-loop feedback driven by assessment. The MST-GRM module includes a Multi-scale Dual-pathway Feature Extraction (MSBP) module, used to capture local teaching dynamics such as the macro-layout of the classroom and blackboard writing; a Semantic-Aware Generative Enhancement (SAGE) module, used to enhance subtle feature differences generated by the gradual changes in teaching semantics such as line-by-line writing on the blackboard; and a Spatio-Temporal Graph Neural Networks (ST-GNN) module, used to model the temporal dependence of teaching and cross-temporal semantic associations. The CHACD module includes a multi-view clustering construction module, a consensus matrix calculation module, and a hierarchical stable grouping and representative frame selection module. The EKQI module includes a temporal monotonicity evaluation module to ensure the temporal logic of the teaching process, a semantic representativeness evaluation module to evaluate the completeness of blackboard writing and knowledge delivery coverage, a visual diversity evaluation module, a redundancy evaluation module, and a temporal coverage evaluation module. Further optimization involves constructing an HCFS collaborative unsupervised keyframe extraction framework, with the following specific steps: First, the MST-GRM module is constructed. Addressing the issue of insufficient feature discriminativeness caused by the fixed scenes in classroom teaching videos resulting in minimal visual differences between adjacent frames, while the teaching semantics, such as blackboard writing and knowledge delivery, continuously evolve, this module collaboratively learns frame-level features with strong discriminative power and high temporal consistency through MSBP, SAGE, and ST-GNN. MSBP employs a heterogeneous dual-branch architecture. The global branch captures macroscopic structures such as the podium layout and blackboard area, while the local branch focuses on extracting local teaching dynamics such as changes in handwriting in the blackboard writing area and teacher's standing area. These are integrated through residual projection and splicing fusion. SAGE addresses the issue that adjacent frames are visually almost identical when writing line by line, but the teaching semantics have evolved from "writing halfway" to "writing completed." It introduces random offsets constrained by teaching semantics into the feature space, predicts offset vectors through a generator network, and enhances them in the form of residuals, thus widening the feature distance between frames with different blackboard completion levels and different explanation stages. ST-GNN constructs a graph model containing temporal edges and semantic edges. Temporal edges ensure the natural evolution order of blackboard writing and knowledge explanation, while semantic edges capture the semantic connections between teaching across time periods (such as the correspondence between the definition at the beginning of class and the summary review at the end). It outputs enhanced frame representations that integrate the teaching spatiotemporal context through graph convolution. Then, the CHACD module is constructed. Addressing the instability of keyframe decisions under unsupervised conditions and the uneven distribution of teaching events such as blackboard explanations and teacher-student interactions—where a blackboard segment lasts several minutes with gradual visual changes, while a question-and-answer session lasts only tens of seconds but involves dramatic semantic shifts—making it difficult for a single cluster to stably divide teaching semantic segments, this module designs a heterogeneous clustering ensemble strategy. Through algorithmic complementarity (partitioning clustering captures single blackboard explanations or example demonstrations, hierarchical clustering captures large-granular structures such as "review and introduction—new knowledge instruction—example explanation—class summary"), parameter perturbation, and feature subspace complementarity, diverse cluster sets are constructed. A consensus matrix is built to quantify stable co-occurrence relationships between frames, overcoming the problem of a single algorithm incorrectly segmenting the same blackboard or incorrectly merging different segments. Based on consensus distance, hierarchical clustering generates semantic groups corresponding to blackboard stage transitions, knowledge point transitions, and interactive segments. Within each group, frames that represent the core content of the teaching event and are located at the event's central time position are selected as keyframes through weighted selection of feature centrality and temporal centrality, ensuring that the decision results match the actual distribution of events in classroom teaching. Next, the EKQI metric was constructed. Addressing the issue that existing evaluation systems cannot reflect the need for complete reproduction of the teaching process in classroom teaching videos, this metric integrates seven dimensions: temporal monotonicity, representativeness score, shot coverage, redundancy, visual diversity, and traditional accuracy. Temporal monotonicity ensures the correct logical sequence of teaching; representativeness score ensures complete coverage of the process of writing on the blackboard from blank to complete, and the process of explaining knowledge points from introduction to in-depth; shot coverage ensures that teaching segments such as explanation, blackboard writing, and interaction are representative; redundancy suppresses repeated sampling of the same blackboard writing state or explanation posture; temporal coverage ensures that keyframes are evenly distributed along the timeline of the entire lesson; visual diversity ensures that keyframes have sufficient differences in blackboard writing completion, teacher posture, etc. EKQI can comprehensively characterize the overall quality of keyframes in terms of teaching semantic coverage, process reproduction, and structural characterization without the need for manual annotation. Finally, a high-dimensional hyperparameter optimization algorithm (HCOA) is used to achieve overall collaborative optimization of the framework. Key hyperparameters of each module are uniformly modeled as configuration vectors, and the EKQI comprehensive score is used as the fitness function for adaptive optimization. This allows the model to adapt to the feature distribution and teaching semantic structure of different classroom scenarios, such as blackboard explanation, interactive discussion, and experimental demonstration, achieving closed-loop collaborative optimization of feature learning, keyframe decision-making, and quality assessment. The technical solution provided by this invention has the following advantages compared with the prior art: First, in response to the characteristics of classroom teaching videos, such as highly fixed scenes, subtle visual changes, and continuous and gradual evolution of teaching semantics, this invention constructs the HCFS collaborative optimization framework. Through the closed-loop collaborative design of "feature-decision-evaluation", it reconciles the contradiction between the gradual change of teaching semantics and the discrete sampling of key frames, so that the key frame sequence can completely connect the teaching logic chain such as blackboard explanation and teacher-student interaction. Second, this invention constructs a heterogeneous robust feature representation learning module MST-GRM, which simultaneously captures macro-structures such as the layout of the podium and the blackboard area, as well as local teaching dynamics such as blackboard writing and teacher-student interaction through MSBP; distinguishes subtle semantic differences such as the gradual addition of blackboard writing and the gradual expansion of knowledge points through SAGE; and integrates teaching temporal dependence and semantic association through ST-GNN to generate feature representations that conform to the evolutionary laws of teaching behavior. Third, this invention constructs an adaptive keyframe decision module CHACD based on clustering consensus. In response to the uneven distribution and ambiguous semantic boundaries of teaching events such as blackboard explanation and teacher-student interaction, it overcomes the problem of unstable grouping by a single algorithm through heterogeneous clustering integration and consensus matrix, so that keyframes accurately cover key teaching nodes such as blackboard stage transition, knowledge point transition, and interactive links. Fourth, this invention constructs an enhanced keyframe quality index (EKQI) that integrates temporal monotonicity, representativeness score, shot coverage, redundancy, temporal coverage, and visual diversity to ensure complete coverage of the blackboard writing process, correct teaching sequence logic, and balanced representation of each teaching segment, thus comprehensively reflecting the overall quality of keyframes in terms of teaching semantic coverage and process restoration. Fifth, this invention achieves overall collaborative optimization through the high-dimensional hyperparameter optimization algorithm HCOA, enabling the model to adapt to different classroom scenarios such as blackboard explanation, interactive discussion, and experimental demonstration, thus solving the problem of insufficient generalization ability of traditional methods across different teaching styles. Sixth, experiments show that the overall quality index of this method is 17.0% higher than that of the second-best method on TVSum20. On the self-built teaching dataset, it effectively preserves key semantic information such as blackboard writing process and knowledge explanation with a compression ratio of 0.36%, which verifies the advantages and lightweight characteristics of this method in capturing the semantics of gradual teaching. Seventh, the method of the present invention can achieve high-quality keyframe extraction without relying on a large amount of manually labeled data, and has good generalization ability and practical application value. It is applicable to multiple fields such as classroom teaching video analysis, intelligent teaching evaluation and multimedia content understanding. This invention is reasonably designed and can effectively improve the stability, representativeness and diversity of key frame extraction from classroom teaching videos, providing reliable data support for teaching behavior analysis and classroom quality assessment. It has good application prospects and promotion value. Attached Figure Description
[0004] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 This diagram illustrates the overall structure of the HCFS collaborative unsupervised keyframe extraction framework proposed in this invention. Figure 2 This diagram illustrates the structure of the MST-GRM heterogeneous robust feature representation learning module. Figure 3 A schematic diagram illustrating the structure of the CHACD adaptive keyframe decision module based on clustering consensus; Figure 4 This diagram illustrates the structure of the EKQI enhanced keyframe quality assessment model. Figure 5 This diagram illustrates the performance comparison between the keyframe extraction method of this invention and existing methods. Figure 6 This is a visual diagram illustrating the keyframe extraction results of the present invention. Figure 7 This diagram illustrates the generalization performance verification results of the method of the present invention on different datasets. Detailed Implementation
[0005] In order to better understand the above-mentioned objectives, features and advantages of the present invention, the solution of the present invention will be further described below; The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings; This invention centers on HCFS, a collaborative unsupervised keyframe extraction framework for classroom teaching videos, integrating the MST-GRM heterogeneous robust feature representation module, the CHACD hierarchical clustering consensus decision-making module, and the EKQI multidimensional keyframe quality assessment module. Classroom teaching videos are characterized by highly fixed scenes, subtle visual changes, and continuous, gradual evolution of teaching semantics. Teaching behaviors such as line-by-line writing on the blackboard and the gradual unfolding of knowledge points exhibit continuous, gradual changes, resulting in minimal visual differences between adjacent frames while significant shifts in teaching semantics may occur. This invention aims to resolve the structural contradiction between the smooth evolution of teaching semantics and discrete keyframe sampling. It achieves multi-scale teaching semantic enhancement and spatiotemporal dependency modeling through MST-GRM, improves decision-making stability in scenarios with gradually changing teaching semantics through CHACD, and provides multi-dimensional quality assessment and optimization guidance for keyframes through EKQI. An adaptive keyframe extraction method for efficient and stable analysis of classroom teaching videos, such as Figure 1 , Figure 2 and Figure 3 As shown, the HCFS keyframe extraction framework is constructed, including the MST-GRM heterogeneous robust feature representation learning module, used to capture the macro-layout of the classroom and the dynamics of local teaching, enhance the differences in the gradual changes of teaching semantic features, and model the temporal dependence and cross-temporal semantic association of teaching; the CHACD clustering consensus-based adaptive keyframe decision module, used to stably divide the continuous teaching semantic flow into semantic groups corresponding to teaching events such as blackboard explanation, slide turning, and interactive Q&A; and the EKQI enhanced keyframe quality assessment module, used to ensure the correctness of the teaching temporal logic, ensure the complete coverage of blackboard and slide content, and balance the representation of each teaching link. The HCOA high-dimensional hyperparameter optimization algorithm is used to achieve collaborative optimization of the parameters of each module, with EKQI as the global objective function to guide the optimization direction, so that the model can adapt to the teaching semantic evolution rhythm of different classroom scenarios, and achieve a comprehensive balance between the integrity of teaching semantics, consistency of process temporality, and low information redundancy of keyframes. MST-GRM includes the MSBP multi-scale dual-path feature extraction submodule, the SAGE semantic perception generative enhancement submodule, and the ST-GNN spatiotemporal graph neural network submodule; CHACD includes the multi-view clustering integration module, the consensus matrix construction module, the hierarchical semantic grouping module, and the representative keyframe selection module; EKQI includes the temporal monotonicity evaluation module, the semantic coverage evaluation module, the visual diversity evaluation module, the redundancy suppression evaluation module, the temporal coverage evaluation module, and the shot coverage evaluation module. First, construct the HCFS collaborative unsupervised keyframe extraction framework; Given a length of Classroom teaching video sequence ,in, Indicates the first Frame images. The objective of this invention is to select a set of keyframes from them under unsupervised conditions. ,in Represents the time index of the selected keyframe, i.e. This ensures that the keyframe sequence fully covers the process of writing on the blackboard and the evolution of knowledge delivery; like Figure 1 As shown, the HCFS framework includes the MST-GRM heterogeneous robust feature representation learning module, the CHACD clustering consensus-based adaptive keyframe decision module, and the EKQI enhanced keyframe quality assessment module, which achieves collaborative optimization through closed-loop feedback. First, the video sequence The MST-GRM module inputs frame by frame, capturing macroscopic structures such as the podium layout and blackboard area, as well as local teaching dynamics such as changes in handwriting, through multi-scale dual-path feature extraction. Semantic perception enhancement widens the feature distance between frames with different levels of blackboard completion, and spatiotemporal graph modeling ensures the evolution sequence of teaching behaviors and captures cross-temporal semantic relationships, resulting in a frame-level feature set that integrates the spatiotemporal context of teaching. ; Then, Inputting into the CHACD module, it constructs inter-frame consensus relationships through multi-view clustering ensemble, overcoming the problem of single algorithms incorrectly segmenting the same blackboard writing process or incorrectly merging different teaching segments. It generates semantic groups corresponding to blackboard writing stage transitions, knowledge point transitions, and interactive segments. Within each group, it selects the frame that best represents the core content of the teaching event, forming a candidate keyframe set. ; Furthermore, Inputting the EKQI module ensures the correctness of the teaching sequence logic through time monotonicity, guarantees complete coverage of blackboard writing and knowledge explanation through representativeness scores, and ensures balanced representation of teaching links such as explanation, blackboard writing, and interaction through camera coverage. After comprehensive evaluation, the final set of keyframes is obtained. ; EKQI evaluation results are fed back to MST-GRM and CHACD as global optimization signals, forming a closed-loop optimization of "feature learning - decision generation - quality assessment", so that the key frame sequence can be completely connected to the logical chain of classroom teaching. Second, a heterogeneous robust feature representation learning module (MST-GRM) is constructed. To model the continuous evolution of teaching semantics and address the insufficient discriminative power of features in classroom teaching videos due to the minimal visual differences between adjacent frames caused by fixed scenes and the continuous gradual changes in teaching semantics such as blackboard writing and knowledge delivery, this invention proposes the MST-GRM module. Through the joint design of MSBP, SAGE, and ST-GNN, the original video frames are mapped to feature representations with higher discriminative power and stronger temporal consistency. For example... Figure 2 As shown, MST-GRM consists of three sequentially connected sub-modules: Multi-Scale Dual-Path Feature Extractor (MSBP), Semantic Aware Generative Augmentation Module (SAGE), and Spatiotemporal Graph Neural Network (ST-GNN). (1) Multi-scale dual-path feature extractor (MSBP) To address the challenge of single-scale features simultaneously capturing both the overall classroom structure (e.g., podium layout, blackboard area) and local teaching details (e.g., handwriting and gestures), this invention designs a heterogeneous dual-branch architecture. The global branch extracts frame-level overall semantics through a two-layer fully connected network, capturing macroscopic structures such as podium layout, blackboard area, and projection screen position. The local branch divides the input frame into a 2×2 spatial grid, extracting local texture and edge features for the blackboard writing area, teacher's standing area, and student interaction area. After mapping by a fusion network, fine-grained visual details such as handwriting variations and teacher gestures are preserved. Let the output of the global branch be... The local branch output is The result is obtained by residual projection and stitching fusion. The frame fusion features are: ; in, The residual projection weight matrix ( ); This indicates a feature concatenation operation. and These are the output layer weights and bias terms, respectively, and this output serves as the input to the subsequent SAGE module; (2) Semantic Aware Generative Enhancement Module (SAGE) In classroom teaching videos, the podium, blackboard, and other scenes are highly fixed, and the visual content of adjacent frames is extremely similar. Even if the content on the blackboard has changed significantly or the knowledge point has progressed to the next stage, frame-level features may still be highly concentrated in the embedding space, making it difficult for the model to distinguish subtle changes in teaching semantics. The SAGE module introduces random feature shifts constrained by teaching semantics into the feature space, effectively widening the feature distance between frames with different levels of blackboard completion and different explanation stages, enabling the model to distinguish subtle differences such as "halfway through writing on the blackboard" versus "complete writing on the blackboard," and "knowledge introduction" versus "knowledge summary." Let the MSBP output be... Introducing low-dimensional noise After concatenation, input into the generator network Predicted offsets are enhanced in the form of residuals: ; in This indicates a splicing operation, which will... and Concatenate them as generator input; It is a lightweight two-layer fully connected parameterized generator network (Offset Prediction Network, or OPN for short) used to predict the feature offset direction during the semantic gradation process in instruction; The residual scaling factor controls the magnitude of feature enhancement to avoid disrupting the visual coherence of the teaching scene; This refers to the final enhanced frame-level features; Through the above methods, the SAGE module effectively widens the feature distance between different state frames in teaching semantics gradual change scenarios such as the writing process on the blackboard, the page turning process of courseware, and the change of the explanation posture. This enables the model to distinguish subtle differences in teaching semantics and ensures the robustness of subsequent graph modeling and clustering decisions in teaching semantics gradual change scenarios. (3) Spatiotemporal Graph Neural Network (ST-GNN) To capture the spatiotemporal dependencies between frames in classroom teaching videos, particularly the correlation of writing states at different points in the blackboard writing process, the correspondence between screen content before and after page turns in the courseware, and the semantic connections between different stages of knowledge delivery, this invention constructs a dynamic multi-relationship graph model. Nodes represent enhanced frame features. Define the adjacency matrix ,in This represents the total number of video frames. Edge relationships fall into two categories: one is temporal adjacency edges (…). One type ensures that the natural evolution of teaching behaviors such as blackboard writing, slide flipping, explanation process, and experimental steps are not disrupted; the other type is teaching semantic similarity edge, which is constructed based on feature cosine similarity and used to capture teaching semantic associations across time periods—such as the correspondence between questions raised during the introduction of the class and answers during the summary of the class, the correspondence between the explanation of experimental principles and the operation demonstration, and the analogical association between knowledge points, etc. Specifically, firstly, regarding the features conduct Normalization is used to calculate the cosine similarity between any two frames. ,in Represent the cosine similarity between any two frames and select the frame with the highest similarity for each node. The connection of non-self nodes allows for the explicit modeling of cross-time-segment teaching logic relationships, such as the start and end frames of blackboard writing, the before and after page turns in courseware, and classroom questions and after-class answers. This is achieved by adding self-connections to the adjacency matrix. After symmetric normalization, the node feature propagation is as follows: ; in, It is a diagonal matrix that satisfies It is used to balance the information dissemination between long-term gradual changes in blackboard writing and short-term teaching events such as teacher-student Q&A; Denotes the first symmetric normalized adjacency matrix. , One element; The weight matrix is learnable and adaptively adjusts the fusion ratio of temporal dependency and semantic association. For bias terms, This represents a non-linear activation function, and the initial node features are set to... , Indicates the first Layer nodes The hidden feature representation is obtained. Through graph convolution, temporal information such as the writing process on the blackboard, the evolution of slide page turning, and changes in the speaker's posture flows along the temporal edge, while semantic associations such as the correspondence between classroom introduction and summary, and the correspondence between experimental principles and operations flow along the semantic edge. Finally, the enhanced frame representation that integrates the teaching spatiotemporal context is output. This design enables the graph structure to not only reflect temporal adjacency relationships but also capture the semantic association network of teaching content, providing frame-level features that conform to the teaching logic for subsequent keyframe decisions; Furthermore, to verify the effectiveness of the feature extraction strategy, a comparative experiment was conducted on a classroom teaching video dataset, and the results are shown in Table 1. Table 1 Comparison of keyframe feature extraction methods MSE↑ PSNR↓ SSIM↓ Entropy↑ MI↓ Fixed interval sampling 103.758 29.046 0.945 7.401 3.147 Random sampling 80.964 31.247 0.957 7.401 3.304 Rare optical flow method 183.980 26.254 0.921 7.402 2.932 Dense Optical Flow Method 139.002 27.613 0.928 7.402 4.035 MSBP 93.307 29.550 0.947 7.401 2.984 MSBP+CNN 98.497 29.164 0.945 7.401 2.447 MSBP+CNN+FOX 103.372 28.812 0.938 7.389 2.360 Sparse optical flow + BP 179.186 26.468 0.923 7.402 2.952 Sparse optical flow + BP + FOX 180.034 26.436 0.923 7.402 2.949 CNN + Color Histogram + FOX 138.560 27.397 0.931 7.402 3.015 Sparse optical flow + MSBP (Ours) 205.394 25.987 0.914 7.403 2.896 As shown in Table 1, the method of the present invention achieves optimal results in both information entropy and mutual information indices, indicating that it can effectively suppress redundant features while preserving key teaching semantics. This advantage is particularly crucial in classroom teaching videos: fixed sampling is prone to generating a large number of visually similar frames in long-term gradual change scenarios such as blackboard writing and page-by-page presentation of courseware, while the present invention can effectively distinguish subtle differences in teaching semantics between different levels of blackboard writing completion and different pages of courseware; To verify the effectiveness of each submodule, ablation experiments were conducted, and the results are shown in Table 2. Table 2 Ablation Experiment Results of MST-GRM Module serial number MSBP SAGE ST-GNN F1 score EKQI Compression ratio (%) Model parameter count (K) 1 √ 0.082 0.399 0.14 21.60 2 √ 0.086 0.419 0.15 5.17 3 √ 0.072 0.413 0.14 13.35 4 √ √ 0.058 0.410 0.08 25.33 5 √ √ 0.078 0.406 0.15 33.51 6 √ √ 0.089 0.425 0.14 17.06 7 √ √ √ 0.240 0.435 0.36 37.22 As shown in Table 2, the performance of a single module is significantly insufficient; when the three modules work together, the EKQI reaches 0.435, the F1-score reaches 0.240, and the compression ratio reaches 0.36%, which verifies the joint gain effect: MSBP simultaneously perceives the macro structure such as the podium layout and projection screen, as well as the local teaching dynamics such as blackboard writing and gestures; SAGE separates the feature distances of different state frames during the gradual transition process such as blackboard writing and courseware page turning; ST-GNN captures the cross-temporal teaching semantic associations such as the correspondence between classroom introduction and summary, and the correspondence between principles and operations. The three work together to completely depict the macro layout and micro evolution of the teaching process. Third, construct an adaptive keyframe decision module (CHACD) based on clustering consensus. For example... Figure 3As shown, after obtaining the enhanced features provided by the MST-GRM module, the main challenge in keyframe decision-making is how to obtain a stable and representative set of discrete keyframes from continuous teaching semantic feature representations under unsupervised settings. Teaching events in classroom videos, such as blackboard explanations, slide flipping, teacher-student interactions, and experimental demonstrations, are unevenly distributed and their semantic changes are slow. Traditional single clustering methods easily lead to keyframe redundancy by incorrectly segmenting the same teaching segment into multiple fragments, or incorrectly merging different teaching segments, resulting in the omission of key teaching nodes. To address the aforementioned problems, this invention proposes an adaptive keyframe decision-making mechanism (CHACD) based on hierarchical consensus clustering. By integrating multi-perspective and multi-granularity clustering information, it constructs a consensus-based similarity metric, thereby improving the robustness of discretized decisions. Given a set of... The enhanced feature set of the classroom teaching video sequence of frames is as follows: CHACD achieves stable keyframe decisions through the following three stages: (1) Multi-perspective clustering and consensus relationship modeling This method designs a heterogeneous clustering ensemble strategy, which integrates multiple significantly different base clustering results to construct stable consensus semantic relationships that are insensitive to initialization, parameters, and noise. The ensemble system is based on the following three-layer complementarity principle: Algorithm complementarity: Combining partitioning clustering and hierarchical clustering, the former clusters frames within the same teaching segment (such as a blackboard explanation, an example demonstration, or a slide presentation) into one class, while the latter captures the hierarchical structure of the teaching process, such as dividing teaching segments like "review and introduction - new knowledge instruction - example explanation - class summary" into layers according to granularity, so that the clustering results conform to the actual organizational structure of classroom teaching; Parameter perturbation: Dynamically sampling key parameters to introduce diversity. For example, the number of clusters. In the interval Internal sampling ( (Target keyframe count) to accommodate the natural differences in the number of teaching events in different classroom scenarios—less frequent but longer-lasting blackboard lectures, and more frequent but shorter-lasting interactive discussion sessions; dimensionality reduction. In the set Random selection is made to capture the teaching semantic structure at different granularities; Feature subspace complementarity: Dimensionality reduction through Principal Component Analysis (PCA) and random feature subset sampling (with a fixed subset size) Clustering is performed in different projection subspaces. Different subspaces correspond to different teaching semantic emphases; some subspaces focus more on changes in the content of the blackboard area and courseware screen, while others focus more on the dynamic features of teacher posture and teacher-student interaction. Through multi-subspace integration, the multi-dimensional semantic relationships of classroom teaching are comprehensively captured. Each algorithm configuration is initialized using a different random seed to ensure independence. Finally, by combining different algorithm types, parameter configurations, and feature views, clustering is derived from six basic configurations. The heterogeneous clustering results are used to form a cluster set. This enables the subsequent consensus matrix to effectively smooth out the deviations caused by a single algorithm incorrectly segmenting the same teaching segment or merging errors from different teaching segments; (2) Hierarchical stable grouping construction Based on clustering sets First, construct the inter-frame consensus distance matrix. For frame pairs... Its consensus distance is defined as: ; in This is an indicator function that takes the value 1 when the condition inside the parentheses is true and 0 when it is false. For the total number of clustering rounds, For clustering round index, , The first Frames in sub-clustering ,frame Cluster labels; for The corresponding consensus distance. This distance matrix, by fusing information from multiple perspectives, effectively smooths out the impact of a single algorithm. The study identifies errors in segmenting the same blackboard or courseware presentation and incorrect merging of different teaching segments. Subsequently, Ward hierarchical clustering is used to aggregate the consensus distance matrix, generating hierarchical semantic groupings corresponding to teaching events such as blackboard stage transitions, courseware page switching, knowledge point transitions, and interactive Q&A. ; (3) Selection of representative keyframes In each stable semantic group Internally, define the first Frame representativeness score It is a weighted combination of eigencenter and temporal centrality: ; in, Indicates the characteristics of group center; The Euclidean distance between the eigenvectors and the group centers; To balance the weights of feature similarity and temporal smoothness, and temporal centrality score for: ; in, , The first in each group Frame, First The frame sequence number and timestamp correspond to the frame. Temporal centrality ensures that the selected frame is at the temporal center of the teaching event; the most representative frames in each group are selected to form a candidate keyframe set, so that the keyframe decision matches the actual event distribution in classroom teaching; Fourth, construct an enhanced keyframe quality assessment module (EKQI). For example... Figure 4 As shown, to address the problems of existing keyframe evaluation methods relying excessively on manual annotation and failing to reflect the specific needs of classroom teaching videos in terms of complete reproduction of the teaching process and the correctness of teaching logic, this invention constructs an enhanced keyframe quality assessment module (EKQI) for unified quantitative evaluation of candidate keyframe sets, serving as a global objective for subsequent collaborative optimization. EKQI is defined as: ; in, As a score for time monotonicity, and as a multiplicative correction factor to ensure the correctness of the teaching sequence logic, key frames must be arranged in the actual order in which the teaching occurs; otherwise, it will disrupt the logical sequence of the blackboard writing and the order of knowledge presentation. To ensure accuracy compared to manual annotation, this is only counted when annotation data is available; otherwise, it is set to 0. To ensure representative scoring, the process of writing on the blackboard from blank to complete, the presentation of courseware from the first page to the last page, and the explanation of knowledge points from introduction to in-depth are fully covered; To ensure adequate camera coverage, each teaching segment, including explanations, blackboard writing, interactions, and multimedia presentations, has a corresponding keyframe. To reduce redundancy, repeated sampling of the same blackboard writing state, the same courseware page, or the same lecturing posture is suppressed; To ensure time coverage, keyframes are evenly distributed across the entire lesson's timeline; To ensure visual diversity, keyframes should vary sufficiently in terms of blackboard completion, presentation slides, and teacher posture. , , Let be the weighting coefficient, satisfying ; In actual assessment, Through tolerance parameter Match the predicted keyframe with the manually labeled data. When the predicted keyframe matches the manually labeled data... When a match is found, it is considered successful. This item follows the standard precision definition, where the true match is... This represents the number of manually labeled predicted keyframes that were successfully matched within the tolerance range; false positives. This indicates the number of predicted keyframes that failed to match successfully. The definitions of each sub-metric are as follows: Representative score Defined as: ; in, This represents the average distance from all original frames to their nearest keyframe. This represents the total number of frames in the video. This is a normalization factor. This indicator ensures that keyframes fully cover the evolution of the teaching content; Lens coverage Defined as: ; in, The first in the video A semantic lens, It is a collection of all semantic shots in the video. and These are the start and end timestamps for the corresponding shots. This refers to the frame number of the keyframe. This metric is used to assess whether the keyframe covers the main semantic segments of the video, ensuring that each teaching segment is representative. Redundancy Defined as: ; in, The cosine similarity function is used. For similarity threshold, This refers to the number of keyframes. This metric effectively suppresses repeated sampling of the same teaching state, ensuring sufficient variation in the keyframe set across aspects such as blackboard completion and explanation stages. Time coverage Defined as: ; in, For the ideal time interval, Total video duration; This represents the standard deviation of the actual time interval. It ensures a uniform temporal distribution of keyframes. Visual diversity Defined as: ; in, This represents the cosine similarity between keyframe pairs, ensuring that keyframes have sufficient differences in terms of blackboard completion, courseware visuals, and teacher posture. In this invention, the weighting coefficient of EKQI is set as follows: This configuration retains the necessary manual annotation and verification capabilities while placing greater emphasis on the semantic fidelity of keyframe extraction and using spatiotemporal distribution as an auxiliary dimension for constraint. To verify the rationality of the weight settings of each evaluation index in EKQI, experiments were conducted on the TVSum20 dataset and the Taiyuan Multi-View TeacherInstruction Dataset (TY-MVTID) under different weight combinations. The results are shown in Tables 3 and 4. Table 3. Weight Sensitivity Experiment on TVSum20 Dataset algorithm Weights (0.3, 0.5, 0.2) Weights (0.5, 0.3, 0.2) Weights (0.2, 0.6, 0.2) Weights (1.0, 0.0, 0.0) AVS 0.325 0.326 0.325 0.192 DR-DSN 0.026 0.026 0.026 0.052 DSNet 0.399 0.404 0.391 0.217 D-KTS 0.149 0.149 0.149 0.282 LMSKE 0.091 0.120 0.071 0.252 AKS 0.182 0.182 0.182 0.132 HCFS (Ours) 0.466 0.426 0.404 0.340 Table 4. Weight Sensitivity Experiments on the TY-MVTID Dataset algorithm Weights (0.3, 0.5, 0.2) Weights (0.5, 0.3, 0.2) Weights (0.2, 0.6, 0.2) Weights (1.0, 0.0, 0.0) AVS 0.312 0.327 0.297 0.234 DR-DSN 0.127 0.127 0.127 0.159 DSNet 0.411 0.419 0.404 0.222 D-KTS 0.367 0.375 0.360 0.139 LMSKE 0.404 0.411 0.394 0.195 AKS 0.182 0.191 0.173 0.178 HCFS (Ours) 0.435 0.420 0.415 0.351 As shown in Tables 3 and 4, different weight configurations have a significant impact on the EKQI index. Among them, the weight configuration (0.3, 0.5, 0.2) achieved the highest EKQI scores on both datasets (0.466 for TVSum20 and 0.435 for TY-MVTID), indicating that this configuration has good discriminative ability in keyframe quality assessment. Meanwhile, this weight configuration exhibits stable performance across different data scenarios, effectively balancing semantic representativeness and temporal distribution characteristics, thereby achieving efficient evaluation of the quality of the keyframe set. Therefore, this invention adopts this weight combination as the final weight configuration for EKQI; Based on the completed weighted sensitivity analysis, several typical methods were selected for comparative experiments, and the results are as follows: Figure 5 As shown, the method of this invention outperforms existing methods in terms of EKQI and F1-score. In particular, in classroom teaching video scenarios, existing methods, due to their lack of specific design for the gradual changes in teaching semantics, suffer from significant deficiencies in blackboard writing coverage, courseware display completeness, and the balance of teaching segments. In contrast, this invention, through multi-scale teaching semantic modeling and clustering consensus decision-making, can fully capture the gradual teaching process, including blackboard writing, courseware page turning, and knowledge point evolution, thus verifying the effectiveness and advancement of this invention. Through the joint modeling of the above multi-dimensional indicators, EKQI can comprehensively characterize the overall quality of keyframes in terms of instructional semantic coverage, process reconstruction and structural characterization without the need for manual annotation. Fifth, a collaborative optimization mechanism is constructed. To avoid the local optima problem caused by the independence between feature representation, keyframe decision-making, and quality assessment, this invention performs unified modeling and joint optimization of the MST-GRM, CHACD, and EKQI modules. Specifically, this invention abstracts the key hyperparameters in each module in a unified manner and constructs a system-level configuration vector. in, , The hidden layer dimension and noise dimension of the MST-GRM module determine the expressive power of the teaching semantic features; , The scaling factor and Dropout rate for feature enhancement control the discriminative power of the gradation features in the whiteboard. The target number of clusters for the CHACD module. , As a weight that balances representativeness and temporal centrality, To maximize the number of clustering rounds, the granularity and stability of grouping teaching events are jointly determined; The inter-frame similarity threshold for the EKQI metric. The time tolerance for labeling and matching affects the strictness of blackboard coverage integrity and redundancy suppression. The EKQI comprehensive score is used as the fitness function. Its expression is: ; in, Indicates configuration with given parameters The optimal configuration is then globally searched within the parameter space using HCOA. Its formal expression is: ; The model adapts to the evolving pace and event distribution characteristics of teaching semantics in different classroom scenarios, such as blackboard explanation, interactive discussion, and experimental demonstration. During the optimization process, each parameter update executes a complete feature extraction, keyframe candidate generation, and quality assessment process, forming a closed-loop optimization of "feature representation - keyframe decision - quality assessment - parameter update". This makes feature learning more conducive to grouping teaching events and keyframe selection more in line with the overall quality goals, thereby improving the stability and overall performance of keyframe extraction. To further demonstrate the practical effects of this invention, the keyframe extraction results are visualized and analyzed, such as... Figure 6As shown, the keyframes extracted by the method of this invention completely cover the transition moments of typical teaching behaviors such as "lecturing—writing on the blackboard—viewing PPT—interactive Q&A," and the temporal logic is completely consistent with the actual evolution sequence of the teaching process; while the comparative methods have problems such as keyframe redundancy, uneven time distribution, or omission of key teaching nodes. Combined with the EKQI index, the visualization results are consistent with the quantitative evaluation, verifying the advantages of this method in capturing gradual teaching semantics and ensuring the integrity of teaching logic; To verify the generalization ability, experiments were conducted on the TVSum20 and SumMe datasets, and the results are shown in Table 5. Figure 7 As shown; Table 5 Generalization experiments on the TVSum20 dataset method F1 score EKQI Compression ratio (%) Model parameter count AVS 0.216 0.325 9.61 1.64M DR-DSN 0.088 0.026 15.05 1.58M DSNet 0.085 0.399 0.09 4.33M D-KTS 0.189 0.149 0.20 5.60M LMSKE 0.204 0.091 0.28 10.63K AKS 0.163 0.182 0.12 151.28M HCFS (Ours) 0.238 0.466 0.36 37.22K On TVSum20, HCFS achieved an EKQI score of 0.466 and an F1-score of 0.238, representing improvements of 17.0% and 180.7% respectively compared to the second-best method, DSNet, with a compression ratio of only 0.36%. Particularly noteworthy is its ability to accurately capture key knowledge delivery points and natural transitions between teaching segments within the instructional videos included in TVSum20, while the comparative methods show significant shortcomings in terms of coverage of blackboard writing and balance of teaching segments. On the SumMe dataset, the results are as follows: Figure 7 As shown, HCFS performs excellently in both representativeness score (0.998) and lens coverage (1.000), with a comprehensive score of 0.517, which is higher than the listed comparison methods. This result demonstrates that even in scenarios lacking a unified supervisory signal, HCFS can still output a set of keyframes with high representativeness of teaching semantics and balanced coverage of teaching segments, validating the adaptability of this method to different classroom scenarios. Experiments show that HCFS maintains stable and competitive performance on both the TVSum20 and SumMe datasets. Especially in the instructional videos included in TVSum20, this method accurately captures key points in knowledge delivery, the blackboard writing process, and the natural transitions between teaching segments. In contrast, the comparative methods show significant shortcomings in blackboard writing coverage and the balance of teaching segments, validating the adaptability of this method to different teaching styles and classroom types. This invention was carried out in an environment with an Intel(R) Core(TM) i5-10400F @ 2.90GHz CPU, NVIDIA GeForceRTX 3060, 8GB of memory, Windows 10 operating system, and PyTorch 2.7.1 (based on Python 3.10.14); This invention is applicable to keyframe extraction tasks in various types of classroom teaching videos, especially those with clear temporal structures and semantic changes in the teaching process, such as blackboard explanation, interactive discussion, and experimental demonstration videos. In blackboard explanation classes, this method can completely capture the writing process from blank to completed blackboard writing; in interactive discussion classes, this method can evenly cover different teaching stages such as teacher explanation, student questioning, and teacher-student interaction; in experimental demonstration classes, this method can accurately record key operation nodes of experimental steps. The application of this invention helps to improve the automation and intelligence level of the keyframe extraction process, reduce the uncertainty caused by manual annotation and reliance on experience, and improve the efficiency of understanding and compressing classroom teaching video content, showing good application prospects. The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the present invention. Although detailed descriptions have been provided with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments, and they should all be covered within the protection scope of the claims.
Claims
1. An adaptive keyframe extraction method for efficient and stable analysis of classroom teaching videos, characterized in that: A Hybrid Collaborative Framework for Stable Keyframe Extraction (HCFS) is constructed to extract keyframes from classroom teaching videos that exhibit stable semantic evolution and subtle visual differences between adjacent frames. The framework includes a multi-scale semantic-aware Spatio-Temporal Graph Representation Module (MST-GRM) heterogeneous robust feature representation learning module, a clustering-based hierarchical adaptive consensus decision module (CHACD), and an enhanced keyframe quality index (EKQI). The MST-GRM employs a multi-scale semantic and spatiotemporal relationship modeling mechanism to learn features from the continuously evolving teaching semantics of the classroom teaching videos. A clustering consensus decision-making mechanism generates candidate keyframes, and a multi-dimensional quality assessment mechanism comprehensively quantifies the teaching semantic coverage and logical consistency of these candidate keyframes. A closed-loop feedback driven by the assessment achieves synergistic optimization of feature learning and decision-making. The MST-GRM includes a multi-scale dual-pathway feature extraction module (MSBP) to capture local teaching dynamics such as the macro-level classroom layout and blackboard writing; a semantic-aware generative enhancement module (SAGE) to enhance subtle feature differences between adjacent frames caused by gradual changes in teaching semantics; and a spatio-temporal graph neural network module. NeuralNetworks (ST-GNN) is used to model the temporal dependencies and semantic associations between frames in classroom teaching. The CHACD includes a multi-view clustering construction module, a consensus matrix calculation module, and a hierarchical stable grouping and representative frame selection module, which are used to stably divide the continuous teaching semantic feature stream into semantic groups corresponding to teaching events. The EKQI includes a temporal monotonicity evaluation module to ensure the temporal logic of the teaching process, a semantic representativeness evaluation module to evaluate the coverage and completeness of blackboard writing and knowledge delivery content, a visual diversity evaluation module, a redundancy evaluation module, and a temporal coverage evaluation module.
2. The adaptive keyframe extraction method for efficient and stable analysis of classroom teaching videos according to claim 1, characterized in that: The specific steps for constructing the HCFS collaborative unsupervised keyframe extraction framework are as follows: First, the input classroom teaching video is sampled at the frame level to construct a video frame sequence. The video frame sequence is then input into the MST-GRM module, where the MSBP module extracts local teaching dynamic features such as the macro-layout of the classroom and blackboard writing. These features are then fused through residual projection and splicing to obtain multi-scale fusion features. Next, the sequence is input into the SAGE module, where random perturbations constrained by teaching semantics are introduced into the feature space. The offset is predicted by a generative network and enhanced in the form of residuals to widen the feature distance between frames with different levels of blackboard writing completion. Finally, the sequence is input into the ST-GNN module, which constructs a graph structure with video frames as nodes. Temporal edges ensure the evolution order of teaching behavior, and semantic edges capture cross-temporal teaching semantic relationships. The enhanced frame-level feature representation is obtained through graph convolution fusion. Then, the enhanced frame-level feature representation is input into the CHACD module. Diverse cluster sets are constructed through algorithm complementarity, parameter perturbation, and feature subspace complementarity. The stable co-occurrence relationship between frames is quantified based on the consensus matrix, and the consensus distance is calculated. Semantic groups corresponding to teaching events such as blackboard stage transitions, knowledge point transitions, and interactive sessions are generated through hierarchical clustering. Within each group, a representative score is calculated by weighting feature centrality and temporal centrality. Frames that can represent the core content of the teaching event and are located at the central time position of the event are selected as representative frames to obtain a set of candidate keyframes. Then, the candidate keyframe set is input into the EKQI module. The correctness of the teaching sequence logic is ensured by the time monotonicity, the complete coverage of the blackboard writing process and knowledge explanation is ensured by the semantic representativeness, and the repeated sampling of the same teaching state is suppressed by the visual diversity and redundancy. The EKQI comprehensive quality score is calculated and the final keyframe set is selected and output accordingly. Finally, the key hyperparameters in the MST-GRM, CHACD, and EKQI modules are uniformly modeled as system-level configuration vectors using the High-dimensional Hyperparameter Optimization Algorithm (HCOA). The EKQI comprehensive quality score is used as the fitness function for global optimization, enabling the model to adapt to different classroom scenarios such as blackboard explanation and interactive discussion. The optimal parameter configuration is selected through population evolution iteration, realizing closed-loop collaborative optimization of feature representation, keyframe decision and quality assessment.
3. The adaptive keyframe extraction method for efficient and stable analysis of classroom teaching videos according to claim 2, characterized in that: The MST-GRM heterogeneous robust feature representation learning module is constructed as follows: First, the input classroom teaching video frames are fed into the MSBP module, which is designed with a heterogeneous dual-branch structure. The global branch extracts frame-level overall semantic features to capture the macro-structure of the classroom, such as the podium layout and blackboard area. The local branch divides the input frames into spatial grids and focuses on extracting local teaching dynamic features from the blackboard writing area and the teacher's standing area. Then, the global and local features are fused through feature splicing and residual projection to obtain a multi-scale teaching semantic feature representation. Then, the multi-scale teaching semantic features are fed into the SAGE module. To address the problem that adjacent frames are visually similar but the teaching semantics change continuously when writing on the blackboard line by line, low-dimensional random noise is introduced and input into a lightweight generator network to predict feature shifts. The original features are enhanced in the form of residuals, thus widening the feature distance between frames with different blackboard completion levels. Finally, the enhanced features are fed into the ST-GNN module to construct a graph structure with classroom teaching video frames as nodes. Temporal adjacency edges are defined to ensure the natural evolution of blackboard writing and knowledge explanation, semantic similarity edges are defined to capture teaching semantic associations across time periods, and the graph complexity is controlled through the Top-K mechanism. On the graph structure, the node features are updated through a graph convolutional network to obtain an enhanced feature representation that integrates the teaching spatiotemporal context.
4. The adaptive keyframe extraction method for efficient and stable analysis of classroom teaching videos according to claim 3, characterized in that: The MSBP multi-scale dual-path feature extraction module is constructed as follows: Let the global branch output be... It is used to capture the macro-structure of the classroom, such as the layout of the podium and the blackboard area. The output after the fusion of local branches is Used to extract local teaching dynamics such as the blackboard writing area and the teacher's standing area, then the first... Frame fusion features for: ； in, This is the residual projection weight matrix; Indicates feature concatenation operation; and These are the output layer weights and bias terms, respectively. This represents a nonlinear activation function; through the above fusion, joint modeling of the macro-scene and micro-level teaching dynamics of classroom teaching videos is achieved.
5. The adaptive keyframe extraction method for efficient and stable analysis of classroom teaching videos according to claim 3, characterized in that: The SAGE semantic-aware generative enhancement module is constructed as follows: The output of the MSBP module Frame features are denoted as To address the issue of visual similarity between adjacent frames during line-by-line writing on the blackboard, but with a continuous and gradual change in pedagogical meaning, a low-dimensional random noise variable is introduced. It follows a distribution After splicing, input into the generative network Predicted feature shifts, enhanced in the form of residuals: ； in, It is a two-layer fully connected generator network; This is the residual scaling factor; by using the above method, the feature distance between frames with different levels of blackboard completion is increased, thereby improving the feature discriminability in teaching semantic transition scenarios.
6. The adaptive keyframe extraction method for efficient and stable analysis of classroom teaching videos according to claim 3, characterized in that: The ST-GNN spatiotemporal graph neural network module is constructed as follows: The frame-level features output by the SAGE module are used as nodes to construct a graph structure, denoted as the total number of video frames. Define the adjacency matrix as The edge relationships in the graph structure include two types: (1) Temporal adjacency edge: When the time indices of two frames satisfy Establish connections in a timely manner to ensure the natural progression of blackboard writing and knowledge explanation; (2) Semantic similarity edges: for features conduct After normalization, the cosine similarity between any two frames is calculated. ,in This represents the cosine similarity between any two frames. For each node, the frame with the highest similarity is selected. By establishing connections between non-self nodes, the logical relationships between teaching elements across time periods, such as the start and end frames of blackboard writing, and the definition introduction and summary review frames, are explicitly modeled; and the adjacency matrix with added self-connections is used to... After symmetric normalization, the propagation of node features is as follows: ； in, Let be a degree matrix, satisfying , Denotes the first symmetric normalized adjacency matrix. , One element; For the learnable first Layer weight matrix, For bias terms, Let be the activation function, and let the initial node features be set to . , Indicates the first Layer nodes The hidden feature representation; through the above graph structure, the information of the blackboard writing gradation process flows along the temporal edge, and the cross-time teaching logic connection flows along the semantic edge, thus obtaining the enhanced feature representation that integrates the teaching spatiotemporal context.
7. The adaptive keyframe extraction method for efficient and stable analysis of classroom teaching videos according to claim 2, characterized in that: The CHACD adaptive keyframe decision module based on clustering consensus is constructed as follows: First, the set of frame-level features output by the ST-GNN module is represented as: ； in, Given the total number of video frames, a diverse cluster set is constructed through algorithmic complementarity, parameter perturbation, and feature subspace complementarity. Partitioning clustering captures compact teaching events such as single blackboard writing or example demonstrations, while hierarchical clustering captures large-granular teaching segments such as review introductions and classroom summaries. A consensus matrix is constructed based on cluster sets, and the consensus distance between frame pairs is defined as: ； in, For indicator functions, , The first Frames in sub-clustering ,frame Cluster labels; The corresponding consensus distance effectively smooths out the deviation of a single algorithm in incorrectly segmenting the same blackboard writing process or incorrectly merging different teaching links; Subsequently, based on consensus distance, a second aggregation is performed using Ward hierarchical clustering to generate semantic groups corresponding to blackboard writing stage transitions, knowledge point transitions, and interactive sessions: ; Finally, in each group Within the framework, a representative score is calculated based on a combination of feature centrality and temporal centrality. Frames that represent both the core content of the teaching event and are located at the event's central time position are selected as representative frames. The scoring definition is as follows: ； in, For frames eigenvectors and grouping centers The squared Euclidean distance corresponds to the feature centrality; the temporal centrality score. The calculation formula is: ； in, , The first in each group Frame, First The frame sequence number and timestamp corresponding to the frame, and the time centrality ensures that the selected frame is at the time center of teaching events such as blackboard explanation or interactive Q&A. To balance the weights of feature similarity and temporal smoothness; Finally, the frame with the highest score in each semantic group is selected as the representative frame to form a candidate keyframe set. This ensures that keyframe decisions align with the actual event distribution in classroom teaching.
8. The adaptive keyframe extraction method for efficient and stable analysis of classroom teaching videos according to claim 2, characterized in that: The EKQI-enhanced keyframe quality assessment module is constructed as follows: For candidate keyframe set Construct a keyframe quality evaluation function EKQI that integrates multi-dimensional evaluation metrics, defined as follows: ； in, The temporal monotonicity score is defined as the strictly increasing proportion of adjacent keyframe pairs, serving as a correction factor as a necessary condition to ensure the correct teaching sequence logic of blackboard writing and knowledge delivery. The accuracy rate based on manual annotation comparison is only included when the annotation data is available; To ensure representative scoring, the entire writing process, from blank to complete, must be fully covered on the blackboard. To ensure adequate camera coverage, each teaching segment, including explanation, blackboard writing, and interactive activities, has a corresponding keyframe. To reduce redundancy, repeated sampling of the same blackboard writing state or explanation posture is suppressed; For time coverage; To ensure visual diversity, keyframes should vary sufficiently in terms of the completeness of the blackboard writing and the teacher's posture. , , Let be the weighting coefficient, satisfying .
9. The adaptive keyframe extraction method for efficient and stable analysis of classroom teaching videos according to claim 8, characterized in that: The definitions of each sub-indicator in the EKQI are as follows: Representative score Defined as: ； in, This represents the average distance from all original frames to their nearest keyframe. As a normalization factor, this ensures that the entire writing process, from blank to complete, is fully covered. Lens coverage Defined as: ； In the formula, For the video number A semantic lens, This refers to the collection of all semantic shots in a video; KF is an abbreviation for keyframe. , These are the start and end timestamps of the shot. The frame number of the keyframe; This refers to the total number of video shots, ensuring that each teaching segment, such as explanation, blackboard writing, and interaction, has a corresponding keyframe. Redundancy Defined as: ； in, For cosine similarity, , This is the number of keyframes; this item suppresses repeated sampling of the same blackboard state or lecturing posture. Time coverage Defined as: ； in, For the ideal time interval, Total video duration; The standard deviation of the actual time interval; Visual diversity Defined as: ； This ensures that keyframes show sufficient variation in the completion of the blackboard writing and the teacher's posture.
10. The adaptive keyframe extraction method for efficient and stable analysis of classroom teaching videos according to claim 2, characterized in that: The system-level configuration vector in the collaborative optimization mechanism is defined as follows: ； in For the predefined parameter search space, This is a system-level configuration vector, where each element corresponds to a hyperparameter in the HCFS framework that can be optimized via HCOA. The physical meaning of each parameter in the formula is as follows: , The hidden layer dimension and noise dimension of the MST-GRM module determine the expressive power of the teaching semantic features; , The scaling factor and Dropout rate for feature enhancement control the discriminative power of the gradation features in whiteboard writing. The target number of clusters for the CHACD module. , As a weight that balances representativeness and temporal centrality, To maximize the number of clustering rounds, the granularity and stability of grouping teaching events are jointly determined; The inter-frame similarity threshold for the EKQI metric. The time tolerance for label matching affects the completeness of blackboard coverage and the strictness of redundancy suppression; The EKQI comprehensive quality score is used as the fitness function: ； in, Indicates in configuration The set of keyframes output by the HCFS framework; further, the optimal parameter configuration is obtained by maximizing the fitness function: This enables the model to adapt to the teaching semantic evolution rhythm of different classroom scenarios, such as blackboard explanation and interactive discussion, and achieves closed-loop collaborative optimization. Through the above optimization process, under the constraint of a unified objective function, a closed-loop feedback mechanism is formed between feature representation quality, clustering decision stability, and keyframe evaluation results. Specifically, the feature quality output by the MST-GRM module affects the grouping stability of the CHACD module, the keyframe set generated by the CHACD module determines the evaluation object of EKQI, and the EKQI evaluation results serve as fitness feedback to guide parameter configuration updates, thereby achieving synergistic optimization of features, decisions, and evaluation.