Student classroom behavior intelligent analysis method based on video semantic understanding

Abstract

The invention discloses a student classroom behavior intelligent analysis method based on video semantic understanding, and relates to the technical field of intelligent education, and the method comprises the steps: firstly collecting a classroom double-view video, and carrying out the cleaning, interception and other processing, and constructing a behavior data set; secondly, extracting RGB images frame by frame from the first visual angle video, and inputting a first model to identify behaviors and hand, body and head postures; extracting skeleton information from the second visual angle video by using Openpose, inputting two models in parallel to output actions and subdivided postures, and performing double-model fusion identification; finally, through multi-target tracking cross-frame matching and integration of double-view results, the student behaviors are deduced and corrected, the problems that in the prior art, video semantic understanding is low in precision, and scattered behavior data are difficult to effectively integrate are solved, automatic and precise analysis of the student classroom behaviors is achieved, and real-time data feedback is provided for teachers.

Description

A Smart Analysis Method for Student Classroom Behavior Based on Video Semantic Understanding Technical Field

[0001] This invention relates to the field of intelligent education technology, specifically to an intelligent analysis method for student classroom behavior based on video semantic understanding. Background Technology

[0002] The current wave of educational informatization is surging, and the drawbacks of traditional classroom analysis methods are becoming increasingly apparent. These methods are time-consuming, inefficient, and highly subjective, failing to meet the demands of educational development. As education moves towards intelligence and personalization, leveraging artificial intelligence for classroom observation is becoming an inevitable trend. Video semantic understanding, as a key AI technology, can deeply mine classroom video information and shows great potential in the field of student classroom behavior analysis. Currently, although challenges remain, such as the lack of standardized coding and large-scale datasets, technologies like deep learning offer breakthroughs. In the future, student classroom behavior analysis based on video semantic understanding will be continuously optimized, achieving more accurate and efficient behavior recognition and classification, providing strong support for teaching practice.

[0003] However, classroom scenarios are complex and diverse, and students' behaviors are varied and subject to many interfering factors, making it difficult for video semantic understanding technology to accurately identify and interpret various student behaviors. Furthermore, the student behavior data obtained from videos is massive and complex, and how to effectively integrate the scattered data remains an urgent problem to be solved.

[0004] Therefore, a method for intelligent analysis of student classroom behavior based on video semantic understanding is needed to solve the above problems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention discloses an intelligent analysis method for student classroom behavior based on video semantic understanding. This method enables automated and precise analysis of student classroom behavior and provides teachers with real-time data feedback to optimize teaching strategies.

[0006] The present invention adopts the following technical solution:

[0007] A method for intelligent analysis of student classroom behavior based on video semantic understanding includes the following steps:

[0008] Step 1: Collect dual-view classroom video data in a classroom setting to obtain first-view and second-view classroom video data. Then, perform data cleaning, image cropping, image preprocessing, image annotation, and dataset partitioning on the collected classroom video data to construct a dataset of student classroom behavior.

[0009] Step 2: For the processed first-person classroom video data, extract RGB images frame by frame from the video stream and input the extracted RGB images into the first student behavior-pose recognition model to recognize student behavior as well as student hand poses, body poses and head poses.

[0010] Step 3: For the processed second-view classroom video data, the Openpose human pose estimation tool is used to extract skeletal information from the video stream. The extracted skeletal information is input in parallel into the skeletal information extraction model and the second student behavior-pose recognition model, which output student actions and hand poses, body poses, left and right head poses and up and down head poses, respectively. The outputs of the two models are fused by dual-model decision to realize the recognition of student actions and poses.

[0011] Step 4: Use a multi-target tracking algorithm to perform cross-frame matching on the identified student targets, and combine the identification results from the two different perspectives in Step 2 and Step 3 to make reasonable inferences and corrections on the student's behavior.

[0012] Furthermore, in step one, the dual-view classroom video data is collected collaboratively by a fixed camera and a schedulable camera to achieve simultaneous capture of the classroom panorama and local details. During data cleaning, abnormal video segments with blurriness exceeding a preset blurriness threshold and empty scene segments without student targets in consecutive frames are removed. During image cropping, the time interval between two adjacent frames is greater than a preset cropping threshold. During image preprocessing, the images are rotated, translated, and color-corrected to expand the sample size. During image annotation, two-person cross-validation is used to record the category and location information of the image or the target within the image. During dataset partitioning, stratified sampling is used to divide the image data into a training set and a test set.

[0013] Furthermore, before training on the input training set, the first student behavior-pose recognition model and the second student behavior-pose recognition model are pre-trained on the database to master general human features, and then transfer training is carried out to adapt to the student classroom behavior recognition scenario. After training is completed, the effect is tested on the test set that did not participate in the training process to evaluate the generalization performance of the model.

[0014] Furthermore, the first student behavior-pose recognition model and the second student behavior-pose recognition model recognize student behavior and pose based on a convolutional neural network, which includes convolutional layers, pooling layers and fully connected layers.

[0015] Furthermore, the Openpose human pose estimation tool completes the topological connection of each joint point according to the human physiological structure, constructs the student human skeleton structure, and performs temporal correlation on the skeleton information of continuous video frames, extracts the temporal motion features of joint displacement changes, motion speed and angle deflection, and forms a student skeleton information set that integrates spatial skeleton topological features and temporal motion features.

[0016] Furthermore, the dual-model decision fusion process specifically involves: performing feature fusion and weight allocation on the output results of the skeleton information extraction model and the second student behavior-pose recognition model. The weight allocation is dynamically adjusted based on the confidence scores or historical accuracy of the two models on the validation set, and the final fusion recognition result is generated through weighted summation or voting mechanisms.

[0017] Furthermore, the multi-target tracking algorithm determines whether the target students identified by the subject in different frames belong to the same entity based on Kalman filtering and Hungarian algorithm. First, the motion trajectory of the target is predicted based on Kalman filtering, and then the predicted result is optimized using Hungarian algorithm. Finally, online tracking mode is used for real-time tracking and identification.

[0018] The beneficial effects of this invention are as follows:

[0019] 1. This invention adopts a dual-view classroom video data acquisition mode, combining visual feature recognition of RGB images from the first viewpoint with action feature recognition of skeletal information from the second viewpoint, to achieve dual capture of visual surface features and deep skeletal features. At the same time, through the dual-model decision fusion of the skeletal information extraction model and the second student behavior-posture recognition model, and the comprehensive inference and correction of cross-view recognition results, it effectively avoids the problems of single-view occlusion, viewpoint bias, and inaccurate judgment of subtle behaviors by single feature recognition. It can not only identify the overall classroom behavior of students, but also finely break down the postures and movements of hands, body, and head (left and right / up and down), which greatly improves the accuracy, comprehensiveness and robustness of classroom behavior recognition, making the classroom behavior analysis results more consistent with real classroom scenarios.

[0020] 2. This invention automates the entire process from classroom video data processing, feature extraction, behavior recognition to result inference through an algorithmic model. It replaces the traditional methods of manual observation and statistics in classroom behavior analysis. On the one hand, it significantly saves the manpower costs of teachers and teaching researchers, freeing them from tedious manual data statistics and allowing them to focus their energy on teaching optimization and student guidance. On the other hand, it replaces subjective human judgment with quantitative technical analysis, effectively avoiding the subjectivity, bias, and randomness of manual observation. This makes the analysis results of student classroom behavior more objective and scientific, providing reliable data support for teaching evaluation and decision-making.

[0021] 3. In the data acquisition phase, this invention constructs a behavior dataset based on real classroom scenarios, removes irrelevant video clips, and defines and labels behaviors in accordance with actual classroom conditions. During model training, preprocessing such as image rotation, translation, and color adjustment is used to expand the sample, improving the model's resistance to complex situations such as changes in classroom lighting, student movement, and environmental interference. Simultaneously, the SORT multi-target tracking algorithm is used to achieve cross-frame matching of student targets, adapting to the needs of large and medium-sized classrooms with multiple students, including primary and secondary schools and universities. The entire technical process is designed based on actual classroom applications, and video acquisition can be completed using existing classroom monitoring equipment, demonstrating strong feasibility and scenario adaptability.

[0022] 4. This invention can accurately capture key information such as students' classroom participation, focus, and learning styles, clearly identifying the characteristics and proportions of various student behaviors, such as listening attentively, looking down at books, interacting and answering questions, and being distracted. It provides teachers with a comprehensive and detailed profile of student classroom performance. Based on the analysis results, teachers can develop personalized teaching strategies for students with different learning states and styles, achieving precise teaching research and personalized instruction, and effectively improving the effectiveness of classroom teaching and students' learning outcomes. Attached Figure Description

[0023] Figure 1 is a schematic diagram of the overall process of the present invention;

[0024] Figure 2 is a flowchart illustrating the student classroom behavior recognition process of this invention;

[0025] Figure 3 is a schematic diagram of the training process of the student behavior-pose recognition model in this invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to Figures 1 to 3. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] This invention discloses an intelligent analysis method for student classroom behavior based on video semantic understanding, as shown in Figure 1, which includes the following steps:

[0028] Step 1: Dual-Perspective Classroom Video Data Acquisition and Processing

[0029] We collected dual-view classroom video data in classroom scenarios to obtain first-view and second-view classroom video data. We then performed data cleaning, image cropping, image preprocessing, image annotation, and dataset partitioning on the collected classroom video data to construct a dataset of student classroom behavior.

[0030] The dual-view classroom video data is collected collaboratively by a fixed camera and a schedulable camera to simultaneously capture the overall classroom view and local details. During data cleaning, abnormal video segments with blur levels exceeding a preset blur threshold and empty scene segments without student targets in consecutive frames are removed. During image cropping, the time interval between two adjacent frames is greater than a preset cropping threshold. During image preprocessing, images are rotated, translated, and color-corrected to expand the sample size. During image annotation, two-person cross-validation is used to record the category and location information of images or targets within images. During dataset partitioning, stratified sampling is used to divide the image data into training and test sets.

[0031] Datasets are the most crucial foundation for algorithm design and validation, playing a vital role in the training effectiveness of neural network models. High-quality datasets are essential for building student behavior recognition models, providing strong support for the subsequent construction of a student behavior analysis platform. However, current research in this field, both domestically and internationally, is still in its infancy, lacking publicly available databases of student behavior data. Therefore, the primary task of this project is to construct a student behavior dataset based on real classroom scenarios. The construction steps mainly cover data collection, data cleaning, image extraction, student behavior definition, data annotation, and dataset partitioning.

[0032] Collect dual-perspective classroom video data, including the teacher's perspective (first-person view) and the student's perspective (second-person view). Perform end-to-end processing on the collected classroom video data:

[0033] Data cleaning: Filtering low-quality frames and setting quality thresholds for motion blur, exposure, etc.

[0034] Image capture: Capture keyframes by time interval or event trigger.

[0035] Image preprocessing includes data augmentation operations such as rotation, translation, and color adjustment.

[0036] Image annotation: Multi-dimensional annotation of behavior categories and pose key points.

[0037] Dataset partitioning: Divide the dataset proportionally into training, validation, and test sets.

[0038] The specific process of constructing a dataset includes data collection, cleaning, extraction, and labeling.

[0039] First-person view camera equipment: installed above the blackboard / electronic whiteboard at the front of the classroom, using a 4K high-definition network camera with a frame rate of 25fps and a downward angle of 15°, used to capture students' facial expressions, hand and desktop activities, and upper body posture.

[0040] Second-view camera equipment: installed at the corner of the ceiling on the side and rear of the classroom, using a wide-angle panoramic camera, 25fps frame rate, 45° downward angle, covering the entire student's field of view, used to capture the full skeletal movement and the spatial position of the group.

[0041] Client-side: The teacher's management interface, used to view behavior analysis results in real time.

[0042] Step 2: Perform student behavior and pose recognition on the first-person RGB image.

[0043] For the processed first-person classroom video data, RGB images are extracted frame by frame from the video stream, and the extracted RGB images are input into the first student behavior-pose recognition model to identify student behavior as well as student hand posture, body posture and head posture.

[0044] Step 3: Perform model fusion recognition on the second-view skeleton information.

[0045] For the processed second-view classroom video data, the Openpose human pose estimation tool is used to extract skeletal information from the video stream. The extracted skeletal information is then input in parallel into the skeletal information extraction model and the second student behavior-pose recognition model, which output student actions and hand poses, body poses, left and right head poses, and up and down head poses, respectively. The outputs of the two models are then fused into a dual-model decision to achieve the recognition of student actions and poses.

[0046] The Openpose human pose estimation tool completes the topological connection of each joint point according to the human physiological structure, constructs the student human skeleton structure, and performs temporal correlation on the skeleton information of continuous video frames to extract the temporal motion features of joint displacement changes, motion speed and angle deflection, forming a student skeleton information set that integrates spatial skeleton topological features and temporal motion features.

[0047] Before training on the input training set, the first student behavior-pose recognition model and the second student behavior-pose recognition model are pre-trained on the database to master general human features, and then transfer training is carried out to adapt to the student classroom behavior recognition scenario. After training is completed, the effect is tested on the test set that did not participate in the training process to evaluate the generalization performance of the model.

[0048] The first student behavior-pose recognition model and the second student behavior-pose recognition model are based on convolutional neural networks to recognize students’ behavior and poses. The convolutional neural network includes convolutional layers, pooling layers and fully connected layers.

[0049] The dual-model decision fusion process specifically involves: performing feature fusion and weight allocation on the output results of the skeleton information extraction model and the second student behavior-pose recognition model. The weight allocation is dynamically adjusted based on the confidence scores or historical accuracy of the two models on the validation set, and the final fusion recognition result is generated through weighted summation or voting mechanisms.

[0050] The student classroom behavior recognition process proposed in steps two and three is detailed in Figure 2. This method first collects classroom video data from the camera's viewpoint set for the students, and then uses RGB images to identify student behavior and posture. Next, it acquires classroom video data again, uses Openpose to extract skeletal information, and merges the RGB image-based recognition model with the skeletal information extraction model to achieve the recognition of student actions and postures. Finally, it integrates the information obtained from the two different perspectives to make reasonable inferences and corrections regarding student behavior.

[0051] Convolutional Neural Networks (CNNs) have achieved remarkable results in numerous fields such as image and video recognition, image classification, and object detection, not only making groundbreaking progress but also gaining extremely wide applications. Traditional neural networks typically have a classic three-layer structure: input layer, hidden layer, and output layer. CNNs differ from traditional neural networks, often using convolutional layers instead of hidden layers. In computer vision, CNNs are arguably the most commonly used core algorithm. This algorithm evolved from classic feedforward neural networks, and its most prominent feature is the use of convolutional operations in the intermediate layers. With the help of this convolutional operation, it can extract local features from images without being limited by the positional relationships between objects, thus more effectively improving the ability and quality of feature extraction. Moreover, after sufficient training, CNNs exhibit excellent stability. Typically, CNNs are mainly composed of convolutional layers, pooling layers, and fully connected layers.

[0052] Figure 3 illustrates the training process for constructing the first and second student behavior-pose recognition models. The process begins with acquiring classroom video data, which has two different perspectives. After acquiring the video, image cropping and cleaning are required. When cropping images, the time interval between adjacent frames should not be too short; when cleaning the data, it is essential to ensure that each teacher's perspective image corresponds to a student's perspective image. Image annotation involves recording information such as the category and location of images or objects within images, aiming to enable the model to learn the relationship between data and labels. Image preprocessing typically involves operations such as rotation, translation, and color adjustment to expand the sample size and improve the model's robustness against interference. Subsequently, the image data is divided into training and testing sets. Before inputting the training set into the model for training, pre-training is usually performed. Pre-training on the model using a large database allows the model to fully grasp the basic features of objects. Pre-training is necessary to avoid poor recognition results due to the model's limited ability to learn basic features when the sample size is small. Once the model has been trained, its performance needs to be evaluated on a test set that was not used in the training process. This will assess the model's generalization performance and enable the creation of a recognition model that can handle complex real-world classroom environments and has strong generalization capabilities.

[0053] Step 4: Cross-perspective matching and behavioral reasoning

[0054] A multi-target tracking algorithm is used to perform cross-frame matching on the identified student targets. The identification results from two different perspectives, steps two and three, are combined to make reasonable inferences and corrections on the students' behavior.

[0055] The multi-target tracking algorithm is based on Kalman filtering and Hungarian algorithm to determine whether the target students identified by the subject in different frames belong to the same entity. First, the motion trajectory of the target is predicted based on Kalman filtering. Then, the predicted result is optimized using Hungarian algorithm. Finally, online tracking mode is used for real-time tracking and recognition.

[0056] SORT (Simple Online and Real-time Tracking) is an online real-time algorithm specifically designed for multi-target tracking scenarios. Although its architecture is relatively simple, it quickly attracted significant attention in the industry upon its initial introduction due to its innovative concept and excellent tracking performance. The algorithm primarily comprises two core modules: target detection and data association. The target detection module is responsible for identifying and detecting specific target categories, such as moving objects, people, and vehicles. The data association module uses Kalman filtering and the Hungarian algorithm to determine whether target objects detected in different frames belong to the same entity. The key to the SORT algorithm lies in its fusion of Kalman filtering and Hungarian matching algorithms. First, Kalman filtering is used to predict the target's trajectory, and then the Hungarian algorithm is used to optimize the predicted result. This algorithm employs an online tracking mode, eliminating the need for information from future frames, and is considered to possess top-tier model performance.

[0057] Statistical analysis of the proportion of student classroom behaviors allows teachers to gain a comprehensive understanding of students' learning performance and habits, enabling them to tailor personalized learning support and instructional guidance. By analyzing the percentage of various student behaviors in the classroom, teachers can obtain crucial information about students' learning preferences and states. This information plays a vital role in accurately diagnosing problems in students' learning processes and developing more targeted educational intervention strategies. For students who participate less in classroom Q&A sessions, teachers can design more engaging and attractive question-and-answer sessions to stimulate their participation, and then further adjust and optimize teaching based on student feedback.

[0058] This study analyzes student behavior patterns using real classroom examples to demonstrate the positive impact of this method on teaching. Three students were selected for analysis, and the frequency of each student's classroom behavior was statistically analyzed. Student #2 listened attentively for 55% of the time, spending the rest of the time reading independently or interacting with the teacher by answering questions. This indicates a moderate level of interest in the lesson content; however, when encountering interesting questions, they actively think and answer, suggesting a self-directed learning style. Student #3 listened attentively for the majority of the lesson (91%), indicating a more passive learning style and a preference for teacher explanations. Their less frequent participation in class Q&A reflects lower engagement, suggesting the teacher may need to encourage more active thinking and participation. Student #10 was not actively engaged in class, spending half the time on their phone. The teacher needs to inquire about the reasons after class and guide the student to improve their learning attitude.

[0059] Statistical analysis of the proportion of student classroom behaviors provides an objective way to measure and evaluate teaching effectiveness, allowing teachers to make targeted adjustments and improvements. By recording and analyzing the percentage of time students spend on various classroom behaviors (such as speaking, asking questions, and participating in interactions), teachers can directly grasp the level of student participation. If the statistical results show high student participation and frequent interaction, it often means that the teaching methods are effective and have successfully stimulated students' learning enthusiasm and motivation. Conversely, if the data shows low student participation or that some students are frequently distracted, then teachers need to reflect on their teaching strategies and consider changing their teaching methods, such as using more interactive teaching activities or introducing more attractive teaching materials to improve student participation.

[0060] Furthermore, by analyzing the dynamic changes in the proportion of specific behaviors, teachers can identify potential problems in certain teaching processes. For example, if the percentage of students standing up to answer questions decreases significantly in a lesson, teachers should consider the possible reasons behind students' reluctance to speak, such as whether the difficulty of the lesson content exceeds their comprehension. Teachers can also use this statistical data to conduct personalized teaching, tailoring customized tutoring plans based on each student's learning style and preferences, amplifying their strengths while specifically helping them address their learning weaknesses. Ultimately, by consistently collecting and analyzing data over the long term, teachers can build a feedback loop for teaching strategies, fine-tuning teaching content and methods to ensure that teaching activities are designed to better meet students' needs, improve teaching quality, and promote students' all-round development.

[0061] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, any equivalent modifications or substitutions made by those skilled in the art to the relevant technical features will fall within the scope of protection of the present invention.

Claims

1. A method for intelligent analysis of student classroom behavior based on video semantic understanding, characterized in that, Includes the following steps: Step 1: Collect dual-view classroom video data in a classroom setting, obtaining first-view and second-view classroom video data. Perform data cleaning, image cropping, image preprocessing, image annotation, and dataset partitioning on the collected classroom video data to construct a student classroom behavior dataset. Step 2: For the processed first-view classroom video data, extract RGB images frame-by-frame from the video stream and input the extracted RGB images into a first student behavior-pose recognition model to identify student behavior, hand poses, body poses, and head poses. Step 3: For the processed second-view classroom video data, use the Openpose human pose estimation tool to extract skeletal information from the video stream and input the extracted skeletal information in parallel into a skeletal information extraction model. In the first model and the second student behavior-pose recognition model, student actions, hand poses, body poses, left-right head poses, and up-down head poses are output respectively. The outputs of the two models are fused using a dual-model decision process to achieve recognition of student actions and poses. Step four: A multi-target tracking algorithm is used to perform cross-frame matching on the identified student targets. The recognition results from the two different perspectives in steps two and three are combined to make reasonable inferences and corrections on the student's behavior. The dual-model decision fusion process specifically involves: feature fusion and weight allocation of the output results of the skeleton information extraction model and the second student behavior-pose recognition model. The weight allocation is dynamically adjusted based on the confidence scores or historical accuracy of the two models on the validation set, and the final fusion recognition result is generated through weighted summation or a voting mechanism.

2. The intelligent analysis method for student classroom behavior based on video semantic understanding according to claim 1, characterized in that, In step one, the dual-view classroom video data is collected collaboratively by a fixed camera and a schedulable camera to achieve simultaneous capture of the classroom panorama and local details. During data cleaning, abnormal video segments with blurriness exceeding a preset blurriness threshold and empty scene segments without student targets in consecutive frames are removed. During image cropping, the time interval between two adjacent frames is greater than a preset cropping threshold. During image preprocessing, the images are rotated, translated, and color-corrected to expand the sample size. During image annotation, two-person cross-validation is used to record the category and location information of the image or the target within the image. During dataset partitioning, stratified sampling is used to divide the image data into a training set and a test set.

3. The intelligent analysis method for student classroom behavior based on video semantic understanding according to claim 1, characterized in that, Before training on the input training set, the first student behavior-pose recognition model and the second student behavior-pose recognition model are pre-trained on the database to master general human features, and then transfer training is carried out to adapt to the student classroom behavior recognition scenario. After training is completed, the effect is tested on the test set that did not participate in the training process to evaluate the generalization performance of the model.

4. The intelligent analysis method for student classroom behavior based on video semantic understanding according to claim 1, characterized in that, The first student behavior-pose recognition model and the second student behavior-pose recognition model are based on convolutional neural networks to recognize students’ behavior and poses. The convolutional neural network includes convolutional layers, pooling layers and fully connected layers.

5. The intelligent analysis method for student classroom behavior based on video semantic understanding according to claim 1, characterized in that, The Openpose human pose estimation tool completes the topological connection of each joint point according to the human physiological structure, constructs the student human skeleton structure, and performs temporal correlation on the skeleton information of continuous video frames to extract the temporal motion features of joint displacement changes, motion speed and angle deflection, forming a student skeleton information set that integrates spatial skeleton topological features and temporal motion features.

6. The intelligent analysis method for student classroom behavior based on video semantic understanding according to claim 1, characterized in that, The multi-target tracking algorithm is based on Kalman filtering and Hungarian algorithm to determine whether the target students identified by the subject in different frames belong to the same entity. First, the motion trajectory of the target is predicted based on Kalman filtering. Then, the predicted result is optimized using Hungarian algorithm. Finally, online tracking mode is used for real-time tracking and recognition.

Images