A student classroom behavior recognition method based on YOLOv5 and CNN fusion
By integrating YOLOv5, CNN, and Transformer models, efficient and accurate identification of student classroom behavior is achieved, solving the problems of low accuracy and poor real-time performance in existing technologies, and improving the real-time performance and accuracy of classroom behavior analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ANHUI POLYTECHNIC UNIV MECHANICAL & ELECTRICAL COLLEGE
- Filing Date
- 2026-03-04
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies for recognizing student classroom behavior suffer from low accuracy and poor real-time performance, especially for complex behaviors. Furthermore, existing methods are costly and prone to interfering with teaching.
A student classroom behavior recognition method that integrates YOLOv5 and CNN is adopted. The YOLOv5 model is used for target detection and preliminary behavior classification, and the CNN model is combined to extract high-level features and use Transformer for temporal modeling. The decision fusion module generates the final behavior recognition result.
It improves the accuracy of recognizing complex behaviors, reduces background interference, enhances the model's generalization ability and real-time performance, and reduces costs.
Smart Images
Figure CN122116478A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method for recognizing student classroom behavior based on the fusion of YOLOv5 and CNN. Background Technology
[0002] Traditional teaching evaluation methods primarily rely on manual methods such as recording teacher and student classroom behavior, including teacher classroom notes and questionnaires, or using wearable devices based on sensors for analysis and evaluation. While manual methods can observe specific teaching behaviors, they are inefficient and subjective. Wearable device methods based on sensors are costly and prone to interfering with normal teaching. With the rapid development of smart campus construction, the integration of education and artificial intelligence has become an inevitable trend. Student classroom behavior analysis can reflect students' learning progress in real time, providing valuable insights for evaluating teaching quality and improving teaching methods. It also offers important evidence for optimizing classroom teaching strategies and improving teaching methods.
[0003] Existing behavioral analysis techniques often employ single models, such as single-stage object detectors like YOLO and SSD. While fast, these methods suffer from low accuracy and are prone to false positives and false negatives when dealing with complex behaviors like sleeping or using a mobile phone. Using time-series models like CNN or LSTM to process entire image sequences is computationally intensive and cannot meet real-time requirements, especially given significant background interference. Current technologies present a trade-off between real-time performance and accuracy, particularly for complex and subtle classroom behaviors, lacking effective solutions. Therefore, designing a low-cost, high-accuracy, and real-time-efficient classroom behavior recognition method is of practical significance. This paper proposes a student classroom behavior recognition method based on the fusion of YOLOv5 and CNN. Summary of the Invention
[0004] The purpose of this invention is to solve the problems in the prior art by proposing a student classroom behavior recognition method based on the fusion of YOLOv5 and CNN.
[0005] A student classroom behavior recognition method based on the fusion of YOLOv5 and CNN includes the following steps: S1. Collect video and image data from classroom scenarios to construct a dataset of student classroom behavior; S2. Preprocess the collected dataset and divide the preprocessed dataset into training set and test set; S3. Construct and train a classroom behavior recognition model; S4. Input the real-time classroom video image data into the trained classroom behavior recognition model and output the final behavior recognition result.
[0006] Preferably, in step S2, the preprocessing of the collected dataset includes: Use annotation tools to annotate the student's behavior in each frame and assign basic behavior labels; Extract short video clips containing complex behaviors from the video and tag the entire clip with behavior tags; Perform at least one of the following data augmentation operations on images and video clips: rotation, scaling, brightness adjustment, and noise addition.
[0007] Preferably, in step S3, the classroom behavior recognition model includes a classroom behavior recognition module and a decision fusion module; The classroom behavior recognition module uses the YOLOv5 model to perform target detection and preliminary behavior classification on video image data. For low-confidence behaviors or suspected complex behaviors output by YOLOv5, the image sequence of the student's region of interest is extracted. The high-level features of each frame in the image sequence are extracted using a CNN model to form a feature vector sequence. The feature vector sequence is then input into the Transformer model for temporal modeling, and the final behavior classification result is output. Decision fusion module: Combines the outputs of YOLOv5 and CNN+Transformer models to generate the final behavior recognition result.
[0008] Preferably, the YOLOv5 model is used for: Real-time detection of student positions in each frame of the image; Output the bounding box coordinates and preliminary behavior classification and confidence score for each student.
[0009] Preferably, the CNN model is a pre-trained convolutional neural network used to extract high-level semantic features from a single frame image.
[0010] Preferably, the Transformer model is used for: Perform time-series modeling on the feature vector sequence output by CNN; Capture the dependencies of behavior over time; Output the classification results for complex behaviors.
[0011] Preferably, the decision fusion module fuses the results according to the following rules: If the confidence score of the behavior output by YOLOv5 is higher than the preset threshold, its classification result is directly adopted; if the confidence score is lower than the preset threshold or the behavior is judged as a complex behavior, the output of the CNN+Transformer model is used as the final behavior recognition result.
[0012] Preferably, the ratio of the training set to the test set in the dataset is 7:3.
[0013] Preferably, the classroom behavior recognition model undergoes no less than 1000 training iterations.
[0014] Compared with existing technologies, the advantages of this invention are: 1. This invention integrates YOLOv5 and CNN. YOLOv5 effectively handles multiple targets, small targets, and occlusion situations, ensuring that no target is missed in the detection. CNN focuses on extracting detailed features and can accurately distinguish similar behaviors. The combination of the two not only improves the recognition speed of high-confidence behaviors, but also improves the recognition accuracy of complex behaviors.
[0015] 2. In this invention, YOLOv5 first extracts students from complex backgrounds, greatly reducing background interference in subsequent classification tasks, allowing CNN and Transformer to focus more on human actions themselves, thereby improving the model's generalization ability in different classroom environments. Attached Figure Description
[0016] Figure 1 This is a flowchart of the present invention.
[0017] Figure 2 This is a schematic diagram of the framework for classroom behavior recognition in this invention.
[0018] Figure 3 This is a flowchart of the training method for the classroom behavior recognition model in this invention. Detailed Implementation
[0019] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.
[0020] Reference Figure 1-3 As shown, a student classroom behavior recognition method based on the fusion of YOLOv5 and CNN includes the following steps: S1. Collect video and image data from classroom scenarios to construct a dataset of student classroom behavior; By collecting a large amount of video image data in classroom scenarios through cameras, ensuring diversity in lighting, angles, and student density, student classroom behaviors such as looking up, raising hands, looking down, lying down, standing, taking notes, and playing with mobile phones are selected. Through recording and analyzing student classroom behaviors, images of various student behaviors are selected to form a dataset.
[0021] S2. Preprocess the collected dataset and divide the preprocessed dataset into training set and test set; Preprocessing of the collected dataset includes: Use annotation tools (such as LabelImg) to annotate the student's behavior in each frame and add basic behavior labels; For complex behaviors such as "sleeping" and "playing on the phone", a 5-second segment is extracted from the video, and the entire segment is labeled with a behavior tag. Perform at least one data augmentation operation on images and video clips, including rotation, scaling, brightness adjustment, and noise addition, to improve the generalization ability of the model; The preprocessed dataset was divided into a training set (70%) and a test set (30%).
[0022] S3. Construct and train a classroom behavior recognition model; The classroom behavior recognition model includes a classroom behavior recognition module and a decision fusion module; The classroom behavior recognition module uses the YOLOv5 model to perform target detection and preliminary behavior classification on video image data. For low-confidence behaviors or suspected complex behaviors output by YOLOv5, the region of interest of the student is cropped from consecutive frames based on the region output by YOLOv5 to form an image sequence. The high-level features of each frame in the image sequence are extracted using a CNN model to form a feature vector sequence. The feature vector sequence is input into the Transformer model for temporal modeling to capture the dependence of behavior in the time dimension and output the final behavior classification result. Decision Fusion Module: Design a decision fusion module to fuse the results of YOLOv5 and CNN+Transformer models. If the confidence score of behaviors such as listening to lectures and taking notes is higher than the threshold (0.9), the result of YOLOv5 will be directly adopted. If the confidence score is lower than the preset threshold or the behavior is judged as a complex behavior, the CNN model will be called to analyze the first N frames of the student's sequence and the output of the CNN+Transformer model will be used as the final judgment of the behavior.
[0023] The entire model training iteration is no less than 1000 times to ensure a reliable classroom behavior recognition model is obtained after training. Next, the network training process is observed, and the optimal training parameters are selected based on the experimental results. The best training model is selected, and finally, it is tested using a test set. The experimental results are observed and analyzed to identify and classify student classroom behaviors.
[0024] S4. Input the real-time classroom video image data into the trained classroom behavior recognition model and output the final behavior recognition result.
[0025] The real-time video stream is input into the trained classroom behavior recognition model, and the YOLOv5 model detects each student. If the confidence score of their behavior (such as "listening to the lesson") is higher than 0.9, the result is output directly. If YOLOv5 detects that a student's behavior has a low confidence level (below 0.9) or is in an abnormal state such as "looking down" for a long time, it will trigger complex behavior analysis. The system will automatically extract the image sequence of the student's region of interest, feed it into the CNN+Transformer model for analysis, and output the behavior results.
[0026] As is known from common technical knowledge, this invention can be implemented through other embodiments that do not depart from its spirit or essential characteristics. Therefore, the disclosed embodiments described above are merely illustrative in all respects and are not the only ones. All modifications within the scope of this invention or its equivalents are included in this invention.
Claims
1. A student classroom behavior recognition method based on the fusion of YOLOv5 and CNN, characterized in that: Includes the following steps: S1. Collect video and image data from classroom scenarios to construct a dataset of student classroom behavior; S2. Preprocess the collected dataset and divide the preprocessed dataset into training set and test set; S3. Construct and train a classroom behavior recognition model; S4. Input the real-time classroom video image data into the trained classroom behavior recognition model and output the final behavior recognition result.
2. The student classroom behavior recognition method based on the fusion of YOLOv5 and CNN as described in claim 1, characterized in that: In step S2, the preprocessing of the collected dataset includes: Use annotation tools to annotate the student's behavior in each frame and assign basic behavior labels; Extract short video clips containing complex behaviors from the video and tag the entire clip with behavior tags; Perform at least one of the following data augmentation operations on images and video clips: rotation, scaling, brightness adjustment, and noise addition.
3. The student classroom behavior recognition method based on the fusion of YOLOv5 and CNN as described in claim 1, characterized in that: In step S3, the classroom behavior recognition model includes a classroom behavior recognition module and a decision fusion module; The classroom behavior recognition module uses the YOLOv5 model to perform target detection and preliminary behavior classification on video image data. For low-confidence behaviors or suspected complex behaviors output by YOLOv5, the image sequence of the student's region of interest is extracted. The high-level features of each frame in the image sequence are extracted using the CNN model to form a feature vector sequence. The feature vector sequence is input into the Transformer model to output the final behavior classification result. Decision fusion module: Combines the outputs of YOLOv5 and CNN+Transformer models to generate the final behavior recognition result.
4. The student classroom behavior recognition method based on the fusion of YOLOv5 and CNN as described in claim 3, characterized in that: The YOLOv5 model is used for: Real-time detection of student positions in each frame of the image; Output the bounding box coordinates and preliminary behavior classification and confidence score for each student.
5. The student classroom behavior recognition method based on the fusion of YOLOv5 and CNN according to claim 3, characterized in that: The Transformer model is used for: Perform time-series modeling on the feature vector sequence output by CNN; Capture the dependencies of behavior over time; Output the classification results for complex behaviors.
6. The student classroom behavior recognition method based on the fusion of YOLOv5 and CNN according to claim 3, characterized in that: The decision fusion module fuses the results according to the following rules: If the confidence level of the behavior output by YOLOv5 is higher than the preset threshold, its classification result will be directly adopted. If the confidence level is lower than the preset threshold or the behavior is judged to be a complex behavior, the output of the CNN+Transformer model is used as the final behavior recognition result.
7. The student classroom behavior recognition method based on the fusion of YOLOv5 and CNN according to claim 1, characterized in that: The ratio of the training set to the test set in the dataset is 7:
3.
8. The student classroom behavior recognition method based on the fusion of YOLOv5 and CNN according to claim 1, characterized in that: The training iterations of the classroom behavior recognition model shall not be less than 1000.