Classroom behavior detection method, device and system and storage medium

Abstract

The invention discloses a classroom behavior detection method, device and system, and a storage medium, and the method comprises the steps: carrying out the classroom behavior detection through employing a YOLOv11 model, recognizing the classroom behaviors of students frame by frame, and recording the related information; through the face detection and matching technology, the identity of each student is accurately recognized, classroom behavior information is associated with student data, and behavior statistical analysis is facilitated; a visual chart is generated by means of a Matplotlib library by counting the occurrence frequency and proportion of various behaviors of students in a specific time period, the proportion of the behaviors of the students in the whole class at a specific time point and the proportion of various classroom behaviors in the whole time period. According to the technical scheme, the accuracy and efficiency of classroom behavior monitoring are improved, and then data support is provided for related education research and practice.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a classroom behavior detection method and device, a system, and a storage medium. Background Art

[0002] In the modern education system, effective monitoring and analysis of classroom behavior is of great significance to improving teaching quality and student learning outcomes. Traditional classroom observation methods often rely on teachers' subjective judgments, making it difficult to capture the behavior of each student in a comprehensive and real-time manner. In addition, it is difficult to match students' identity information with classroom behavior, which makes it challenging for teachers to understand the learning status and behavioral characteristics of each student. At the same time, due to the complexity and dynamism of the classroom environment, traditional methods often seem powerless when dealing with a large number of students and diverse behaviors. These problems urgently need to be solved through advanced technical means to achieve accurate detection and analysis of classroom behavior. The classroom behavior detection system for wide-angle lenses is a system that detects and statistically analyzes students' classroom behavior, which is one of the key applications of artificial intelligence + education. The technologies required mainly include video image processing technology and deep learning technology.

[0003] Video image processing technology: The classroom behavior detection system for wide-angle lenses mainly obtains classroom image information through wide-angle surveillance cameras, and pre-processes and analyzes the images through video image processing technology, thereby realizing functions such as classroom behavior detection, face detection, face matching, and statistics. Technologies widely used in visual algorithms include target detection, behavior recognition, feature extraction, and face matching.

[0004] Deep learning technology: In order to improve the accuracy of classroom behavior recognition and the identification and matching of behavior target identity information, the classroom behavior detection system for wide-angle lenses uses various deep learning technologies. Common machine learning algorithms include support vector machines, decision trees, random forests, etc.; while deep learning algorithms mainly use convolutional neural networks, transformers and other methods. These technologies can not only improve system performance, but also realize more functions, such as face recognition, face feature extraction, gender recognition, age recognition, etc.

[0005] Among them, classroom behavior detection, face detection, and face matching are key links. Classroom behaviors in images are identified based on the YOLOv11 classroom behavior detection model to obtain target detection results. The detection result information is used to locate the location where the classroom behavior occurs, and face detection is performed at that location to obtain the face location of the initiator of the classroom behavior, so as to facilitate subsequent personnel identity matching and filter out some false detection targets. Through face positioning, faces are obtained, facial features are extracted, and the acquired facial features are matched with the facial feature information in the established facial feature library to obtain the identity information of the initiator of the behavior, thereby achieving matching of classroom behavior and facial information, which is convenient for subsequent statistical analysis. However, most traditional methods have significant limitations in classroom behavior detection and analysis, including:

[0006] Low recognition accuracy: Many traditional behavior detection technologies rely on simple rules or template matching, which cannot fully utilize the powerful characteristics of deep learning models, resulting in low recognition accuracy in complex environments and prone to missed detection. Especially in classrooms where students have diverse behavioral performance, traditional methods often have difficulty accurately identifying complex behavior patterns.

[0007] Difficulty in matching identity and behavior: Traditional observation methods usually rely on teachers’ subjective judgment, making it difficult to accurately match specific behaviors with specific student identities in real time. Teachers may not be able to pay comprehensive and continuous attention to each student’s performance, which greatly limits the in-depth analysis of individual student behavior and thus affects educational decision-making.

[0008] Low data processing efficiency: Traditional methods usually use video playback for post-class analysis, which cannot process data in real time and efficiently. Teachers need to spend a lot of time reviewing videos, lack instant feedback on classroom dynamics, and cannot quickly adjust teaching strategies.

[0009] More manual intervention and poor objectivity: Traditional methods that rely on manual observation are easily affected by the observer's personal subjective factors, resulting in data deviation. This subjective intervention not only affects the objectivity and consistency of the results, but may also lead to a decrease in the reliability of the analysis results.

[0010] Difficulty in responding to dynamic changes: The classroom environment is highly dynamic, and traditional methods often have difficulty adapting to unexpected classroom situations, such as sudden changes in student behavior and different teachers’ teaching styles, making it difficult to fully track and analyze classroom behavior. Summary of the invention

[0011] The technical problem to be solved by the present invention is to provide a classroom behavior detection method and device, system, and storage medium.

[0012] To achieve the above object, the present invention adopts the following technical solution:

[0013] A classroom behavior detection method, comprising:

[0014] Step S1, collecting classroom video data;

[0015] Step S2, associating the target person information with the classroom identification detection information, so as to associate each frame and the person target identity information in each frame with the behavior detection information;

[0016] Step S3: Perform data statistics on the associated data, count the behavior of each student in each frame of the video in a specific time period, count the proportion of various behaviors of students in a time period, and count the proportion of various behaviors in each frame for visualization in the next stage.

[0017] Preferably, in step S1, classroom behavior recognition is performed on the data information and the detection results are recorded; based on the detection results, face recognition is performed on the target to obtain the target face location and perform identity matching.

[0018] Preferably, in step S3, statistical data are used to generate a bar graph of the proportion of each student's behavior in a specific time period, a pie chart of the proportion of students' behaviors in the entire class at a specific time point, and a line graph of the changes in the proportion of various types of classroom behaviors in the entire time period; paired associated data are used to draw the boundary box of the target area in the video image, and the identity of the people and classroom behavior are marked.

[0019] The present invention also provides a classroom behavior detection device, comprising:

[0020] The first processing module is used to collect classroom video data;

[0021] The second processing module associates the target person information with the classroom identification detection information to associate each frame and the person target identity information in each frame with the behavior detection information;

[0022] The third processing module performs data statistics on the associated data, counting the behavior of each student in each frame of the video in a specific time period, counting the proportion of various behaviors of students in a time period, and counting the proportion of various behaviors in each frame for visualization in the next stage.

[0023] Preferably, the first processing module comprises:

[0024] A detection unit is used to identify classroom behaviors based on data information and record the detection results;

[0025] The recognition unit is used to perform face recognition on the target based on the detection results, obtain the target face location, and perform identity matching.

[0026] Preferably, the third processing module comprises:

[0027] The first processing unit is used to generate a bar graph of the proportion of each student's behavior in a specific time period using statistical data, a fan graph of the proportion of students' behavior in the entire class at a specific time point, and a line graph of the change in the proportion of various types of classroom behaviors in the entire time period;

[0028] The second processing unit is used to draw the boundary box of the target area in the video image by using the paired associated data, and to mark the identity of the person and the classroom behavior.

[0029] The present invention also provides a classroom behavior detection system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes the classroom behavior detection method when executed by the processor.

[0030] An embodiment of the present invention further provides a storage medium, on which a computer program is stored, and the computer program executes the classroom behavior detection method when running.

[0031] The present invention uses the YOLOv11 model to detect classroom behavior, identify students' classroom behavior frame by frame and record relevant information. At the same time, through face detection and matching technology, the identity of each student is accurately identified, and the classroom behavior information is associated with student data to facilitate behavioral statistical analysis. Finally, by counting the number of occurrences and proportions of various behaviors of students in a specific time period, as well as the proportion of occurrences of various behaviors in a single frame, a visual chart is generated with the help of the Matplotlib library, and the video is annotated and saved at the same time, so as to facilitate subsequent analysis and research. This system provides an effective tool for improving classroom management and teaching efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work.

[0033] Figure 1 This is a flow chart of a classroom behavior detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Embodiment 1:

[0037] like Figure 1 As shown, a classroom behavior detection method according to an embodiment of the present invention includes:

[0038] Step 1: Data Collection

[0039] A fixed wide-angle camera is installed to monitor the target area and the OpenCV library is used to obtain the video stream for the next step of classroom behavior detection.

[0040] Specifically, a wide-angle camera should be installed in the area where student classroom behavior analysis is required, such as above the classroom blackboard. The camera should be installed to ensure that its monitoring range can completely cover the area where classroom behavior analysis is required, so as to ensure that complete video information can be collected and accurate identification and recording of student classroom behavior can be achieved.

[0041] Step 2: Train YOLOv11 to detect and record classroom behavior

[0042] Specifically, the YOLOv11 model is used to train the self-built wide-angle student classroom behavior detection dataset to solve the student classroom behavior detection task. This step requires the following sub-steps: data preparation, model training, model evaluation, and model application.

[0043] Step 2-1 Data preparation: First, obtain the public student behavior perspective image data from the Internet, and then use the script file to perform wide-angle distortion on the collected student classroom behavior perspective image data. At the same time, use a wide-angle camera to record videos in real classrooms, perform frame-by-frame segmentation, and filter out blurred images.

[0044] The labelimg tool is then used to calibrate the data. Labeling usually refers to marking the position and category of the target in the image. Labeling usually uses the yolo format, which contains information such as the target category (reading, looking around, listening, raising hands, standing up, lying on the table, playing with mobile phones and other behaviors), border position, etc.

[0045] Step 2-2 Model training: The YOLOv11 model needs to be trained using the prepared data set. The training process usually includes steps such as model initialization, forward propagation, back propagation, and gradient update. During the training process, it is necessary to select appropriate hyperparameters, loss functions, optimizers, etc. to improve the accuracy and generalization ability of the model. The batch_size of the experiment of the present invention is set to 64, the initial learning rate is set to 0.001, the final learning rate is set to 0.01, and the learning rate momentum is set to 0.9. In order to improve the generalization of the model, the online data enhancement method is introduced in the experiment: scale transformation (scale is set to 0.5), mosaic enhancement (mosaic is set to 0.2), color perturbation (hsv_h, hsv_s, hsv_v are set to 0.015, 0.7, 0.4 respectively), and the optimizer selects AdamW. The overall loss function is:

[0046] L total =L bbox +L cls

[0047] The boundary regression loss function is:

[0048] L bbox =λ CIOU L CIOU +λ DFL L DFL

[0049] Where L CIOU is CIOU loss, L DFL is the DFL loss, λ CIOU and λ DFL As the hyperparameter is used to balance the impact of these two losses, this experiment sets them to 1.0 and 0.5 respectively.

[0050] L CIOU It is a loss function used to evaluate the overlap between the predicted bounding box and the true bounding box. Different from the traditional IoU (Intersection over Union), CIOU adds factors such as center point distance, aspect ratio, and angle direction, providing a more comprehensive evaluation. Its expression is as follows:

[0051]

[0052] Among them, IoU(b pred ,b ture ) is the intersection-over-union ratio between the predicted bounding box and the true bounding box, d is the distance from the center of the predicted bounding box to the center of the true bounding box, c is the minimum diagonal length of the bounding box, α is the weight term used to weight the difference in aspect ratio, and v is the penalty term for aspect ratio, which is usually expressed as Among them (w true ,htrue ) and (w pred ,h pred ) represent the width and height of the predicted bounding box and the true bounding box respectively.

[0053] L DFL It is mainly used to solve the problem of class imbalance, especially the scarce categories in large data sets. DFL guides the model to better learn the target by weighting the difficult samples (i.e. the samples with high loss values). Its expression is as follows:

[0054]

[0055] in is the true category probability (usually 1 or 0, indicating whether the target exists or not), p i is the predicted probability.

[0056] L cls The classification loss uses binary cross entropy (BCE) to calculate the confidence that the target is of this category, and its expression is as follows:

[0057]

[0058] Where C is the total number of categories, is the true label, indicating the authenticity (1 or 0) of the i-th target box belonging to the j-th category. ij is the probability that the i-th target box predicted by the model belongs to the j-th class.

[0059] The training process is completed on the 3090Ti GPU.

[0060] Step 2-3 Model evaluation: After training is completed, the performance of the model needs to be evaluated. Generally, accuracy, precision, recall, mean average precision (mAP), etc. can be used. The evaluation can be performed through cross-validation, test set validation, etc. The evaluation results can be used to adjust model parameters, improve training data, etc., to improve the performance and generalization ability of the model. The main evaluation indicators used in the target detection process of the experiment of the present invention are Precision, Reacll, and mAP.

[0061] Step 2-4 Model application: Apply the trained model to downstream tasks. In this invention, the downstream task is student classroom behavior detection.

[0062] After completing the student classroom behavior test, the input and output results of the student classroom behavior test are recorded for subsequent analysis and application. The information recorded by the experiment of the present invention mainly includes the following aspects:

[0063] 1. Target location information: Record the precise location information of the target in the image by recording the bounding box position, center point position, etc.

[0064] 2. Target category: output the confidence of all target categories and select the category with the largest confidence as the target category.

[0065] 3. The detection results (target location information, target category) obtained after model detection will be used for subsequent analysis and application.

[0066] Step 3: Face Detection

[0067] Through the results of classroom behavior detection, the student target position can be located, and the target image information of the position can be copied. The copied target image is used for face detection using the face positioning method in the face_recognition library in the Python library. If there is a face target, it is retained. If there is no face target, the target is filtered out because the specific student information cannot be determined and it is very likely to be a wrong detection. The output face target position information, as well as the target position information and target category information of the classroom behavior detection in step 2, will be used for subsequent analysis and application.

[0068] Step 4: Face matching

[0069] Face matching uses the face_recognition library in Python to perform face matching, which is mainly divided into two steps:

[0070] Step 4-1: Constructing the facial feature library: Obtain the student’s facial image for identity registration. First, input the student’s facial image and student ID number, use the facial encoding method in the face_recognition library to obtain the facial feature vector, and construct a dictionary. After the input is completed, generate a Json file for subsequent facial feature library loading.

[0071] Step 4-2 Face matching: Load the Json file of the face feature library to generate a face feature library dictionary. Copy the face target location information obtained in step 3 to the target face image. Use the face feature comparison method in the face_recognition library to perform similarity matching between the target face and the face features in the face feature library dictionary. It returns a Boolean list and obtains the target information of the person whose value is True in the list.

[0072] The location information of the target detected in step 2, the target category and the person target information (student number) obtained by face matching in step 4 will be used for subsequent analysis and application.

[0073] Step 5: Student Behavior Association Records

[0074] Using the position information of the paired targets in step 4, the target category and the person target information (student number) obtained by face matching are generated and saved in a Json file.

[0075] Step 6: Behavior Statistics

[0076] Read the generated student behavior association record Json file, count the frequency of occurrence of their classroom behaviors according to their student ID, and calculate the proportion of each classroom behavior in all frames to determine the proportion of different classroom behaviors of students in a specific time period. The formula is as follows:

[0077]

[0078] The number of times the behavior appears is the number of times a specific behavior appears in all frames in which the student participates. The total number of classroom frames of the student is the total number of frames in which the student appears in the data.

[0079] By calculating the proportion of each behavior in a single frame, the changes in student behavior in the classroom can be determined. The formula is as follows:

[0080]

[0081] Among them, the number of occurrences of a single type of behavior is the number of students with a specific behavior in a single frame. The total number of all students detected is the total number of students that can be detected in a single frame.

[0082] These behavioral statistics will be used for subsequent analysis and application.

[0083] Step 7: Data Visualization

[0084] The behavior statistics calculated in step 6 can be used to generate visual charts using the Matplotlib library, including bar charts, pie charts, and line charts. These charts will intuitively show the proportion of each student's various behaviors in class, the frequency of each category of behavior in a specific time period, and the changes in the number of different categories in different time periods. These visual analyses will help to gain a deeper understanding of key factors such as classroom efficiency.

[0085] Using the position information of the paired targets in step 4, the target category and the person target information (student number) obtained by face matching, draw a target bounding box in each frame of the video sequence to display the target identity information and behavior category, and save the video sequence in the format of classroom behavior analysis video of the xth class on xx / xx / xx.mp4 for subsequent analysis.

[0086] Embodiment 2:

[0087] The embodiment of the present invention also provides a classroom behavior detection device, including:

[0088] The first processing module is used to collect classroom video data;

[0089] The second processing module associates the target person information with the classroom identification detection information to associate each frame and the person target identity information in each frame with the behavior detection information;

[0090] The third processing module performs data statistics on the associated data, counting the behaviors of each student in each frame of the video in a specific time period, counting the proportion of various behaviors of students in a time period, and counting the proportion of various behaviors in each frame for visualization in the next stage.

[0091] As an implementation of the embodiment of the present invention, the first processing module includes:

[0092] A detection unit is used to identify classroom behaviors based on data information and record the detection results;

[0093] The recognition unit is used to perform face recognition on the target based on the detection results, obtain the target face location, and perform identity matching.

[0094] As an implementation of the embodiment of the present invention, the third processing module includes:

[0095] The first processing unit is used to generate a bar graph of the proportion of each student's behavior in a specific time period using statistical data, a fan graph of the proportion of students' behavior in the entire class at a specific time point, and a line graph of the change in the proportion of various types of classroom behaviors in the entire time period;

[0096] The second processing unit is used to draw the boundary box of the target area in the video image by using the paired associated data, and to mark the identity of the person and the classroom behavior.

[0097] Embodiment 3:

[0098] An embodiment of the present invention further provides a classroom behavior detection system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a classroom behavior detection method when executed by the processor.

[0099] Embodiment 4:

[0100] An embodiment of the present invention further provides a storage medium, on which a computer program is stored, and the computer program executes the classroom behavior detection method when running.

[0101] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A classroom behavior detection method, characterized in that: include: Step S1, collecting classroom video data; Step S2, associating the target person information with the classroom identification detection information, so as to associate each frame and the person target identity information in each frame with the behavior detection information; Step S3: Perform data statistics on the associated data, count the behavior of each student in each frame of the video in a specific time period, count the proportion of various behaviors of students in a time period, and count the proportion of various behaviors in each frame for visualization in the next stage.

2. The classroom behavior detection method according to claim 1, characterized in that: In step S1, classroom behavior recognition is performed on the data information and the detection results are recorded; based on the detection results, face recognition is performed on the target to obtain the target face location and perform identity matching.

3. The classroom behavior detection method as claimed in claim 2, characterized in that: In step S3, the statistical data is used to generate a bar graph of the proportion of each student's behavior in a specific time period, a pie chart of the proportion of the entire class of students' behavior at a specific time point, and a line graph of the change in the proportion of various types of classroom behaviors in the entire time period; The paired association data is used to draw the bounding box of the target area in the video image and annotate the identity of the person and the classroom behavior.

4. A classroom behavior detection device, characterized in that: include: The first processing module is used to collect classroom video data; The second processing module associates the target person information with the classroom identification detection information to associate each frame and the person target identity information in each frame with the behavior detection information; The third processing module performs data statistics on the associated data, counting the behavior of each student in each frame of the video in a specific time period, counting the proportion of various behaviors of students in a time period, and counting the proportion of various behaviors in each frame for visualization in the next stage.

5. The classroom behavior detection device as claimed in claim 4, characterized in that: The first processing module includes: A detection unit is used to identify classroom behaviors based on data information and record the detection results; The recognition unit is used to perform face recognition on the target based on the detection results, obtain the target face location, and perform identity matching.

6. The classroom behavior detection device as claimed in claim 5, characterized in that: The third processing module includes: The first processing unit is used to generate a bar graph of the proportion of each student's behavior in a specific time period using statistical data, a fan graph of the proportion of students' behavior in the entire class at a specific time point, and a line graph of the change in the proportion of various types of classroom behaviors in the entire time period; The second processing unit is used to draw the boundary box of the target area in the video image by using the paired associated data, and to mark the identity of the person and the classroom behavior.

7. A classroom behavior detection system, characterized in that: include: A memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the classroom behavior detection method as described in any one of claims 1 to 3 is executed.

8. A storage medium, characterized in that: The storage medium stores a computer program, which executes the classroom behavior detection method as described in any one of claims 1 to 3 when running.

Images