Classroom condition analysis system and method based on computer vision

Abstract

The invention discloses a classroom condition analysis system and method based on computer vision, and relates to the technical field of computer vision, the system comprises a student information acquisition module, a course creation module, a classroom attendance module, an abnormal behavior detection module and a classroom condition analysis module; by integrating an advanced computer vision technology, comprehensive and accurate analysis of classroom conditions is realized, powerful data support is provided for education managers and teachers, firstly, the system adopts a dynamic face feature fusion algorithm to carry out classroom attendance checking, the attendance checking accuracy is improved, and the classroom attendance checking efficiency is improved; in addition, by combining static features and historical time sequence features of a current frame, identification errors caused by factors such as student posture changes and poor illumination conditions in a traditional attendance checking mode are effectively solved, in addition, a data enhancement strategy is further introduced in the system, and by conducting rotation and brightness adjustment processing on an original image, the attendance checking accuracy is improved. The diversity of the face feature vectors is further enriched, and the generalization ability of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and specifically provides a classroom situation analysis system and method based on computer vision. Background Art

[0002] With the continuous advancement of educational informatization, the traditional classroom management mode is facing an urgent need for transformation and upgrading. In the context of the information age, how to use modern technical means to improve classroom management efficiency and achieve real-time monitoring and accurate analysis of students' learning status has become a research hotspot in the education field. As an important branch of the field of artificial intelligence, computer vision technology has made remarkable progress in recent years. Its powerful image processing and analysis capabilities provide new ideas and methods for classroom management. Through computer vision technology, it is possible to automatically identify, record, and analyze the behaviors of students in the classroom, thereby providing teachers with more comprehensive and objective teaching feedback and helping to improve the quality of education.

[0003] Traditional classroom management mainly relies on manual observation and recording. This method has many limitations. First of all, manual observation is easily affected by subjective factors. Different teachers may have different judgment criteria for students' behaviors, resulting in the lack of objectivity and consistency in evaluation results. Secondly, the manual recording method is inefficient and difficult to achieve real-time monitoring and recording of the behaviors of a large number of students, let alone in-depth data analysis. In addition, there are also many inconveniences in attendance management in the traditional way. For example, proxy signing and missed signing problems occur frequently, seriously affecting the accuracy of attendance data. Therefore, the traditional classroom management method has been difficult to meet the needs of modern education management, and there is an urgent need to introduce new technical means for innovation.

[0004] Therefore, developing a classroom situation analysis system and method based on computer vision will effectively improve classroom management efficiency, promote the improvement of teaching quality, and drive the development of the educational informatization process. Summary of the Invention

[0005] The purpose of the present invention is to make up for the deficiencies of the existing technology and provide a classroom situation analysis system and method based on computer vision. The present invention integrates student information collection, course creation, classroom attendance, abnormal behavior detection, and classroom situation analysis. The system realizes accurate attendance and abnormal behavior recognition through a dynamic face feature fusion algorithm, key point detection, and the YOLOv10 object detection algorithm, integrates attendance and behavior data, and generates a quantitative evaluation report.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: On the one hand, a classroom situation analysis system based on computer vision, the system includes: a student information collection module, a course creation module, a classroom attendance module, an abnormal behavior detection module, and a classroom situation analysis module; The student information collection module: Connects to the school's educational administration system, collects basic information such as the class, student ID, and name of the student. At the same time, it captures a face image through a camera, extracts a face feature vector, and stores the student's basic information and the face feature vector in the database; The course creation module: Used for users to enter course information, associates with class information, and supports graphical interaction to create and import data into the course scheduling system; The classroom attendance module: Obtains the real-time video stream of the current class students through the classroom camera, uses the dynamic face feature fusion algorithm to extract the face feature vector in the video stream, compares it with the student face feature vector in the database for identification, confirms the identity of the attending students, and automatically stores the attendance status and time information in the database; The abnormal behavior detection module: Takes the video stream of the classroom camera as input, uses the key point detection algorithm and the YOLOv10 object detection algorithm to detect the abnormal behavior of students, and records data such as the basic information, abnormal type, occurrence time, and duration of the abnormal students; The classroom situation analysis module: Integrates the attendance data and the abnormal behavior data, quantitatively evaluates the classroom situation through the comprehensive classroom situation analysis algorithm, and generates a comprehensive report according to the analysis results. The report supports export in PDF and Word formats and interactive preview on the front-end interface.

[0007] Furthermore, in the student information collection module, data augmentation strategies are adopted for face feature vector extraction, including rotating the original image, adjusting the brightness, and performing Gaussian blur processing. Three enhanced versions are generated for each sample and combined with the original image for feature extraction.

[0008] Even further, in the classroom attendance module, the dynamic face feature fusion algorithm is used to extract the face feature vector in the video stream. The face feature vector fusion function is defined as , and the calculation formula is: , where: is the fused face feature vector, is the static visual feature vector extracted from the current frame, is the temporal feature vector of the t-th frame in the past n frames, is the dynamic weight coefficient, n is the number of past frames, t is the index variable for the summation operation, is the index of a certain face detected in the current frame.

[0009] Even further, in the classroom attendance module, the dynamic face matching algorithm is used to compare and identify the face feature vector extracted from the video stream with the student face feature vector in the database to confirm the identity of the attending students. The calculation formula is: , where, is the The feature matching score of a student, The index of the student in the database, Is the reference feature vector of the j-th student in the database, Is to calculate the cosine similarity between the fused feature and the reference feature, Is the fused face feature vector, Represents the relevant index of the face feature extracted in the current frame, Is the distance attenuation coefficient, Is the spatio-temporal continuity factor, and the formula is: , Represents the time window size of the past frame, Represents the past The fused feature vector related to the j-th student in the frame, Is the offset expression of the time index, Is the rate of change of the distance between the face in the current frame and the camera, Is the distance attenuation coefficient, when , confirm the student's identity, Is the dynamic threshold of the current frame.

[0010] Furthermore, the dynamic threshold of the current frame in the classroom attendance module The calculation formula is: , where confidence Is the confidence of the current frame feature extraction, and the value range is 0-1, Is the time or frame index of the current frame.

[0011] Furthermore, the specific steps for detecting drowsiness behavior based on the key point detection algorithm in the abnormal behavior detection module are as follows: Perform initialization detection and record the reference EAR when each student's eyes are fully open; Set the dynamic threshold to 60% of the reference EAR; Perform real-time EAR calculation for each student in each frame of the image. If the EAR of a certain student in the current frame is less than the dynamic threshold, update the abnormal state and record the current time as last_update; When it is detected that the student's eyes are closed for more than 3 seconds, record the start time and continuously track the duration of the abnormal behavior; The drowsiness behavior of the current student is not detected in consecutive frames of images, and last_update is not updated within the tolerance time of 5 seconds. If the tolerance time is exceeded, mark the end of the abnormality; When stopping detection or performing regular checks, perform timeout judgment on all unended abnormal records and update the end time.

[0012] Furthermore, the adjustment of the dynamic threshold for dozing detection in the abnormal behavior detection module: Fatigue accumulation: Record the historical dozing frequency of students. When the number of dozing times in a single week exceeds 3 times, the dynamic threshold is reduced by 5%. Head pose compensation: By detecting the head pitch angle, when the angle exceeds 15°, the EAR threshold is automatically adjusted to 55% of the reference value.

[0013] Furthermore, the specific steps for detecting the behavior of playing with mobile phones based on the YOLOv10 model in the abnormal behavior detection module are as follows: Use web crawler technology to crawl mobile phone image samples from the network, clean the image samples that do not meet the requirements, and divide them into training samples and test samples; Use the labelimg tool to label the mobile phones in the cleaned training samples and test samples. The annotation content is the rectangular boundary box where the mobile phone is located in the image, and label these boundary boxes as playphone; for each image after annotation, a YOLO format file is generated, which contains the class ID of each object in the image and the coordinates of the corresponding boundary box; Use the PyTorch framework and the YOLOv10 model to train the annotated training samples. The input of the YOLOv10 model is a 3-channel color image of 320×320 pixels, and the number of iterations is 100 times. Then use the test samples to detect the YOLOv10 model; Use the trained YOLOv10 model to detect whether there is a behavior of students playing with mobile phones in the classroom. By calculating the spatial center distance between the mobile phone boundary box and the face boundary box, associate the mobile phone playing detection result with the student, and introduce the judgment of abnormal behavior persistence to avoid misjudgment of instantaneous behaviors.

[0014] Furthermore, in the classroom situation analysis module, the classroom situation is quantitatively evaluated through the classroom situation comprehensive analysis algorithm. The formula is: CAI AttendRate TrendScore, where: CAI is the comprehensive evaluation value of the classroom situation, AttendRate is the attendance rate, is the th duration of abnormal behavior, is the total duration of abnormal behavior, TrendScore is the behavior trend score, is the total classroom time, 、 、 are the weight coefficients, and , and the weight is automatically adjusted according to the course type.

[0015] On the other hand, a method for analyzing classroom situations based on computer vision, the analysis method comprising the following steps: S100, data preparation and initialization: The system interfaces with the school's educational administration system to import student information, collects face images through classroom cameras, and extracts face feature vectors and stores them in the database; the user creates courses through the graphical interface and associates class information, or imports the course scheduling system data to complete course configuration; S200, real-time classroom data collection: The system automatically activates the camera to obtain the video stream, preprocesses the video stream by decoding, scaling, and denoising, performs rotation, brightness adjustment, and Gaussian blur processing, and uses the dynamic face feature fusion algorithm to extract the face feature vectors in the video stream; S300, multi-dimensional behavior analysis: Based on the comparison of face feature vectors with the database for attendance recognition, a dynamic threshold mechanism is adopted to improve the recognition accuracy; the key point detection and YOLOv10 algorithms are run synchronously, and the dozing behavior is detected based on the EAR value, and the mobile phone playing behavior is recognized through object detection and face space association, and the types, times, and durations of abnormal behaviors are recorded; S400, data fusion and quantitative evaluation: Integrate the attendance and abnormal behavior data, automatically assign weights according to the course type to calculate the comprehensive index of the classroom situation, analyze the concentration and distribution characteristics of abnormal behaviors, and generate a behavior heat map to intuitively display the problem areas; S500, visualization presentation and report generation: Realize data visualization, provide a real-time monitoring large screen, trend analysis charts, and student portrait interactive components; automatically generate a report containing a summary, detailed analysis, comparison data, and intelligent suggestions based on a template, support PDF / Word export and integration with the teaching management system.

[0016] Compared with the prior art, the computer vision-based classroom situation analysis system and method have the following beneficial effects: First, through the integration of advanced computer vision technology, the present invention realizes a comprehensive and accurate analysis of classroom situations, providing strong data support for education administrators and teachers. First, the system uses the dynamic face feature fusion algorithm for classroom attendance, which not only improves the accuracy of attendance, but also effectively solves the recognition errors caused by factors such as student posture changes and poor lighting conditions in traditional attendance methods by combining the static features of the current frame and the historical time series features. In addition, the system also introduces a data enhancement strategy, and by rotating and adjusting the brightness of the original image, it further enriches the diversity of face feature vectors and improves the generalization ability of the model. This new attendance method not only reduces the workload of teachers, but also improves the authenticity and reliability of attendance data, providing a strong basis for the evaluation of teaching quality.

[0017] II. By combining the key point detection algorithm and the YOLOv10 object detection algorithm, the present invention realizes the accurate recognition and recording of abnormal behaviors of students in the classroom. Specifically, the system can monitor the dozing and mobile phone playing behaviors of students in real time, and through the dynamic threshold mechanism and spatial correlation technology, effectively avoids misjudgment of instantaneous behaviors and improves the detection accuracy. Especially for the detection of dozing behavior, the system not only records the occurrence time and duration of the abnormal state, but also dynamically adjusts the detection threshold through the fatigue accumulation and head pose compensation mechanisms, further enhancing the detection sensitivity and adaptability. This new abnormal behavior detection method not only helps teachers to timely discover and correct the bad behaviors of students, but also provides a quantitative evaluation basis for classroom discipline for education managers, providing strong support for improving teaching methods and enhancing teaching quality.

[0018] Other advantages, objectives and features of the present invention will be described to some extent in the subsequent specification, and to some extent, will be obvious to those skilled in the art based on the study of the following text, or can be taught from the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0020] Figure 1 It is an overall architecture diagram of a classroom situation analysis system based on computer vision; Figure 2 It is a working flow chart of the student information acquisition module; Figure 3 It is a working flow chart of the classroom attendance module; Figure 4 It is a working flow chart of the abnormal behavior detection module; Figure 5 It is a working flow chart of the classroom situation analysis module; Figure 6 It is a flow chart of a classroom situation analysis method based on computer vision. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] To further elaborate on the technical means and effects adopted by the present invention to achieve the predetermined invention objective, the following will, in conjunction with the accompanying drawings and preferred embodiments, describe in detail the specific embodiments, structures, features and their effects of the present invention as follows. Embodiment

[0022] An embodiment of a classroom situation analysis system based on computer vision.

[0023] Classroom teaching of the "Principles of Control Systems" course for the automation major in colleges and universities Student information collection module: In the first week of the semester, the system administrator completes the docking of the system with the school's educational administration system through background configuration. At this time, the system automatically synchronizes the basic information of the student IDs, names, and classes of 45 students in this class to form an initial data list. At the same time, the high-definition camera installed in the front of the classroom is activated, guiding the students to perform face collection in the designated area in turn. After the camera captures the frontal image of the student, the system automatically performs rotation, brightness adjustment, and data enhancement processing on the original image to make the features more obvious. Subsequently, the face feature vector is extracted, and this information is corresponded to the basic information of the students one by one and stored in the encrypted database, as Figure 2 shown.

[0024] Course creation module: After the course teacher logs in to the system and enters the course creation module, considering that "Principles of Control Systems" is a core professional course, the teacher manually enters the course name and class hour arrangement information through the graphical interface and associates it with the class information. The drag-and-drop operation on the interface is like a jigsaw puzzle, which is convenient for the teacher to quickly complete the binding of the course and the class; of course, the teacher can also choose to import the Excel data file of this course from the school course scheduling system, and the system will automatically parse and fill in the relevant information, saving the work of repeated entry.

[0025] Classroom attendance module: 10 minutes before the start of the "Principles of Control Systems" course, the system automatically activates the classroom camera according to the preset time. At this time, the camera obtains the real-time video stream at a speed of 25 frames per second, and the picture covers the entire classroom seating area. The video stream is first preprocessed to remove noise and adjust the size to make the picture clearer. Subsequently, rotation, brightness adjustment, and Gaussian blur processing are performed, and the dynamic face feature fusion algorithm is used to extract the face feature vector in the video stream. Define the face feature vector fusion function as , and the calculation formula is: , where: is the fused face feature vector, is the static visual feature vector extracted from the current frame, is the temporal feature vector of the t-th frame in the past n frames, is the dynamic weight coefficient, n is the number of past frames, and t is the index variable for the summation operation. is the index of a certain face detected in the current frame, and the static visual features (such as the relative positions of eyes and nose) of the student's face in the current frame are fused with the temporal features (such as the trajectory of head rotation) of several past frames to generate a more stable feature vector.

[0026] The fused feature vectors will be compared with the student face feature vectors in the database to identify the students present. Here, a dynamic face matching algorithm is used, which comprehensively considers the cosine similarity of the feature vectors, spatio-temporal continuity, and the distance change factor between the face and the camera to calculate the matching score between each detected face and the students in the database. The calculation formula is as follows: , where is the feature matching score of the th student, is the index of the student in the database, is the reference feature vector of the jth student in the database, is to calculate the cosine similarity between the fused feature and the reference feature, is the fused face feature vector, represents the index related to the face feature extracted in the current frame, is the distance attenuation coefficient, is the spatio-temporal continuity factor, and the formula is: , represents the time window size of the past frames, represents the past fused feature vectors related to the jth student in the frames, is the offset expression of the time index, is the distance change rate between the face in the current frame and the camera, is the distance attenuation coefficient. The system will compare the calculated feature matching score with the dynamic threshold of the current frame. When , the student identity is confirmed. The dynamic threshold is automatically adjusted, and the calculation formula is: , where confidence is the confidence of the feature extraction in the current frame, and its value range is 0 - 1, is the time or frame index of the current frame, ensuring that the identity of the students present can be accurately identified even when the students' sitting postures change or the light is insufficient. The recognition results will be displayed on the teacher's terminal screen in real time, and at the same time, the attendance status and specific timestamps will be stored in the database to form an attendance record log, as shown in Figure 3 .

[0027] Abnormal behavior detection module: As shown in Figure 4 ; Dozing behavior detection: When the course reaches the 30th minute, some students may become fatigued. The system first initializes the detection of each student and records the baseline EAR value when their eyes are fully open. In class, the system calculates the student's EAR value frame by frame. When a student's EAR value is less than the dynamic threshold set based on the baseline value, the system will immediately mark the student as entering an abnormal state and record the current time. If the student closes his eyes for more than 3 seconds continuously, like the set "lateness judgment time", the system will formally record the start time of the dozing behavior and continue to track its duration. In addition, the system will intelligently learn from students' historical behavior. When a student dozes off more than 3 times in a single week, the dynamic threshold will be automatically lowered to increase detection sensitivity. At the same time, if the student's head pitch angle is detected to exceed 15°, it means that he may be in a drowsy state with his head down, and the EAR threshold will be adjusted accordingly to make the detection more accurate.

[0028] Mobile phone playing behavior detection: During the course, the system runs a trained YOLOv10 model in the background, constantly scanning the mobile phone target in the picture. The model is trained using the PyTorch framework by crawling thousands of image samples of mobile phone playing from the Internet, cleaning and annotating them. It has a high accuracy rate for mobile phone recognition in classroom environments. When the model detects a mobile phone target, it calculates the spatial center distance between the mobile phone bounding box and the face bounding box, just like measuring the positional relationship between two points, thereby determining the identity of the student playing with the mobile phone. In order to avoid misjudgment of students taking out their mobile phones for a short time, the system will introduce abnormal behavior persistence judgment. Only when the mobile phone continues to appear in the picture for more than a certain period of time will it be confirmed as abnormal mobile phone playing behavior, and the abnormal type, time and duration will be recorded.

[0029] Classroom situation analysis module: After the get out of class bell rings, the system automatically stops data collection and starts to integrate the attendance data and abnormal behavior data of the entire class. Through the classroom situation comprehensive analysis algorithm, the algorithm automatically assigns weights to the attendance rate, the total duration of abnormal behavior, and the behavior trend scoring factors according to the course type (for example, theoretical classes pay more attention to attendance and have a higher weight), and calculates the comprehensive evaluation value of the classroom situation. The formula is: CAI AttendRate TrendScore, where: CAI is the comprehensive evaluation value of the classroom situation, AttendRate is the attendance rate, For the Duration of abnormal behavior is the sum of the duration of abnormal behavior, TrendScore is the behavior trend score, is the total class time, , , is the weight coefficient, and The weights are automatically adjusted according to the course type. Meanwhile, the system analyzes the distribution patterns of abnormal behaviors, such as which time periods have a higher number of students dozing off and which seating areas are concentrated on mobile phone usage.

[0030] Based on the analysis results, the system automatically generates a multi-dimensional comprehensive report. The report cover shows the course name, date, and class information. The main body of the report includes an attendance statistics table (marking the list of students who are late and absent), details of abnormal behaviors (such as a certain student starting to doze off at the 25th minute and lasting for 5 minutes), a quantitative evaluation score, and a radar chart analysis. The report supports export in PDF and Word formats, and teachers can directly download it for teaching summary. At the same time, the front-end interface provides an interactive preview function. Teachers can click on the chart to view the detailed behavior records of specific students, as Figure 5 shown.

[0031] To sum up, in the courses of the automation major in colleges and universities, this system first connects to the educational administration system to collect student information and extract facial features, creates courses and associates them with classes. In the classroom, it obtains video streams through cameras, completes attendance using dynamic facial feature fusion algorithms and dynamic face matching algorithms, combines key point detection algorithms and YOLOv10 algorithms to detect dozing off and mobile phone usage behaviors, and finally integrates the data, uses a comprehensive classroom situation analysis algorithm to evaluate and generate reports, achieving a comprehensive analysis and management of classroom situations, as Figure 1 shown. Embodiment

[0032] Embodiment of the classroom situation analysis method based on computer vision Mid-term review class of the "Mechanical Drawing" course for the mechatronics major in vocational colleges Data preparation and initialization: Before the start of the mid-term review week, the system administrator communicates with the school's educational administration department in advance to obtain the student roster of Class 2 of the mechatronics major, and imports the student ID numbers and names through the system interface. To ensure the accuracy of facial features, a wide-angle camera is installed in the training classroom, and students are organized for centralized facial collection during break time. During the collection, the system performs data enhancement processing such as rotation and brightness adjustment on each image, and then extracts facial feature vectors, which are stored in a dedicated review class database to establish a basic data pool for classroom analysis of the review class, as Figure 6 shown.

[0033] When creating the "Mechanical Drawing" mid-term review class in the system, considering that the review class needs to specifically explain error-prone knowledge points, the teacher uploads the outline of the key chapters of this review through the graphical interface and associates the course with the class information. In addition, the teacher also imports the original class schedule data of this course from the school's educational administration system. The system automatically identifies that this is a review class type, providing a basis for subsequent weight allocation to more accurately analyze the classroom situation.

[0034] Classroom real-time data collection: At the start of the review class, the system automatically activates the cameras in the training classroom according to preset rules. The cameras capture video streams in real time at 1080P resolution, clearly showing the desktops and upper body postures of each student. After the video stream enters the preprocessing module, it first undergoes decoding to decompose the video into single-frame images, followed by scaling and denoising operations to make subsequent processing more efficient. Then, a dynamic face feature fusion algorithm is used to extract the face feature vectors from the video stream. The calculation formula is: , providing a more reliable basis for attendance recognition.

[0035] Multi-dimensional behavior analysis: Attendance recognition: The system compares the fused feature vectors with the student face features in the database through a dynamic face matching algorithm to confirm the identities of the attending students. The calculation formula is: , This algorithm comprehensively considers the similarity of the feature vectors, the continuity of the student's position changes in the classroom, and the changes in the distance of the face from the camera to ensure the accuracy of attendance recognition. During the comparison process, the system compares the feature matching score calculated from the current frame features with the dynamic threshold of the current frame. When , the student's identity is confirmed. For example, when the light is dim and the confidence level of feature extraction is low, the threshold will be appropriately relaxed to avoid misjudgment due to environmental factors. The recognition results are displayed in real time on the attendance panel at the teacher's end. A green indicator means signed in, and a red indicator means not recognized, facilitating the teacher to quickly grasp the attendance situation.

[0036] Abnormal behavior detection: The system runs the keypoint detection algorithm and the YOLOv10 algorithm simultaneously. Through the keypoint detection algorithm, the system continuously monitors the eye opening and closing degree (EAR value) of students. When the EAR value of a certain student is lower than the dynamic threshold set based on their baseline value, the closing eye time starts to be recorded. If it exceeds 3 seconds, it is determined as a dozing behavior. For the behavior of playing with mobile phones, the YOLOv10 model scans the mobile phone targets in the picture in real time. When a mobile phone is detected, by calculating the spatial position relationship between the mobile phone and the student's face, the specific student is determined, and whether it is an abnormal behavior is judged in combination with the duration. For example, a certain student takes out the mobile phone from the pocket at the 40th minute and operates it continuously for 2 minutes. The system will record the behavior of this student playing with the mobile phone during this time period.

[0037] Data Fusion and Quantitative Evaluation: After the review class, the system automatically aggregates the attendance data and abnormal behavior data of the entire class. Since this is a review class, the system automatically adjusts the weight coefficients of the comprehensive classroom situation analysis algorithm, increases the weight of the behavior trend score, and pays more attention to the students' concentration on the review content. The algorithm comprehensively calculates factors such as attendance rate, total duration of abnormal behavior (e.g., 12 minutes of cumulative dozing off, 5 minutes of cumulative mobile phone use), and behavior trend (e.g., an increase in abnormal behavior in the second half of the class, indicating an increase in students' fatigue), and obtains the comprehensive classroom situation index for this review class. The formula is: CAI AttendRate TrendScore.

[0038] At the same time, the system analyzes abnormal behaviors in the spatial and temporal dimensions and generates a behavior heat map. For example, the heat map shows a darker area in the right rear area of the classroom from the 60th to the 70th minute of the review class, indicating that the abnormal behaviors of students in this area are more concentrated, probably because they are farther away from the teacher's podium and their attention is easily distracted.

[0039] Visualization Presentation and Report Generation: The system generates a visualization dashboard on the teacher's interface. On the left is a real-time attendance statistics chart (a bar chart showing changes in the number of attendees), in the middle is a dynamic classroom behavior monitoring screen (marking the seat numbers and types of students with abnormal behaviors), and on the right is a line chart of abnormal behavior trends (showing the number of abnormalities in different time periods). When the teacher clicks on a record of an abnormal behavior, a detailed behavior log of the student will pop up, including the specific time, behavior type, and duration.

[0040] The report generation module automatically generates an analysis report for the mid-term review class based on a preset template. The beginning of the report summarizes the key content of the course and the attendance overview. The middle part details the distribution characteristics of abnormal behaviors (e.g., dozing off is concentrated around 3 pm, which coincides with the biological clock fatigue period), and compares with the abnormal data of the previous 3 lectures of this course, showing that the number of abnormal behaviors in the review class has decreased by 15%, indicating that the pertinence of the review content has had an effect. Finally, the report gives intelligent suggestions, such as suggesting that the teacher increase the interactive question-and-answer session around 3 pm to improve students' attention. The report supports one-click export to PDF format and can also be synchronized to the school's teaching management platform through the API interface for teaching supervisors and class teachers to view.

[0041] In summary, in the review class of the mechatronics major in vocational colleges, this method first imports student information, collects facial features, and creates a course. During class, it activates the camera to collect video streams, extracts facial features with an algorithm after preprocessing, performs attendance and abnormal behavior detection through a dynamic facial feature fusion algorithm, then fuses data and evaluates, and finally visualizes the data and generates a report, providing a complete solution for classroom teaching situation analysis.

[0042] The above are only the preferred embodiments of the present invention, and do not impose any form of limitation on the present invention. Although the present invention has been disclosed above with the preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to equivalent embodiments with equivalent changes within the scope of the technical solution of the present invention. However, as long as it does not depart from the content of the technical solution of the present invention, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A classroom situation analysis system based on computer vision, characterized in that, The system includes: a student information collection module, a course creation module, a classroom attendance module, an abnormal behavior detection module, and a classroom situation analysis module; The student information collection module: interfaces with the school's educational administration system, collects basic information such as the class, student ID, and name of the students. At the same time, it captures face images through a camera and extracts face feature vectors, and stores the student's basic information and face feature vectors in the database; The course creation module: is used for users to input course information, associates with class information, and supports graphical interaction to create and import data into the class scheduling system; The classroom attendance module: obtains the real-time video stream of the current class students through the classroom camera, uses the dynamic face feature fusion algorithm to extract the face feature vectors in the video stream, compares and identifies them with the student face feature vectors in the database, confirms the identities of the attending students, and automatically stores the attendance status and time information in the database; The abnormal behavior detection module: takes the video stream of the classroom camera as input, uses the key point detection algorithm and the YOLOv10 object detection algorithm to detect the abnormal behavior of students, and records data such as the basic information, abnormal type, occurrence time, and duration of the abnormal students; The classroom situation analysis module: integrates the attendance data and abnormal behavior data, quantitatively evaluates the classroom situation through the comprehensive classroom situation analysis algorithm, and generates a comprehensive report according to the analysis results. The report supports export in PDF and Word formats and interactive preview on the front-end interface.

2. The classroom situation analysis system based on computer vision according to claim 1, wherein In the student information collection module, a data augmentation strategy is adopted for face feature vector extraction, including rotating the original image, adjusting the brightness, and performing Gaussian blur processing. Three enhanced versions are generated for each sample and combined with the original image for feature extraction.

3. The classroom situation analysis system based on computer vision according to claim 1, characterized in that In the classroom attendance module, the dynamic face feature fusion algorithm is used to extract the face feature vector in the video stream. The face feature vector fusion function is defined as , and the calculation formula is: , where: is the fused face feature vector, is the static visual feature vector extracted from the current frame, is the temporal feature vector of the t-th frame in the past n frames, is the dynamic weight coefficient, n is the number of past frames, t is the index variable of the summation operation, is the index of a certain face detected in the current frame.

4. The classroom situation analysis system based on computer vision according to claim 3, characterized in that, In the classroom attendance module, the face feature vectors in the video stream are extracted through a dynamic face matching algorithm and compared with the student face feature vectors in the database to identify the identity of the attending students. The calculation formula is as follows: , where is the feature matching score of the th student, is the index of the student in the database, is the reference feature vector of the jth student in the database, is to calculate the cosine similarity between the fused feature and the reference feature, is the fused face feature vector, represents the index related to the face feature extracted in the current frame, is the distance attenuation coefficient, is the spatio-temporal continuity factor, and the formula is: , represents the time window size of the past frame, represents the fused feature vector related to the jth student in the past frames, is the offset expression of the time index, is the distance change rate between the current frame face and the camera, is the distance attenuation coefficient. When , the student identity is confirmed, is the dynamic threshold of the current frame.

5. The classroom situation analysis system based on computer vision according to claim 4, characterized in that, The dynamic threshold of the current frame in the classroom attendance module is calculated using the formula: , where confidence is the confidence of feature extraction for the current frame, with a value range of 0 - 1, is the time or frame index of the current frame.

6. The classroom situation analysis system based on computer vision according to claim 1, characterized in that, The specific steps for detecting drowsy behavior based on the key point detection algorithm in the abnormal behavior detection module are as follows: (1) Perform initialization detection and record the baseline EAR when each student's eyes are fully open; (2) Set the dynamic threshold to 60% of the baseline EAR; (3) Calculate the real-time EAR for each student in each frame of the image. If the EAR of a certain student in the current frame is less than the dynamic threshold, update the abnormal status and record the current time as last_update; (4) When it is detected that a student's eyes are closed for more than 3 seconds, record the start time and continuously track the duration of the abnormal behavior; (5) If the drowsy behavior of the current student is not detected in consecutive multiple frames of images, last_update is not updated within the tolerance time of 5 seconds. If the tolerance time is exceeded, the abnormal situation is marked as ended; (6) When stopping detection or performing a timed check, perform a timeout judgment on all unended abnormal records and update the end time.

7. The classroom situation analysis system based on computer vision according to claim 6, characterized in that, Adjustment of the dynamic threshold for drowsy detection in the abnormal behavior detection module: Fatigue accumulation: Record the historical drowsy frequency of students. When the number of drowsy times in a single week exceeds 3 times, the dynamic threshold is reduced by 5%; Head pose compensation: By detecting the head pitch angle, when the angle exceeds 15°, the EAR threshold is automatically adjusted to 55% of the baseline value.

8. The classroom situation analysis system based on computer vision according to claim 1, characterized in that The specific steps for detecting the behavior of playing with mobile phones based on the YOLOv10 model in the abnormal behavior detection module are as follows: (1) Use web crawler technology to crawl mobile phone image samples from the network, clean the image samples that do not meet the requirements, and divide them into training samples and test samples; (2) Use the labelimg tool to annotate the mobile phones in the cleaned training samples and test samples. The annotation content is the rectangular bounding box where the mobile phone is located in the image, and the label playphone is assigned to these bounding boxes; for each image, a YOLO-format file is generated after annotation, and the file contains the class ID of each object in the image and the coordinates of the corresponding bounding box; (3) Use the PyTorch framework and the YOLOv10 model to train the annotated training samples. The input of the YOLOv10 model is a 3-channel color image of 320×320 pixels, and the number of iterations is 100 times. Then use the test samples to detect the YOLOv10 model; (4) Use the trained YOLOv10 model to detect whether there are students playing with mobile phones in the classroom. By calculating the spatial center distance between the mobile phone bounding box and the face bounding box, associate the mobile phone playing detection result with the student, and introduce the judgment of the persistence of abnormal behavior to avoid misjudgment of instantaneous behavior.

9. A classroom situation analysis system based on computer vision according to claim 1, characterized in that In the classroom situation analysis module, the classroom situation is quantitatively evaluated through the comprehensive classroom situation analysis algorithm. The formula is: CAI AttendRate TrendScore, where: CAI is the comprehensive classroom situation evaluation value, AttendRate is the attendance rate, is the duration of the th abnormal behavior, is the total duration of abnormal behaviors, TrendScore is the behavior trend score, is the total classroom time, 、 、 are weight coefficients, and , and the weight is automatically adjusted according to the course type.

10. A method for analyzing classroom situations based on computer vision, which is applicable to a computer vision-based classroom situation analysis system according to any one of claims 1-9, characterized in that, The specific steps of this analysis method are as follows: S100, Data preparation and initialization: The system is connected to the school's educational administration system to import student information, collect face images through classroom cameras, and extract face feature vectors and store them in the database; users create courses through the graphical interface and associate class information, or import data from the course scheduling system to complete course configuration; S200, Classroom real-time data collection: The system automatically activates the camera to obtain the video stream, performs preprocessing such as decoding, scaling, and denoising on the video stream, performs rotation, brightness adjustment, and Gaussian blur processing, and uses the dynamic face feature fusion algorithm to extract the face feature vectors in the video stream; S300, Multi-dimensional behavior analysis: Based on the comparison of face feature vectors with the database for attendance recognition, adopt a dynamic threshold mechanism to improve the recognition accuracy; synchronously run the key point detection and YOLOv10 algorithms, detect drowsiness behavior based on the EAR value respectively, and identify the behavior of playing with mobile phones through object detection and face space association, and record the type, time, and duration of abnormal behavior; S400, Data fusion and quantitative evaluation: Integrate attendance and abnormal behavior data, automatically assign weights according to the course type to calculate the comprehensive index of the classroom situation, analyze the concentration and distribution characteristics of abnormal behavior, and generate a behavior heat map to intuitively display the problem area; S500, Visualization presentation and report generation: Realize data visualization, provide real-time monitoring large screen, trend analysis charts, and student portrait interactive components; automatically generate a report containing a summary, detailed analysis, comparison data, and intelligent suggestions based on a template, support PDF / Word export and integration with the teaching management system.

Images