Intelligent classroom student state monitoring system based on deep learning

Abstract

The invention discloses an intelligent classroom student state monitoring system based on deep learning. The intelligent classroom student state monitoring system comprises the following functions: student attendance realizes statistics of attendance students; through cheating detection, classroom test behavior supervision of students is realized; and through concentration degree analysis, the class attending state of the students in the class is quantitatively evaluated. Compared with a method of manually supervising the class states of the students by teachers, the method is convenient for efficiently counting the attendance number of the students in the class, standardizing the behaviors of the students in the class test and quantifying the learning concentration of the students in the class, thereby further improving the teaching efficiency of the teachers and the learning efficiency of the students.

Description

Technical Field

[0001] The present invention relates to the field of learning technology, and in particular to a smart classroom student status monitoring system based on deep learning. Background Art

[0002] In traditional classroom teaching, due to the obvious gap between the number of teachers and students, it is difficult for teachers to effectively supervise all students in the classroom. In order to effectively improve the quality of teaching, quantitative evaluation of students' learning process in the classroom has become a hot research issue. Different researchers have different research methods for the status of students in the classroom. Currently, the main research methods used are questionnaire surveys, interviews, and classroom actual inspections to study the status of students in the classroom. The questionnaire method is simple to obtain data, but there are certain problems with the authenticity of the data obtained, resulting in inaccurate results of students' learning status; the data obtained by the interview method is greatly influenced by the subjective factors of the interviewees; the actual classroom inspection method is time-consuming and labor-intensive, and the number of people observed at a single time is limited, so the obtained student classroom status is not convincing.

[0003] With the rapid development of deep learning, face detection and posture estimation technologies are becoming increasingly mature. As an important biological feature, portraits are widely used in many aspects of production and life. The improvement of hardware performance and processing performance has better promoted the application and development of image recognition technology. Therefore, image processing technology is very important in today's applications and research. Summary of the invention

[0004] The technical problem to be solved by the present invention is to propose a smart classroom student status monitoring system based on deep learning in response to the deficiencies of the above-mentioned prior art, which can realize student attendance and concentration analysis in class and cheating detection in exams, so as to better quantify and score students' classroom learning, build a more scientific teaching system, set up more reasonable teaching plans, and further improve the teaching quality.

[0005] To achieve the above object, the present invention provides the following technical solution: a smart classroom student status monitoring system based on deep learning, comprising the following method:

[0006] S1: Student attendance;

[0007] S2: Cheating detection;

[0008] S3: Classroom concentration analysis;

[0009] Preferably, the S1 student attendance realizes dynamic roll call, updates the number of attendance and absence at a certain time; reads information in the face library; uses the camera as the video source to read information in the video source; processes the streaming data of the video source in sequence, including the tasks of face detection, face feature extraction, and face matching; creates a sign-in task list based on known face data; each sign-in task consists of a name, a face image, and a corresponding list of signed-in and non-signed-in students, and counts the number of signed-in students and non-signed-in students; when a frame is not successfully read, reopens the video source.

[0010] Preferably, the content of the S2 cheating detection includes: passing notes, peeking, and looking around; taking pictures of the cheating scene, counting the number of people passing notes, peeking, and looking around at different time points, and taking pictures of the cheating scene.

[0011] Preferably, the S3 classroom concentration analysis: monitors and classifies students' behaviors, facial states, and head postures in class; behaviors include sitting upright, raising hands, relaxing, resting, and activities; faces include occluding, active, general, and negative; head postures include occluding, lowering the head in class, raising the head in class, lowering the head in self-study, raising the head in self-study, and looking up; dividing classes and self-study by comparing the number of students lowering their heads and the number of students raising their heads; constructing a behavior fuzzy matrix, a facial fuzzy matrix, and a head posture fuzzy matrix; classifying the recognition results of behavior, face, and head posture, and multiplying them with the corresponding fuzzy matrices to obtain the behavior level, facial level, and head posture level respectively; calculating the action information entropy, facial information entropy, and head posture information entropy, and using the softmax function to calculate the main factors of concentration, the main factors of concentration are a matrix composed of action information entropy, facial information entropy, and head information entropy; performing matrix multiplication on the levels of the main factors of concentration and the secondary factors of concentration to obtain the level of the main factors of concentration.

[0012] A smart classroom student status monitoring system based on deep learning, including the following methods:

[0013] S1. Learning behavior analysis: The system monitors students’ learning behaviors such as class participation, interaction frequency, and homework completion, analyzes students’ learning habits and preferences, and provides teachers with targeted teaching suggestions;

[0014] S2 Emotional change monitoring: With the help of biometric technologies, such as facial expression recognition and voice analysis, the system can monitor students' emotional changes in real time, helping teachers to promptly identify students' confusion or dissatisfaction and adjust teaching strategies;

[0015] S3. Concentration: Through technologies such as eye tracking and head posture monitoring, the system can assess students’ concentration and remind them to adjust their state to improve their learning efficiency.

[0016] S4. Learning effect evaluation: The system regularly tests students, collects students’ answer data, analyzes students’ learning effects and knowledge mastery, and provides data support for teaching evaluation;

[0017] S5. Personalized learning suggestions: Based on students’ learning data and performance, the system recommends personalized learning resources and learning paths to help students learn more effectively;

[0018] S6. Real-time feedback mechanism: The system can provide instant feedback to students and teachers, including learning progress, grade changes, learning difficulties, etc., to help students adjust their learning strategies in time and teachers improve their teaching methods;

[0019] S7. Adjustment of learning path: Based on the students’ learning progress and results, the system can dynamically adjust the students’ learning path to ensure that students learn along the path that best suits them.

[0020] S8. Intelligent decision support: The system provides teachers with intelligent decision support through big data analysis, such as analysis of the overall learning status of the class, suggestions for optimizing teaching strategies, etc., to help teachers make more scientific teaching decisions.

[0021] Compared with the prior art, the beneficial effects of the present invention are: classroom attendance of students through face recognition saves teachers the time-consuming manual roll call, cheating detection effectively simplifies students' cheating behavior during classroom tests, and concentration analysis helps teachers pay attention to students' learning status in class and improve teaching methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of the present invention;

[0023] Figure 2 The student attendance graph of the present invention;

[0024] Figure 3 is a cheating detection diagram of the present invention;

[0025] Figure 4 This is a classroom concentration analysis diagram of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Embodiment 1: Figure 1-4As shown, a smart classroom student status monitoring system based on deep learning includes the following methods:

[0028] S1: Student attendance;

[0029] S2: Cheating detection;

[0030] S3: Classroom concentration analysis;

[0031] Preferably, the S1 student attendance realizes dynamic roll call, updates the number of attendance and absence at a certain time; reads information in the face library; uses the camera as the video source to read information in the video source; processes the streaming data of the video source in sequence, including the tasks of face detection, face feature extraction, and face matching; creates a sign-in task list based on known face data; each sign-in task consists of a name, a face image, and a corresponding list of signed-in and non-signed-in students, and counts the number of signed-in students and non-signed-in students; when a frame is not successfully read, reopens the video source.

[0032] Preferably, the content of the S2 cheating detection includes: passing notes, peeking, and looking around; taking pictures of the cheating scene, counting the number of people passing notes, peeking, and looking around at different time points, and taking pictures of the cheating scene.

[0033] Preferably, the S3 classroom concentration analysis: monitors and classifies students' behaviors, facial states, and head postures in class; behaviors include sitting upright, raising hands, relaxing, resting, and activities; faces include occluding, active, general, and negative; head postures include occluding, lowering the head in class, raising the head in class, lowering the head in self-study, raising the head in self-study, and looking up; dividing classes and self-study by comparing the number of students lowering their heads and the number of students raising their heads; constructing a behavior fuzzy matrix, a facial fuzzy matrix, and a head posture fuzzy matrix; classifying the recognition results of behavior, face, and head posture, and multiplying them with the corresponding fuzzy matrices to obtain the behavior level, facial level, and head posture level respectively; calculating the action information entropy, facial information entropy, and head posture information entropy, and using the softmax function to calculate the main factors of concentration, the main factors of concentration are a matrix composed of action information entropy, facial information entropy, and head information entropy; performing matrix multiplication on the levels of the main factors of concentration and the secondary factors of concentration to obtain the level of the main factors of concentration.

[0034] A smart classroom student status monitoring system based on deep learning, including the following methods:

[0035] S1. Learning behavior analysis: The system monitors students’ learning behaviors such as class participation, interaction frequency, and homework completion, analyzes students’ learning habits and preferences, and provides teachers with targeted teaching suggestions;

[0036] S2 Emotional change monitoring: With the help of biometric technologies, such as facial expression recognition and voice analysis, the system can monitor students' emotional changes in real time, helping teachers to promptly identify students' confusion or dissatisfaction and adjust teaching strategies;

[0037] S3. Concentration: Through technologies such as eye tracking and head posture monitoring, the system can assess students’ concentration and remind them to adjust their state to improve their learning efficiency.

[0038] S4. Learning effect evaluation: The system regularly tests students, collects students’ answer data, analyzes students’ learning effects and knowledge mastery, and provides data support for teaching evaluation;

[0039] S5. Personalized learning suggestions: Based on students’ learning data and performance, the system recommends personalized learning resources and learning paths to help students learn more effectively;

[0040] S6. Real-time feedback mechanism: The system can provide instant feedback to students and teachers, including learning progress, grade changes, learning difficulties, etc., to help students adjust their learning strategies in time and teachers improve their teaching methods;

[0041] S7. Adjustment of learning path: Based on the students’ learning progress and results, the system can dynamically adjust the students’ learning path to ensure that students learn along the path that best suits them.

[0042] S8. Intelligent decision support: The system provides teachers with intelligent decision support through big data analysis, such as analysis of the overall learning status of the class, suggestions for optimizing teaching strategies, etc., to help teachers make more scientific teaching decisions.

[0043] Embodiment 2:

[0044] The present invention includes the functions of face registration, student attendance, cheating detection and classroom concentration analysis. To realize the face registration function, the frame data returned by the camera is read through the camera, and the face detection algorithm is used to obtain the face data. If multiple faces are detected, only the largest face is detected.

[0045] Preferably, a silent liveness detection algorithm is used to perform liveness detection on the face before face registration. When the silent liveness detection algorithm determines that it is a real face, the face position is correct, and the head posture is correct, feature extraction is performed on the face;

[0046] Preferably, when performing face registration, the information to be entered includes face picture, code, name, and class information; update, enter, and delete faces, and store the entered faces in the face database; to realize the student attendance function, read the frame data returned by the camera through the camera; calculate the distance between the face coding data, perform face matching based on the distance value, and determine the label and matching probability of each face; recognize multiple faces at the same time, find the face with the highest matching degree by comparing the distance values ​​between different faces, and generate the corresponding label and matching probability;

[0047] Preferably, the face recognition in the video stream is repeated multiple times, and the attendance list of students at different time points is dynamically updated;

[0048] Preferably, the attendance list and the absent student list are counted and updated in real time, and a list is generated for display;

[0049] It can monitor students' attendance in real time to avoid students skipping classes after completing a single attendance sign-in, and ensure students' learning time in class;

[0050] In order to realize the cheating detection function, the cheating content is judged including:

[0051] Students are passing notes, peeking, and looking around;

[0052] The posture estimation algorithm is used to classify student behaviors into normal sitting still, normal bowing head to write, normal stretching, lowering right hand, lowering left hand, raising right hand, raising left hand, standing up, raising hand, right hand stretching, left hand stretching, right hand stretching 2, left hand stretching 2, right turn, left turn, right head turn, left head turn, sleeping in class, and serious bowing head;

[0053] Preferably, the behaviors of right hand extension, left hand extension, right hand extension 2, and left hand extension 2 are classified as passing notes, right turn, left turn, right head turn, and left head turn are classified as looking around, severe head lowering is classified as looking down, and normal sitting still, normal head lowering to write, normal stretching, lowering the right hand, lowering the left hand, raising the right hand, raising the left hand, standing up, raising hands, and sleeping in class are classified as normal;

[0054] Preferably, the number of times cheating content appears at different time points is counted respectively, and a statistical graph showing the number of cheating students changing over time is generated;

[0055] To realize the function of classroom concentration analysis, the concentration of students in the classroom within the screen range;

[0056] The primary level of concentration is equal to the secondary level of concentration multiplied by the primary factor of concentration;

[0057] The main factors of concentration are obtained by normalizing the matrix consisting of action information entropy, facial information entropy, and head posture information entropy through the softmax function;

[0058] The secondary level of concentration is composed of behavioral level, facial level, and head posture level in columns to form a new two-dimensional array;

[0059] The behavior level is equal to the action classification multiplied by the action fuzzy matrix, the face level is equal to the face classification multiplied by the face fuzzy matrix, and the head posture level is equal to the head posture classification multiplied by the head posture fuzzy matrix;

[0060] Preferably, the original action recognition results are reclassified to obtain action classifications and the number of each category, the original face recognition results are reclassified to obtain face classifications and the number of each category, and the original head posture recognition results are used to obtain head posture classifications and the number of each category; the action classifications include: sitting upright, raising hands, relaxing, resting, and activities; the face classifications include: occlusion, positive, general, and negative; the head posture classifications include: occlusion, lowering the head in class, raising the head in class, lowering the head in self-study, raising the head in self-study, and looking up;

[0061] Preferably, when the number of people lowering their heads is greater than the number of people raising their heads, it is a self-study state, and when the number of people lowering their heads is less than the number of people raising their heads, it is a class state; the average value of the secondary level is calculated, the average value of the primary level is calculated and drawn on the interface image; in the above content, Yolov5 algorithm is used for image recognition, and AlphaPose is used to estimate the posture; in order to complete the above task scheduling and execution functions, a task scheduling framework is built, and task data can be processed by defining different modules;

[0062] Embodiment three:

[0063] The present invention also discloses a computer device, comprising: a memory for storing a computer program; a processor for executing the computer program to implement any of the above methods;

[0064] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above items is implemented;

[0065] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A smart classroom student status monitoring system based on deep learning, including the following methods: S1: Student attendance; S2: Cheating detection; S3: Analysis of classroom concentration.

2. According to a deep learning-based smart classroom student status monitoring system according to claim 1, it is characterized by: The S1 student attendance system implements dynamic roll call, updates the number of attendance and absences at a certain time; reads information from the face database; uses the camera as a video source to read information from the video source; processes the stream data of the video source in sequence, including tasks such as face detection, face feature extraction, and face matching; creates a sign-in task list based on known face data; each sign-in task consists of a name, a face image, and a corresponding list of signed-in and non-signed-in students, and counts the number of signed-in students and non-signed-in students; when a frame is not successfully read, the video source is reopened.

3. According to a deep learning-based smart classroom student status monitoring system according to claim 1, it is characterized by: S2: Cheating detection includes: passing notes, peeking, and looking around; taking pictures of the cheating scene, counting the number of people passing notes, peeking, and looking around at different time points, and taking pictures of the cheating scene.

4. According to a deep learning-based smart classroom student status monitoring system according to claim 1, it is characterized by: The S3 classroom concentration analysis: monitors and classifies students' behaviors, facial states, and head postures in class; behaviors include sitting upright, raising hands, relaxing, resting, and activities; faces include covering, positive, general, and negative; head postures include covering, lowering heads in class, raising heads in class, lowering heads during self-study, raising heads during self-study, and looking up; The students in class and those in self-study are divided by comparing the number of students who lower their heads and the number of students who raise their heads; the behavior fuzzy matrix, facial fuzzy matrix and head posture fuzzy matrix are constructed; the recognition results of behavior, face and head posture are classified and multiplied with the corresponding fuzzy matrices to obtain the behavior level, face level and head posture level respectively; the action information entropy, facial information entropy and head posture information entropy are calculated, and the softmax function is used to calculate the main factors of concentration. The main factors of concentration are a matrix composed of action information entropy, facial information entropy and head information entropy; the main factors of concentration and the secondary factors of concentration are matrix multiplied to obtain the main factor level of concentration.

5. A smart classroom student status monitoring system based on deep learning, including the following methods: S1. Learning behavior analysis: The system monitors students’ learning behaviors such as class participation, interaction frequency, and homework completion, analyzes students’ learning habits and preferences, and provides teachers with targeted teaching suggestions; S2 Emotional change monitoring: With the help of biometric technologies, such as facial expression recognition and voice analysis, the system can monitor students' emotional changes in real time, helping teachers to promptly identify students' confusion or dissatisfaction and adjust teaching strategies; S3. Concentration: Through technologies such as eye tracking and head posture monitoring, the system can assess students’ concentration and remind them to adjust their state to improve their learning efficiency. S4. Learning effect evaluation: The system regularly tests students, collects students’ answer data, analyzes students’ learning effects and knowledge mastery, and provides data support for teaching evaluation; S5. Personalized learning suggestions: Based on students’ learning data and performance, the system recommends personalized learning resources and learning paths to help students learn more effectively; S6. Real-time feedback mechanism: The system can provide instant feedback to students and teachers, including learning progress, grade changes, learning difficulties, etc., to help students adjust their learning strategies in time and teachers improve their teaching methods; S7. Adjustment of learning path: Based on the students’ learning progress and results, the system can dynamically adjust the students’ learning path to ensure that the students learn along the path that best suits them. S8. Intelligent decision support: The system provides teachers with intelligent decision support through big data analysis, such as analysis of the overall learning status of the class, suggestions for optimizing teaching strategies, etc., to help teachers make more scientific teaching decisions.

Images