A system and method for intelligent analysis and judgment of students' classroom behavior

Abstract

The present invention discloses a system and method for intelligent analysis and judgment of student classroom behavior, which relates to the technical field of classroom behavior recognition, and includes a data acquisition and storage module, a behavior analysis module, a monitoring time control module and an after-class feedback module; a target recognition algorithm, a labeling technology and a face recognition technology are used to generate a boundary box of the student position in a frame image per second, identify the student identity and record the coordinate range and name of the corresponding boundary box; a behavior recognition model is constructed through the labeling technology and the key point technology, and the corresponding student behavior is analyzed when different analysis signals are received; the face features of the teacher in the classroom are monitored through the face recognition technology to adjust the classroom behavior monitoring cycle; the number and total number of abnormal behaviors of each student are analyzed, and the designated students are specially marked; the present invention realizes intelligent behavior recognition of different types of students, and intelligent adjustment of the monitoring cycle under different circumstances.

Description

Technical Field

[0001] The present invention relates to the technical field of classroom behavior recognition, and in particular to a system and method for intelligently analyzing and judging students' classroom behavior. Background Art

[0002] Student classroom behavior refers to the various actions, reactions and attitudes that students display during classroom teaching in an educational environment; these behaviors not only affect students' learning outcomes, but also have an important impact on classroom atmosphere and teaching quality;

[0003] In the modern education system, classroom teaching is the core link of knowledge transfer and skill cultivation. Its quality and effect are directly related to students' learning outcomes and future development. However, with the continuous expansion of the scale of education, the number of students in a class has increased significantly. Traditional manual observation and evaluation methods can no longer meet the needs of comprehensive, real-time and objective classroom behavior analysis. Therefore, exploring and applying intelligent technology to assist classroom behavior analysis has become an important research direction in the field of education.

[0004] Existing student classroom behavior analysis often monitors the behavior of all students in the classroom at the same time, which makes the data very complex and difficult to process. At the same time, it is also impossible to use different monitoring methods for different types of students according to the actual situation of the students (such as location, whether abnormal movements often occur), which may not accurately reflect the actual situation of each student; in addition, most of the existing student classroom behavior analysis uses a fixed monitoring cycle to monitor the student's behavior in a class, and does not dynamically adjust the monitoring cycle according to whether the teacher is facing the students. That is, when the teacher is facing the students, the students' abnormal behavior usually decreases; and when the teacher is not facing the students, the students' abnormal behavior usually increases. The use of a fixed monitoring cycle ignores the impact of teacher behavior on student behavior. Summary of the invention

[0005] Technical issues solved

[0006] In response to the technical problems in the background technology, the present invention proposes a system and method for intelligent analysis and judgment of students' classroom behavior, which respectively set the first analysis signal attribute, the second analysis signal attribute and the third analysis signal attribute, and perform behavior detection operations on the corresponding students according to the type of signal received in the behavior detection unit; dynamically adjust the monitoring period interval of students' classroom behavior based on whether the identified teacher is facing the students; thereby solving the technical problems recorded in the background technology.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A student classroom behavior intelligent analysis and judgment system, comprising:

[0010] The data acquisition and storage module is used to obtain the video data captured by two high-definition cameras in the classroom, and after pre-processing, store the frame-per-second images of the video data captured by the high-definition cameras configured in front of the classroom in real time into the monitoring image table of this course; it is also used to store the identified abnormal behaviors into the abnormal behavior library of the corresponding students;

[0011] Behavior analysis module, including target recognition unit and behavior detection unit;

[0012] The target recognition unit is used to generate a bounding box representing the position of the students, and identify the students in each bounding box, and store the coordinate range data of the extracted bounding box in the corresponding frame image and the name data of the matching students in the temporary student information table;

[0013] The behavior detection unit trains a behavior recognition model based on the behavior of the corresponding student in each boundary box and the key point structure diagram; and recognizes the behavior of the corresponding student according to the behavior recognition model and in combination with different analysis signals;

[0014] The monitoring time control module is used to perform real-time detection of the frames per second image captured by the high-definition camera installed at the back of the classroom, determine whether a face appears and whether the facial features of the face match the facial features of the class teacher, and adjust the classroom behavior monitoring cycle based on the results;

[0015] The after-class feedback module analyzes the number of abnormal behaviors of students and the total number of students with abnormal behaviors at the end of the class, adds or removes special marks for students, and determines whether to send corresponding information to the class teacher and subject teachers.

[0016] Specifically, the monitoring image table stores frame images per second that are continuously updated as the length of video data captured by a high-definition camera located in front of the classroom increases, as well as time attribute data, first analysis signal attribute data, second analysis signal attribute data, and third analysis signal attribute data.

[0017] Specifically, the values ​​of the first analysis signal attribute, the second analysis signal attribute and the third analysis signal attribute are "T" or "F", "T" means sending the corresponding analysis signal, and "F" means not sending the corresponding analysis signal; when a new frame-per-second image is acquired, starting from the first analysis signal attribute, the values ​​of the first analysis signal attribute, the second analysis signal attribute and the third analysis signal attribute are detected in turn, and the corresponding signal with the value "T" that first appears is sent to the behavior detection unit.

[0018] Specifically, when the target recognition unit receives the in-position instruction, it obtains the pre-processed next second frame image from the video data shot by the high-definition camera configured in front of the classroom; identifies the position information of all students in the image based on the YOLOv8 algorithm and generates a corresponding bounding box, and records the coordinate range data of the bounding box in the image;

[0019] The students in each bounding box are identified based on face recognition technology. The similarity between the facial features of the students in the bounding box and the facial features in all the student photos in the class is calculated in turn to determine the identity of the student who best matches the current facial features, and the name data of the student corresponding to each bounding box is recorded.

[0020] Further, when the first analysis signal is received, the behavior of all students in the classroom is identified;

[0021] When the second analysis signal is received, the behavior of students sitting in the back row and sitting by the window in the classroom is identified;

[0022] When the third analysis signal is received, the behavior of the specially marked students in the classroom is identified;

[0023] After receiving the analysis signal, based on the coordinate range data of the bounding boxes corresponding to several students to be identified in the image, several sub-images corresponding to each bounding box area are split from the frame image corresponding to the current moment, and behavior recognition operations are performed on the students in the sub-images based on the behavior recognition model.

[0024] Specifically, the OpenPose model is used to annotate the corresponding key points of the students in each bounding box in all the frame-per-second images in the historical data obtained, and the key points include 12 feature points, namely, nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist and mouth. All the annotated key points are connected by line segments to obtain a key point structure diagram; each bounding box in all the frame-per-second images in the historical data obtained is annotated with the corresponding student behavior type; all the bounding box data are summarized, and the behavior recognition model is obtained by establishing a training set, a validation set and a test set for training.

[0025] Specifically, the frame-per-second images captured by the high-definition camera at the back of the classroom are acquired in real time, and the OpenCV software is used to perform facial recognition on the frame-per-second images acquired in real time. If a face is detected in the image, the area containing the entire face is cropped out; the area containing the entire face is sequentially compared with the photo files of all the teachers in the current class. If the recognized facial features match the facial features of one of the teachers in the current class, the classroom student behavior is monitored and identified according to the standard monitoring plan, specifically: T1=2*T2 and T2>2; T3=1; the unit of the monitoring period is seconds and they are all positive integers, where T1, T2, and T3 are the first, second, and third monitoring period intervals, respectively;

[0026] If the recognized facial features do not match the facial features of all teachers in the current class, or if it is detected that no face appears in the frame-per-second image taken by the high-definition camera configured at the back of the classroom, the first and second monitoring cycle intervals are adjusted to T1=T2=T3=1.

[0027] Specifically, at the end of each class, the total number of abnormal behaviors of each student in that class is counted. i , set the abnormal behavior threshold n0, and compare the total number of abnormal behaviors of each student in this class n i The relationship between the number of abnormal behaviors and the threshold value n0, if n i n0, then the student will be specially marked; if n i If n0 and the student has been specially marked in this class, the special mark of the student will be cleared; if a student has been specially marked for n consecutive classes, a reminder message will be sent to his class teacher.

[0028] Furthermore, at the end of each class, the total number of students with abnormal behavior in the class is counted. If m m0 / 2, then strengthen management information will be sent to the teacher of the class, where m0 represents the total number of students in the current class.

[0029] A method for intelligent analysis and judgment of students' classroom behavior, comprising:

[0030] Step 1: Obtain the frame-per-second images of the videos shot by two high-definition cameras in the classroom, and after pre-processing, store the frame-per-second images of the video data shot by the high-definition camera configured in front of the classroom in real time into the monitoring image table of this course;

[0031] Step 2: Generate a bounding box representing the student's location based on the YOLOv8 algorithm and Labelme annotation technology, and identify the students in each bounding box based on face recognition technology, and store the coordinate range data of the extracted bounding box in the corresponding frame image and the name data of the matching student in the temporary student information table;

[0032] Step 3: Based on the Labelme annotation technology, the bounding box corresponding to each student is annotated, and the key point structure diagram of each student is obtained based on some key points of COCO and the OpenPose model. After training, a behavior recognition model is obtained; the behavior recognition of the corresponding student is performed based on the behavior recognition model and combined with different analysis signals;

[0033] Step 4: Based on face recognition technology, the frame-per-second image captured by the high-definition camera at the back of the classroom is detected in real time to determine whether a face appears and whether the facial features of the face match the facial features of the class teacher, and the classroom behavior monitoring cycle is adjusted according to the results;

[0034] Step 5: At the end of the class, analyze the number of abnormal behaviors of each student and the total number of students with abnormal behaviors, add and remove special marks for students, and determine whether to send corresponding information to the class teacher and subject teachers.

[0035] (III) Beneficial effects

[0036] The present invention provides a system and method for intelligent analysis and judgment of students' classroom behavior, which has the following beneficial effects:

[0037] 1. Through the deployment of high-definition cameras, the facial features and hand movements of students in the classroom, as well as the activities of teachers, can be fully and accurately captured, providing a high-quality data basis for subsequent behavior analysis; by setting up a classroom monitoring video library and a student abnormal behavior library, comprehensive monitoring and recording of classroom behavior is achieved, which is convenient for teachers to understand students' classroom performance in a timely manner and discover potential problems; the flexible analysis signal sending mechanism can dynamically adjust the analysis strategy according to actual needs, improving the practicality and flexibility of the system;

[0038] 2. By integrating the YOLOv8 algorithm, Labelme annotation technology, face recognition technology and OpenPose model, it can efficiently and accurately identify each student in the classroom and monitor and analyze their behavior in real time. This not only improves the accuracy and real-time performance of behavior recognition, but also greatly enhances the practicality and reliability of the system. The system can flexibly identify the behavior of all students, students in specific areas or specially marked students based on the different analysis signals received, meeting the needs of classroom behavior monitoring for different types of students.

[0039] 3. By real-time monitoring of the frames per second image taken by the high-definition camera at the back of the classroom, and using face recognition technology to determine whether the teacher's face appears in the frames per second image, the system can intelligently adjust the classroom behavior monitoring cycle; when the teacher's face is recognized, periodic monitoring is carried out according to the preset standard monitoring plan, which not only ensures comprehensive monitoring of student behavior, but also avoids unnecessary real-time detection, thereby saving computing resources; when the teacher's face is not recognized or the recognized face does not match any teacher in the current class, the system can quickly adjust the monitoring cycle and realize real-time detection, ensuring that in the absence of direct teacher supervision, students' abnormal behavior can be discovered and handled in a timely manner;

[0040] 4. Through the after-class feedback module, it can automatically analyze the abnormal behavior data of students in each class, and specially mark or clear the students according to the preset threshold. This helps teachers quickly identify students who need special attention and adjust teaching strategies in time; when the number of students with abnormal behavior in a class exceeds half of the total number of students in the class, the system will automatically send strengthened management information to the teacher in charge, reminding the teacher to strengthen control and management of the class, thereby effectively reducing abnormal behavior in the classroom and improving teaching quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of the structure of a student classroom behavior intelligent analysis and judgment system provided by the present invention;

[0042] Figure 2 The present invention provides a flowchart for the implementation of a method for intelligent analysis and judgment of students' classroom behavior. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] refer to Figure 1 The present invention provides a system for intelligent analysis and judgment of student classroom behavior, comprising:

[0045] The data acquisition and storage module obtains the video data captured by two high-definition cameras in the classroom, and after pre-processing, stores the frame-per-second images of the video data captured by the high-definition camera configured in front of the classroom in real time into the monitoring image table of this course; it is also used to store the identified abnormal behaviors into the abnormal behavior library of the corresponding students;

[0046] A high-definition camera is respectively arranged at the front and back sides of the classroom, and the installation positions of the two high-definition cameras are determined; the high-definition camera arranged at the front of the classroom is placed above the front blackboard, and the horizontal position is the center of the front blackboard, and the height of the high-definition camera is adjusted so that the shooting range of the high-definition camera can completely cover the entire facial features and hand movements of each student in the class; the high-definition camera arranged at the back of the classroom is placed above the back blackboard, and the horizontal position is the center of the back blackboard, and the height of the high-definition camera is adjusted so that the shooting range of the high-definition camera can completely cover the entire front blackboard; wherein, the high-definition camera arranged at the front of the classroom is used to shoot the facial features and hand movements of each student in the classroom, and the high-definition camera arranged at the back of the classroom is used to shoot the teacher in the classroom;

[0047] Establish a classroom monitoring video library to store the frame images per second of the video data captured by the high-definition camera installed in front of the classroom; it should be noted that the frame image per second refers to a single frame image corresponding to every second in the video;

[0048] When a class begins, a monitoring image table of this class is established in the classroom monitoring video library. As the duration of the video data captured by the high-definition camera installed in front of the classroom increases, the frame-per-second images in the video are stored in the monitoring image table of this class in real time. That is, when the class is in the first minute, 60 frame-per-second images have been stored in the monitoring image table of this class, corresponding to single-frame images of the video data captured by the high-definition camera installed in front of the classroom at the first second, the second second, ... and the 60th second of the class; in addition, each frame-per-second image in the monitoring image table also corresponds to a unique time attribute, a first analysis signal attribute, a second analysis signal attribute and a third analysis signal attribute;

[0049] It should be noted that when the frame-per-second image is obtained, the image is preprocessed, including data cleaning, denoising and enhancement operations, to retain the key information in the image (here, the student part in the image), while reducing interference factors and improving the accuracy of abnormal behavior recognition;

[0050] It should be noted that the time of each frame image per second is recorded in the form of how long after the class starts, such as the time of a single frame image 30 seconds after the class starts is 30, and the time of a single frame image 20 minutes after the class starts is 1200;

[0051] It should be noted that the first analysis signal attribute, the second analysis signal attribute and the third analysis signal attribute respectively indicate whether to send a corresponding analysis signal to the behavior detection unit in the behavior analysis module, and the values ​​are "T" or "F", "T" indicates sending a corresponding analysis signal, and "F" indicates not sending a corresponding analysis signal; when a new frame-per-second image is acquired, starting from the first analysis signal attribute, the values ​​of the first analysis signal attribute, the second analysis signal attribute and the third analysis signal attribute are detected in turn, and the corresponding signal with the value "T" appearing first is sent to the behavior analysis module; for example: if the values ​​of the first analysis signal attribute and the third analysis signal attribute are both "T", and the value of the second analysis signal attribute is "F", the first analysis signal is sent to the behavior detection unit in the behavior analysis module;

[0052] A database of students' abnormal behaviors is established, which includes a statistical table of abnormal behaviors of each student. Abnormal behaviors refer to abnormal behaviors performed by students during class, including whispering, lying down, turning heads, eating, etc., which are specifically identified by the abnormal behavior analysis module; each student's abnormal behavior statistical table stores the abnormal behaviors identified by the student in class, as well as the class subject and time corresponding to each abnormal behavior; if a student whispers at the 40th second of the third class on September 1, and the class is a Chinese class, the whispering behavior will be recorded in the student's abnormal behavior statistical table, and the abnormal behavior attribute will be recorded as "whispering", the time attribute will be recorded as "the 40th second of the third class on September 1", and the class subject attribute will be recorded as "Chinese".

[0053] Through the deployment of high-definition cameras, the facial features and hand movements of students in the classroom, as well as the activities of teachers, can be fully and accurately captured, providing a high-quality data basis for subsequent behavioral analysis; by setting up a classroom monitoring video library and a student abnormal behavior library, comprehensive monitoring and recording of classroom behavior is achieved, which is convenient for teachers to understand students' classroom performance in a timely manner and discover potential problems; the flexible analysis signal sending mechanism can dynamically adjust the analysis strategy according to actual needs, improving the practicality and flexibility of the system.

[0054] The behavior analysis module includes a target recognition unit and a behavior detection unit. The target recognition unit generates a bounding box representing the student's position based on the YOLOv8 algorithm and the Lableme annotation technology, and identifies the student in each bounding box based on the face recognition technology, and stores the coordinate range data of the extracted bounding box in the corresponding frame image and the name data of the matching student in the temporary student information table. The behavior detection unit performs behavior annotation on the bounding box corresponding to each student based on the Lableme annotation technology, obtains the key point structure diagram of each student based on some key points of COCO and the OpenPose model, and obtains the behavior recognition model after training. The behavior recognition model is used to identify the corresponding student's behavior in combination with different analysis signals.

[0055] The behavior analysis module includes the following steps:

[0056] Step 201: At the beginning of each class, after the teacher confirms that all students are in their seats, the mobile terminal sends a take-place instruction, and the target recognition unit in the behavior analysis module is responsible for receiving the take-place instruction; after the target recognition unit receives the take-place instruction, it obtains the pre-processed next second frame image from the video data taken by the high-definition camera configured in front of the classroom, and identifies the position information of all students in the image based on the YOLOv8 algorithm, generates a corresponding bounding box, and records the coordinate range data of the bounding box in the image;

[0057] The YOLOv8 algorithm is a target detection algorithm that uses a single-stage detection mechanism to transform the target detection problem into a regression problem. The location and category of the target can be predicted through a single forward propagation. It has the advantages of fast speed, good real-time performance, and high accuracy. In the present invention, it is used to identify each student in the classroom, specifically:

[0058] Randomly obtain several history classroom video data from the classroom monitoring video library, requiring that the subjects, classrooms, and students corresponding to each video data are not exactly the same, and obtain several corresponding frames per second images from the monitoring image table of these history classroom video data. Each frame per second image may contain different student information and different student postures; use Labelme image annotation software to annotate each frame per second image. In the target detection of students, only the area where each student is located in each frame per second image needs to be marked with a rectangular bounding box, and the annotated frame per second image file is converted into a training mode; summarize all the converted frame per second image files, randomly divide them into training set, validation set and test set in a ratio of 6:2:2, and use the frame per second image files in the training set for training. During the training process, adjust the model parameters such as learning rate, batch size, number of training rounds, etc. according to the performance of the validation set; use the test set to evaluate the performance of the trained model, and the evaluation indicators include accuracy P, recall R, F1 score and mean average precision mAP; according to the evaluation results, tune the model, such as adjusting the model architecture, increasing training data, improving data enhancement strategy, etc., to obtain the final target recognition model;

[0059] Step 202, identifying the students in each bounding box based on face recognition technology; face recognition technology is based on the photos in the class student list, and the identity of the students in each bounding box is recognized through steps such as face detection, alignment, feature extraction and identity recognition; the photo files of all students are obtained from the student list of the current class, and imported into the face recognition software; the OpenCV software is used to identify the faces of the students from each generated bounding box and crop out the area containing the entire facial features; the extracted area containing the entire facial features is sequentially compared with the photo files of each student in the current class, and the identity of the student who best matches the current facial features is determined by calculating the similarity between the facial features; the name data of the student corresponding to each bounding box is recorded;

[0060] It should be noted that the coordinate range data of the extracted bounding box in the image and the corresponding student name data are stored in the temporary student information table. Whenever a new class starts, the data in the temporary student information table is cleared;

[0061] Step 203: Perform corresponding behavior analysis operations according to the received signal type, specifically:

[0062] When the first analysis signal is received, the behavior of all students in the classroom is identified based on the behavior identification model;

[0063] When the second analysis signal is received, the behavior of students sitting in the back row and sitting by the window in the classroom is identified based on the behavior recognition model;

[0064] When the third analysis signal is received, the behavior of the specially marked students in the classroom is identified based on the behavior recognition model;

[0065] It should be noted that the behavior recognition model is used to analyze the behavior of each student in the classroom to identify the abnormal behavior of students during class. Specifically, the behavior type at the corresponding moment is obtained based on the key point structure diagram of each student in the frame image per second. The specific steps of constructing the behavior recognition model are:

[0066] Step 2031, obtain a number of frames per second images obtained from the monitoring image table of the history classroom video data in step 201 and marked with bounding boxes, and use Labelme image annotation software again to annotate each bounding box in each frame per second image, and specifically mark the behavior type of the corresponding student in each bounding box, including "raising head to listen to class", "lowering head to write", "lowering head to read", "whispering", "lying down", "looking out the window", "eating" and "lowering head to play with mobile phone", a total of 8 behaviors, of which the first 3 behaviors are normal behaviors, and the last 5 behaviors are abnormal behaviors. These abnormal behaviors To represent students; in addition, with the help of some key points in COCO key points, including nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, and a total of 12 feature points of the mouth, the OpenPose model is used to annotate the corresponding key points of the students in each bounding box, and all the annotated key points are connected by line segments to obtain a key point structure diagram, and the line segments between the key points represent joints; the OpenPose model is a human posture estimation framework based on deep learning, which can detect and track human key points in images or videos in real time;

[0067] Step 2032: Summarize all bounding box data, each of which contains a key point structure diagram and a corresponding behavior type, and convert all bounding box data into a file mode for training; randomly divide the converted bounding box file data into a training set, a validation set, and a test set in a ratio of 6:2:2, and use the bounding box file data in the training set for training. During the training process, adjust model parameters such as learning rate, batch size, number of training rounds, etc. according to the performance of the validation set to optimize the performance of the model; use the test set to evaluate the performance of the trained model, and the evaluation indicators include accuracy P, recall rate R, F1 score, and mean average precision mAP; according to the evaluation results, tune the model, such as adjusting the model architecture, increasing training data, improving data enhancement strategy, etc., to obtain the final behavior recognition model;

[0068] It should be noted that when the first analysis signal is received, it means that in the frame image corresponding to the current moment, the value of the first analysis signal attribute is "T", which means that the current moment is the moment for behavior recognition of all students in the classroom; specifically, based on the coordinate range data of the bounding box corresponding to each student in step 201 in the image, a sub-image of the corresponding coordinate range is split from the frame image corresponding to the current moment, and a behavior recognition operation is performed; the behavior recognition operation is specifically: using the OpenPose model to annotate the key points of the students in each split sub-image, and connecting all the annotated key points through line segments to obtain a key point structure diagram in each sub-image, that is, a key point structure diagram of each student, and performing behavior recognition on the key point structure diagram of each student based on the behavior recognition model;

[0069] When the second analysis signal is received, it indicates that in the frame image corresponding to the current moment, the value of the second analysis signal attribute is "T" and the value of the first analysis signal attribute is "F", that is, it indicates that the current moment is the moment for behavior recognition of students in the back row and by the window in the classroom, where the back row refers to the last two rows, and by the window refers to the innermost two columns (i.e., the two columns closest to the inner windows, and the inner windows are windows on the side without doors); the name data of the students whose seats are in the last two rows and the innermost two columns are obtained from the class student table, and the coordinate range data of the corresponding student's boundary box in the image is obtained from the student information table of the current class; the sub-images corresponding to these coordinate ranges are split from the frame image corresponding to the current moment, and the behavior recognition operation is performed;

[0070] When the third analysis signal is received, it indicates that in the frame image corresponding to the current moment, the value of the third analysis signal attribute is "T", the value of the second analysis signal attribute is "F", and the value of the first analysis signal attribute is "F", which means that the current moment is the moment for behavior recognition of the specially marked student; all student name data in the table are obtained from the special student table of the current class, and the coordinate range data of the corresponding student's boundary box in the image is obtained from the student information table of the current class; sub-images corresponding to these coordinate ranges are split from the frame image corresponding to the current moment, and behavior recognition operations are performed;

[0071] Step 204: If it is detected that the behavior type of a student is abnormal behavior, the corresponding abnormal behavior type and current time are recorded in the abnormal behavior statistics table of the student.

[0072] By integrating the YOLOv8 algorithm, Lableme annotation technology, face recognition technology and OpenPose model, it is possible to efficiently and accurately identify each student in the classroom and monitor and analyze their behavior in real time. This not only improves the accuracy and real-time performance of behavior recognition, but also greatly enhances the practicality and reliability of the system. The system can flexibly identify the behavior of all students, students in specific areas or specially marked students based on the different analysis signals received, meeting the classroom behavior monitoring needs for different types of students.

[0073] The monitoring time control module uses face recognition technology to perform real-time detection of the frames per second image captured by the high-definition camera at the back of the classroom, determine whether a face appears and whether the facial features of the face match the facial features of the class teacher, and adjust the behavior monitoring cycle based on the results;

[0074] The frame-per-second images captured by the high-definition camera at the back of the classroom are acquired in real time, and the OpenCV software is used to perform facial recognition on the frame-per-second images acquired in real time. If a face is detected in the image, the area containing the entire face is cropped; the area containing the entire face is compared with the photo files of all the teachers in the current class in turn, and the similarity between the facial features is calculated to determine whether the recognized facial features match the facial features of one of the teachers in the current class; if the recognized facial features match the facial features of one of the teachers in the current class, it means that the substitute teacher is facing the students at this time, and there is no need to detect the student behavior in real time. The standard monitoring plan is used to monitor and identify the student behavior in the classroom, specifically:

[0075] Set the first monitoring period and the interval as T1, that is, the monitoring period for behavior recognition of all students in the classroom. Whenever entering a monitoring time point, set the value of the first analysis signal attribute of the frame image at the corresponding time point to "T";

[0076] Set the second monitoring period and the interval is T2, that is, the monitoring period for behavior recognition of the last two rows and the innermost two columns of students in the classroom. Whenever entering the monitoring time point, set the value of the second analysis signal attribute of the frame image at the corresponding time point to "T";

[0077] A third monitoring period is set with an interval of T3, that is, a monitoring period for behavior recognition of specially marked students in the classroom. Whenever a monitoring time point is entered, the value of the third analysis signal attribute of the frame image at the corresponding time point is set to "T";

[0078] Wherein, T1=2*T2 and T2>2; T3=1; the unit of the monitoring period is seconds and is a positive integer. The specific value of the second monitoring period T2 is determined by the head teacher of each class;

[0079] If the recognized facial features do not match the facial features of all teachers in the current class, or if it is detected that no face appears in the frame-per-second image taken by the high-definition camera installed at the back of the classroom, it is necessary to detect the student's classroom behavior in real time and adjust the settings of the first monitoring period T1 and the second monitoring period T2. At this time, T1=T2=T3=1.

[0080] By real-time monitoring of the frames per second image taken by the high-definition camera installed at the back of the classroom, and using face recognition technology to determine whether the teacher's face appears in the frames per second image, the system can intelligently adjust the student classroom behavior monitoring cycle; when the teacher's face is recognized, periodic monitoring is carried out according to the preset standard monitoring plan, which not only ensures comprehensive monitoring of student behavior, but also avoids unnecessary real-time detection, thereby saving computing resources; when the teacher's face is not recognized or the recognized face does not match that of all the teachers in the current class, the system can quickly adjust the monitoring cycle and realize real-time detection, ensuring that in the absence of direct teacher supervision, students' abnormal behavior can be discovered and handled in a timely manner.

[0081] The post-class feedback module analyzes the number of abnormal behaviors of students and the total number of students with abnormal behaviors at the end of the class, adds or removes special marks for students, and determines whether to send corresponding information to the head teacher and the subject teacher;

[0082] At the end of each class, obtain all abnormal behavior data that occurred in the class from each student's abnormal behavior statistics table, and count the total number of abnormal behaviors that occurred in each student in the class n i , each class teacher sets the abnormal behavior threshold n0 by himself, and compares the total number of abnormal behaviors of each student in the class n i The relationship between the number of abnormal behaviors and the threshold value n0, if n i n0, the student will be specially marked for more frequent behavior monitoring in the next class; if n i If n0 and the student has been specially marked in this class, the special mark of the student will be cleared; if a student has been specially marked for n consecutive classes, a reminder message will be sent to his class teacher, and n is also set by the class teacher of each class;

[0083] At the end of each class, obtain all abnormal behavior data that occurred in the class from each student's abnormal behavior statistics table, and count the total number of students with abnormal behavior in the class m. If m m0 / 2 means that more than half of the students in the class have abnormal behavior, and a strengthening management message is sent to the teacher of the class, where m0 represents the total number of students in the current class.

[0084] Through the after-class feedback module, it can automatically analyze the abnormal behavior data of students in each class, and specially mark or clear the students according to the preset threshold. This helps teachers quickly identify students who need special attention and adjust teaching strategies in time; when the number of students with abnormal behavior in a class exceeds half of the total number of students in the class, the system will automatically send strengthened management information to the teacher in charge, reminding the teacher to strengthen control and management of the class, thereby effectively reducing abnormal behavior in the classroom and improving teaching quality.

[0085] refer to Figure 2 The present invention also provides a method for intelligently analyzing and judging students' classroom behavior, comprising:

[0086] Step 1: Obtain the frame-per-second images of the videos shot by two high-definition cameras in the classroom, and after pre-processing, store the frame-per-second images of the video data shot by the high-definition camera configured in front of the classroom in real time into the monitoring image table of this course;

[0087] Step 2: Generate a bounding box representing the student's location based on the YOLOv8 algorithm and Labelme annotation technology, and identify the students in each bounding box based on face recognition technology, and store the coordinate range data of the extracted bounding box in the corresponding frame image and the name data of the matching student in the temporary student information table;

[0088] Step 3: Based on the Labelme annotation technology, the bounding box corresponding to each student is annotated, and the key point structure diagram of each student is obtained based on some key points of COCO and the OpenPose model. After training, a behavior recognition model is obtained; the behavior recognition of the corresponding student is performed according to the behavior recognition model and combined with different analysis signals, and the identified abnormal behavior is stored in the abnormal behavior library of the corresponding student;

[0089] Step 4: Based on face recognition technology, the frame-per-second image captured by the high-definition camera at the back of the classroom is detected in real time to determine whether a face appears and whether the facial features of the face match the facial features of the class teacher, and the classroom behavior monitoring cycle is adjusted according to the results;

[0090] Step 5: At the end of the class, analyze the number of abnormal behaviors of each student and the total number of students with abnormal behaviors, add and remove special marks for students, and determine whether to send corresponding information to the class teacher and subject teachers.

[0091] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer storage medium or transmitted via a computer storage medium.

[0092] Computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. Computer storage media can be any available media that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state drives (SSDs)).

[0093] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An intelligent analysis and judgment system for students' classroom behavior, characterized by: include: The data acquisition and storage module is used to obtain the video data captured by two high-definition cameras in the classroom, pre-process the video data, and store the frame-per-second images of the video data captured by the high-definition cameras configured in front of the classroom in real time into the monitoring image table of this course; It is also used to store the identified abnormal behavior in the abnormal behavior library of the corresponding student; Behavior analysis module, including target recognition unit and behavior detection unit; The target recognition unit generates a bounding box representing the student's position, and identifies the student in each bounding box, and stores the coordinate range data of the extracted bounding box in the corresponding frame image and the name data of the matching student in the temporary student information table; The behavior detection unit trains a behavior recognition model based on the behavior of the corresponding student in each bounding box and the key point structure diagram; and recognizes the behavior of the corresponding student according to the behavior recognition model and in combination with different analysis signals; The monitoring time control module is used to perform real-time detection of the frames per second image taken by the high-definition camera installed in front of the classroom, determine whether a face appears and whether the facial features of the face match the facial features of the class teacher, and adjust the classroom behavior monitoring cycle according to the judgment results; The frames per second image taken by the high-definition camera configured at the back of the classroom is acquired in real time, and the OpenCV software is used to perform facial recognition on the frames per second image acquired in real time. If a face is detected in the image, the area containing the entire face is cropped; the area containing the entire face is compared with the photo files of all the teachers in the current class in turn. If the recognized facial features match the facial features of one of the teachers in the current class, the classroom student behavior is monitored and identified according to the standard monitoring plan, specifically: T1=2*T2 and T2>2; T3=1; the unit of the monitoring cycle is seconds and they are all positive integers, where T1, T2, and T3 are the first, second, and third monitoring cycle intervals respectively; the first monitoring cycle interval refers to the monitoring cycle interval for behavior recognition of all students in the classroom; the second monitoring cycle interval refers to the monitoring cycle interval for behavior recognition of students in the last two rows and the innermost two columns in the classroom; the third monitoring cycle interval refers to the monitoring cycle interval for behavior recognition of specially marked students in the classroom; If the recognized facial features do not match the facial features of all teachers in the current class, or if it is detected that no face appears in the frame-per-second image taken by the high-definition camera configured at the back of the classroom, the first and second monitoring cycle intervals are adjusted to T1=T2=T3=1; The post-class feedback module is used to analyze the number of abnormal behaviors of students and the total number of students with abnormal behaviors at the end of the class, add and remove special marks for students, and determine whether to send corresponding information; At the end of each class, count the total number of times each student has abnormal behavior in that class n i , set the abnormal behavior threshold n0, and compare the total number of abnormal behaviors of each student in this class n i The relationship between the number of abnormal behaviors and the threshold value n0, if n i n0, then the student will be specially marked; if n i If n0 and the student has been specially marked in this class, the special mark of the student will be cleared; if a student has been specially marked for n consecutive classes, a reminder message will be sent to his class teacher.

2. The intelligent analysis and judgment system for student classroom behavior according to claim 1, characterized in that: The monitoring image table stores frame images per second that are continuously updated as the length of the video data captured by the high-definition camera located in front of the classroom increases, time attribute data, first analysis signal attribute data, second analysis signal attribute data, and third analysis signal attribute data.

3. A student classroom behavior intelligent analysis and judgment system as claimed in claim 2, characterized in that: The values ​​of the first analysis signal attribute, the second analysis signal attribute and the third analysis signal attribute are "T" or "F", "T" means sending the corresponding analysis signal, and "F" means not sending the corresponding analysis signal; when a new frame-per-second image is acquired, starting from the first analysis signal attribute, the values ​​of the first analysis signal attribute, the second analysis signal attribute and the third analysis signal attribute are detected in turn, and the corresponding signal with the value "T" that first appears is sent to the behavior detection unit.

4. The intelligent analysis and judgment system for student classroom behavior as claimed in claim 3, characterized in that: When the target recognition unit receives the in-position instruction, it obtains the preprocessed next-second frame image from the video data taken by the high-definition camera configured in front of the classroom; identifies the position information of all students in the image based on the YOLOv8 algorithm, generates a corresponding bounding box, and records the coordinate range data of the bounding box in the image; The students in each bounding box are identified based on face recognition technology. The similarity between the facial features of the students in the bounding box and the facial features in all the student photos in the class is calculated in turn to determine the identity of the student who best matches the current facial features, and the name data of the student corresponding to each bounding box is recorded.

5. The intelligent analysis and judgment system for student classroom behavior according to claim 4, characterized in that: When the first analysis signal is received, the behavior of all students in the classroom is identified; When the second analysis signal is received, the behavior of students sitting in the back row and sitting by the window in the classroom is identified; When the third analysis signal is received, the behavior of the specially marked students in the classroom is identified; After receiving the analysis signal, based on the coordinate range data of the bounding boxes corresponding to several students to be identified in the image, several sub-images corresponding to each bounding box area are split from the frame image corresponding to the current moment, and behavior recognition operations are performed on the students in the sub-images based on the behavior recognition model.

6. A student classroom behavior intelligent analysis and judgment system as claimed in claim 5, characterized in that: The OpenPose model is used to annotate the corresponding key points of the students in each bounding box in all the frame-per-second images in the historical data obtained. The key points include 12 feature points, namely, nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist and mouth. All the annotated key points are connected by line segments to obtain a key point structure diagram. The corresponding student behavior type is annotated for each bounding box in all the frame-per-second images in the historical data obtained. All the bounding box data are summarized, and the behavior recognition model is obtained by establishing a training set, a validation set and a test set for training.

7. The intelligent analysis and judgment system for student classroom behavior according to claim 1, characterized in that: At the end of each class, count the total number of students with abnormal behavior in that class m, if m m0 / 2, then strengthen management information will be sent to the teacher of the class, where m0 represents the total number of students in the current class.

8. A method for intelligent analysis and judgment of students' classroom behavior, characterized in that: include: Step 1: Obtain the frame-per-second images of the videos shot by two high-definition cameras in the classroom, and after pre-processing, store the frame-per-second images of the video data shot by the high-definition camera configured in front of the classroom in real time into the monitoring image table of this course; Step 2: Generate a bounding box representing the student's location based on the YOLOv8 algorithm and Labelme annotation technology, and identify the students in each bounding box based on face recognition technology, and store the coordinate range data of the extracted bounding box in the corresponding frame image and the name data of the matching student in the temporary student information table; Step 3: Based on the Labelme annotation technology, the bounding box corresponding to each student is annotated, and the key point structure diagram of each student is obtained based on some key points of COCO and the OpenPose model. After training, a behavior recognition model is obtained; the behavior recognition of the corresponding student is performed based on the behavior recognition model and combined with different analysis signals; Step 4: Based on face recognition technology, the frame-per-second image captured by the high-definition camera at the back of the classroom is detected in real time to determine whether a face appears and whether the facial features of the face match the facial features of the class teacher, and the classroom behavior monitoring cycle is adjusted according to the results; The frames per second image taken by the high-definition camera configured at the back of the classroom is acquired in real time, and the OpenCV software is used to perform facial recognition on the frames per second image acquired in real time. If a face is detected in the image, the area containing the entire face is cropped; the area containing the entire face is compared with the photo files of all the teachers in the current class in turn. If the recognized facial features match the facial features of one of the teachers in the current class, the classroom student behavior is monitored and identified according to the standard monitoring plan, specifically: T1=2*T2 and T2>2; T3=1; the unit of the monitoring cycle is seconds and they are all positive integers, where T1, T2, and T3 are the first, second, and third monitoring cycle intervals respectively; the first monitoring cycle interval refers to the monitoring cycle interval for behavior recognition of all students in the classroom; the second monitoring cycle interval refers to the monitoring cycle interval for behavior recognition of students in the last two rows and the innermost two columns in the classroom; the third monitoring cycle interval refers to the monitoring cycle interval for behavior recognition of specially marked students in the classroom; If the recognized facial features do not match the facial features of all teachers in the current class, or if it is detected that no face appears in the frame-per-second image taken by the high-definition camera configured at the back of the classroom, the first and second monitoring cycle intervals are adjusted to T1=T2=T3=1; Step 5: At the end of the class, analyze the number of abnormal behaviors of each student and the total number of students with abnormal behaviors, add and remove special marks for students, and determine whether to send corresponding information; At the end of each class, count the total number of times each student has abnormal behavior in that class n i , set the abnormal behavior threshold n0, and compare the total number of abnormal behaviors of each student in this class n i The relationship between the number of abnormal behaviors and the threshold value n0, if n i n0, then the student will be specially marked; if n i If n0 and the student has been specially marked in this class, the special mark of the student will be cleared; if a student has been specially marked for n consecutive classes, a reminder message will be sent to his class teacher.

Images