Behavior detection method based on deep learning and tablet computer

Abstract

The invention belongs to the field of image processing, and provides a behavior detection method based on deep learning and a tablet personal computer, and the method comprises the steps: obtaining an overall classroom video, processing the video into continuous image frames, obtaining the position information of a face bounding box of each person and the coordinates of a face mark, and tracking the position information and the coordinates of the face mark; the method comprises the steps of obtaining scores of positive emotional states or negative emotional states based on a deep learning model, estimating scores of positive and negative attention postures based on each face mark coordinate in continuous image frames, and measuring the head displacement distance of each face between the continuous image frames based on the coordinates of the centroid. The score of the positive or negative head movement is obtained; assigning different weights to the scores in the emotion, attention and head movement corresponding to each face, and then performing fusion addition to obtain the behavior participation degree of the individual and the participation rate of the whole class; the problem that in a traditional machine learning method, the thought of student behavior analysis is single, and the calculation amount is large can be solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a behavior detection method based on deep learning and a tablet computer. Background Art

[0002] In recent years, the number of students has continued to rise, and teachers' teaching management work has become increasingly arduous. With the application of artificial intelligence technology in various aspects, the education industry has also ushered in an intelligent era. Classroom teaching activities have always been the focus of educational research. In classroom teaching, students are the main body of learning activities, and their behavior status is a direct reflection of the quality of teaching. It is becoming increasingly important to realize intelligent detection and analysis of students' classroom behavior to analyze students' participation in teaching.

[0003] In traditional machine learning methods, research on student classroom behavior detection mainly uses methods such as human skeleton vectors, optical flow features, and global motion direction features for feature extraction, and then combines classifiers such as naive Bayes or support vector machines for behavior recognition. However, these methods require a lot of manual work to extract features, have a large amount of calculation, and are not suitable for running on lightweight devices.

[0004] In traditional machine learning methods, the idea of ​​analyzing students' behaviors is single, usually a single skeletal movement, a single facial expression, etc., without combining multiple behaviors, and the analysis results are not accurate enough. Summary of the invention

[0005] In view of the above technical problems, the present invention provides a behavior detection method and a tablet computer based on deep learning to solve the problems of single thinking and large amount of calculation in traditional machine learning methods for analyzing students' behavior.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present invention, a behavior detection method based on deep learning is disclosed, the method comprising: Get the whole classroom video and process it into continuous image frames; Performing face detection on the image frame to obtain the position information of each person's facial bounding box and facial marker coordinates, assigning an ID to each facial bounding box and calculating the centroid for tracking; Based on the pre-trained deep learning model, the emotional state of the human face in each facial boundary frame of the continuous image frames is classified to obtain a score of a positive emotional state or a negative emotional state, and the scores are accumulated in the positive or negative emotion accumulator corresponding to the individual; based on the facial marker coordinates of each human face in the continuous image frames, the corresponding yaw angle, pitch angle and roll angle are estimated, and the head posture is estimated by calculating the translation and rotation matrix to obtain a score of a positive attention posture or a negative attention posture, and the scores are accumulated in the positive or negative attention accumulator corresponding to the individual; based on the coordinates of the center of mass, the head displacement distance of each human face between the continuous image frames is measured, and the head displacement distance is compared with a preset empirical threshold to obtain a score of a positive head movement or a negative head movement, and the scores are accumulated in the positive or negative head movement accumulator corresponding to the individual; Different weights are assigned to the positive or negative scores in the emotion accumulator, the attention accumulator, and the head movement accumulator corresponding to each face, and then the scores are fused and added to obtain the individual's behavioral participation degree, and the participation rate of the entire class is obtained based on the behavioral participation degree of each individual.

[0008] Furthermore, the facial bounding box and facial marker coordinates are obtained based on a multi-task cascade convolutional network, and the position information of the facial bounding box includes the coordinates, width and height of the upper right pixel. When classifying the emotional state, emotion detection is performed after the face image is cropped from the image frame based on the facial bounding box. The marker point positions of the facial marker coordinates are: nose tip, eyes, mouth corners, and chin.

[0009] Furthermore, the step of assigning an ID to each facial bounding box and calculating the centroid before tracking includes: S1, assigning a unique ID to each of the facial bounding boxes in the image frame, and calculating the centroid of the facial bounding box; S2, calculating a new centroid of each facial bounding box of the next image frame; S3, based on the Euclidean distance between the new centroid obtained in S2 and the old centroid obtained in S1, associates the unique ID of the old centroid with the new centroid with the minimum distance; S4. When any new centroid is independent of the unique IDs of all old centroids in the image frame of the previous frame, a new unique ID is assigned to the new centroid and the operation is restarted from S1.

[0010] Furthermore, in the unique ID assignment process, when a face is obscured, a unique ID is assigned to it based on dynamic facial recognition. The dynamic facial recognition is based on FaceNet. A database is dynamically constructed during the teaching process. The database stores each individual's high-level facial feature vector and unique ID. When a face is obscured, the same unique ID is reallocated to the face that cannot be tracked by the centroid by mapping the high-level facial feature vector.

[0011] Furthermore, the deep learning model is modified based on the MobileNetV2 architecture. During the modification, the 1000-class classifier of MobileNetV2 is replaced by a classifier consisting of a GlobalAveragePooling layer, a dropout layer, a batch normalization layer, and a dense layer consisting of seven neurons, and the neurons use a Softmax function to output class probabilities.

[0012] Furthermore, when performing head posture estimation, the yaw angle represents the angle of horizontal movement of the head, the pitch angle represents the angle of vertical movement of the head, the roll angle represents the angle of circular movement of the head, and the translation and rotation matrix is ​​based on the association between the facial marker coordinates and their respective 3D environment coordinates, and estimates the output head posture as one of a positive attention posture or a negative attention posture.

[0013] According to another aspect of the present invention, a behavior detection tablet computer based on deep learning is disclosed, and the tablet computer is used to visualize classroom videos, the behavior participation of each individual in the classroom and the participation rate of the entire class, and the behavior participation and the participation rate are obtained based on the above method.

[0014] The technical solution disclosed in this disclosure has the following beneficial effects: The present disclosure analyzes students' classroom participation from multiple aspects based on the combination of emotional state classification, head posture estimation, and head movement estimation, and then combines them to obtain the participation rate of the entire class, and the analysis results are accurate; Compared with traditional machine learning feature extraction methods, the entire behavior detection method has less computational complexity, most steps can be run on lightweight devices, and has broader application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flowchart of a behavior detection method based on deep learning in an embodiment of this specification; Figure 2 This is a structural block diagram of a behavior detection tablet computer based on deep learning in an embodiment of this specification. DETAILED DESCRIPTION

[0016] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0017] In addition, the accompanying drawings are only schematic illustrations of the present disclosure. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0018] In one embodiment, if Figure 1 As shown, the present specification provides a behavior detection method based on deep learning, and the execution subject of the method can be a computer, a server, a mobile phone, a tablet computer, etc. The method can specifically include steps S101~S104, and S101-S104 can be simplified as: video acquisition, individual tracking, multi-channel feature extraction, and participation estimation. In the video acquisition stage, a high-definition camera is used to record the classroom video and process it into image frames, and then the frames are input into the individual tracking stage. In the individual tracking stage, the face of each student is detected and then tracked by assigning a unique ID. Next, these tracked faces are input into the multi-channel feature extraction stage, and facial expressions, head postures, and head displacements are analyzed to obtain scores for emotions, attention, and head movements, respectively. In the participation estimation stage, the three scores obtained are fused using a weighted fusion method to calculate a single fused participation score, and the participation scores are summarized to obtain the overall class participation rate.

[0019] Specifically, steps S101 to S104: In step S101, the entire classroom video is obtained and processed into continuous image frames.

[0020] Among them, a camera with more than five megapixels can be used to record videos of students in a classroom at a frame rate of more than 30 frames per second and a resolution of 1920 x 1080 pixels. The camera can be installed in the center above the blackboard to sample the input video with a sampling interval of 0.20 seconds to omit redundant frame processing, thereby reducing computational overhead.

[0021] In step S102, face detection is performed on the image frame to obtain the position information of each person's facial bounding box and facial marker coordinates, and each facial bounding box is assigned an ID and tracked after calculating the centroid.

[0022] Among them, the facial bounding box and facial marker coordinates are obtained based on a multi-task cascade convolutional network (MTCNN), the position information of the facial bounding box includes the coordinates, width and height of the upper right pixel, and when classifying the emotional state, emotion detection is performed after the face image is cropped from the image frame based on the facial bounding box, and the marker point positions of the facial marker coordinates are: nose tip, eyes, mouth corners, and chin.

[0023] In step S103, based on the pre-trained deep learning model, the emotional state of the faces in each facial boundary frame of the continuous image frames is classified to obtain scores of positive emotional state or negative emotional state, and the scores are accumulated in the positive or negative emotion accumulator corresponding to the individual; based on the facial marker coordinates of each face in the continuous image frames, the corresponding yaw angle, pitch angle and roll angle are estimated, and the head posture is estimated by calculating the translation and rotation matrix to obtain the score of positive attention posture or negative attention posture, and the scores are accumulated in the positive or negative attention accumulator corresponding to the individual; based on the coordinates of the center of mass, the head displacement distance of each face between the continuous image frames is measured, and the head displacement distance is compared with a preset empirical threshold to obtain the score of positive head movement or negative head movement, and the scores are accumulated in the positive or negative head movement accumulator corresponding to the individual.

[0024] Among them, the deep learning model is modified based on the MobileNetV2 architecture. During the modification, the 1000-class classifier of MobileNetV2 is replaced by a classifier consisting of a GlobalAveragePooling layer, a dropout layer, a batch normalization layer, and a dense layer consisting of seven neurons, and the neurons use the Softmax function to output class probabilities. MobileNetV2 is a variant of MobileNet, which is an ultra-lightweight architecture based on deep CNN developed by Google. It has improved performance and efficiency. It consists of 158 layers, including 16 fixed feature extraction blocks and a 1000-class classification block. It adopts three structures: deep separable convolution, inverted residual, and linear bottleneck. Compared with standard convolution operations, this greatly reduces the computational cost without affecting accuracy. Therefore, it is very suitable for developing mobile or embedded applications for visual tasks such as object detection and facial expression recognition. In this embodiment, its 1000-class classifier is replaced by a custom neural network classifier to further improve its lightness so that it can be run on mobile devices such as tablets. Specifically, preprocessed color facial images are used as input to the deep learning model, and their learned emotional states are predicted as output labels. Then, the output labels are further classified into positive or negative emotional states to explain the positive and negative emotional engagement dimensions of students in classroom learning. Positive emotional states include happiness and concentration, while negative emotional states include boredom, confusion, frustration, sleepiness, and yawning. The emotional state of each student is estimated based on the student's consecutive facial images, and the estimated positive and negative emotional labels are accumulated into the positive emotion accumulator and negative emotion accumulator of the student, respectively, and the emotional score is obtained using the maximum accumulator index, such as: ; is the sentiment score, is a positive emotion accumulator, It is a negative emotion accumulator.

[0025] The head is an important non-verbal visual indicator of students' attention in class, explaining the behavioral dimension of engagement. Head-related features include two forms: head pose and head movement. The head pose and movement of inattentive students are significantly different from those of attentive students. Both forms of head clues are estimated and fused with emotional states to more reliably estimate students' engagement. Specifically, the PnP (Perspective-n-Point) head pose estimation technique can be used to estimate head pose using facial landmark coordinates. The head pose can be frontal, right tilted, left tilted, head up or head down. The head pose is represented by Euler angles (i.e., yaw, pitch, and roll). The yaw angle represents the angle of horizontal movement of the head, the pitch angle represents the angle of vertical movement of the head, and the roll angle represents the angle of circular movement of the head. PnP estimates the head pose by calculating the translation and rotation matrix, which can be based on the association of six 2D facial landmark coordinates with their respective 3D environment coordinates. The estimated head pose output is further divided into positive attention pose or negative attention pose. For example, the front head pose is regarded as a positive attention pose, while all other non-front head poses are regarded as negative attention poses. The estimated positive and negative attention poses are then accumulated into the positive attention accumulator and negative attention accumulator of the student, respectively, and the attention score is obtained using the maximum accumulator index, such as: ; Score for attention, is the positive attention accumulator, is the negative attention accumulator.

[0026] In head estimation, a single centroid is used to measure the head displacement distance of each pupil between consecutive image frames. The centroid is represented as the x and y coordinate pixel position in the image frame. The head displacement distance is measured by calculating the absolute difference between the coordinates of the centroid in the current image frame and the coordinates of the centroid in the previous image frame. Then, the normalized average head displacement value is calculated by considering all image frames in a single video. If the average head displacement value is less than the empirical threshold, the head movement is regarded as positive attention head movement, and the score in the positive head movement accumulator is increased by one. If it is greater than the empirical threshold, it is regarded as negative attention head movement, and the score in the negative head movement accumulator is increased by one, and then the maximum accumulator index is used to obtain the head movement score, such as: ; is the head movement score, is the positive head motion accumulator, is the negative head motion accumulator.

[0027] In step S104, different weights are assigned to the positive or negative scores in the emotion accumulator, the attention accumulator, and the head movement accumulator corresponding to each face, and then the scores are fused and added to obtain the individual's behavioral participation degree, and the participation rate of the entire class is obtained based on the behavioral participation degree of each individual.

[0028] Among them, the weighted and fusion method can be used to fuse the three scores representing the student's emotional and behavioral engagement dimensions to calculate a single fusion engagement score, that is, the individual's behavioral engagement, such as: ; is the individual's behavioral engagement, To fuse the weights, according to the experimental calculation, a pairwise comparison matrix can be generated by the comparison values ​​between the three directions (emotion, head movement, head posture), the normalized comparison matrix is ​​calculated by dividing all elements by the sum of the columns, and then the fusion weights are obtained by calculating the average of each row. In addition, based on the individual's behavioral engagement, the individual student engagement label is estimated as "engaged" or "unengaged", and these estimated individual behavioral engagement labels are processed to estimate the overall class engagement.

[0029] In one embodiment, assigning an ID to each facial bounding box and calculating the centroid and then tracking the facial bounding box includes: S1, assigning a unique ID to each of the facial bounding boxes in the image frame, and calculating the centroid of the facial bounding box; S2, calculating a new centroid of each facial bounding box of the next image frame; S3, based on the Euclidean distance between the new centroid obtained in S2 and the old centroid obtained in S1, associates the unique ID of the old centroid with the new centroid with the minimum distance; S4. When any new centroid is independent of the unique IDs of all old centroids in the image frame of the previous frame, a new unique ID is assigned to the new centroid and the operation is restarted from S1.

[0030] Among them, in the process of unique ID allocation, when a face is obscured, a unique ID is allocated to it based on dynamic facial recognition. The dynamic facial recognition is based on FaceNet. A database is dynamically constructed during the teaching process. The database stores the high-level facial feature vector and unique ID of each individual. When the face is obscured, the same unique ID is reallocated to the face that cannot be tracked by the centroid by mapping the high-level facial feature vector.

[0031] Additionally, FaceNet is one of the SOTA-based face recognition technologies used for face recognition.

[0032] Based on the same idea, Figure 2 As shown, an exemplary embodiment of the present disclosure also provides a behavior detection tablet computer based on deep learning, which is used to visualize classroom videos, the behavioral participation of each individual in the classroom and the participation rate of the entire class, and the behavioral participation and the participation rate are obtained based on the above method.

[0033] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary implementation of the present disclosure.

[0034] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiment of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0035] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A behavior detection method based on deep learning, characterized in that: The method comprises: Get the whole classroom video and process it into continuous image frames; Performing face detection on the image frame to obtain the position information of each person's facial bounding box and facial marker coordinates, assigning an ID to each facial bounding box and calculating the centroid for tracking; Based on the pre-trained deep learning model, the emotional state of the human face in each facial boundary frame of the continuous image frames is classified to obtain a score of a positive emotional state or a negative emotional state, and the scores are accumulated in the positive or negative emotion accumulator corresponding to the individual; based on the facial marker coordinates of each human face in the continuous image frames, the corresponding yaw angle, pitch angle and roll angle are estimated, and the head posture is estimated by calculating the translation and rotation matrix to obtain a score of a positive attention posture or a negative attention posture, and the scores are accumulated in the positive or negative attention accumulator corresponding to the individual; based on the coordinates of the center of mass, the head displacement distance of each human face between the continuous image frames is measured, and the head displacement distance is compared with a preset empirical threshold to obtain a score of a positive head movement or a negative head movement, and the scores are accumulated in the positive or negative head movement accumulator corresponding to the individual; Different weights are assigned to the positive or negative scores in the emotion accumulator, the attention accumulator, and the head movement accumulator corresponding to each face, and then the scores are fused and added to obtain the individual's behavioral participation degree, and the participation rate of the entire class is obtained based on the behavioral participation degree of each individual.

2. A behavior detection method based on deep learning according to claim 1, characterized in that: The facial bounding box and facial marker coordinates are obtained based on a multi-task cascade convolutional network, where the position information of the facial bounding box includes the coordinates, width and height of the upper right pixel. When classifying the emotional state, emotion detection is performed after the face image is cropped from the image frame based on the facial bounding box. The marker point positions of the facial marker coordinates are: nose tip, eyes, mouth corners, and chin.

3. The behavior detection method based on deep learning according to claim 1, characterized in that: The step of assigning an ID to each facial bounding box and calculating the centroid for tracking includes: S1, assigning a unique ID to each of the facial bounding boxes in the image frame, and calculating the centroid of the facial bounding box; S2, calculating a new centroid of each facial bounding box of the next image frame; S3, based on the Euclidean distance between the new centroid obtained in S2 and the old centroid obtained in S1, associates the unique ID of the old centroid with the new centroid with the minimum distance; S4. When any new centroid is independent of the unique IDs of all old centroids in the image frame of the previous frame, a new unique ID is assigned to the new centroid and the operation is restarted from S1.

4. The behavior detection method based on deep learning according to claim 3 is characterized in that: In the unique ID assignment process, when a face is obscured, a unique ID is assigned to it based on dynamic facial recognition. The dynamic facial recognition is based on FaceNet. A database is dynamically constructed during the teaching process. The database stores each individual's high-level facial feature vector and unique ID. When a face is obscured, the same unique ID is reallocated to the face that cannot be tracked by the centroid by mapping the high-level facial feature vector.

5. The behavior detection method based on deep learning according to claim 1, characterized in that: The deep learning model is modified based on the MobileNetV2 architecture. During the modification, the 1000-class classifier of MobileNetV2 is replaced by a classifier consisting of a GlobalAveragePooling layer, a dropout layer, a batch normalization layer, and a dense layer consisting of seven neurons, and the neurons use a Softmax function to output class probabilities.

6. The behavior detection method based on deep learning according to claim 1, characterized in that: When performing head posture estimation, the yaw angle represents the angle of horizontal movement of the head, the pitch angle represents the angle of vertical movement of the head, the roll angle represents the angle of circular movement of the head, and the translation and rotation matrix is ​​based on the association between the facial marker coordinates and their respective 3D environment coordinates, and the output head posture is estimated to be one of a positive attention posture or a negative attention posture.

7. A behavior detection tablet computer based on deep learning, characterized in that: The tablet computer is used for visually displaying classroom videos, the behavioral participation of each individual in the classroom and the participation rate of the entire class, and the behavioral participation and the participation rate are obtained based on the method described in any one of claims 1-6.

Images