Face-based student abnormal behavior monitoring method and system, and mobile terminal

Abstract

The invention provides a face-based student abnormal behavior monitoring method and system and a mobile terminal. The monitoring method comprises the following steps: acquiring image information; face information in the image information is extracted; based on an LBF algorithm, eye features and mouth features of the face information are extracted; based on a face expression recognition algorithm of an improved Transform, extracting a face expression behavior of the face information, and determining a face expression score according to the face expression behavior; determining whether abnormal behaviors exist according to the eye features and the mouth features; according to the eye features, the mouth features and the facial expression scores, a comprehensive quantitative score is determined, and the comprehensive quantitative score is used for representing the classroom learning concentration degree of the students. According to the face expression recognition algorithm based on the improved Transform, features of face information can be focused in key and key areas, irrelevant features are ignored, and therefore the problem that face expression recognition is difficult due to face shielding and non-frontal head posture change in a real environment is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of facial expression recognition, and in particular to a method, system and mobile terminal for monitoring abnormal behavior of students based on faces. Background Art

[0002] With the rapid development of Internet technology, online education has gradually become an important teaching model, and more and more students are receiving education through online network platforms. However, in traditional face-to-face teaching, teachers can perceive students' emotional states through their facial expressions and adjust teaching strategies in time. In online teaching, due to the lack of face-to-face communication, teachers cannot directly observe students' facial expressions, which brings challenges to the quality and effectiveness of teaching. Therefore, the introduction of facial expression recognition technology can make up for this deficiency in online teaching and help capture students' emotional states in real time.

[0003] However, the existing facial expression recognition technology is sensitive to common interference factors such as lighting changes, posture changes and occlusion. In the real environment, facial occlusion and non-frontal head posture changes may occur, which will make facial expression recognition difficult. Summary of the invention

[0004] In order to overcome the above technical defects, the present invention provides a method, system and mobile terminal for monitoring abnormal behavior of students based on face, which can solve the problem of difficulty in recognition caused by facial occlusion and non-frontal head posture changes in real environment.

[0005] In order to solve the above problems, the present invention is implemented according to the following scheme:

[0006] A method for monitoring abnormal student behavior based on human face is provided, which is characterized by comprising:

[0007] Get image information;

[0008] Extracting facial information from the image information;

[0009] Extracting eye features and mouth features of the face information based on the LBF algorithm;

[0010] A facial expression recognition algorithm based on an improved Transformer is used to extract facial expression behaviors of the face information, and a facial expression score is determined according to the facial expression behaviors;

[0011] determining whether there is abnormal behavior according to the eye features and the mouth features;

[0012] A comprehensive quantitative score is determined based on the eye features, the mouth features and the facial expression score, and the comprehensive quantitative score is used to represent the student's classroom learning concentration.

[0013] Compared with the prior art, the beneficial effects of the face-based student abnormal behavior monitoring method provided by the present invention are as follows: the face facial expression recognition algorithm based on the improved Transformer can focus the features of facial information on key areas and ignore irrelevant features, thereby solving the problem of difficulty in facial expression recognition caused by facial occlusion and non-frontal head posture changes in real environments.

[0014] Optionally, extracting facial information from the image information includes:

[0015] Combine the Focal Loss loss function and the EIOU function to determine the target loss function;

[0016] The original loss function of the YOLOv5n algorithm is replaced with the target loss function to determine an improved YOLOv5n algorithm;

[0017] Based on the improved YOLOv5n algorithm, the facial information is extracted.

[0018] Optionally, the face facial expression recognition algorithm based on the improved Transformer extracts facial expression behavior of the face information, including:

[0019] Extracting features of different levels of the facial information;

[0020] Determining a multi-scale feature map of the face information according to the features at different levels;

[0021] The multi-scale feature map is divided into three paths, each of which passes through three convolution kernels with holes introduced;

[0022] splicing the output feature maps of the convolution kernels to determine a fusion feature map of the face information;

[0023] Pass the fused feature map through the channel attention branch to determine a channel attention map;

[0024] Passing the fused feature map through the spatial attention branch to determine a spatial attention map;

[0025] Determine an output feature map according to the channel attention map and the spatial attention map;

[0026] The facial expression behavior of the face information is extracted according to the output feature map.

[0027] Optionally, the face facial expression recognition algorithm based on the improved Transformer includes:

[0028] Mini-Swin module, feature fusion module and feature enhancement module;

[0029] The Mini-Swin module is obtained by Swin Transformer after model compression, and the Mini-Swin module includes weight conversion and weight distillation. The Mini-Swin module is used to extract features of different levels of the face information and determine the multi-scale feature map;

[0030] The feature fusion module includes feature fusion, and the feature fusion module is used to fuse the global features and local features of the face information to determine the fusion feature map;

[0031] The feature enhancement module includes an attention mechanism, and the feature enhancement module is used to extract key features of the facial information and determine the output feature map.

[0032] Optionally, determining a facial expression score according to the facial expression behavior includes:

[0033] Classifying the facial expression behaviors into positive emotions, negative emotions, and neutral emotions;

[0034] The positive emotions include happiness and surprise, the negative emotions include fear, sadness, anger and disgust, and the neutral emotion is calmness;

[0035] Calculating the probabilities of the positive emotion, the negative emotion, and the neutral emotion respectively;

[0036] The facial expression score is determined according to the probabilities of the positive emotion, the negative emotion, and the neutral emotion.

[0037] Optionally, determining whether there is abnormal behavior based on the eye features and the mouth features includes:

[0038] determining an eye behavior based on the eye characteristics;

[0039] determining a mouth behavior based on the mouth features;

[0040] The eye behavior and the mouth behavior are compared with the abnormal behavior determination criteria respectively. If they are the same, abnormal behavior exists; if they are different, abnormal behavior does not exist.

[0041] Optionally, the abnormal behavior determination criterion is an abnormal behavior dataset of students in an online class.

[0042] Optionally, determining a comprehensive quantitative score according to the eye features, the mouth features and the facial expression score includes:

[0043] Determining an eye quantification score according to the eye characteristics;

[0044] Determining a mouth quantization score according to the mouth features;

[0045] Determining a facial expression quantification score according to the facial expression score;

[0046] The comprehensive quantitative score is determined according to the eye quantitative score, the mouth quantitative score and the facial expression quantitative score.

[0047] A system for monitoring abnormal student behavior based on human face is also provided, which is applied to the method for monitoring abnormal student behavior based on human face, and comprises:

[0048] An image acquisition module, used for acquiring image information;

[0049] A face detection module, used to extract face information from the image information;

[0050] A key point detection module, used for extracting eye features and mouth features of the face information based on the LBF algorithm;

[0051] A facial expression analysis module, which is used to extract facial expression behaviors of the face information based on an improved Transformer-based facial expression recognition algorithm, and determine a facial expression score according to the facial expression behaviors;

[0052] an abnormal behavior analysis module, used to determine whether there is abnormal behavior based on the eye features and the mouth features;

[0053] The behavior quantification module is used to determine a comprehensive quantification score according to the eye features, the mouth features and the facial expression score.

[0054] A mobile terminal is also provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the student abnormal behavior monitoring system based on human face. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is the overall flow chart of the monitoring method in the present invention;

[0056] Figure 2 It is the overall framework diagram of the facial expression recognition algorithm based on the improved Transformer in the present invention;

[0057] Figure 3 It is a framework diagram of the weight reuse strategy in the present invention;

[0058] Figure 4 It is a structural diagram of weight conversion in the present invention;

[0059] Figure 5 is the abnormal behavior determination standard in the present invention;

[0060] Figure 6 It is the overall framework diagram of the monitoring system in the present invention; DETAILED DESCRIPTION

[0061] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0062] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0063] See also Figure 1 As shown, the present invention provides a method for monitoring abnormal student behavior based on human face, comprising:

[0064] S1: Acquire image information, where the image information is image data of each frame of the video stream.

[0065] S2: extracting face information from the image information, including:

[0066] Combine the Focal Loss loss function and the EIOU function to determine the target loss function L Focal-EIOU , which is expressed as:

[0067] L Focal-EIOU =IOU γ *L EIOU

[0068] Among them, IOU is the indicator of target detection task performance, γ is the degree of adjusting outlier suppression, and L EIOU is the EIOU function.

[0069] Replace the original loss function of the YOLOv5n algorithm with the target loss function L Focal-EIOU , determine the improved YOLOv5n algorithm;

[0070] Based on the improved YOLOv5n algorithm, the facial information is extracted.

[0071] S3: Extracting the eye features and mouth features of the facial information based on the LBF algorithm.

[0072] S4: Based on the improved Transformer facial expression recognition algorithm, the facial expression behavior of the face information is extracted, and the facial expression score is determined according to the facial expression behavior.

[0073] See also Figure 2 The overall framework of the face facial expression recognition algorithm based on the improved Transformer is shown in the figure. The face facial expression recognition algorithm based on the improved Transformer includes:

[0074] Mini-Swin module, feature fusion module and feature enhancement module;

[0075] The Mini-Swin module is obtained by compressing the Swin Transformer model, which greatly reduces the number of parameters of the Swin Transformer.

[0076] The core idea of ​​the Mini-Swin module is to share weights in the multiplexed continuous transformer blocks through a weight reuse strategy to improve the training stability and model performance during the weight sharing process, and at the same time extract all relevant features of the face information, and achieve the purpose of multi-scale feature fusion by fusing features from different Transformer levels.

[0077] See also Figure 3 As shown in the framework diagram of the weight reuse strategy, the weight reuse strategy includes two parts: weight conversion and weight distillation. The weight reuse strategy enables the Mini-Swin module to better achieve parameter reduction and improve training effects while maintaining performance.

[0078] The calculation formula of the weight reuse strategy is as follows:

[0079] Z i+1 =f(Z i ;θ,θ i ′),i=0,…,L-1

[0080] f is the converter block, Z iis the feature embedding of the sequence in the i-th layer, i is the total number of layers, θ is the shared weight of the transformer block across all layers, and θ i ′ is the weight of the transform block in the i-th transformer layer, and θ i The number of parameters of ′ is much lower than that of θ.

[0081] See also Figure 4 As shown in the structural diagram of weight conversion, the key idea of ​​weight conversion is to impose conversion on shared weights so that different layers have slightly different weights, which diversifies parameters and makes the training process more stable. Specifically, the W-MSA (SW-MSA) module is mainly used for local feature extraction and modeling, reducing computational complexity and capturing dependencies within local regions; the MLP module is used to further transform and integrate features processed by the W-MSA (SW-MSA) module, that is, to apply simple linear conversion to the W-MSA (SW-MSA) module and the MLP module of each weight-sharing transformer layer, so that each layer includes a separate conversion matrix, so that the parameters of each layer are different, increasing the parameter diversification capability. The conversion of the W-MSA (SW-MSA) module is obtained by inserting tiny linear layers before and after the softmax layer and applying weight conversion to these linear layers.

[0082] Compared with the original W-MSA module, the calculation formula of the attention weight and output feature of the W-MSA module equipped with linear transformation is:

[0083]

[0084]

[0085] Z ijkl is the final output feature matrix, Q m Km represents the query and key, which is used to calculate the attention weight. T means transposition. Represents the linear transformation kernel after the softmax layer, which is used to calculate the attention weight matrix A n In the process of linear transformation, is the linear transformation kernel before the softmax layer, A n represents the attention weight matrix, F (1) , F (2) They are the linear transformation kernels before and after the softmax layer. This linear transformation can make each attention weight matrix A n Different, W K represents the weight matrix, Z is the final output feature matrix, V ij(m+n-1)l Represents the value vector of the (m+n-1)th row and lth column of the (i,j)th window.

[0086] Among them, the linear transformation of the MLP module is achieved by introducing d linear transformations and depthwise convolutions. By using depthwise convolutions to sparsify and share the weights in each weight matrix, the number of parameters can be reduced from N 2 d parameters to K 2 d parameters (where K is the kernel size of the convolution, K << N). The calculation formula for the output of the MLP module is as follows:

[0087] Y′ = [C (1) y1, …, C (d) y d

[0088] H = σ(Y′W (1) + b (1) )W (2) + b (2)

[0089] H is the output of the MLP module, y1 is the first position of the embedding vectors of all tokens, C (1) , C (2) are the independent weight matrices of the linear layer, y d is the d-th position of the embedding vectors of all tokens, C (d) is the independent weight matrix of the linear layer used to perform a linear transformation on the embedding vector y d at the corresponding d-th position. Y′ is the input of the MLP module introducing d linear transformations, W (1) and b (1) are the weight matrix and bias term in the first linear transformation, W (2) and b (2) : are the weight matrix and bias term in the second linear transformation.

[0090] Among them, the core idea of weight distillation is to transfer the knowledge contained in a large pre-trained model to a small model with weight sharing, so as to achieve compression of the large pre-trained model and solve the problem of performance degradation caused by weight sharing. The weight distillation method includes three different types of distillation, namely prediction logit distillation, self-attention distillation, and hidden state distillation. The specific calculation formulas are as follows:

[0091]

[0092]

[0093] is prediction logit distillation, CE represents the cross-entropy loss, z s is the logit predicted by the student model, z t ​is the logit predicted by the teacher model, T is the temperature value that controls the smoothness of the logit, and the default setting is T=1.

[0094] is the self-attention distillation loss, CE represents the cross entropy loss, and R ij,n Represents R ij The nth row, R ij Represents a relationship matrix.

[0095] is the hidden state distillation loss based on the relationship matrix, R H,n Represents R H The nth row, R H Represents a relationship matrix.

[0096] is the final distillation target score, β and γ are hyperparameters, the default value of β is 1, and the default value of γ is 0.1.

[0097] The Mini-Swin module is used to extract features at different levels of facial information and determine multi-scale feature maps.

[0098] The feature fusion module includes feature fusion, and the feature fusion module is used to fuse the global features and local features of face information to determine a fusion feature map.

[0099] The feature fusion module has the ability to fuse global features and local features of the input image. The hole convolution expands the receptive field without losing resolution and aggregates multi-scale information.

[0100] First, the multi-scale feature map currently output from the Mini-Swin module is evenly divided into three paths. In order to reduce parameters and prevent overfitting, these three paths are each passed through a sub-network containing three identical dilated convolutional kernels with 3 convolutional kernels.

[0101] Then the three outputs are spliced ​​together to merge the global and local features of the acquired facial information, thereby achieving the purpose of fusing the global and local features of the facial information.

[0102] Among them, the specific calculation formula for calculating the receptive field is as follows:

[0103] k′=k+(k-1)×(d-1)

[0104] F i+1 =F i +(k′-1)×S i

[0105]

[0106] The convolution kernel k in the dilated convolution subnetwork is 3, the dilation rate d is 3, the step size S is 1, and F i+1 represents the receptive field of the current layer, F i represents the receptive field of the previous layer, S i It represents the product of the stride of all previous layers (excluding this layer), k′ represents the changed convolution kernel, and Stride represents the stride.

[0107] The ACmix module is introduced into the spatial attention branch to obtain a feature enhancement module. Therefore, the feature enhancement module includes an attention mechanism. The feature enhancement module is used to extract the key features of face information and determine the output feature map.

[0108] Through the feature enhancement module, the input features (fused feature map of facial information) can be focused on key areas, ignoring irrelevant features of facial information, thereby solving the problem of difficulty in recognizing facial occlusion and non-frontal head posture changes in real environments.

[0109] The input features pass through the channel attention branch and the spatial attention branch respectively to obtain the channel attention map and the spatial attention map, and add the two together. The result of the addition is multiplied element-by-element with the input features, and then added to the input features to obtain the final output features (output feature map).

[0110] Among them, the calculation formula for calculating the attention map M(X) and the output feature X′ is as follows:

[0111] M(X)=sigmod(M c (X)+M s (X)

[0112]

[0113] M c (X) represents the channel attention map, M s (X) represents the spatial attention map, and X represents the input feature.

[0114] Among them, the purpose of the channel attention branch is to focus on the features in each channel.

[0115] Perform global average pooling on the input feature X to obtain the feature map in each channel, and then pass it through a multi-layer perceptron (MLP) with a hidden layer, and finally through a normalization layer (BN) to adjust the output ratio. The channel attention map calculation formula is as follows:

[0116] M c (X) = BN(MLP(AvgPool(X)))

[0117] Among them, the spatial attention branch emphasizes the features of important spatial positions and suppresses the features of unimportant spatial positions, thereby solving the problem of difficulty in recognizing facial occlusion and non-frontal head posture changes in real environments.

[0118] The ACmix module is inserted into the spatial attention branch to increase the receptive field, thereby aggregating contextual information well. The spatial attention map calculation formula is as follows:

[0119]

[0120] f represents the convolution operation, the superscript represents the size of the convolution kernel, and the subscript represents the number of convolution layers.

[0121] Among them, ACmix can enjoy the advantages of self-attention and convolution at the same time with minimal computational overhead. ACmix consists of two stages.

[0122] The first stage is the process of feature learning:

[0123] The input features are passed through three convolution kernels of size 1*1 to generate intermediate feature maps respectively;

[0124] The intermediate feature map is then placed into the self-attention path and the convolution path for operation, thus entering the second stage.

[0125] The second stage is the process of feature aggregation:

[0126] In the self-attention path, the three intermediate feature maps of each group of inputs are used as query, key and value, and then aggregated into a feature map through the multi-head self-attention module. Finally, the feature maps of N groups are connected and aggregated to generate X att .

[0127] In the convolution path, the input of each group of paths is k2 intermediate feature maps, and then a feature map is generated through shift and sum operations. Finally, the feature maps of N groups are connected and aggregated to generate X conv .

[0128] Finally, the outputs of the two paths are added together to obtain the new output feature map X out , the strength of the two paths is controlled by α and β, and the calculation formula is as follows:

[0129] X out =αX att +βX conv

[0130] Finally, the facial expression behavior of the face information is extracted based on the output feature map.

[0131] A facial expression score is determined based on facial expression behaviors, including:

[0132] Classify facial expression behaviors into positive emotions, negative emotions, and neutral emotions;

[0133] Positive emotions include happiness and surprise, negative emotions include fear, sadness, anger and disgust, and neutral emotions include calmness;

[0134] Calculate the probabilities of positive emotions, negative emotions, and neutral emotions respectively. The specific calculation formulas are as follows;

[0135] P pos =P h +P sup (0<=P pos <=1)

[0136] P neu =P n (0<=P neu <=1)

[0137] P neg =P f +P a +P sad +P d (0<=P neg <=1)

[0138] P pos The probability of expressing positive emotions, P h represents the probability of being happy, P sup The probability of expressing surprise, P neu The probability of expressing neutral sentiment, P n represents the probability of calm, P neg The probability of expressing negative emotions, P f represents the probability of negativity, P a The probability of expressing fear, P sad Indicates the probability of being angry, P d The probability of expressing disgust.

[0139] According to the probability of positive emotions, negative emotions and neutral emotions, the facial expression score is determined. The specific calculation formula is as follows:

[0140] S=80·P n +50·P h +60·P sup -40×P d -30×P f -20×P a -30×P sad

[0141] S5: Determine whether there is abnormal behavior according to the eye features and the mouth features;

[0142] Eye features are specifically the horizontal and vertical coordinates of the key points of the left and right eyes. Eye behavior is determined based on the eye aspect ratio (EAR).

[0143] The mouth features are specifically the horizontal and vertical coordinates of the mouth key points, and the mouth behavior is determined based on the mouth aspect ratio (MAR).

[0144] See also Figure 5 The abnormal behavior determination criteria shown are used to determine whether a student has exhibited abnormal behavior (eg, absence, talking, yawning, sleeping in class).

[0145] Among them, the eye aspect ratio (EAR) calculation formula is as follows:

[0146]

[0147] y 38 ,y 39 ,y 41 ,y 42 Represents the ordinate of the left eye key point, x 37 、x 40 Represents the horizontal coordinate of the left eye key point.

[0148] y 44 ,y 45 ,y 47 ,y 48 Represents the vertical coordinate of the key point of the right eye, x 43 、x 46 Represents the horizontal coordinate of the key point of the right eye.

[0149] The calculation formula of mouth aspect ratio (MAR) is as follows:

[0150]

[0151] y 51 ,y 53 ,y 59 ,y 57 Represents the ordinate of the mouth key point, x 55 、x 49 Represents the horizontal axis.

[0152] S6: Determine a comprehensive quantitative score based on the eye features, the mouth features and the facial expression score, where the comprehensive quantitative score is used to represent the student's concentration in classroom learning.

[0153] Based on the eye features, mouth features and facial expression scores, a comprehensive quantitative score is determined, including:

[0154] Based on the eye characteristics, the eye quantitative score is determined; for eye behavior, abnormal behavior includes closing the eyes (sleeping).Eyes is the quantitative score of abnormal eye behavior, and the calculation formula is as follows:

[0155]

[0156] S Eyes =1-F EC

[0157] F EC It represents the cumulative time percentage of the eye aspect ratio (EAR) less than the preset threshold of 0.12 within 30 frames, that is, the cumulative time percentage of eye closure per unit time, ∑f EC It indicates the number of frames less than the preset threshold 0.12 within 30 consecutive frames, and M is 30 frames.

[0158] According to the mouth features, the mouth quantification score is determined; for mouth behavior, abnormal behavior includes yawning and talking. Therefore, the abnormal behavior is quantified, and the quantification scores are expressed as S Yawn , S Speak , the calculation formula is as follows:

[0159]

[0160] S Speak =1-F Speak

[0161] S Mouth =min(S Yawn , S Speak )

[0162] S Mouth is the comprehensive quantitative score of abnormal mouth behavior, S Yawn is the quantitative score of yawning behavior, f Yawn is the number of yawns within 30 frames, S Speak is the quantitative score of speaking behavior, F Speak The frequency of speaking within 30 frames.

[0163] According to the facial expression score, the facial expression quantitative score is determined; for facial expression behavior, abnormal behavior refers to negative expression, S Emotion is the quantitative score of abnormal facial expression behavior, and the calculation formula is as follows:

[0164]

[0165] S is the facial expression score determined in step S4 according to the facial expression behavior.

[0166] The comprehensive quantitative score is determined based on the quantitative scores of the eyes, mouth and facial expressions. That is, the final comprehensive quantitative score of the student is obtained by combining the quantitative scores of abnormal mouth behavior, abnormal eye behavior and abnormal facial expression behavior. The comprehensive quantitative score is used to judge the student's classroom concentration. The calculation formula is as follows:

[0167]

[0168] See also Figure 6 As shown, the present invention also provides a student abnormal behavior monitoring system based on human face, comprising:

[0169] An image acquisition module, used for acquiring image information;

[0170] A face detection module, used to extract face information from the image information;

[0171] A key point detection module, used for extracting eye features and mouth features of the face information based on the LBF algorithm;

[0172] A facial expression analysis module, which is used to extract facial expression behaviors of the face information based on an improved Transformer-based facial expression recognition algorithm, and determine a facial expression score according to the facial expression behaviors;

[0173] an abnormal behavior analysis module, used to determine whether there is abnormal behavior based on the eye features and the mouth features;

[0174] The behavior quantification module is used to determine a comprehensive quantification score according to the eye features, the mouth features and the facial expression score.

[0175] The present invention also provides a mobile terminal, including a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the above-mentioned face-based student abnormal behavior monitoring system.

[0176] Specifically, the mobile terminal may be an Android phone or tablet, and the image acquisition module may be a camera of the mobile terminal.

[0177] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0178] The memory can be used to store the computer program or module, and the processor implements various functions of the monitoring method by running or executing the computer program or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0179] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for monitoring abnormal student behavior based on human face, characterized in that: include: Get image information; Extracting facial information from the image information; Extracting eye features and mouth features of the face information based on the LBF algorithm; A facial expression recognition algorithm based on an improved Transformer is used to extract facial expression behaviors of the face information, and a facial expression score is determined according to the facial expression behaviors; determining whether there is abnormal behavior according to the eye features and the mouth features; A comprehensive quantitative score is determined based on the eye features, the mouth features and the facial expression score, and the comprehensive quantitative score is used to represent the student's classroom learning concentration.

2. The method for monitoring abnormal student behavior based on face according to claim 1, characterized in that: The extracting of face information from the image information includes: Combine the Focal Loss loss function and the EIOU function to determine the target loss function; The original loss function of the YOLOv5n algorithm is replaced with the target loss function to determine an improved YOLOv5n algorithm; Based on the improved YOLOv5n algorithm, the facial information is extracted.

3. The method for monitoring abnormal student behavior based on face according to claim 1, characterized in that: The improved Transformer-based facial expression recognition algorithm extracts facial expression behaviors of the face information, including: Extracting features of different levels of the facial information; Determining a multi-scale feature map of the face information according to the features at different levels; The multi-scale feature map is divided into three paths, each of which passes through three convolution kernels with holes introduced; splicing the output feature maps of the convolution kernels to determine a fusion feature map of the face information; Pass the fused feature map through the channel attention branch to determine a channel attention map; Passing the fused feature map through the spatial attention branch to determine a spatial attention map; Determine an output feature map according to the channel attention map and the spatial attention map; The facial expression behavior of the face information is extracted according to the output feature map.

4. The method for monitoring abnormal student behavior based on face according to claim 3 is characterized in that: The improved Transformer-based facial expression recognition algorithm includes: Mini-Swin module, feature fusion module and feature enhancement module; The Mini-Swin module is obtained by Swin Transformer after model compression, and the Mini-Swin module includes weight conversion and weight distillation. The Mini-Swin module is used to extract features of different levels of the face information and determine the multi-scale feature map; The feature fusion module includes feature fusion, and the feature fusion module is used to fuse the global features and local features of the face information to determine the fusion feature map; The feature enhancement module includes an attention mechanism, and the feature enhancement module is used to extract key features of the facial information and determine the output feature map.

5. The method for monitoring abnormal student behavior based on face according to claim 1, characterized in that: Determining a facial expression score according to the facial expression behavior comprises: Classifying the facial expression behaviors into positive emotions, negative emotions, and neutral emotions; The positive emotions include happiness and surprise, the negative emotions include fear, sadness, anger and disgust, and the neutral emotion is calmness; Calculating the probabilities of the positive emotion, the negative emotion, and the neutral emotion respectively; The facial expression score is determined according to the probabilities of the positive emotion, the negative emotion, and the neutral emotion.

6. The method for monitoring abnormal student behavior based on face according to claim 1, characterized in that: The determining whether there is abnormal behavior according to the eye features and the mouth features includes: determining an eye behavior based on the eye characteristics; determining a mouth behavior based on the mouth features; The eye behavior and the mouth behavior are compared with the abnormal behavior determination criteria respectively. If they are the same, abnormal behavior exists; if they are different, abnormal behavior does not exist.

7. The method for monitoring abnormal student behavior based on face according to claim 6, characterized in that: The abnormal behavior determination standard is an abnormal behavior dataset of students in online classes.

8. The method for monitoring abnormal student behavior based on face according to claim 1, characterized in that: Determining a comprehensive quantitative score according to the eye features, the mouth features and the facial expression score includes: Determining an eye quantification score according to the eye characteristics; Determining a mouth quantization score according to the mouth features; Determining a facial expression quantification score according to the facial expression score; The comprehensive quantitative score is determined according to the eye quantitative score, the mouth quantitative score and the facial expression quantitative score.

9. A student abnormal behavior monitoring system based on human face, applied to a student abnormal behavior monitoring method based on human face as claimed in claims 1-8, characterized in that: include: An image acquisition module, used for acquiring image information; A face detection module, used to extract face information from the image information; A key point detection module, used for extracting eye features and mouth features of the face information based on the LBF algorithm; A facial expression analysis module, which is used to extract facial expression behaviors of the face information based on an improved Transformer-based facial expression recognition algorithm, and determine a facial expression score according to the facial expression behaviors; an abnormal behavior analysis module, used to determine whether there is abnormal behavior based on the eye features and the mouth features; The behavior quantification module is used to determine a comprehensive quantification score according to the eye features, the mouth features and the facial expression score.

10. A mobile terminal, characterized in that: The mobile terminal includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the monitoring system as described in claim 9.

Images